diff options
Diffstat (limited to 'src/spdk/dpdk/doc/guides/sample_app_ug')
80 files changed, 28365 insertions, 0 deletions
diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/bbdev_app.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/bbdev_app.rst new file mode 100644 index 000000000..405e706a4 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/bbdev_app.rst @@ -0,0 +1,133 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017 Intel Corporation + +.. bbdev_app: + +Loop-back Sample Application using Baseband Device (bbdev) +========================================================== + +The baseband sample application is a simple example of packet processing using +the Data Plane Development Kit (DPDK) for baseband workloads using Wireless +Device abstraction library. + +Overview +-------- + +The Baseband device sample application performs a loop-back operation using a +baseband device capable of transceiving data packets. +A packet is received on an ethernet port -> enqueued for downlink baseband +operation -> dequeued from the downlink baseband device -> enqueued for uplink +baseband operation -> dequeued from the baseband device -> then the received +packet is compared with the baseband operations output. Then it's looped back to +the ethernet port. + +* The MAC header is preserved in the packet + +Limitations +----------- + +* Only one baseband device and one ethernet port can be used. + +Compiling the Application +------------------------- + +#. DPDK needs to be built with ``baseband_turbo_sw`` PMD driver enabled along + with ``FLEXRAN SDK`` Libraries. Refer to *SW Turbo Poll Mode Driver* + documentation for more details on this. + +#. Go to the example directory: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK}/examples/bbdev_app + +#. Set the target (a default target is used if not specified). For example: + + .. code-block:: console + + export RTE_TARGET=x86_64-native-linux-gcc + + See the *DPDK Getting Started Guide* for possible RTE_TARGET values. + +#. Build the application: + + .. code-block:: console + + make + +Running the Application +----------------------- + +The application accepts a number of command line options: + +.. code-block:: console + + $ ./build/bbdev [EAL options] -- [-e ENCODING_CORES] [-d DECODING_CORES] / + [-p ETH_PORT_ID] [-b BBDEV_ID] + +where: + +* ``e ENCODING_CORES``: hexmask for encoding lcores (default = 0x2) +* ``d DECODING_CORES``: hexmask for decoding lcores (default = 0x4) +* ``p ETH_PORT_ID``: ethernet port ID (default = 0) +* ``b BBDEV_ID``: BBDev ID (default = 0) + +The application requires that baseband devices is capable of performing +the specified baseband operation are available on application initialization. +This means that HW baseband device/s must be bound to a DPDK driver or +a SW baseband device/s (virtual BBdev) must be created (using --vdev). + +To run the application in linux environment with the turbo_sw baseband device +using the whitelisted port running on 1 encoding lcore and 1 decoding lcore +issue the command: + +.. code-block:: console + + $ ./build/bbdev --vdev='baseband_turbo_sw' -w <NIC0PCIADDR> -c 0x38 --socket-mem=2,2 \ + --file-prefix=bbdev -- -e 0x10 -d 0x20 + +where, NIC0PCIADDR is the PCI address of the Rx port + +This command creates one virtual bbdev devices ``baseband_turbo_sw`` where the +device gets linked to a corresponding ethernet port as whitelisted by +the parameter -w. +3 cores are allocated to the application, and assigned as: + + - core 3 is the master and used to print the stats live on screen, + + - core 4 is the encoding lcore performing Rx and Turbo Encode operations + + - core 5 is the downlink lcore performing Turbo Decode, validation and Tx + operations + + +Refer to the *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + +Using Packet Generator with baseband device sample application +-------------------------------------------------------------- + +To allow the bbdev sample app to do the loopback, an influx of traffic is required. +This can be done by using DPDK Pktgen to burst traffic on two ethernet ports, and +it will print the transmitted along with the looped-back traffic on Rx ports. +Executing the command below will generate traffic on the two whitelisted ethernet +ports. + +.. code-block:: console + + $ ./pktgen-3.4.0/app/x86_64-native-linux-gcc/pktgen -c 0x3 \ + --socket-mem=1,1 --file-prefix=pg -w <NIC1PCIADDR> -- -m 1.0 -P + +where: + +* ``-c COREMASK``: A hexadecimal bitmask of cores to run on +* ``--socket-mem``: Memory to allocate on specific sockets (use comma separated values) +* ``--file-prefix``: Prefix for hugepage filenames +* ``-w <NIC1PCIADDR>``: Add a PCI device in white list. The argument format is <[domain:]bus:devid.func>. +* ``-m <string>``: Matrix for mapping ports to logical cores. +* ``-P``: PROMISCUOUS mode + + +Refer to *The Pktgen Application* documents for general information on running +Pktgen with DPDK applications. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/cmd_line.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/cmd_line.rst new file mode 100644 index 000000000..6deb6c811 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/cmd_line.rst @@ -0,0 +1,158 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Command Line Sample Application +=============================== + +This chapter describes the Command Line sample application that +is part of the Data Plane Development Kit (DPDK). + +Overview +-------- + +The Command Line sample application is a simple application that +demonstrates the use of the command line interface in the DPDK. +This application is a readline-like interface that can be used +to debug a DPDK application, in a Linux* application environment. + +.. note:: + + The rte_cmdline library should not be used in production code since + it is not validated to the same standard as other DPDK libraries. + See also the "rte_cmdline library should not be used in production code due to limited testing" item + in the "Known Issues" section of the Release Notes. + +The Command Line sample application supports some of the features of the GNU readline library such as, completion, +cut/paste and some other special bindings that make configuration and debug faster and easier. + +The application shows how the rte_cmdline application can be extended to handle a list of objects. +There are three simple commands: + +* add obj_name IP: Add a new object with an IP/IPv6 address associated to it. + +* del obj_name: Delete the specified object. + +* show obj_name: Show the IP associated with the specified object. + +.. note:: + + To terminate the application, use **Ctrl-d**. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling` + +The application is located in the ``cmd_line`` sub-directory. + +Running the Application +----------------------- + +To run the application in linux environment, issue the following command: + +.. code-block:: console + + $ ./build/cmdline -l 0-3 -n 4 + +Refer to the *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. + +EAL Initialization and cmdline Start +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The first task is the initialization of the Environment Abstraction Layer (EAL). +This is achieved as follows: + +.. code-block:: c + + int main(int argc, char **argv) + { + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_panic("Cannot init EAL\n"); + +Then, a new command line object is created and started to interact with the user through the console: + +.. code-block:: c + + cl = cmdline_stdin_new(main_ctx, "example> "); + cmdline_interact(cl); + cmdline_stdin_exit(cl); + +The cmd line_interact() function returns when the user types **Ctrl-d** and in this case, +the application exits. + +Defining a cmdline Context +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A cmdline context is a list of commands that are listed in a NULL-terminated table, for example: + +.. code-block:: c + + cmdline_parse_ctx_t main_ctx[] = { + (cmdline_parse_inst_t *) &cmd_obj_del_show, + (cmdline_parse_inst_t *) &cmd_obj_add, + (cmdline_parse_inst_t *) &cmd_help, + NULL, + }; + +Each command (of type cmdline_parse_inst_t) is defined statically. +It contains a pointer to a callback function that is executed when the command is parsed, +an opaque pointer, a help string and a list of tokens in a NULL-terminated table. + +The rte_cmdline application provides a list of pre-defined token types: + +* String Token: Match a static string, a list of static strings or any string. + +* Number Token: Match a number that can be signed or unsigned, from 8-bit to 32-bit. + +* IP Address Token: Match an IPv4 or IPv6 address or network. + +* Ethernet* Address Token: Match a MAC address. + +In this example, a new token type obj_list is defined and implemented +in the parse_obj_list.c and parse_obj_list.h files. + +For example, the cmd_obj_del_show command is defined as shown below: + +.. code-block:: c + + struct cmd_obj_add_result { + cmdline_fixed_string_t action; + cmdline_fixed_string_t name; + struct object *obj; + }; + + static void cmd_obj_del_show_parsed(void *parsed_result, struct cmdline *cl, __rte_unused void *data) + { + /* ... */ + } + + cmdline_parse_token_string_t cmd_obj_action = TOKEN_STRING_INITIALIZER(struct cmd_obj_del_show_result, action, "show#del"); + + parse_token_obj_list_t cmd_obj_obj = TOKEN_OBJ_LIST_INITIALIZER(struct cmd_obj_del_show_result, obj, &global_obj_list); + + cmdline_parse_inst_t cmd_obj_del_show = { + .f = cmd_obj_del_show_parsed, /* function to call */ + .data = NULL, /* 2nd arg of func */ + .help_str = "Show/del an object", + .tokens = { /* token list, NULL terminated */ + (void *)&cmd_obj_action, + (void *)&cmd_obj_obj, + NULL, + }, + }; + +This command is composed of two tokens: + +* The first token is a string token that can be show or del. + +* The second token is an object that was previously added using the add command in the global_obj_list variable. + +Once the command is parsed, the rte_cmdline application fills a cmd_obj_del_show_result structure. +A pointer to this structure is given as an argument to the callback function and can be used in the body of this function. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/compiling.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/compiling.rst new file mode 100644 index 000000000..6f04743c8 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/compiling.rst @@ -0,0 +1,108 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015 Intel Corporation. + +Compiling the Sample Applications +================================= + +This section explains how to compile the DPDK sample applications. + +To compile all the sample applications +-------------------------------------- + +Set the path to DPDK source code if its not set: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + +Go to DPDK source: + + .. code-block:: console + + cd $RTE_SDK + +Build DPDK: + + .. code-block:: console + + make defconfig + make + +Build the sample applications: + + .. code-block:: console + + export RTE_TARGET=build + make -C examples + +For other possible ``RTE_TARGET`` values and additional information on +compiling see +:ref:`Compiling DPDK on Linux <linux_gsg_compiling_dpdk>` or +:ref:`Compiling DPDK on FreeBSD <building_from_source>`. +Applications are output to: ``$RTE_SDK/examples/app-dir/build`` or +``$RTE_SDK/examples/app-dir/$RTE_TARGET``. + + +In the example above the compiled application is written to the ``build`` subdirectory. +To have the applications written to a different location, +the ``O=/path/to/build/directory`` option may be specified in the make command. + + .. code-block:: console + + make O=/tmp + +To build the applications for debugging use the ``DEBUG`` option. +This option adds some extra flags, disables compiler optimizations and +sets verbose output. + + .. code-block:: console + + make DEBUG=1 + + +To compile a single application +------------------------------- + +Set the path to DPDK source code: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + +Go to DPDK source: + + .. code-block:: console + + cd $RTE_SDK + +Build DPDK: + + .. code-block:: console + + make defconfig + make + +Go to the sample application directory. Unless otherwise specified the sample +applications are located in ``$RTE_SDK/examples/``. + + +Build the application: + + .. code-block:: console + + export RTE_TARGET=build + make + +To cross compile the sample application(s) +------------------------------------------ + +For cross compiling the sample application(s), please append 'CROSS=$(CROSS_COMPILER_PREFIX)' to the 'make' command. +In example of AARCH64 cross compiling: + + .. code-block:: console + + export RTE_TARGET=build + export RTE_SDK=/path/to/rte_sdk + make -C examples CROSS=aarch64-linux-gnu- + or + make CROSS=aarch64-linux-gnu- diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/dist_app.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/dist_app.rst new file mode 100644 index 000000000..90270e3a5 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/dist_app.rst @@ -0,0 +1,152 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Distributor Sample Application +============================== + +The distributor sample application is a simple example of packet distribution +to cores using the Data Plane Development Kit (DPDK). It also makes use of +Intel Speed Select Technology - Base Frequency (Intel SST-BF) to pin the +distributor to the higher frequency core if available. + +Overview +-------- + +The distributor application performs the distribution of packets that are received +on an RX_PORT to different cores. When processed by the cores, the destination +port of a packet is the port from the enabled port mask adjacent to the one on +which the packet was received, that is, if the first four ports are enabled +(port mask 0xf), ports 0 and 1 RX/TX into each other, and ports 2 and 3 RX/TX +into each other. + +This application can be used to benchmark performance using the traffic +generator as shown in the figure below. + +.. _figure_dist_perf: + +.. figure:: img/dist_perf.* + + Performance Benchmarking Setup (Basic Environment) + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``distributor`` sub-directory. + +Running the Application +----------------------- + +#. The application has a number of command line options: + + .. code-block:: console + + ./build/distributor_app [EAL options] -- -p PORTMASK + + where, + + * -p PORTMASK: Hexadecimal bitmask of ports to configure + +#. To run the application in linux environment with 10 lcores, 4 ports, + issue the command: + + .. code-block:: console + + $ ./build/distributor_app -l 1-9,22 -n 4 -- -p f + +#. Refer to the DPDK Getting Started Guide for general information on running + applications and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The distributor application consists of four types of threads: a receive +thread (``lcore_rx()``), a distributor thread (``lcore_dist()``), a set of +worker threads (``lcore_worker()``), and a transmit thread(``lcore_tx()``). +How these threads work together is shown in :numref:`figure_dist_app` below. +The ``main()`` function launches threads of these four types. Each thread +has a while loop which will be doing processing and which is terminated +only upon SIGINT or ctrl+C. + +The receive thread receives the packets using ``rte_eth_rx_burst()`` and will +enqueue them to an rte_ring. The distributor thread will dequeue the packets +from the ring and assign them to workers (using ``rte_distributor_process()`` API). +This assignment is based on the tag (or flow ID) of the packet - indicated by +the hash field in the mbuf. For IP traffic, this field is automatically filled +by the NIC with the "usr" hash value for the packet, which works as a per-flow +tag. The distributor thread communicates with the worker threads using a +cache-line swapping mechanism, passing up to 8 mbuf pointers at a time +(one cache line) to each worker. + +More than one worker thread can exist as part of the application, and these +worker threads do simple packet processing by requesting packets from +the distributor, doing a simple XOR operation on the input port mbuf field +(to indicate the output port which will be used later for packet transmission) +and then finally returning the packets back to the distributor thread. + +The distributor thread will then call the distributor api +``rte_distributor_returned_pkts()`` to get the processed packets, and will enqueue +them to another rte_ring for transfer to the TX thread for transmission on the +output port. The transmit thread will dequeue the packets from the ring and +transmit them on the output port specified in packet mbuf. + +Users who wish to terminate the running of the application have to press ctrl+C +(or send SIGINT to the app). Upon this signal, a signal handler provided +in the application will terminate all running threads gracefully and print +final statistics to the user. + +.. _figure_dist_app: + +.. figure:: img/dist_app.* + + Distributor Sample Application Layout + + +Intel SST-BF Support +-------------------- + +In DPDK 19.05, support was added to the power management library for +Intel-SST-BF, a technology that allows some cores to run at a higher +frequency than others. An application note for Intel SST-BF is available, +and is entitled +`Intel Speed Select Technology – Base Frequency - Enhancing Performance <https://builders.intel.com/docs/networkbuilders/intel-speed-select-technology-base-frequency-enhancing-performance.pdf>`_ + +The distributor application was also enhanced to be aware of these higher +frequency SST-BF cores, and when starting the application, if high frequency +SST-BF cores are present in the core mask, the application will identify these +cores and pin the workloads appropriately. The distributor core is usually +the bottleneck, so this is given first choice of the high frequency SST-BF +cores, followed by the rx core and the tx core. + +Debug Logging Support +--------------------- + +Debug logging is provided as part of the application; the user needs to uncomment +the line "#define DEBUG" defined in start of the application in main.c to enable debug logs. + +Statistics +---------- + +The main function will print statistics on the console every second. These +statistics include the number of packets enqueued and dequeued at each stage +in the application, and also key statistics per worker, including how many +packets of each burst size (1-8) were sent to each worker thread. + +Application Initialization +-------------------------- + +Command line parsing is done in the same way as it is done in the L2 Forwarding Sample +Application. See :ref:`l2_fwd_app_cmd_arguments`. + +Mbuf pool initialization is done in the same way as it is done in the L2 Forwarding +Sample Application. See :ref:`l2_fwd_app_mbuf_init`. + +Driver Initialization is done in same way as it is done in the L2 Forwarding Sample +Application. See :ref:`l2_fwd_app_dvr_init`. + +RX queue initialization is done in the same way as it is done in the L2 Forwarding +Sample Application. See :ref:`l2_fwd_app_rx_init`. + +TX queue initialization is done in the same way as it is done in the L2 Forwarding +Sample Application. See :ref:`l2_fwd_app_tx_init`. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ethtool.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ethtool.rst new file mode 100644 index 000000000..8f7fc6ca6 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ethtool.rst @@ -0,0 +1,116 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015 Intel Corporation. + +Ethtool Sample Application +========================== + +The Ethtool sample application shows an implementation of an +ethtool-like API and provides a console environment that allows +its use to query and change Ethernet card parameters. The sample +is based upon a simple L2 frame reflector. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ethtool`` sub-directory. + +Running the Application +----------------------- + +The application requires an available core for each port, plus one. +The only available options are the standard ones for the EAL: + +.. code-block:: console + + ./ethtool-app/ethtool-app/${RTE_TARGET}/ethtool [EAL options] + +Refer to the *DPDK Getting Started Guide* for general information on +running applications and the Environment Abstraction Layer (EAL) +options. + +Using the application +--------------------- + +The application is console-driven using the cmdline DPDK interface: + +.. code-block:: console + + EthApp> + +From this interface the available commands and descriptions of what +they do as follows: + +* ``drvinfo``: Print driver info +* ``eeprom``: Dump EEPROM to file +* ``module-eeprom``: Dump plugin module EEPROM to file +* ``link``: Print port link states +* ``macaddr``: Gets/sets MAC address +* ``mtu``: Set NIC MTU +* ``open``: Open port +* ``pause``: Get/set port pause state +* ``portstats``: Print port statistics +* ``regs``: Dump port register(s) to file +* ``ringparam``: Get/set ring parameters +* ``rxmode``: Toggle port Rx mode +* ``stop``: Stop port +* ``validate``: Check that given MAC address is valid unicast address +* ``vlan``: Add/remove VLAN id +* ``quit``: Exit program + + +Explanation +----------- + +The sample program has two parts: A background `packet reflector`_ +that runs on a slave core, and a foreground `Ethtool Shell`_ that +runs on the master core. These are described below. + +Packet Reflector +~~~~~~~~~~~~~~~~ + +The background packet reflector is intended to demonstrate basic +packet processing on NIC ports controlled by the Ethtool shim. +Each incoming MAC frame is rewritten so that it is returned to +the sender, using the port in question's own MAC address as the +source address, and is then sent out on the same port. + +Ethtool Shell +~~~~~~~~~~~~~ + +The foreground part of the Ethtool sample is a console-based +interface that accepts commands as described in `using the +application`_. Individual call-back functions handle the detail +associated with each command, which make use of the functions +defined in the `Ethtool interface`_ to the DPDK functions. + +Ethtool interface +----------------- + +The Ethtool interface is built as a separate library, and implements +the following functions: + +- ``rte_ethtool_get_drvinfo()`` +- ``rte_ethtool_get_regs_len()`` +- ``rte_ethtool_get_regs()`` +- ``rte_ethtool_get_link()`` +- ``rte_ethtool_get_eeprom_len()`` +- ``rte_ethtool_get_eeprom()`` +- ``rte_ethtool_set_eeprom()`` +- ``rte_ethtool_get_module_info()`` +- ``rte_ethtool_get_module_eeprom()`` +- ``rte_ethtool_get_pauseparam()`` +- ``rte_ethtool_set_pauseparam()`` +- ``rte_ethtool_net_open()`` +- ``rte_ethtool_net_stop()`` +- ``rte_ethtool_net_get_mac_addr()`` +- ``rte_ethtool_net_set_mac_addr()`` +- ``rte_ethtool_net_validate_addr()`` +- ``rte_ethtool_net_change_mtu()`` +- ``rte_ethtool_net_get_stats64()`` +- ``rte_ethtool_net_vlan_rx_add_vid()`` +- ``rte_ethtool_net_vlan_rx_kill_vid()`` +- ``rte_ethtool_net_set_rx_mode()`` +- ``rte_ethtool_get_ringparam()`` +- ``rte_ethtool_set_ringparam()`` diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/eventdev_pipeline.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/eventdev_pipeline.rst new file mode 100644 index 000000000..dc7972aa9 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/eventdev_pipeline.rst @@ -0,0 +1,145 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017 Intel Corporation. + +Eventdev Pipeline Sample Application +==================================== + +The eventdev pipeline sample application is a sample app that demonstrates +the usage of the eventdev API using the software PMD. It shows how an +application can configure a pipeline and assign a set of worker cores to +perform the processing required. + +The application has a range of command line arguments allowing it to be +configured for various numbers worker cores, stages,queue depths and cycles per +stage of work. This is useful for performance testing as well as quickly testing +a particular pipeline configuration. + + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``examples`` sub-directory. + + + +Running the Application +----------------------- + +The application has a lot of command line options. This allows specification of +the eventdev PMD to use, and a number of attributes of the processing pipeline +options. + +An example eventdev pipeline running with the software eventdev PMD using +these settings is shown below: + + * ``-r1``: core mask 0x1 for RX + * ``-t1``: core mask 0x1 for TX + * ``-e4``: core mask 0x4 for the software scheduler + * ``-w FF00``: core mask for worker cores, 8 cores from 8th to 16th + * ``-s4``: 4 atomic stages + * ``-n0``: process infinite packets (run forever) + * ``-c32``: worker dequeue depth of 32 + * ``-W1000``: do 1000 cycles of work per packet in each stage + * ``-D``: dump statistics on exit + +.. code-block:: console + + ./build/eventdev_pipeline --vdev event_sw0 -- -r1 -t1 -e4 -w FF00 -s4 -n0 -c32 -W1000 -D + +The application has some sanity checking built-in, so if there is a function +(e.g.; the RX core) which doesn't have a cpu core mask assigned, the application +will print an error message: + +.. code-block:: console + + Core part of pipeline was not assigned any cores. This will stall the + pipeline, please check core masks (use -h for details on setting core masks): + rx: 0 + tx: 1 + +Configuration of the eventdev is covered in detail in the programmers guide, +see the Event Device Library section. + + +Observing the Application +------------------------- + +At runtime the eventdev pipeline application prints out a summary of the +configuration, and some runtime statistics like packets per second. On exit the +worker statistics are printed, along with a full dump of the PMD statistics if +required. The following sections show sample output for each of the output +types. + +Configuration +~~~~~~~~~~~~~ + +This provides an overview of the pipeline, +scheduling type at each stage, and parameters to options such as how many +flows to use and what eventdev PMD is in use. See the following sample output +for details: + +.. code-block:: console + + Config: + ports: 2 + workers: 8 + packets: 0 + priorities: 1 + Queue-prio: 0 + qid0 type: atomic + Cores available: 44 + Cores used: 10 + Eventdev 0: event_sw + Stages: + Stage 0, Type Atomic Priority = 128 + Stage 1, Type Atomic Priority = 128 + Stage 2, Type Atomic Priority = 128 + Stage 3, Type Atomic Priority = 128 + +Runtime +~~~~~~~ + +At runtime, the statistics of the consumer are printed, stating the number of +packets received, runtime in milliseconds, average mpps, and current mpps. + +.. code-block:: console + + # consumer RX= xxxxxxx, time yyyy ms, avg z.zzz mpps [current w.www mpps] + +Shutdown +~~~~~~~~ + +At shutdown, the application prints the number of packets received and +transmitted, and an overview of the distribution of work across worker cores. + +.. code-block:: console + + Signal 2 received, preparing to exit... + worker 12 thread done. RX=4966581 TX=4966581 + worker 13 thread done. RX=4963329 TX=4963329 + worker 14 thread done. RX=4953614 TX=4953614 + worker 0 thread done. RX=0 TX=0 + worker 11 thread done. RX=4970549 TX=4970549 + worker 10 thread done. RX=4986391 TX=4986391 + worker 9 thread done. RX=4970528 TX=4970528 + worker 15 thread done. RX=4974087 TX=4974087 + worker 8 thread done. RX=4979908 TX=4979908 + worker 2 thread done. RX=0 TX=0 + + Port Workload distribution: + worker 0 : 12.5 % (4979876 pkts) + worker 1 : 12.5 % (4970497 pkts) + worker 2 : 12.5 % (4986359 pkts) + worker 3 : 12.5 % (4970517 pkts) + worker 4 : 12.5 % (4966566 pkts) + worker 5 : 12.5 % (4963297 pkts) + worker 6 : 12.5 % (4953598 pkts) + worker 7 : 12.5 % (4974055 pkts) + +To get a full dump of the state of the eventdev PMD, pass the ``-D`` flag to +this application. When the app is terminated using ``Ctrl+C``, the +``rte_event_dev_dump()`` function is called, resulting in a dump of the +statistics that the PMD provides. The statistics provided depend on the PMD +used, see the Event Device Drivers section for a list of eventdev PMDs. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/fips_validation.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/fips_validation.rst new file mode 100644 index 000000000..2953fddeb --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/fips_validation.rst @@ -0,0 +1,132 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +Federal Information Processing Standards (FIPS) CryptoDev Validation +==================================================================== + +Overview +-------- + +Federal Information Processing Standards (FIPS) are publicly announced standards +developed by the United States federal government for use in computer systems by +non-military government agencies and government contractors. + +This application is used to parse and perform symmetric cryptography +computation to the NIST Cryptographic Algorithm Validation Program (CAVP) test +vectors. + +For an algorithm implementation to be listed on a cryptographic module +validation certificate as an Approved security function, the algorithm +implementation must meet all the requirements of FIPS 140-2 and must +successfully complete the cryptographic algorithm validation process. + +Limitations +----------- + +* Only NIST CAVP request files are parsed by this application. +* The version of request file supported is ``CAVS 21.0`` +* If the header comment in a ``.req`` file does not contain a Algo tag + i.e ``AES,TDES,GCM`` you need to manually add it into the header comment for + example:: + + # VARIABLE KEY - KAT for CBC / # TDES VARIABLE KEY - KAT for CBC + +* The application does not supply the test vectors. The user is expected to + obtain the test vector files from `NIST + <https://csrc.nist.gov/projects/cryptographic-algorithm-validation- + program/block-ciphers>`_ website. To obtain the ``.req`` files you need to + email a person from the NIST website and pay for the ``.req`` files. + The ``.rsp`` files from the site can be used to validate and compare with + the ``.rsp`` files created by the FIPS application. + +* Supported test vectors + * AES-CBC (128,192,256) - GFSbox, KeySbox, MCT, MMT + * AES-GCM (128,192,256) - EncryptExtIV, Decrypt + * AES-CCM (128) - VADT, VNT, VPT, VTT, DVPT + * AES-CMAC (128) - Generate, Verify + * HMAC (SHA1, SHA224, SHA256, SHA384, SHA512) + * TDES-CBC (1 Key, 2 Keys, 3 Keys) - MMT, Monte, Permop, Subkey, Varkey, + VarText + +Application Information +----------------------- + +If a ``.req`` is used as the input file after the application is finished +running it will generate a response file or ``.rsp``. Differences between the +two files are, the ``.req`` file has missing information for instance if doing +encryption you will not have the cipher text and that will be generated in the +response file. Also if doing decryption it will not have the plain text until it +finished the work and in the response file it will be added onto the end of each +operation. + +The application can be run with a ``.rsp`` file and what the outcome of that +will be is it will add a extra line in the generated ``.rsp`` which should be +the same as the ``.rsp`` used to run the application, this is useful for +validating if the application has done the operation correctly. + + +Compiling the Application +------------------------- + +* Compile Application + + .. code-block:: console + + make -C examples/fips_validation + +* Run ``dos2unix`` on the request files + + .. code-block:: console + + dos2unix AES/req/* + dos2unix AES_GCM/req/* + dos2unix CCM/req/* + dos2unix CMAC/req/* + dos2unix HMAC/req/* + dos2unix TDES/req/* + +Running the Application +----------------------- + +The application requires a number of command line options: + + .. code-block:: console + + ./fips_validation [EAL options] + -- --req-file FILE_PATH/FOLDER_PATH + --rsp-file FILE_PATH/FOLDER_PATH + [--cryptodev DEVICE_NAME] [--cryptodev-id ID] [--path-is-folder] + +where, + * req-file: The path of the request file or folder, separated by + ``path-is-folder`` option. + + * rsp-file: The path that the response file or folder is stored. separated by + ``path-is-folder`` option. + + * cryptodev: The name of the target DPDK Crypto device to be validated. + + * cryptodev-id: The id of the target DPDK Crypto device to be validated. + + * path-is-folder: If presented the application expects req-file and rsp-file + are folder paths. + + +To run the application in linux environment to test one AES FIPS test data +file for crypto_aesni_mb PMD, issue the command: + +.. code-block:: console + + $ ./fips_validation --vdev crypto_aesni_mb -- + --req-file /PATH/TO/REQUEST/FILE.req --rsp-file ./PATH/TO/RESPONSE/FILE.rsp + --cryptodev crypto_aesni_mb + +To run the application in linux environment to test all AES-GCM FIPS test +data files in one folder for crypto_aesni_gcm PMD, issue the command: + +.. code-block:: console + + $ ./fips_validation --vdev crypto_aesni_gcm0 -- + --req-file /PATH/TO/REQUEST/FILE/FOLDER/ + --rsp-file ./PATH/TO/RESPONSE/FILE/FOLDER/ + --cryptodev-id 0 --path-is-folder diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/flow_classify.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/flow_classify.rst new file mode 100644 index 000000000..dc40b4d6f --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/flow_classify.rst @@ -0,0 +1,544 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017 Intel Corporation. + +Flow Classify Sample Application +================================ + +The Flow Classify sample application is based on the simple *skeleton* example +of a forwarding application. + +It is intended as a demonstration of the basic components of a DPDK forwarding +application which uses the Flow Classify library API's. + +Please refer to the +:doc:`../prog_guide/flow_classify_lib` +for more information. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``flow_classify`` sub-directory. + +Running the Application +----------------------- + +To run the example in a ``linux`` environment: + +.. code-block:: console + + cd ~/dpdk/examples/flow_classify + ./build/flow_classify -c 4 -n 4 -- --rule_ipv4="../ipv4_rules_file.txt" + +Please refer to the *DPDK Getting Started Guide*, section +:doc:`../linux_gsg/build_sample_apps` +for general information on running applications and the Environment Abstraction +Layer (EAL) options. + + +Sample ipv4_rules_file.txt +-------------------------- + +.. code-block:: console + + #file format: + #src_ip/masklen dst_ip/masklen src_port : mask dst_port : mask proto/mask priority + # + 2.2.2.3/24 2.2.2.7/24 32 : 0xffff 33 : 0xffff 17/0xff 0 + 9.9.9.3/24 9.9.9.7/24 32 : 0xffff 33 : 0xffff 17/0xff 1 + 9.9.9.3/24 9.9.9.7/24 32 : 0xffff 33 : 0xffff 6/0xff 2 + 9.9.8.3/24 9.9.8.7/24 32 : 0xffff 33 : 0xffff 6/0xff 3 + 6.7.8.9/24 2.3.4.5/24 32 : 0x0000 33 : 0x0000 132/0xff 4 + +Explanation +----------- + +The following sections provide an explanation of the main components of the +code. + +All DPDK library functions used in the sample code are prefixed with ``rte_`` +and are explained in detail in the *DPDK API Documentation*. + +ACL field definitions for the IPv4 5 tuple rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following field definitions are used when creating the ACL table during +initialisation of the ``Flow Classify`` application.. + +.. code-block:: c + + enum { + PROTO_FIELD_IPV4, + SRC_FIELD_IPV4, + DST_FIELD_IPV4, + SRCP_FIELD_IPV4, + DSTP_FIELD_IPV4, + NUM_FIELDS_IPV4 + }; + + enum { + PROTO_INPUT_IPV4, + SRC_INPUT_IPV4, + DST_INPUT_IPV4, + SRCP_DESTP_INPUT_IPV4 + }; + + static struct rte_acl_field_def ipv4_defs[NUM_FIELDS_IPV4] = { + /* first input field - always one byte long. */ + { + .type = RTE_ACL_FIELD_TYPE_BITMASK, + .size = sizeof(uint8_t), + .field_index = PROTO_FIELD_IPV4, + .input_index = PROTO_INPUT_IPV4, + .offset = sizeof(struct rte_ether_hdr) + + offsetof(struct rte_ipv4_hdr, next_proto_id), + }, + /* next input field (IPv4 source address) - 4 consecutive bytes. */ + { + /* rte_flow uses a bit mask for IPv4 addresses */ + .type = RTE_ACL_FIELD_TYPE_BITMASK, + .size = sizeof(uint32_t), + .field_index = SRC_FIELD_IPV4, + .input_index = SRC_INPUT_IPV4, + .offset = sizeof(struct rte_ether_hdr) + + offsetof(struct rte_ipv4_hdr, src_addr), + }, + /* next input field (IPv4 destination address) - 4 consecutive bytes. */ + { + /* rte_flow uses a bit mask for IPv4 addresses */ + .type = RTE_ACL_FIELD_TYPE_BITMASK, + .size = sizeof(uint32_t), + .field_index = DST_FIELD_IPV4, + .input_index = DST_INPUT_IPV4, + .offset = sizeof(struct rte_ether_hdr) + + offsetof(struct rte_ipv4_hdr, dst_addr), + }, + /* + * Next 2 fields (src & dst ports) form 4 consecutive bytes. + * They share the same input index. + */ + { + /* rte_flow uses a bit mask for protocol ports */ + .type = RTE_ACL_FIELD_TYPE_BITMASK, + .size = sizeof(uint16_t), + .field_index = SRCP_FIELD_IPV4, + .input_index = SRCP_DESTP_INPUT_IPV4, + .offset = sizeof(struct rte_ether_hdr) + + sizeof(struct rte_ipv4_hdr) + + offsetof(struct rte_tcp_hdr, src_port), + }, + { + /* rte_flow uses a bit mask for protocol ports */ + .type = RTE_ACL_FIELD_TYPE_BITMASK, + .size = sizeof(uint16_t), + .field_index = DSTP_FIELD_IPV4, + .input_index = SRCP_DESTP_INPUT_IPV4, + .offset = sizeof(struct rte_ether_hdr) + + sizeof(struct rte_ipv4_hdr) + + offsetof(struct rte_tcp_hdr, dst_port), + }, + }; + +The Main Function +~~~~~~~~~~~~~~~~~ + +The ``main()`` function performs the initialization and calls the execution +threads for each lcore. + +The first task is to initialize the Environment Abstraction Layer (EAL). +The ``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` +function. The value returned is the number of parsed arguments: + +.. code-block:: c + + int ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); + +It then parses the flow_classify application arguments + +.. code-block:: c + + ret = parse_args(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid flow_classify parameters\n"); + +The ``main()`` function also allocates a mempool to hold the mbufs +(Message Buffers) used by the application: + +.. code-block:: c + + mbuf_pool = rte_mempool_create("MBUF_POOL", + NUM_MBUFS * nb_ports, + MBUF_SIZE, + MBUF_CACHE_SIZE, + sizeof(struct rte_pktmbuf_pool_private), + rte_pktmbuf_pool_init, NULL, + rte_pktmbuf_init, NULL, + rte_socket_id(), + 0); + +mbufs are the packet buffer structure used by DPDK. They are explained in +detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*. + +The ``main()`` function also initializes all the ports using the user defined +``port_init()`` function which is explained in the next section: + +.. code-block:: c + + RTE_ETH_FOREACH_DEV(portid) { + if (port_init(portid, mbuf_pool) != 0) { + rte_exit(EXIT_FAILURE, + "Cannot init port %" PRIu8 "\n", portid); + } + } + +The ``main()`` function creates the ``flow classifier object`` and adds an ``ACL +table`` to the flow classifier. + +.. code-block:: c + + struct flow_classifier { + struct rte_flow_classifier *cls; + }; + + struct flow_classifier_acl { + struct flow_classifier cls; + } __rte_cache_aligned; + + /* Memory allocation */ + size = RTE_CACHE_LINE_ROUNDUP(sizeof(struct flow_classifier_acl)); + cls_app = rte_zmalloc(NULL, size, RTE_CACHE_LINE_SIZE); + if (cls_app == NULL) + rte_exit(EXIT_FAILURE, "Cannot allocate classifier memory\n"); + + cls_params.name = "flow_classifier"; + cls_params.socket_id = socket_id; + + cls_app->cls = rte_flow_classifier_create(&cls_params); + if (cls_app->cls == NULL) { + rte_free(cls_app); + rte_exit(EXIT_FAILURE, "Cannot create classifier\n"); + } + + /* initialise ACL table params */ + table_acl_params.name = "table_acl_ipv4_5tuple"; + table_acl_params.n_rule_fields = RTE_DIM(ipv4_defs); + table_acl_params.n_rules = FLOW_CLASSIFY_MAX_RULE_NUM; + memcpy(table_acl_params.field_format, ipv4_defs, sizeof(ipv4_defs)); + + /* initialise table create params */ + cls_table_params.ops = &rte_table_acl_ops, + cls_table_params.arg_create = &table_acl_params, + cls_table_params.type = RTE_FLOW_CLASSIFY_TABLE_ACL_IP4_5TUPLE; + + ret = rte_flow_classify_table_create(cls_app->cls, &cls_table_params); + if (ret) { + rte_flow_classifier_free(cls_app->cls); + rte_free(cls); + rte_exit(EXIT_FAILURE, "Failed to create classifier table\n"); + } + +It then reads the ipv4_rules_file.txt file and initialises the parameters for +the ``rte_flow_classify_table_entry_add`` API. +This API adds a rule to the ACL table. + +.. code-block:: c + + if (add_rules(parm_config.rule_ipv4_name)) { + rte_flow_classifier_free(cls_app->cls); + rte_free(cls_app); + rte_exit(EXIT_FAILURE, "Failed to add rules\n"); + } + +Once the initialization is complete, the application is ready to launch a +function on an lcore. In this example ``lcore_main()`` is called on a single +lcore. + +.. code-block:: c + + lcore_main(cls_app); + +The ``lcore_main()`` function is explained below. + +The Port Initialization Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The main functional part of the port initialization used in the Basic +Forwarding application is shown below: + +.. code-block:: c + + static inline int + port_init(uint8_t port, struct rte_mempool *mbuf_pool) + { + struct rte_eth_conf port_conf = port_conf_default; + const uint16_t rx_rings = 1, tx_rings = 1; + struct rte_ether_addr addr; + int retval; + uint16_t q; + + /* Configure the Ethernet device. */ + retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf); + if (retval != 0) + return retval; + + /* Allocate and set up 1 RX queue per Ethernet port. */ + for (q = 0; q < rx_rings; q++) { + retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE, + rte_eth_dev_socket_id(port), NULL, mbuf_pool); + if (retval < 0) + return retval; + } + + /* Allocate and set up 1 TX queue per Ethernet port. */ + for (q = 0; q < tx_rings; q++) { + retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE, + rte_eth_dev_socket_id(port), NULL); + if (retval < 0) + return retval; + } + + /* Start the Ethernet port. */ + retval = rte_eth_dev_start(port); + if (retval < 0) + return retval; + + /* Display the port MAC address. */ + retval = rte_eth_macaddr_get(port, &addr); + if (retval < 0) + return retval; + printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8 + " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n", + port, + addr.addr_bytes[0], addr.addr_bytes[1], + addr.addr_bytes[2], addr.addr_bytes[3], + addr.addr_bytes[4], addr.addr_bytes[5]); + + /* Enable RX in promiscuous mode for the Ethernet device. */ + retval = rte_eth_promiscuous_enable(port); + if (retval != 0) + return retval; + + return 0; + } + +The Ethernet ports are configured with default settings using the +``rte_eth_dev_configure()`` function and the ``port_conf_default`` struct. + +.. code-block:: c + + static const struct rte_eth_conf port_conf_default = { + .rxmode = { .max_rx_pkt_len = RTE_ETHER_MAX_LEN } + }; + +For this example the ports are set up with 1 RX and 1 TX queue using the +``rte_eth_rx_queue_setup()`` and ``rte_eth_tx_queue_setup()`` functions. + +The Ethernet port is then started: + +.. code-block:: c + + retval = rte_eth_dev_start(port); + + +Finally the RX port is set in promiscuous mode: + +.. code-block:: c + + retval = rte_eth_promiscuous_enable(port); + +The Add Rules function +~~~~~~~~~~~~~~~~~~~~~~ + +The ``add_rules`` function reads the ``ipv4_rules_file.txt`` file and calls the +``add_classify_rule`` function which calls the +``rte_flow_classify_table_entry_add`` API. + +.. code-block:: c + + static int + add_rules(const char *rule_path) + { + FILE *fh; + char buff[LINE_MAX]; + unsigned int i = 0; + unsigned int total_num = 0; + struct rte_eth_ntuple_filter ntuple_filter; + + fh = fopen(rule_path, "rb"); + if (fh == NULL) + rte_exit(EXIT_FAILURE, "%s: Open %s failed\n", __func__, + rule_path); + + fseek(fh, 0, SEEK_SET); + + i = 0; + while (fgets(buff, LINE_MAX, fh) != NULL) { + i++; + + if (is_bypass_line(buff)) + continue; + + if (total_num >= FLOW_CLASSIFY_MAX_RULE_NUM - 1) { + printf("\nINFO: classify rule capacity %d reached\n", + total_num); + break; + } + + if (parse_ipv4_5tuple_rule(buff, &ntuple_filter) != 0) + rte_exit(EXIT_FAILURE, + "%s Line %u: parse rules error\n", + rule_path, i); + + if (add_classify_rule(&ntuple_filter) != 0) + rte_exit(EXIT_FAILURE, "add rule error\n"); + + total_num++; + } + + fclose(fh); + return 0; + } + + +The Lcore Main function +~~~~~~~~~~~~~~~~~~~~~~~ + +As we saw above the ``main()`` function calls an application function on the +available lcores. +The ``lcore_main`` function calls the ``rte_flow_classifier_query`` API. +For the Basic Forwarding application the ``lcore_main`` function looks like the +following: + +.. code-block:: c + + /* flow classify data */ + static int num_classify_rules; + static struct rte_flow_classify_rule *rules[MAX_NUM_CLASSIFY]; + static struct rte_flow_classify_ipv4_5tuple_stats ntuple_stats; + static struct rte_flow_classify_stats classify_stats = { + .stats = (void *)&ntuple_stats + }; + + static __rte_noreturn void + lcore_main(cls_app) + { + uint16_t port; + + /* + * Check that the port is on the same NUMA node as the polling thread + * for best performance. + */ + RTE_ETH_FOREACH_DEV(port) + if (rte_eth_dev_socket_id(port) > 0 && + rte_eth_dev_socket_id(port) != (int)rte_socket_id()) { + printf("\n\n"); + printf("WARNING: port %u is on remote NUMA node\n", + port); + printf("to polling thread.\n"); + printf("Performance will not be optimal.\n"); + + printf("\nCore %u forwarding packets. \n", + rte_lcore_id()); + printf("[Ctrl+C to quit]\n + } + + /* Run until the application is quit or killed. */ + for (;;) { + /* + * Receive packets on a port and forward them on the paired + * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc. + */ + RTE_ETH_FOREACH_DEV(port) { + + /* Get burst of RX packets, from first port of pair. */ + struct rte_mbuf *bufs[BURST_SIZE]; + const uint16_t nb_rx = rte_eth_rx_burst(port, 0, + bufs, BURST_SIZE); + + if (unlikely(nb_rx == 0)) + continue; + + for (i = 0; i < MAX_NUM_CLASSIFY; i++) { + if (rules[i]) { + ret = rte_flow_classifier_query( + cls_app->cls, + bufs, nb_rx, rules[i], + &classify_stats); + if (ret) + printf( + "rule [%d] query failed ret [%d]\n\n", + i, ret); + else { + printf( + "rule[%d] count=%"PRIu64"\n", + i, ntuple_stats.counter1); + + printf("proto = %d\n", + ntuple_stats.ipv4_5tuple.proto); + } + } + } + + /* Send burst of TX packets, to second port of pair. */ + const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0, + bufs, nb_rx); + + /* Free any unsent packets. */ + if (unlikely(nb_tx < nb_rx)) { + uint16_t buf; + for (buf = nb_tx; buf < nb_rx; buf++) + rte_pktmbuf_free(bufs[buf]); + } + } + } + } + +The main work of the application is done within the loop: + +.. code-block:: c + + for (;;) { + RTE_ETH_FOREACH_DEV(port) { + + /* Get burst of RX packets, from first port of pair. */ + struct rte_mbuf *bufs[BURST_SIZE]; + const uint16_t nb_rx = rte_eth_rx_burst(port, 0, + bufs, BURST_SIZE); + + if (unlikely(nb_rx == 0)) + continue; + + /* Send burst of TX packets, to second port of pair. */ + const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0, + bufs, nb_rx); + + /* Free any unsent packets. */ + if (unlikely(nb_tx < nb_rx)) { + uint16_t buf; + for (buf = nb_tx; buf < nb_rx; buf++) + rte_pktmbuf_free(bufs[buf]); + } + } + } + +Packets are received in bursts on the RX ports and transmitted in bursts on +the TX ports. The ports are grouped in pairs with a simple mapping scheme +using the an XOR on the port number:: + + 0 -> 1 + 1 -> 0 + + 2 -> 3 + 3 -> 2 + + etc. + +The ``rte_eth_tx_burst()`` function frees the memory buffers of packets that +are transmitted. If packets fail to transmit, ``(nb_tx < nb_rx)``, then they +must be freed explicitly using ``rte_pktmbuf_free()``. + +The forwarding loop can be interrupted and the application closed using +``Ctrl-C``. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/flow_filtering.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/flow_filtering.rst new file mode 100644 index 000000000..5e5a6cd8a --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/flow_filtering.rst @@ -0,0 +1,515 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2017 Mellanox Technologies, Ltd + +Basic RTE Flow Filtering Sample Application +=========================================== + +The Basic RTE flow filtering sample application is a simple example of a +creating a RTE flow rule. + +It is intended as a demonstration of the basic components RTE flow rules. + + +Compiling the Application +------------------------- + +To compile the application export the path to the DPDK source tree and go to +the example directory: + +.. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + + cd ${RTE_SDK}/examples/flow_filtering + +Set the target, for example: + +.. code-block:: console + + export RTE_TARGET=x86_64-native-linux-gcc + +See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values. + +Build the application as follows: + +.. code-block:: console + + make + + +Running the Application +----------------------- + +To run the example in a ``linux`` environment: + +.. code-block:: console + + ./build/flow -l 1 -n 1 + +Refer to *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + + +Explanation +----------- + +The example is built from 2 files, +``main.c`` which holds the example logic and ``flow_blocks.c`` that holds the +implementation for building the flow rule. + +The following sections provide an explanation of the main components of the +code. + +All DPDK library functions used in the sample code are prefixed with ``rte_`` +and are explained in detail in the *DPDK API Documentation*. + + +The Main Function +~~~~~~~~~~~~~~~~~ + +The ``main()`` function located in ``main.c`` file performs the initialization +and runs the main loop function. + +The first task is to initialize the Environment Abstraction Layer (EAL). The +``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` +function. The value returned is the number of parsed arguments: + +.. code-block:: c + + int ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); + + +The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers) +used by the application: + +.. code-block:: c + + mbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", 4096, 128, 0, + RTE_MBUF_DEFAULT_BUF_SIZE, + rte_socket_id()); + +Mbufs are the packet buffer structure used by DPDK. They are explained in +detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*. + +The ``main()`` function also initializes all the ports using the user defined +``init_port()`` function which is explained in the next section: + +.. code-block:: c + + init_port(); + +Once the initialization is complete, we set the flow rule using the +following code: + +.. code-block:: c + + /* create flow for send packet with */ + flow = generate_ipv4_flow(port_id, selected_queue, + SRC_IP, EMPTY_MASK, + DEST_IP, FULL_MASK, &error); + if (!flow) { + printf("Flow can't be created %d message: %s\n", + error.type, + error.message ? error.message : "(no stated reason)"); + rte_exit(EXIT_FAILURE, "error in creating flow"); + } + +In the last part the application is ready to launch the +``main_loop()`` function. Which is explained below. + + +.. code-block:: c + + main_loop(); + +The Port Initialization Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The main functional part of the port initialization used in the flow filtering +application is shown below: + +.. code-block:: c + + init_port(void) + { + int ret; + uint16_t i; + struct rte_eth_conf port_conf = { + .rxmode = { + .split_hdr_size = 0, + }, + .txmode = { + .offloads = + DEV_TX_OFFLOAD_VLAN_INSERT | + DEV_TX_OFFLOAD_IPV4_CKSUM | + DEV_TX_OFFLOAD_UDP_CKSUM | + DEV_TX_OFFLOAD_TCP_CKSUM | + DEV_TX_OFFLOAD_SCTP_CKSUM | + DEV_TX_OFFLOAD_TCP_TSO, + }, + }; + struct rte_eth_txconf txq_conf; + struct rte_eth_rxconf rxq_conf; + struct rte_eth_dev_info dev_info; + + printf(":: initializing port: %d\n", port_id); + ret = rte_eth_dev_configure(port_id, + nr_queues, nr_queues, &port_conf); + if (ret < 0) { + rte_exit(EXIT_FAILURE, + ":: cannot configure device: err=%d, port=%u\n", + ret, port_id); + } + + rte_eth_dev_info_get(port_id, &dev_info); + rxq_conf = dev_info.default_rxconf; + rxq_conf.offloads = port_conf.rxmode.offloads; + /* only set Rx queues: something we care only so far */ + for (i = 0; i < nr_queues; i++) { + ret = rte_eth_rx_queue_setup(port_id, i, 512, + rte_eth_dev_socket_id(port_id), + &rxq_conf, + mbuf_pool); + if (ret < 0) { + rte_exit(EXIT_FAILURE, + ":: Rx queue setup failed: err=%d, port=%u\n", + ret, port_id); + } + } + + txq_conf = dev_info.default_txconf; + txq_conf.offloads = port_conf.txmode.offloads; + + for (i = 0; i < nr_queues; i++) { + ret = rte_eth_tx_queue_setup(port_id, i, 512, + rte_eth_dev_socket_id(port_id), + &txq_conf); + if (ret < 0) { + rte_exit(EXIT_FAILURE, + ":: Tx queue setup failed: err=%d, port=%u\n", + ret, port_id); + } + } + + ret = rte_eth_promiscuous_enable(port_id); + if (ret != 0) { + rte_exit(EXIT_FAILURE, + ":: cannot enable promiscuous mode: err=%d, port=%u\n", + ret, port_id); + } + + ret = rte_eth_dev_start(port_id); + if (ret < 0) { + rte_exit(EXIT_FAILURE, + "rte_eth_dev_start:err=%d, port=%u\n", + ret, port_id); + } + + assert_link_status(); + + printf(":: initializing port: %d done\n", port_id); + } + +The Ethernet port is configured with default settings using the +``rte_eth_dev_configure()`` function and the ``port_conf_default`` struct: + +.. code-block:: c + + struct rte_eth_conf port_conf = { + .rxmode = { + .split_hdr_size = 0, + }, + .txmode = { + .offloads = + DEV_TX_OFFLOAD_VLAN_INSERT | + DEV_TX_OFFLOAD_IPV4_CKSUM | + DEV_TX_OFFLOAD_UDP_CKSUM | + DEV_TX_OFFLOAD_TCP_CKSUM | + DEV_TX_OFFLOAD_SCTP_CKSUM | + DEV_TX_OFFLOAD_TCP_TSO, + }, + }; + + ret = rte_eth_dev_configure(port_id, nr_queues, nr_queues, &port_conf); + if (ret < 0) { + rte_exit(EXIT_FAILURE, + ":: cannot configure device: err=%d, port=%u\n", + ret, port_id); + } + rte_eth_dev_info_get(port_id, &dev_info); + rxq_conf = dev_info.default_rxconf; + rxq_conf.offloads = port_conf.rxmode.offloads; + +For this example we are configuring number of rx and tx queues that are connected +to a single port. + +.. code-block:: c + + for (i = 0; i < nr_queues; i++) { + ret = rte_eth_rx_queue_setup(port_id, i, 512, + rte_eth_dev_socket_id(port_id), + &rxq_conf, + mbuf_pool); + if (ret < 0) { + rte_exit(EXIT_FAILURE, + ":: Rx queue setup failed: err=%d, port=%u\n", + ret, port_id); + } + } + + for (i = 0; i < nr_queues; i++) { + ret = rte_eth_tx_queue_setup(port_id, i, 512, + rte_eth_dev_socket_id(port_id), + &txq_conf); + if (ret < 0) { + rte_exit(EXIT_FAILURE, + ":: Tx queue setup failed: err=%d, port=%u\n", + ret, port_id); + } + } + +In the next step we create and apply the flow rule. which is to send packets +with destination ip equals to 192.168.1.1 to queue number 1. The detail +explanation of the ``generate_ipv4_flow()`` appears later in this document: + +.. code-block:: c + + flow = generate_ipv4_flow(port_id, selected_queue, + SRC_IP, EMPTY_MASK, + DEST_IP, FULL_MASK, &error); + +We are setting the RX port to promiscuous mode: + +.. code-block:: c + + ret = rte_eth_promiscuous_enable(port_id); + if (ret != 0) { + rte_exit(EXIT_FAILURE, + ":: cannot enable promiscuous mode: err=%d, port=%u\n", + ret, port_id); + } + +The last step is to start the port. + +.. code-block:: c + + ret = rte_eth_dev_start(port_id); + if (ret < 0) { + rte_exit(EXIT_FAILURE, "rte_eth_dev_start:err%d, port=%u\n", + ret, port_id); + } + + +The main_loop function +~~~~~~~~~~~~~~~~~~~~~~ + +As we saw above the ``main()`` function calls an application function to handle +the main loop. For the flow filtering application the main_loop function +looks like the following: + +.. code-block:: c + + static void + main_loop(void) + { + struct rte_mbuf *mbufs[32]; + struct rte_ether_hdr *eth_hdr; + uint16_t nb_rx; + uint16_t i; + uint16_t j; + + while (!force_quit) { + for (i = 0; i < nr_queues; i++) { + nb_rx = rte_eth_rx_burst(port_id, + i, mbufs, 32); + if (nb_rx) { + for (j = 0; j < nb_rx; j++) { + struct rte_mbuf *m = mbufs[j]; + + eth_hdr = rte_pktmbuf_mtod(m, + struct rte_ether_hdr *); + print_ether_addr("src=", + ð_hdr->s_addr); + print_ether_addr(" - dst=", + ð_hdr->d_addr); + printf(" - queue=0x%x", + (unsigned int)i); + printf("\n"); + rte_pktmbuf_free(m); + } + } + } + } + /* closing and releasing resources */ + rte_flow_flush(port_id, &error); + rte_eth_dev_stop(port_id); + rte_eth_dev_close(port_id); + } + +The main work of the application is reading the packets from all +queues and printing for each packet the destination queue: + +.. code-block:: c + + while (!force_quit) { + for (i = 0; i < nr_queues; i++) { + nb_rx = rte_eth_rx_burst(port_id, i, mbufs, 32); + if (nb_rx) { + for (j = 0; j < nb_rx; j++) { + struct rte_mbuf *m = mbufs[j]; + eth_hdr = rte_pktmbuf_mtod(m, struct rte_ether_hdr *); + print_ether_addr("src=", ð_hdr->s_addr); + print_ether_addr(" - dst=", ð_hdr->d_addr); + printf(" - queue=0x%x", (unsigned int)i); + printf("\n"); + rte_pktmbuf_free(m); + } + } + } + } + + +The forwarding loop can be interrupted and the application closed using +``Ctrl-C``. Which results in closing the port and the device using +``rte_eth_dev_stop`` and ``rte_eth_dev_close`` + +The generate_ipv4_flow function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The generate_ipv4_flow function is responsible for creating the flow rule. +This function is located in the ``flow_blocks.c`` file. + +.. code-block:: c + + static struct rte_flow * + generate_ipv4_flow(uint8_t port_id, uint16_t rx_q, + uint32_t src_ip, uint32_t src_mask, + uint32_t dest_ip, uint32_t dest_mask, + struct rte_flow_error *error) + { + struct rte_flow_attr attr; + struct rte_flow_item pattern[MAX_PATTERN_NUM]; + struct rte_flow_action action[MAX_ACTION_NUM]; + struct rte_flow *flow = NULL; + struct rte_flow_action_queue queue = { .index = rx_q }; + struct rte_flow_item_ipv4 ip_spec; + struct rte_flow_item_ipv4 ip_mask; + + memset(pattern, 0, sizeof(pattern)); + memset(action, 0, sizeof(action)); + + /* + * set the rule attribute. + * in this case only ingress packets will be checked. + */ + memset(&attr, 0, sizeof(struct rte_flow_attr)); + attr.ingress = 1; + + /* + * create the action sequence. + * one action only, move packet to queue + */ + action[0].type = RTE_FLOW_ACTION_TYPE_QUEUE; + action[0].conf = &queue; + action[1].type = RTE_FLOW_ACTION_TYPE_END; + + /* + * set the first level of the pattern (ETH). + * since in this example we just want to get the + * ipv4 we set this level to allow all. + */ + pattern[0].type = RTE_FLOW_ITEM_TYPE_ETH; + + /* + * setting the second level of the pattern (IP). + * in this example this is the level we care about + * so we set it according to the parameters. + */ + memset(&ip_spec, 0, sizeof(struct rte_flow_item_ipv4)); + memset(&ip_mask, 0, sizeof(struct rte_flow_item_ipv4)); + ip_spec.hdr.dst_addr = htonl(dest_ip); + ip_mask.hdr.dst_addr = dest_mask; + ip_spec.hdr.src_addr = htonl(src_ip); + ip_mask.hdr.src_addr = src_mask; + pattern[1].type = RTE_FLOW_ITEM_TYPE_IPV4; + pattern[1].spec = &ip_spec; + pattern[1].mask = &ip_mask; + + /* the final level must be always type end */ + pattern[2].type = RTE_FLOW_ITEM_TYPE_END; + + int res = rte_flow_validate(port_id, &attr, pattern, action, error); + if(!res) + flow = rte_flow_create(port_id, &attr, pattern, action, error); + + return flow; + } + +The first part of the function is declaring the structures that will be used. + +.. code-block:: c + + struct rte_flow_attr attr; + struct rte_flow_item pattern[MAX_PATTERN_NUM]; + struct rte_flow_action action[MAX_ACTION_NUM]; + struct rte_flow *flow; + struct rte_flow_error error; + struct rte_flow_action_queue queue = { .index = rx_q }; + struct rte_flow_item_ipv4 ip_spec; + struct rte_flow_item_ipv4 ip_mask; + +The following part create the flow attributes, in our case ingress. + +.. code-block:: c + + memset(&attr, 0, sizeof(struct rte_flow_attr)); + attr.ingress = 1; + +The third part defines the action to be taken when a packet matches +the rule. In this case send the packet to queue. + +.. code-block:: c + + action[0].type = RTE_FLOW_ACTION_TYPE_QUEUE; + action[0].conf = &queue; + action[1].type = RTE_FLOW_ACTION_TYPE_END; + +The fourth part is responsible for creating the pattern and is built from +number of steps. In each step we build one level of the pattern starting with +the lowest one. + +Setting the first level of the pattern ETH: + +.. code-block:: c + + pattern[0].type = RTE_FLOW_ITEM_TYPE_ETH; + +Setting the second level of the pattern IP: + +.. code-block:: c + + memset(&ip_spec, 0, sizeof(struct rte_flow_item_ipv4)); + memset(&ip_mask, 0, sizeof(struct rte_flow_item_ipv4)); + ip_spec.hdr.dst_addr = htonl(dest_ip); + ip_mask.hdr.dst_addr = dest_mask; + ip_spec.hdr.src_addr = htonl(src_ip); + ip_mask.hdr.src_addr = src_mask; + pattern[1].type = RTE_FLOW_ITEM_TYPE_IPV4; + pattern[1].spec = &ip_spec; + pattern[1].mask = &ip_mask; + +Closing the pattern part. + +.. code-block:: c + + pattern[2].type = RTE_FLOW_ITEM_TYPE_END; + +The last part of the function is to validate the rule and create it. + +.. code-block:: c + + int res = rte_flow_validate(port_id, &attr, pattern, action, &error); + if (!res) + flow = rte_flow_create(port_id, &attr, pattern, action, &error); diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/hello_world.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/hello_world.rst new file mode 100644 index 000000000..46f997a7d --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/hello_world.rst @@ -0,0 +1,94 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Hello World Sample Application +============================== + +The Hello World sample application is an example of the simplest DPDK application that can be written. +The application simply prints an "helloworld" message on every enabled lcore. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``helloworld`` sub-directory. + +Running the Application +----------------------- + +To run the example in a linux environment: + +.. code-block:: console + + $ ./build/helloworld -l 0-3 -n 4 + +Refer to *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of code. + +EAL Initialization +~~~~~~~~~~~~~~~~~~ + +The first task is to initialize the Environment Abstraction Layer (EAL). +This is done in the main() function using the following code: + +.. code-block:: c + + int + + main(int argc, char **argv) + + { + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_panic("Cannot init EAL\n"); + +This call finishes the initialization process that was started before main() is called (in case of a Linux environment). +The argc and argv arguments are provided to the rte_eal_init() function. +The value returned is the number of parsed arguments. + +Starting Application Unit Lcores +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the EAL is initialized, the application is ready to launch a function on an lcore. +In this example, lcore_hello() is called on every available lcore. +The following is the definition of the function: + +.. code-block:: c + + static int + lcore_hello(__rte_unused void *arg) + { + unsigned lcore_id; + + lcore_id = rte_lcore_id(); + printf("hello from core %u\n", lcore_id); + return 0; + } + +The code that launches the function on each lcore is as follows: + +.. code-block:: c + + /* call lcore_hello() on every slave lcore */ + + RTE_LCORE_FOREACH_SLAVE(lcore_id) { + rte_eal_remote_launch(lcore_hello, NULL, lcore_id); + } + + /* call it on master lcore too */ + + lcore_hello(NULL); + +The following code is equivalent and simpler: + +.. code-block:: c + + rte_eal_mp_remote_launch(lcore_hello, NULL, CALL_MASTER); + +Refer to the *DPDK API Reference* for detailed information on the rte_eal_mp_remote_launch() function. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/client_svr_sym_multi_proc_app.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/client_svr_sym_multi_proc_app.png Binary files differnew file mode 100644 index 000000000..abd3ef342 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/client_svr_sym_multi_proc_app.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/dist_app.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/dist_app.svg new file mode 100644 index 000000000..f19ad9b77 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/dist_app.svg @@ -0,0 +1,474 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2014-2017 Intel Corporation --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="526.94379" + height="379.53668" + id="svg4090" + version="1.1" + inkscape:version="0.92.1 r15371" + sodipodi:docname="dist_app.svg"> + <defs + id="defs4092"> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mend" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path10501" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4017" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4019" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4021" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4023" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4025" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4027" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4029" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4031" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4033" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4035" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mstart" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path10498" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4039" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4041" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4043" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4045" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4047" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4049" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1" + inkscape:cx="401.32873" + inkscape:cy="130.13572" + inkscape:document-units="px" + inkscape:current-layer="layer1" + showgrid="false" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" + inkscape:window-width="1920" + inkscape:window-height="1137" + inkscape:window-x="1912" + inkscape:window-y="-8" + inkscape:window-maximized="1" /> + <metadata + id="metadata4095"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title /> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-35.078263,-28.308125)"> + <rect + style="fill:none;stroke:#000000;stroke-width:0.81890059;stroke-opacity:0.98412697" + id="rect10443" + width="152.96732" + height="178.99617" + x="124.50176" + y="128.95552" /> + <rect + style="fill:none;stroke:#000000;stroke-width:0.99999988;stroke-opacity:0.98412697" + id="rect10445-2" + width="124.71397" + height="46.675529" + x="437.00507" + y="133.06113" /> + <rect + style="fill:none;stroke:#000000;stroke-width:0.99999988;stroke-opacity:0.98412697" + id="rect10445-0" + width="124.71397" + height="46.675529" + x="436.80811" + y="193.87207" /> + <rect + style="fill:none;stroke:#000000;stroke-width:0.99999988;stroke-opacity:0.98412697" + id="rect10445-9" + width="124.71397" + height="46.675529" + x="436.80811" + y="256.06277" /> + <rect + style="fill:none;stroke:#000000;stroke-width:0.99999988;stroke-opacity:0.98412697" + id="rect10445-7" + width="124.71397" + height="46.675529" + x="135.7057" + y="360.66928" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99566948;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow1Mstart)" + d="M 278.89497,147.51907 436.5713,146.78234" + id="path10486-2" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99290925;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow1Mstart)" + d="m 279.37092,206.8834 156.80331,-0.73671" + id="path10486-1" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99379504;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow1Mstart)" + d="m 279.19738,270.88669 157.15478,-0.73638" + id="path10486-4" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99820405;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="m 277.17846,166.20347 158.11878,-0.73842" + id="path10486-0-4" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99410033;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="m 277.47049,225.92925 157.32298,-0.73606" + id="path10486-0-7" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99566948;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="M 277.70474,289.26714 435.38107,288.5304" + id="path10486-0-77" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="345.02322" + y="134.82103" + id="text11995" + transform="scale(0.93992339,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997" + x="345.02322" + y="134.82103" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">Request burst</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="346.38663" + y="164.76628" + id="text11995-7" + transform="scale(0.93992339,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3" + x="346.38663" + y="164.76628" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">Mbuf Pointers</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="502.36844" + y="151.66222" + id="text11995-7-3" + transform="scale(0.93992339,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-5" + x="502.36844" + y="151.66222" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">WorkerThread1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="499.40103" + y="207.94502" + id="text11995-7-3-9" + transform="scale(0.93992339,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-5-9" + x="499.40103" + y="207.94502" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">WorkerThread2</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="500.1918" + y="266.59644" + id="text11995-7-3-82" + transform="scale(0.9399234,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-5-6" + x="500.1918" + y="266.59644" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">WorkerThreadN</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="193.79703" + y="362.85193" + id="text11995-7-3-6" + transform="scale(0.93992342,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-5-0" + x="193.79703" + y="362.85193" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">TX thread</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="175.78905" + y="207.26257" + id="text11995-7-3-3" + transform="scale(0.9399234,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-5-8" + x="175.78905" + y="207.26257" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">Distributor Thread</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.75945646;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="m 49.600127,54.625621 85.546363,-0.79004" + id="path10486-0-4-5" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.75945646;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:1;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="m 135.70569,384.00706 -85.546361,0.79003" + id="path10486-0-4-5-7" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="73.342712" + y="44.196564" + id="text11995-7-8" + transform="scale(0.9399234,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-3" + x="73.342712" + y="44.196564" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">Mbufs In</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="83.4814" + y="352.62543" + id="text11995-7-8-5" + transform="scale(0.93992342,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-3-1" + x="83.4814" + y="352.62543" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">Mbufs Out</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:1.01068497;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:0.98412697" + d="m 171.68192,308.06701 0.21464,27.84908 -8.6322,0.37082 -11.33877,0.17876 25.75778,13.51792 23.25799,10.16096 18.87014,-6.69841 31.83305,-15.77889 -10.75831,-0.30147 -10.37586,-0.40509 -0.22443,-28.8261 z" + id="path12188" + inkscape:connector-curvature="0" + inkscape:transform-center-y="7.0247597" + sodipodi:nodetypes="cccccccccccc" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="193.68871" + y="309.26349" + id="text11995-7-3-6-2" + transform="scale(0.93992342,1.0639165)"><tspan + sodipodi:role="line" + x="193.68871" + y="309.26349" + id="tspan12214" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">SW Ring</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:1.02106845;stroke-miterlimit:3;stroke-dasharray:none;stroke-opacity:0.98412697" + d="m 173.27214,75.568236 0.21464,28.424254 -8.6322,0.37848 -11.33877,0.18245 25.75778,13.79709 23.25799,10.37083 18.87013,-6.83675 31.83305,-16.10478 -10.75831,-0.30769 -10.37586,-0.41345 -0.22443,-29.421453 z" + id="path12188-5" + inkscape:connector-curvature="0" + inkscape:transform-center-y="7.1698404" + sodipodi:nodetypes="cccccccccccc" /> + <rect + style="fill:none;stroke:#000000;stroke-width:0.99999988;stroke-opacity:0.98412697" + id="rect10445-7-7" + width="124.71397" + height="46.675529" + x="138.18427" + y="28.832333" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="190.80019" + y="51.17778" + id="text11995-7-3-6-6" + transform="scale(0.93992339,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-5-0-4" + x="190.80019" + y="51.17778" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">RX thread</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="196.38097" + y="90.224785" + id="text11995-7-3-6-2-9" + transform="scale(0.93992339,1.0639165)"><tspan + sodipodi:role="line" + x="196.38097" + y="90.224785" + id="tspan12214-8" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">SW Ring</tspan></text> + <rect + style="fill:none;stroke:#000000;stroke-width:0.99999988;stroke-opacity:0.98412697" + id="rect10445-7-7-5" + width="124.71397" + height="46.675529" + x="327.86566" + y="29.009106" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;line-height:0%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + x="387.27209" + y="45.36227" + id="text11995-7-3-6-6-3" + transform="scale(0.93992339,1.0639165)"><tspan + sodipodi:role="line" + id="tspan11997-3-5-0-4-4" + x="387.27209" + y="45.36227" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif">Stats thread</tspan><tspan + sodipodi:role="line" + x="387.27209" + y="57.016178" + style="font-size:9.32312489px;line-height:1.25;font-family:sans-serif" + id="tspan165">(to console)</tspan></text> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/dist_perf.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/dist_perf.svg new file mode 100644 index 000000000..d5b2a2be0 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/dist_perf.svg @@ -0,0 +1,462 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2014 Intel Corporation --> + + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="524.65625" + height="387.59375" + id="svg4116" + version="1.1" + inkscape:version="0.48.5 r10040" + sodipodi:docname="New document 2"> + <defs + id="defs4118"> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mstart" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path10498" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mend" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path10501" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4038" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4040" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4042" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4044" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4046" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4048" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4050" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4052" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4054" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4056" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4058" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4060" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4062" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4064" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4066" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4068" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker4070" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4072" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="marker4074" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path4076" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.4,0,0,0.4,4,0)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1" + inkscape:cx="354.46875" + inkscape:cy="78.904643" + inkscape:document-units="px" + inkscape:current-layer="layer1" + showgrid="false" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" + inkscape:window-width="1920" + inkscape:window-height="1017" + inkscape:window-x="-8" + inkscape:window-y="-8" + inkscape:window-maximized="1" /> + <metadata + id="metadata4121"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-20.53125,-22.84375)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-opacity:0.98412697" + id="rect10443" + width="165.52779" + height="376.84436" + x="21.023544" + y="24.286175" /> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-opacity:0.98412697" + id="rect10445" + width="156.95697" + height="386.59042" + x="387.73376" + y="23.352676" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Mstart)" + d="m 188.27904,66.970932 195.99264,0.833121" + id="path10486" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="M 188.05945,91.53983 384.0521,90.566545" + id="path10486-0" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="460.4104" + y="292.91855" + id="text11995" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + x="460.4104" + y="292.91855" + id="tspan12218">Port2</tspan></text> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="459.06958" + y="59.738571" + id="text11995-7" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + id="tspan11997-3" + x="459.06958" + y="59.738571">Port0</tspan></text> + <text + xml:space="preserve" + style="font-size:15.28272438px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="474.06027" + y="184.77933" + id="text11995-7-3" + sodipodi:linespacing="125%" + transform="scale(0.8986678,1.1127582)"><tspan + sodipodi:role="line" + id="tspan11997-3-5" + x="474.06027" + y="184.77933">DPDK board</tspan></text> + <text + xml:space="preserve" + style="font-size:16.0002594px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="54.009655" + y="171.28656" + id="text11995-7-3-3" + sodipodi:linespacing="125%" + transform="scale(0.81894062,1.2210897)"><tspan + sodipodi:role="line" + id="tspan11997-3-5-8" + x="54.009655" + y="171.28656">Traffic Generator</tspan></text> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="459.46732" + y="91.195976" + id="text11995-7-8" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + id="tspan11997-3-3" + x="459.46732" + y="91.195976">Port1</tspan></text> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="460.15295" + y="326.05963" + id="text11995-7-3-6-2" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + x="460.15295" + y="326.05963" + id="tspan12214">Port3</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99199999, 1.98399994;stroke-dashoffset:0;marker-start:url(#Arrow1Mstart);display:inline" + d="m 389.46894,68.26125 12.5232,0 c 1.5,0 3,1.5 3,3 l 0,21.198419" + id="path10486-06" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="3" + sodipodi:nodetypes="ccc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99999997, 1.99999998;stroke-dashoffset:0" + d="m 398.43415,91.043274 -11.52714,0 0.98804,0" + id="path12267" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Mstart)" + d="m 190.33387,103.00575 195.99267,0.97328" + id="path10486-43" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="M 187.50304,56.857383 383.49569,55.884111" + id="path10486-0-9" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99199997, 1.98399998;stroke-dashoffset:0;marker-start:url(#Arrow1Mstart);display:inline" + d="m 388.42391,103.27876 27.61666,0 c 1.5,0 3,-1.5 3,-3 l 0,-41.462569" + id="path10486-06-7" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="3" + sodipodi:nodetypes="ccc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99999997, 1.99999994;stroke-dashoffset:0" + d="m 417.31173,56.402625 -26.65144,0 2.2844,0" + id="path12267-7" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Mstart)" + d="m 186.54351,319.92933 195.99264,0.83313" + id="path10486-07" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="m 185.45351,344.49822 195.99262,-0.97328" + id="path10486-0-3" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99199997, 1.98399993;stroke-dashoffset:0;marker-start:url(#Arrow1Mstart);display:inline" + d="m 387.7334,321.21965 12.52321,0 c 1.5,0 3,1.5 3,3 l 0,21.19843" + id="path10486-06-4" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="3" + sodipodi:nodetypes="ccc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99999997, 1.99999994;stroke-dashoffset:0" + d="m 396.69862,344.00166 -11.52714,0 0.98804,0" + id="path12267-9" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Mstart)" + d="m 190.33918,355.96416 195.99266,0.97327" + id="path10486-43-6" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:none;marker-start:none;marker-mid:none;marker-end:url(#Arrow1Mend)" + d="m 185.76751,309.8158 195.99266,-0.97331" + id="path10486-0-9-5" + inkscape:connector-type="polyline" + inkscape:connector-curvature="3" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.99199992;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99199997, 1.98399994;stroke-dashoffset:0;marker-start:url(#Arrow1Mstart);display:inline" + d="m 386.68838,356.23716 27.61666,0 c 1.5,0 3,-1.5 3,-3 l 0,-41.46255" + id="path10486-06-7-1" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="3" + sodipodi:nodetypes="ccc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:3;stroke-opacity:1;stroke-dasharray:0.99999995, 1.99999991;stroke-dashoffset:0" + d="m 415.57618,309.36103 -26.65143,0 2.28441,0" + id="path12267-7-0" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="168.01332" + y="295.95398" + id="text11995-9" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + x="168.01332" + y="295.95398" + id="tspan12218-9">Port2</tspan></text> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="166.67253" + y="62.774006" + id="text11995-7-6" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + id="tspan11997-3-8" + x="166.67253" + y="62.774006">Port0</tspan></text> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="167.07025" + y="94.231415" + id="text11995-7-8-3" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + id="tspan11997-3-3-4" + x="167.07025" + y="94.231415">Port1</tspan></text> + <text + xml:space="preserve" + style="font-size:11.9913578px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="167.75591" + y="329.09506" + id="text11995-7-3-6-2-8" + sodipodi:linespacing="125%" + transform="scale(0.91971036,1.0872988)"><tspan + sodipodi:role="line" + x="167.75591" + y="329.09506" + id="tspan12214-4">Port3</tspan></text> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/example_rules.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/example_rules.png Binary files differnew file mode 100644 index 000000000..4019b7dfa --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/example_rules.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/exception_path_example.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/exception_path_example.svg new file mode 100644 index 000000000..c19c5808e --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/exception_path_example.svg @@ -0,0 +1,102 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2014 Intel Corporation --> + +<svg xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="330.16" width="568.88" version="1.0" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <g transform="matrix(0.136 0 0 .13439 -6.7377 -69.183)"> + <path d="m3004.4 514.8v2456.8h1228v-2456.8h-1228z" fill-rule="evenodd" fill="#fcd5b5"/> + <path d="m1359.6 514.8v2456.8h1644.8v-2456.8h-1644.8z" fill-rule="evenodd" fill="#d9d9d9"/> + <path d="m1096 1460v188.8h415.2v-188.8h-415.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m1089.2 1460c0-3.7 3.1-6.8 6.8-6.8h415.2c3.8 0 6.8 3.1 6.8 6.8v188.8c0 3.8-3 6.8-6.8 6.8h-415.2c-3.7 0-6.8-3-6.8-6.8zm13.6 188.8-6.8-6.8h415.2l-6.8 6.8v-188.8l6.8 6.8h-415.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1594.3033" x="1175.8816" font-family="Arial" fill="#ffffff">Port0</text> + <path d="m1096 1762.4v188.8h415.2v-188.8h-415.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m1089.2 1762.4c0-3.7 3.1-6.8 6.8-6.8h415.2c3.8 0 6.8 3.1 6.8 6.8v188.8c0 3.8-3 6.8-6.8 6.8h-415.2c-3.7 0-6.8-3-6.8-6.8zm13.6 188.8-6.8-6.8h415.2l-6.8 6.8v-188.8l6.8 6.8h-415.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1896.6992" x="1175.8816" font-family="Arial" fill="#ffffff">Port1</text> + <path d="m1096 2177.6v189.6h415.2v-189.6h-415.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m1089.2 2177.6c0-3.7 3.1-6.8 6.8-6.8h415.2c3.8 0 6.8 3.1 6.8 6.8v189.6c0 3.8-3 6.8-6.8 6.8h-415.2c-3.7 0-6.8-3-6.8-6.8zm13.6 189.6-6.8-6.8h415.2l-6.8 6.8v-189.6l6.8 6.8h-415.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="2312.4934" x="1167.0984" font-family="Arial" fill="#ffffff">PortN</text> + <path d="m2060 892.8v151.2h491.2v-151.2h-491.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2053.2 892.8c0-3.7 3.1-6.8 6.8-6.8h491.2c3.8 0 6.8 3.1 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-491.2c-3.7 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h491.2l-6.8 6.8v-151.2l6.8 6.8h-491.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1008.4115" x="2124.8179" font-family="Arial" fill="#ffffff">CoreA0</text> + <path d="m2060 1157.6v151.2h491.2v-151.2h-491.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2053.2 1157.6c0-3.7 3.1-6.8 6.8-6.8h491.2c3.8 0 6.8 3.1 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-491.2c-3.7 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h491.2l-6.8 6.8v-151.2l6.8 6.8h-491.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1273.0079" x="2124.8179" font-family="Arial" fill="#ffffff">CoreA1</text> + <path d="m2060 1535.2v151.2h491.2v-151.2h-491.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2053.2 1535.2c0-3.7 3.1-6.8 6.8-6.8h491.2c3.8 0 6.8 3.1 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-491.2c-3.7 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h491.2l-6.8 6.8v-151.2l6.8 6.8h-491.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1651.0026" x="2116.0015" font-family="Arial" fill="#ffffff">CoreAN</text> + <path d="m2060 2026.4v151.2h491.2v-151.2h-491.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2053.2 2026.4c0-3.7 3.1-6.8 6.8-6.8h491.2c3.8 0 6.8 3.1 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-491.2c-3.7 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h491.2l-6.8 6.8v-151.2l6.8 6.8h-491.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="2142.3958" x="2124.8179" font-family="Arial" fill="#ffffff">CoreB0</text> + <path d="m2060 2291.2v151.2h491.2v-151.2h-491.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2053.2 2291.2c0-3.7 3.1-6.8 6.8-6.8h491.2c3.8 0 6.8 3.1 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-491.2c-3.7 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h491.2l-6.8 6.8v-151.2l6.8 6.8h-491.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="2406.9922" x="2124.8179" font-family="Arial" fill="#ffffff">CoreB1</text> + <path d="m2060 2669.6v151.2h491.2v-151.2h-491.2z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2053.2 2669.6c0-3.7 3.1-6.8 6.8-6.8h491.2c3.8 0 6.8 3.1 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-491.2c-3.7 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h491.2l-6.8 6.8v-151.2l6.8 6.8h-491.2l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="2784.9868" x="2116.0015" font-family="Arial" fill="#ffffff">CoreBN</text> + <path d="m2834.4 892.8v151.2h453.6v-151.2h-453.6z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2827.6 892.8c0-3.6 3.2-6.8 6.8-6.8h453.6c3.8 0 6.8 3.2 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-453.6c-3.6 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h453.6l-6.8 6.8v-151.2l6.8 6.8h-453.6l6.8-6.8z" stroke="#385d8a" stroke-width=".2px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1008.4115" x="2921.5908" font-family="Arial" fill="#ffffff">tapA0</text> + <path d="m2834.4 1157.6v151.2h453.6v-151.2h-453.6z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2827.6 1157.6c0-3.6 3.2-6.8 6.8-6.8h453.6c3.8 0 6.8 3.2 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-453.6c-3.6 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h453.6l-6.8 6.8v-151.2l6.8 6.8h-453.6l6.8-6.8z" stroke="#385d8a" stroke-width=".2px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1273.0079" x="2921.5908" font-family="Arial" fill="#ffffff">tapA1</text> + <path d="m2834.4 1535.2v151.2h453.6v-151.2h-453.6z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2827.6 1535.2c0-3.6 3.2-6.8 6.8-6.8h453.6c3.8 0 6.8 3.2 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-453.6c-3.6 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h453.6l-6.8 6.8v-151.2l6.8 6.8h-453.6l6.8-6.8z" stroke="#385d8a" stroke-width=".2px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1651.0026" x="2912.8074" font-family="Arial" fill="#ffffff">tapAN</text> + <path d="m2834.4 2026.4v151.2h453.6v-151.2h-453.6z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2827.6 2026.4c0-3.6 3.2-6.8 6.8-6.8h453.6c3.8 0 6.8 3.2 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-453.6c-3.6 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h453.6l-6.8 6.8v-151.2l6.8 6.8h-453.6l6.8-6.8z" stroke="#385d8a" stroke-width=".2px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="2142.3958" x="2921.5908" font-family="Arial" fill="#ffffff">t</text> + <text xml:space="preserve" font-size="106.4px" y="2142.3958" x="2951.1909" font-family="Arial" fill="#ffffff">apB0</text> + <path d="m2834.4 2291.2v151.2h453.6v-151.2h-453.6z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2827.6 2291.2c0-3.6 3.2-6.8 6.8-6.8h453.6c3.8 0 6.8 3.2 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-453.6c-3.6 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h453.6l-6.8 6.8v-151.2l6.8 6.8h-453.6l6.8-6.8z" stroke="#385d8a" stroke-width=".2px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="2406.9922" x="2921.5908" font-family="Arial" fill="#ffffff">tapB1</text> + <path d="m2834.4 2669.6v151.2h453.6v-151.2h-453.6z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m2827.6 2669.6c0-3.6 3.2-6.8 6.8-6.8h453.6c3.8 0 6.8 3.2 6.8 6.8v151.2c0 3.8-3 6.8-6.8 6.8h-453.6c-3.6 0-6.8-3-6.8-6.8zm13.6 151.2-6.8-6.8h453.6l-6.8 6.8v-151.2l6.8 6.8h-453.6l6.8-6.8z" stroke="#385d8a" stroke-width=".2px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="2784.9868" x="2912.8074" font-family="Arial" fill="#ffffff">tapBN</text> + <path stroke-linejoin="round" d="m1510.7 1543.2 9-0.4 7.8-1.2 8.1-1.9 7.9-2.6 8-3.3 7.9-3.9 8-4.7 7.9-5.3 7.7-5.9 7.7-6.6 7.6-7.1 7.5-7.7 7.2-8.2 7.1-8.8 6.8-9.2 6.6-9.7 6.4-10.2 6.1-10.5 5.8-10.9 5.4-11.3 5.1-11.6 4.8-11.9 4.4-12.1 4-12.4 3.7-12.6 3.1-12.7 2.8-13 2.2-13.1 1.8-13.1 1.3-13.2 0.8-13.2 0.2-13.4 0.6-13.9 1.6-14.5 2.6-14.4 3.6-14.4 4.6-14.2 5.5-14 6.4-13.8 7.2-13.5 8-13.3 8.7-13 9.5-12.7 10.2-12.4 10.8-12.1 11.6-11.6 12-11.3 12.7-10.8 13.1-10.4 13.6-9.9 14.2-9.4 14.5-8.9 15-8.3 15.3-7.8 15.7-7.1 15.9-6.5 16.3-5.9c0.6-0.2 1.2-0.4 1.9-0.5l12.2-2.1 4 22.8-12.2 2.2 1.9-0.5-15.3 5.4-15.1 6.2-14.8 6.8-14.6 7.3-14.1 7.8-13.8 8.4-13.3 8.9-12.9 9.3-12.5 9.8-11.9 10.2-11.4 10.6-10.7 10.9-10.2 11.2-9.5 11.5-8.8 11.8-8.2 12.1-7.4 12.2-6.6 12.4-5.8 12.6-5 12.6-4.2 12.7-3.3 12.8-2.4 12.9-1.5 12.8-0.5 13.5-0.3 14.2-0.8 14.2-1.4 14.1-1.9 13.9-2.5 13.9-2.9 13.7-3.4 13.6-3.8 13.3-4.3 13.2-4.6 12.9-5.1 12.6-5.5 12.3-5.9 12-6.2 11.7-6.5 11.3-6.8 10.8-7.2 10.5-7.4 10-7.7 9.5-8 9.1-8.2 8.5-8.5 7.8-8.6 7.4-9 6.8-9.1 6-9.3 5.3-9.5 4.8-9.6 3.8-9.9 3.1-9.8 2.2-10.1 1.4-9 0.4zm428.8-589.4 120.1 15-108.6 53.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m1510.6 2261.2 9.5-0.5 8.3-1.3 8.3-2.1 8.3-3 8.4-3.7 8.4-4.5 8.4-5.2 8.3-6 8.2-6.7 8.1-7.4 8-8.1 7.8-8.7 7.7-9.4 7.5-10 7.2-10.4 6.9-11 6.8-11.5 6.4-12 6.1-12.4 5.7-12.7 5.5-13.2 5-13.4 4.7-13.9 4.2-14 3.8-14.3 3.3-14.5 2.9-14.6 2.4-14.8 1.9-15 1.4-15 0.8-15 0.3-15.1 0.5-15.7 1.6-16.2 2.5-16.1 3.5-16 4.4-15.8 5.3-15.6 6.1-15.5 6.9-15.2 7.8-14.9 8.5-14.6 9.2-14.3 9.9-13.9 10.5-13.6 11.2-13.1 11.7-12.7 12.3-12.2 12.8-11.7 13.2-11.2 13.8-10.6 14.2-10.1 14.5-9.4 14.9-8.8 15.3-8.1 15.6-7.3 26.4-8.6 7.1 22.1-23.6 7.4-14.6 7-14.4 7.6-14.1 8.2-13.8 8.9-13.4 9.4-12.9 10-12.6 10.6-12.1 11-11.6 11.6-11.1 11.9-10.5 12.4-9.9 12.7-9.3 13.1-8.6 13.3-8 13.7-7.3 13.9-6.4 14.1-5.8 14.3-4.9 14.4-4.1 14.6-3.2 14.7-2.4 14.7-1.4 14.7-0.5 15.3-0.3 15.9-0.9 15.9-1.5 15.9-2 15.6-2.5 15.6-3.1 15.5-3.5 15.2-4 15-4.5 14.8-4.9 14.4-5.3 14.3-5.7 13.8-6.1 13.5-6.5 13.1-6.8 12.7-7.1 12.2-7.5 11.8-7.8 11.2-8.1 10.7-8.3 10.1-8.6 9.6-8.9 8.9-9.1 8.3-9.3 7.6-9.6 6.8-9.7 6-10 5.3-10 4.4-10.4 3.5-10.5 2.5-10.5 1.5-9.4 0.5-1.3-23.1zm428.5-661.4 120.5 11.4-106.9 56.9-13.6-68.3z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2059.2 2114.1-9.7-0.4-10.5-1.2-10.6-2.1-10.4-2.8-10.2-3.6-10.1-4.4-9.9-5-9.7-5.6-9.6-6.2-9.3-6.9-9.1-7.4-8.8-8-8.6-8.3-8.3-9-8-9.3-7.7-9.9-7.4-10.1-7-10.6-6.7-11-6.3-11.2-5.9-11.6-5.5-11.8-5.1-12.2-4.6-12.3-4.2-12.6-3.6-12.8-3.2-12.8-2.6-13.1-2.1-13.2-1.5-13.2-0.9-13.4-0.4-13.3-0.4-12.7-1.4-11.9-2.3-12-3.2-11.8-4-11.9-4.7-11.8-5.6-11.7-6.3-11.5-7.1-11.4-7.8-11.3-8.5-11-9.1-10.7-9.7-10.5-10.3-10.2-10.9-9.9-11.4-9.5-11.9-9.2-12.4-8.7-12.8-8.3-13.2-7.8-13.6-7.4-13.8-6.8-14.3-6.3-14.4-5.7-21.4-5.5 5.9-22.4 24 6.3 15.4 6.1 15 6.7 14.7 7.3 14.4 7.8 14 8.3 13.6 8.8 13 9.2 12.7 9.8 12.1 10.2 11.6 10.5 11.1 10.9 10.3 11.3 9.9 11.6 9.1 11.9 8.4 12.2 7.7 12.5 7 12.7 6.1 13 5.3 13.1 4.4 13.3 3.5 13.6 2.5 13.5 1.5 13.7 0.5 13 0.3 12.4 0.8 12.2 1.4 12.3 1.9 12.2 2.5 12.1 2.9 12.1 3.4 11.9 3.8 11.6 4.4 11.6 4.6 11.3 5.1 11.1 5.5 10.8 5.9 10.5 6.2 10.2 6.5 9.8 6.8 9.4 7.1 9.1 7.4 8.6 7.6 8.2 7.8 7.7 8 7.3 8.2 6.7 8.3 6.1 8.4 5.6 8.6 5 8.6 4.3 8.7 3.8 8.6 3.1 8.8 2.5 8.7 1.8 8.8 1.1 9.6 0.4-0.9 23.1zm-439.5-505.7-108.5-54 120.3-14.6-11.8 68.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2059.3 2378.7-10.4-0.3-11.2-1.2-11-1.9-11-2.6-10.8-3.3-10.7-4-10.6-4.7-10.2-5.1-10.3-5.8-9.9-6.4-9.7-6.8-9.4-7.4-9.2-7.8-8.9-8.3-8.6-8.8-8.2-9.1-7.9-9.5-7.6-9.9-7.1-10.2-6.8-10.5-6.4-10.8-5.9-11.2-5.4-11.3-5-11.5-4.5-11.8-3.9-11.9-3.4-12.2-2.9-12.2-2.2-12.3-1.7-12.5-1-12.5-0.3-12.6-0.4-11.8-1.4-11-2.2-11-3-11.1-3.7-11-4.6-10.8-5.3-10.9-6.1-10.7-6.7-10.6-7.4-10.4-8.1-10.2-8.7-10-9.3-9.7-9.9-9.5-10.4-9.1-10.9-8.9-11.4-8.5-11.8-8.1-12.2-7.8-12.6-7.2-13-6.8-13.3-6.4-13.6-5.9-13.9-5.3 1.6 0.4-17.4-4 5.2-22.6 17.5 4c0.5 0.2 1 0.3 1.6 0.5l14.6 5.7 14.4 6.3 14.1 6.7 13.7 7.3 13.5 7.8 12.9 8.2 12.7 8.6 12.1 9.1 11.6 9.4 11.2 9.9 10.5 10.2 10 10.5 9.4 10.9 8.7 11 8.2 11.5 7.4 11.6 6.6 11.9 5.9 12.1 5.1 12.3 4.3 12.5 3.3 12.6 2.4 12.7 1.5 12.9 0.5 12.1 0.3 11.3 0.8 11.4 1.5 11.3 2 11.3 2.6 11.2 3.1 11.1 3.7 11 4 10.8 4.6 10.7 5 10.4 5.5 10.3 5.8 10 6.2 9.8 6.7 9.4 6.9 9.1 7.3 8.8 7.6 8.5 7.9 8 8.1 7.6 8.4 7.2 8.5 6.7 8.9 6.3 8.9 5.7 9 5.3 9.4 4.6 9.2 4.2 9.4 3.5 9.5 2.9 9.5 2.4 9.5 1.7 9.5 1 10.3 0.4zm-439.7-467.7-108.4-54.2 120.3-14.4z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2059.3 2756.9-10.4-0.3-11.1-1.1-10.9-1.7-11-2.4-10.8-3.1-10.7-3.7-10.4-4.2-10.4-4.9-10.1-5.3-9.9-5.9-9.7-6.3-9.5-6.9-9.1-7.2-9-7.8-8.5-8-8.3-8.4-8-8.9-7.5-9.2-7.2-9.4-6.8-9.9-6.4-10-5.9-10.2-5.5-10.6-5-10.8-4.4-10.9-4-11.2-3.5-11.2-2.8-11.5-2.3-11.5-1.6-11.6-1.1-11.7-0.3-11.7-0.4-10.9-1.4-10-2.2-10.1-2.9-10.1-3.7-10.1-4.6-10-5.3-9.9-6-10-6.7-9.7-7.4-9.6-8-9.4-8.7-9.2-9.3-9-9.8-8.8-10.4-8.5-10.9-8.1-11.4-7.9-11.8-7.5-12.3-7.2-12.6-6.7-13-6.3-13.3-6-13.6-5.4-13.9-4.9-15.7-3.3 4.8-22.7 18.8 4.1 14.6 5.3 14.4 5.7 14.1 6.3 13.7 6.7 13.5 7.2 12.9 7.6 12.6 8 12.1 8.4 11.6 8.8 11.2 9.1 10.6 9.4 10 9.8 9.4 10.1 8.8 10.3 8.2 10.6 7.4 10.8 6.7 11.1 5.9 11.3 5.1 11.5 4.3 11.6 3.4 11.8 2.4 11.9 1.5 12.1 0.5 11.2 0.3 10.4 0.8 10.4 1.5 10.4 2 10.3 2.5 10.3 3.1 10.2 3.6 10.2 4.1 9.9 4.6 9.9 4.9 9.6 5.5 9.5 5.8 9.2 6.2 9 6.6 8.8 6.9 8.4 7.3 8.1 7.6 7.8 7.9 7.4 8 7.1 8.5 6.7 8.5 6.2 8.8 5.8 8.9 5.3 9.2 4.8 9.2 4.4 9.4 3.8 9.4 3.3 9.5 2.7 9.5 2.2 9.6 1.5 9.6 1 10.3 0.4-0.7 23.1zm-439.1-431.2-109-52.9 120.1-15.8-11.1 68.7z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2551.2 957.2h26.5 12.9l12.7 0.1h12.2 11.7 11.1l10.4 0.1h9.5 8.5l7.7 0.1h3.4 2.9 2.8l2.6 0.1h1.9 1.7l1.8 0.1 0.6-0.1 1.6 0.1h-0.8 0.2l1.6 0.1v-0.1h1.4l2 0.1h1.8 2.6 2.5 3.1l3.6 0.1h18.4v23.2h-18.8l-3.2-0.1h-3.1-2.9-2.2-2.4l-1.4-0.1h-1.4-2 0.4-0.2-1.2l0.4-0.1h-1.4-1-1.7-2.3l-2.2-0.1h-2.8-3.2-3.3l-7.5-0.1h-8.6-9.5l-10.3-0.1h-11.2-11.6-12.3l-12.6-0.1h-12.9-26.5v-23.2zm167.4-22.5 115.9 35-116.1 34.6 0.2-69.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2551.2 1221.2h26.5 12.9l12.7 0.1h12.2 11.7 11.1l10.4 0.1h9.5 8.5l7.7 0.1h3.4 2.9 2.8l2.6 0.1h1.9 1.7l1.8 0.1 0.6-0.1 1.6 0.1h-0.8 0.2l1.6 0.1v-0.1h1.4l2 0.1h1.8 2.6 2.5 3.1l3.6 0.1h18.4v23.2h-18.8l-3.2-0.1h-3.1-2.9-2.2-2.4l-1.4-0.1h-1.4-2 0.4-0.2-1.2l0.4-0.1h-1.4-1-1.7-2.3l-2.2-0.1h-2.8-3.2-3.3l-7.5-0.1h-8.6-9.5l-10.3-0.1h-11.2-11.6-12.3l-12.6-0.1h-12.9-26.5v-23.2zm167.4-22.5 115.9 35-116.1 34.6 0.2-69.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2551.2 1599.6h26.5 12.9l12.7 0.1h12.2 11.7 11.1l10.4 0.1h9.5 8.5l7.7 0.1h3.4 2.9 2.8l2.6 0.1h1.9 1.7l1.8 0.1 0.6-0.1 1.6 0.1h-0.8 0.2l1.6 0.1v-0.1h1.4l2 0.1h1.8 2.6 2.5 3.1l3.6 0.1h18.4v23.2h-18.8l-3.2-0.1h-3.1-2.9-2.2-2.4l-1.4-0.1h-1.4-2 0.4-0.2-1.2l0.4-0.1h-1.4-1-1.7-2.3l-2.2-0.1h-2.8-3.2-3.3l-7.5-0.1h-8.6-9.5l-10.3-0.1h-11.2-11.6-12.3l-12.6-0.1h-12.9-26.5v-23.2zm167.4-22.5 115.9 35-116.1 34.6 0.2-69.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2834.5 2114.9h-26.4-13l-12.6-0.1h-12.2-11.7-11.2-10.3l-9.5-0.1h-8.6-7.5l-3.5-0.1h-3-2.7-2.6-1.9l-2-0.1h-1.1-1-1.6 0.8-0.2l-1.6-0.1h-1.8-1.3-2.1l-2.6-0.1h-2.6-3.1-3.5l-18.5-0.1 0.1-23.2 18.8 0.1h3.1 3.1 3l2.2 0.1h2.1 2.1 1l2 0.1h-0.4 0.2 1.2-0.4 1l1.7 0.1h1.4 2.3 2.2 2.7 3.3l3.2 0.1h7.6 8.6l9.4 0.1h10.4 11.1 11.7 12.2l12.6 0.1h13 26.4v23.2zm-167.3 22.5-116-35 116.1-34.6-0.1 69.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2834.5 2379.7h-26.4-13l-12.6-0.1h-12.2-11.7-11.2-10.3l-9.5-0.1h-8.6-7.5l-3.5-0.1h-3-2.7-2.6-1.9l-2-0.1h-1.1-1-1.6 0.8-0.2l-1.6-0.1h-1.8-1.3-2.1l-2.6-0.1h-2.6-3.1-3.5l-18.5-0.1 0.1-23.2 18.8 0.1h3.1 3.1 3l2.2 0.1h2.1 2.1 1l2 0.1h-0.4 0.2 1.2-0.4 1l1.7 0.1h1.4 2.3 2.2 2.7 3.3l3.2 0.1h7.6 8.6l9.4 0.1h10.4 11.1 11.7 12.2l12.6 0.1h13 26.4v23.2zm-167.3 22.5-116-35 116.1-34.6-0.1 69.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m2834.5 2757.3h-26.4-13l-12.6-0.1h-12.2-11.7-11.2-10.3l-9.5-0.1h-8.6-7.5l-3.5-0.1h-3-2.7-2.6-1.9l-2-0.1h-1.1-1-1.6 0.8-0.2l-1.6-0.1h-1.8-1.3-2.1l-2.6-0.1h-2.6-3.1-3.5l-18.5-0.1 0.1-23.2 18.8 0.1h3.1 3.1 3l2.2 0.1h2.1 2.1 1l2 0.1h-0.4 0.2 1.2-0.4 1l1.7 0.1h1.4 2.3 2.2 2.7 3.3l3.2 0.1h7.6 8.6l9.4 0.1h10.4 11.1 11.7 12.2l12.6 0.1h13 26.4v23.2zm-167.3 22.5-116-35 116.1-34.6-0.1 69.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m3288.2 957.2 16.2 0.4 16.6 0.8 16.6 1.2 16.6 1.8 16.4 2.2 16.4 2.8 16.4 3.2 16.4 3.6 16.2 4.2 16.2 4.6 16 5 16 5.4 16 6 15.8 6.2 15.6 6.8 15.4 7 15.4 7.6 15.4 7.8 15 8.2 15 8.6 14.8 9 14.6 9.2 14.4 9.8 14.2 9.8 14 10.4 14 10.6 13.6 10.8 13.4 11.4 26 23 25.2 24.4 24.2 25.6 23 26.4 11.2 13.6 10.8 13.6 10.6 14 10 14.4 10 14.2 9.4 14.6 9.2 14.8 9 14.8 8.4 15.2 8.2 15.2 7.8 15.4 7.4 15.6 7 15.6 6.6 15.8 6.2 16 5.8 16 5.4 16 5 16.2 4.6 16.4 4 16.4 3.8 16.4 3 16.6 2.8 16.6 2.2 16.6 1.8 16.6 1.2 16.8 0.8 16.6 0.2 16.8v16.8l-0.8 16.8-1.2 16.8-1.8 16.6-2.2 16.8-2.8 16.6-3.2 16.4-3.6 16.6-4 16.4-4.6 16.2-5 16.2-5.4 16.2-5.8 16-6.2 16-6.6 15.8-7 15.6-7.4 15.6-7.8 15.4-8 15.4-8.6 15-8.8 15-9.2 14.6-9.6 14.6-9.8 14.4-10.2 14.2-10.4 14-10.8 13.8-11.2 13.6-22.8 26.2-24.2 25.6-25.2 24.4-26 23.2-13.6 11.2-13.4 11-14 10.6-14 10.4-14.2 9.8-14.4 9.8-14.6 9.2-14.8 9-14.8 8.6-15.2 8.2-15.2 7.8-15.4 7.6-15.4 7-15.8 6.8-15.6 6.2-16 5.8-15.8 5.6-16.2 5-16 4.6-16.4 4-26 4.6-3.8-22.8 24.2-4.2 15.6-4 15.6-4.4 15.4-4.8 15.4-5.2 15.4-5.8 15.2-6 15-6.4 15-7 15-7.2 14.8-7.6 14.6-7.8 14.4-8.4 14.4-8.8 14-9 14-9.4 13.8-9.6 13.8-10 13.4-10.2 13.2-10.8 13-10.8 25.4-22.6 24.4-23.6 23.4-24.8 22.6-25.8 10.8-13.2 10.4-13.2 10.2-13.6 9.8-13.8 9.6-14 9.4-14.2 8.8-14.2 8.6-14.6 8.2-14.6 8-14.8 7.4-15 7.2-15 6.8-15.2 6.4-15.2 6-15.4 5.6-15.6 5.2-15.6 4.8-15.6 4.4-15.6 4-15.8 3.4-16 3.2-15.8 2.6-16 2.2-16 1.6-16 1.2-16 0.8-15.8v-16.4l-0.2-16-0.6-16.2-1.2-15.8-1.8-16.2-2.2-15.8-2.6-16-3-15.8-3.4-16-4-15.8-4.4-15.8-4.8-15.6-5.2-15.6-5.6-15.4-6-15.4-6.4-15.4-6.8-15-7.2-15.2-7.4-14.8-8-14.8-8.2-14.6-8.6-14.6-9-14.4-9.2-14-9.6-14-9.8-13.8-10.2-13.6-10.6-13.4-10.6-13-22.4-25.8-23.6-24.6-24.4-23.8-25.6-22.8-13-10.8-13.2-10.6-13.4-10.4-13.6-10-14-9.6-13.8-9.4-14.2-9-14.4-8.6-14.4-8.4-14.8-8-14.6-7.6-15-7.2-15-7-15.2-6.4-15.2-6-15.2-5.8-15.4-5.2-15.6-4.8-15.6-4.6-15.6-3.8-15.6-3.6-15.8-3.2-15.8-2.6-15.8-2.2-15.8-1.6-15.8-1.4-15.8-0.6-16.2-0.4 0.2-23.2zm119.8 1432.2-119.2-22.2 111.6-47 7.6 69.2z" stroke="#c0504d" stroke-width=".2px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m3288.2 1221.2 10.8 0.2 11.2 0.6 11 0.8 11 1 22 3.2 21.8 4.2 21.8 5.6 21.4 6.6 21.2 7.4 20.8 8.8 20.6 9.6 20 10.4 19.6 11.4 19.2 12.4 18.8 13 18 13.8 17.4 14.6 16.8 15.4 16.2 15.8 15.4 16.6 14.8 17 13.8 17.8 13 18 12 18.6 11.2 19 10.2 19.6 9.2 19.6 8 20.2 7 20.2 5.8 20.6 2.6 10.4 2 10.6 1.8 10.4 1.6 10.4 1.2 10.6 0.8 10.4 0.6 10.6 0.2 10.6v10.6l-0.6 10.8-0.8 10.6-1.2 10.4-1.6 10.4-1.8 10.6-2.2 10.4-2.4 10.4-5.6 20.4-7 20.4-8 20-9.2 19.8-10.2 19.6-11 19-12.2 18.6-13 18.2-13.8 17.6-14.6 17.2-15.4 16.6-16 15.8-16.8 15.4-17.6 14.6-18 13.8-18.6 13-19 12.4-19.8 11.4-20 10.6-20.4 9.6-20.8 8.6-21.2 7.6-25.6 6-5.2-22.6 23-5.2 20.2-7.2 19.8-8.2 19.4-9.2 19.2-10 18.8-11 18.2-11.8 18-12.4 17.2-13.2 16.8-14 16-14.6 15.4-15.2 14.8-15.8 14-16.4 13.2-17 12.4-17.2 11.6-17.8 10.6-18 9.6-18.4 8.6-18.8 7.6-18.8 6.6-19.2 5.4-19.6 2.4-9.8 2-9.6 1.8-9.8 1.4-9.8 1-9.8 0.8-9.8 0.6-9.6v-10.2l-0.2-10-0.4-9.6-0.8-10-1.2-9.6-1.4-9.8-1.6-9.8-2-9.8-2.4-9.6-5.4-19.2-6.4-19.2-7.6-19-8.8-18.8-9.6-18.4-10.6-18-11.4-17.8-12.6-17.4-13-16.8-14-16.4-14.8-15.8-15.6-15.4-16-14.6-16.8-14-17.2-13.2-17.8-12.6-18.4-11.6-18.8-11-19.2-10-19.4-9.2-19.8-8.2-20.2-7.4-20.4-6.2-20.4-5.2-20.8-4-20.8-3-10.4-1.2-10.4-0.6-10.6-0.6-10.6-0.2 0.2-23.2zm120.6 897.8-120-17 109.4-51.8 10.6 68.8z" stroke="#c0504d" stroke-width=".2px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m3288.2 1599.6 14.8 0.2 14.8 0.8 15 1 14.8 1.4 15 1.8 14.8 2.2 14.8 2.6 14.6 3 14.6 3.4 14.8 3.8 14.4 4 14.4 4.4 14.4 4.8 14.2 5.2 14 5.4 14.2 5.6 13.8 6.2 13.8 6.4 27 13.6 26.6 14.8 25.8 15.8 25.2 17 24.4 18 23.6 18.8 22.8 19.8 22 20.8 20.8 21.4 19.8 22.2 18.6 22.8 9 12 8.8 11.8 8.2 12 8 12.2 7.8 12.2 7.4 12.4 7 12.6 6.6 12.6 6.4 12.8 6 12.8 5.6 13 5.2 13 5 13.2 4.6 13.2 4 13.4 3.8 13.2 3.2 13.6 3 13.4 2.4 13.6 2 13.4 1.6 13.6 1.2 13.8 0.6 13.6 0.4 13.6-0.2 13.8-0.6 13.8-1 13.6-1.8 13.8-2 13.4-2.4 13.6-3 13.6-3.2 13.4-3.8 13.4-4 13.4-4.6 13.2-4.8 13.2-5.2 13-5.8 13-5.8 12.8-6.4 12.8-6.8 12.8-7 12.4-7.4 12.4-7.6 12.4-8 12.2-8.4 12-17.4 23.4-18.8 23-19.6 22.2-21 21.4-21.8 20.8-22.6 19.8-23.6 18.8-24.4 18-25.2 17-25.8 16-26.6 14.8-27 13.6-14 6.4-13.8 6.2-14 5.6-14.2 5.4-14.2 5.2-14.2 4.8-14.4 4.4-14.6 4-14.6 3.8-14.6 3.2-13.8 1.8-2.6-23 11.2-1.4 14.2-3.2 14-3.4 14-4 13.8-4.2 13.8-4.6 13.8-5 13.6-5.2 13.4-5.6 13.6-5.8 13.2-6.2 26.2-13.2 25.6-14.2 25-15.4 24.4-16.4 23.6-17.4 23-18.4 22-19.2 21-20 20.2-20.6 19.2-21.6 18-22 17-22.8 8-11.6 7.8-11.8 7.4-11.8 7-11.8 6.8-12.2 6.4-12 6-12.2 5.8-12.4 5.4-12.4 5-12.6 4.8-12.4 4.2-12.8 4-12.6 3.4-12.8 3.2-12.8 2.8-12.6 2.4-13 1.8-12.8 1.6-12.8 1-13 0.6-12.6 0.2-13.4-0.2-12.8-0.8-13-1-12.8-1.4-12.8-2-13-2.4-12.8-2.6-12.8-3.2-12.6-3.6-12.8-3.8-12.8-4.4-12.6-4.6-12.6-5-12.4-5.4-12.4-5.8-12.4-6-12.2-6.4-12.2-6.8-12-7.2-11.8-7.4-12-7.6-11.6-8-11.6-8.4-11.4-8.4-11.2-18.2-22.2-19-21.4-20.2-20.8-21.2-20-22-19.2-22.8-18.4-23.8-17.4-24.2-16.4-25.2-15.4-25.6-14.4-26.4-13.2-13.2-6.2-13.6-5.8-13.4-5.6-13.8-5.2-13.6-5-13.8-4.6-14-4.2-13.8-4-14.2-3.6-14-3.2-14.2-2.8-14-2.6-14.4-2.2-14.2-1.8-14.2-1.2-14.4-1.2-14.2-0.6-14.6-0.2 0.2-23.2zm119.6 1167.6-119.2-22.2 111.6-47 7.6 69.2z" stroke="#c0504d" stroke-width=".2px" fill="#c0504d"/> + <text xml:space="preserve" font-size="106.4px" y="642.81787" x="3128.3906" font-family="Arial" fill="#000000">Kernel</text> + <text xml:space="preserve" font-size="106.4px" y="642.81787" x="3436.3906" font-family="Arial" fill="#000000">-</text> + <text xml:space="preserve" font-size="106.4px" y="642.81787" x="3471.5906" font-family="Arial" fill="#000000">space: </text> + <text xml:space="preserve" font-size="106.4px" y="770.81787" x="3128.3906" font-family="Arial" fill="#000000">bridging/forwarding</text> + <path d="m56.8 1422.4v906.4h604.8v-906.4h-604.8z" fill-rule="evenodd" fill="#4f81bd"/> + <path stroke-linejoin="round" d="m50 1422.4c0-3.7 3.1-6.8 6.8-6.8h604.8c3.8 0 6.8 3.1 6.8 6.8v906.4c0 3.8-3 6.8-6.8 6.8h-604.8c-3.7 0-6.8-3-6.8-6.8zm13.6 906.4-6.8-6.8h604.8l-6.8 6.8v-906.4l6.8 6.8h-604.8l6.8-6.8z" stroke="#385d8a" stroke-width=".1px" fill="#385d8a"/> + <text xml:space="preserve" font-size="106.4px" y="1851.599" x="213.47821" font-family="Arial" fill="#ffffff">Traffic </text> + <text xml:space="preserve" font-size="106.4px" y="1979.599" x="119.07821" font-family="Arial" fill="#ffffff">Generator</text> + <path stroke-linejoin="round" d="m1510.7 1845 9.9-0.5 8.8-1.2 8.9-2.1 9-2.7 8.8-3.6 8.8-4.2 8.9-5 8.7-5.7 8.6-6.4 8.6-6.9 8.3-7.7 8.2-8.2 8.1-8.8 7.8-9.4 7.5-9.8 7.3-10.4 7.1-10.8 6.7-11.3 6.3-11.6 6.1-12 5.6-12.4 5.3-12.6 4.8-13 4.4-13.2 4-13.4 3.5-13.6 3-13.8 2.5-13.9 2-14 1.5-14.1 0.8-14.1 0.3-14.1 0.5-14.8 1.5-15.3 2.4-15.3 3.5-15.1 4.2-15 5.3-14.8 5.9-14.6 6.8-14.4 7.5-14.1 8.3-13.8 9-13.5 9.6-13.1 10.3-12.8 10.8-12.4 11.4-12 11.9-11.5 12.5-11.1 12.9-10.5 13.4-10.1 13.8-9.5 14.1-8.8 14.6-8.3 14.9-7.6 15.1-7l1.5-0.6 21.5-6.3 6.6 22.2-21.5 6.4 1.6-0.6-14.3 6.5-14 7.2-13.7 7.7-13.3 8.4-13 8.9-12.6 9.4-12.2 10-11.7 10.3-11.3 10.9-10.8 11.2-10.2 11.7-9.6 11.9-9 12.4-8.4 12.6-7.7 12.8-7 13.1-6.3 13.2-5.6 13.5-4.7 13.5-4 13.7-3.2 13.8-2.2 13.7-1.4 13.9-0.5 14.4-0.3 15.1-1 15.1-1.5 14.9-2.1 14.9-2.7 14.8-3.2 14.6-3.7 14.4-4.3 14.2-4.6 14-5.2 13.7-5.6 13.3-6 13.2-6.4 12.8-6.8 12.4-7.2 11.9-7.5 11.6-7.8 11.1-8.2 10.7-8.5 10-8.8 9.6-8.9 9-9.4 8.4-9.4 7.7-9.8 7.2-10 6.4-10.1 5.7-10.4 4.9-10.6 4.1-10.6 3.3-10.8 2.3-10.9 1.5-9.9 0.4zm428.4-623.6 120.5 11.4-106.9 56.9z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <text xml:space="preserve" font-size="106.4px" y="642.81787" x="1503.0131" font-family="Arial" fill="#000000">User</text> + <text xml:space="preserve" font-size="106.4px" y="642.81787" x="1727.8131" font-family="Arial" fill="#000000">-</text> + <text xml:space="preserve" font-size="106.4px" y="642.81787" x="1763.0131" font-family="Arial" fill="#000000">space:</text> + <text xml:space="preserve" font-size="106.4px" y="770.81787" x="1503.0131" font-family="Arial" fill="#000000">DPDK application</text> + <path stroke-linejoin="round" d="m768.1 1596.7 215-31.4 3.4 22.9-215.1 31.5-3.3-23zm18.2 44.3-119.9-17.7 109.8-51.2 10.1 68.9zm182-97 119.8 17.6-109.7 51.3-10.1-68.9z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m772.4 1853.1 212.9-0.4v23.2l-212.8 0.4-0.1-23.2zm11.7 46.4-116.1-34.6 116-35 0.1 69.6zm189.5-70 116.1 34.5-115.9 35.1-0.2-69.6z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path stroke-linejoin="round" d="m775.2 2204.9 216.5 41.1-4.3 22.8-216.5-41.1 4.3-22.8zm2.7 47.8-107.5-55.9 120.5-12.5-13 68.4zm206.7-31.7 107.5 55.8-120.4 12.6 12.9-68.4z" stroke="#c0504d" stroke-width=".1px" fill="#c0504d"/> + <path d="m1266 1988.4v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m1266 2064.4v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m1266 2139.6v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m2286 1346v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m2286 1422v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m2286 1497.2v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m2286 2479.6v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m2286 2555.6v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m2286 2630.8v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m3042 1346v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m3042 1422v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m3042 1497.2v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m3042 2479.6v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m3042 2555.6v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + <path d="m3042 2630.8v24h24.8v-24h-24.8z" fill-rule="evenodd" fill="#1f497d"/> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/ipsec_endpoints.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ipsec_endpoints.svg new file mode 100644 index 000000000..e4aba4ccb --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ipsec_endpoints.svg @@ -0,0 +1,850 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:osb="http://www.openswatchbook.org/uri/2009/osb" + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="155.68507mm" + height="76.061203mm" + viewBox="0 0 551.64003 269.50821" + id="svg2" + version="1.1" + inkscape:version="0.91 r13725" + sodipodi:docname="endpoints.svg"> + <defs + id="defs4"> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker18451" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path18453" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker18273" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path18275" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10612" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Send"> + <path + transform="matrix(-0.3,0,0,-0.3,0.69,0)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#cfce37;fill-opacity:1;fill-rule:evenodd;stroke:#cfce37;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path10614" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10270" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lstart"> + <path + transform="matrix(0.8,0,0,0.8,10,0)" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path10272" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Sstart" + orient="auto" + refY="0" + refX="0" + id="marker9102" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path9104" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#035002;fill-opacity:1;fill-rule:evenodd;stroke:#035002;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.2,0,0,0.2,1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Sstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Sstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5945" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#d4ce00;fill-opacity:1;fill-rule:evenodd;stroke:#d4ce00;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.2,0,0,0.2,1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <linearGradient + id="linearGradient11398" + osb:paint="gradient"> + <stop + style="stop-color:#000000;stop-opacity:1;" + offset="0" + id="stop11400" /> + <stop + style="stop-color:#000000;stop-opacity:0;" + offset="1" + id="stop11402" /> + </linearGradient> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9540" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path9542" + inkscape:connector-curvature="0" /> + </marker> + <linearGradient + id="linearGradient9336" + osb:paint="solid"> + <stop + style="stop-color:#dc181a;stop-opacity:1;" + offset="0" + id="stop9338" /> + </linearGradient> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="marker7350" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path7352" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path4871" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker7092" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path7094" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5822" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path5824" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker5369" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path5371" + style="fill:#035002;fill-opacity:1;fill-rule:evenodd;stroke:#035002;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="TriangleOutL" + orient="auto" + refY="0" + refX="0" + id="TriangleOutL" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5013" + d="m 5.77,0 -8.65,5 0,-10 8.65,5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="scale(0.8,0.8)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Tail" + orient="auto" + refY="0" + refX="0" + id="Tail" + style="overflow:visible" + inkscape:isstock="true"> + <g + id="g4907" + transform="scale(-1.2,-1.2)" + style="fill:#528ac6;fill-opacity:1;stroke:#000000;stroke-opacity:1"> + <path + id="path4909" + d="M -3.8048674,-3.9585227 0.54352094,0" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.80000001;stroke-linecap:round;stroke-opacity:1" + inkscape:connector-curvature="0" /> + <path + id="path4911" + d="M -1.2866832,-3.9585227 3.0617053,0" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.80000001;stroke-linecap:round;stroke-opacity:1" + inkscape:connector-curvature="0" /> + <path + id="path4913" + d="M 1.3053582,-3.9585227 5.6537466,0" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.80000001;stroke-linecap:round;stroke-opacity:1" + inkscape:connector-curvature="0" /> + <path + id="path4915" + d="M -3.8048674,4.1775838 0.54352094,0.21974226" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.80000001;stroke-linecap:round;stroke-opacity:1" + inkscape:connector-curvature="0" /> + <path + id="path4917" + d="M -1.2866832,4.1775838 3.0617053,0.21974226" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.80000001;stroke-linecap:round;stroke-opacity:1" + inkscape:connector-curvature="0" /> + <path + id="path4919" + d="M 1.3053582,4.1775838 5.6537466,0.21974226" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.80000001;stroke-linecap:round;stroke-opacity:1" + inkscape:connector-curvature="0" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="Arrow2Send" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4904" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0" + refX="0" + id="Arrow1Send" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4886" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#528ac6;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.2,0,0,-0.2,-1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path4874" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5734-4" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path5736-7" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5734-4-3" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path5736-7-7" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5734-4-4" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path5736-7-1" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker5369-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5371-1" + style="fill:#d4ce00;fill-opacity:1;fill-rule:evenodd;stroke:#d4ce00;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker5369-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5371-9" + style="fill:#38782d;fill-opacity:1;fill-rule:evenodd;stroke:#38782d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker5369-3-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5371-1-0" + style="fill:#38782d;fill-opacity:1;fill-rule:evenodd;stroke:#38782d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker18451-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path18453-7" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-7" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path4871-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="marker7350-9" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path7352-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="2.8" + inkscape:cx="258.01346" + inkscape:cy="122.99559" + inkscape:document-units="cm" + inkscape:current-layer="layer1" + showgrid="false" + inkscape:window-width="1920" + inkscape:window-height="1017" + inkscape:window-x="1592" + inkscape:window-y="-8" + inkscape:window-maximized="1" + fit-margin-top="3" + fit-margin-right="1" + fit-margin-bottom="1" + fit-margin-left="1" /> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(194.35761,4.2637303)"> + <rect + style="opacity:1;fill:#528ac6;fill-opacity:1;stroke:none;stroke-opacity:1" + id="rect4140-5" + width="131.15564" + height="102.30846" + x="-179.57692" + y="140.22072" + ry="7.155364" /> + <rect + style="opacity:1;fill:#528ac6;fill-opacity:1;stroke:none;stroke-opacity:1" + id="rect4140-5-1" + width="131.15564" + height="102.30846" + x="213.55591" + y="140.22072" + ry="6.5590839" /> + <rect + style="opacity:1;fill:#528ac6;fill-opacity:1;stroke:none;stroke-opacity:1" + id="rect4140-5-2" + width="131.15564" + height="46.258114" + x="-179.57692" + y="6.3661909" + ry="5.3665228" /> + <rect + style="opacity:1;fill:#528ac6;fill-opacity:1;stroke:none;stroke-opacity:1" + id="rect4140-5-2-7" + width="131.15564" + height="46.258114" + x="213.55591" + y="6.3661909" + ry="6.5590839" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.07670021px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="-182.36929" + y="188.59424" + id="text4210" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan4212" + x="-182.36929" + y="188.59424">ep0</tspan><tspan + sodipodi:role="line" + x="-182.36929" + y="208.69011" + id="tspan4214" /><tspan + sodipodi:role="line" + x="-182.36929" + y="228.786" + id="tspan4216" /></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.07670021px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="324.621" + y="188.49612" + id="text4218" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan4220" + x="324.621" + y="188.49612">ep1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.07670021px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="-159.97678" + y="32.860863" + id="text4222" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan4224" + x="-159.97678" + y="32.860863">traffic gen</tspan><tspan + sodipodi:role="line" + x="-159.97678" + y="52.956738" + id="tspan4226" /></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.07670021px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="249.77068" + y="32.860863" + id="text4222-1" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan4224-1" + x="249.77068" + y="32.860863">traffic gen</tspan><tspan + sodipodi:role="line" + x="249.77068" + y="52.956738" + id="tspan4226-6" /></text> + <path + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.60767007px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend)" + d="m -48.997848,184.52811 c 260.957198,-0.5963 260.957198,-0.5963 260.957198,-0.5963" + id="path5305" + inkscape:connector-curvature="0" /> + <path + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.60767007px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker10270)" + d="M -47.641399,212.81116 C 213.3158,212.21487 213.3158,212.21487 213.3158,212.21487" + id="path5305-2" + inkscape:connector-curvature="0" /> + <path + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.60767007px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker7350)" + d="m 262.8239,53.615939 c 0,85.864371 0,85.864371 0,85.864371" + id="path5922-0-6" + inkscape:connector-curvature="0" /> + <path + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.60767007px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart)" + d="m -146.8568,54.80849 c 0,85.86437 0,85.86437 0,85.86437" + id="path5922-0-1" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:12.5px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="246.78571" + y="500.57648" + id="text7750" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan7752" + x="246.78571" + y="500.57648" /></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="-158.9595" + y="151.84753" + id="text7754" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7756" + x="-158.9595" + y="151.84753">2</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="-110.66317" + y="151.84753" + id="text7758" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7760" + x="-110.66317" + y="151.84753">3</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="267.07309" + y="151.84753" + id="text7762" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7764" + x="267.07309" + y="151.84753">2</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="318.81448" + y="151.84753" + id="text7766" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7768" + x="318.81448" + y="151.84753">3</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="225.86406" + y="183.73483" + id="text7770" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7772" + x="225.86406" + y="183.73483">0</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="226.18214" + y="211.56969" + id="text7774" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7776" + x="226.18214" + y="211.56969">1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:11.25px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="311.78571" + y="505.93362" + id="text7778" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan7780" + x="311.78571" + y="505.93362" /></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="-65.239075" + y="183.73483" + id="text7782" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7784" + x="-65.239075" + y="183.73483">0</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:18.08628845px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="-64.921005" + y="211.56969" + id="text7786" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7788" + x="-64.921005" + y="211.56969">1</tspan></text> + <path + style="fill:none;fill-rule:evenodd;stroke:#dc181a;stroke-width:1.60606241;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:3.21212486, 9.63637456;stroke-dashoffset:0;stroke-opacity:1" + d="M -190.00162,248.33118 C -74.638406,174.62535 10.240093,56.871479 81.499676,57.515772 176.72224,59.019701 245.16177,197.27679 352.92565,246.54234" + id="path7850" + inkscape:connector-curvature="0" + sodipodi:nodetypes="ccc" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.07670021px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:center;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:middle;fill:#dc181a;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="84.985992" + y="98.449951" + id="text7876" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7878" + x="84.985992" + y="98.449951">UNPROTECTED</tspan><tspan + sodipodi:role="line" + x="84.985992" + y="118.54583" + id="tspan7886">cipher-text</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.07670021px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:center;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:middle;fill:#dc181a;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="84.899643" + y="20.937273" + id="text7880" + sodipodi:linespacing="125%" + inkscape:transform-center-x="48.100163" + inkscape:transform-center-y="3.577681" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan7882" + x="84.899643" + y="20.937273">PROTECTED</tspan><tspan + sodipodi:role="line" + x="84.899643" + y="41.03315" + id="tspan7884">clear-text</tspan></text> + <path + style="fill:none;fill-opacity:1;fill-rule:evenodd;stroke:#035002;stroke-width:3.42781782;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker5369)" + d="m -101.57521,164.71986 c -0.0232,13.22578 17.528492,19.5572 32.486212,20.12768" + id="path9370" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#d4ce00;stroke-width:3.08351111;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker5369-3)" + d="m 231.46374,184.56991 c 19.27508,-1.72507 24.47287,-7.77038 31.06883,-19.63365" + id="path9370-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.8206749px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="60.45417" + y="251.78688" + id="text14635" + sodipodi:linespacing="125%" + transform="scale(0.9630889,1.0383257)"><tspan + sodipodi:role="line" + id="tspan14637" + x="60.45417" + y="251.78688">outbound</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.07670021px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="62.073208" + y="232.23116" + id="text14635-4" + sodipodi:linespacing="125%" + transform="scale(0.96291595,1.0385122)"><tspan + sodipodi:role="line" + id="tspan14637-2" + x="62.073208" + y="232.23116">inbound</tspan></text> + <path + style="fill:none;fill-rule:evenodd;stroke:#d4ce00;stroke-width:3.8278625;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow1Sstart)" + d="m -144.07995,171.20523 c 15.90052,30.60035 40.80354,39.04445 75.670299,43.29761" + id="path9370-1" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;fill-opacity:1;fill-rule:evenodd;stroke:#035002;stroke-width:3.73783302;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#marker9102)" + d="m 236.99654,212.30897 c 40.08586,-4.29157 62.39356,-19.33069 76.11102,-48.84339" + id="path9370-8-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#d4ce00;stroke-width:4.01917505;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + d="m 32.274837,235.80819 c 18.037567,-0.2236 20.456397,-0.29813 20.456397,-0.29813" + id="path10604" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#035002;stroke-width:4.01917505;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + d="m 32.274837,256.82674 c 18.037567,-0.22362 20.456397,-0.29814 20.456397,-0.29814" + id="path10604-4" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.60767007px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-7)" + d="m -100.78618,138.21108 c 0,-85.864363 0,-85.864363 0,-85.864363" + id="path5922-0-1-5" + inkscape:connector-curvature="0" /> + <path + style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.60767007px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker7350-9)" + d="m 312.42811,137.49681 c 0,-85.864382 0,-85.864382 0,-85.864382" + id="path5922-0-6-9" + inkscape:connector-curvature="0" /> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/ipv4_acl_rule.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ipv4_acl_rule.png Binary files differnew file mode 100644 index 000000000..017e88150 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ipv4_acl_rule.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/kernel_nic.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/kernel_nic.png Binary files differnew file mode 100644 index 000000000..bffb5c1db --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/kernel_nic.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_benchmark_setup.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_benchmark_setup.svg new file mode 100644 index 000000000..470226fe8 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_benchmark_setup.svg @@ -0,0 +1,520 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2011 Intel Corporation --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="766.95715" + height="494.77206" + id="svg2" + version="1.1"> + <defs + id="defs4" /> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + showgrid="false" + fit-margin-top="0.1" + fit-margin-left="0.1" + fit-margin-right="0.1" + fit-margin-bottom="0.1" + units="cm" /> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title /> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-135.8992,-434.6983)"> + <path + inkscape:connector-curvature="0" + id="4" + style="fill:none;stroke:#000000;stroke-width:2.12513781;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + d="m 140.50508,922.27019 123.45,0 0,-482.96601 -123.45,0 0,482.96601 z" /> + <text + xml:space="preserve" + x="157.20508" + y="692.83075" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;font-family:Arial;-inkscape-font-specification:Arial" + id="text4026" + sodipodi:linespacing="103%" /> + <text + id="6" + xml:space="preserve" + x="256.40509" + y="692.83075" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;font-family:Arial;-inkscape-font-specification:Arial" + sodipodi:linespacing="103%" /> + <path + inkscape:connector-curvature="0" + id="130" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 265.37162,514.23178 508.66341,0" /> + <path + inkscape:connector-curvature="0" + id="131" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 273.60035,514.35422 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <text + xml:space="preserve" + style="font-size:20px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="162.14893" + y="885.45764" + id="text4028" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + id="tspan4030" + x="162.14893" + y="885.45764" + style="font-size:20px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;font-family:Arial;-inkscape-font-specification:Arial">Traffic</tspan><tspan + sodipodi:role="line" + x="162.14893" + y="906.05762" + id="tspan4032" + style="font-size:20px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;font-family:Arial;-inkscape-font-specification:Arial">Generator</tspan></text> + <path + inkscape:connector-curvature="0" + id="4-6" + style="fill:none;stroke:#000000;stroke-width:2.12513781;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + d="m 774.8005,924.8645 123.45,0 0,-482.96601 -123.45,0 0,482.96601 z" /> + <text + xml:space="preserve" + style="font-size:20px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="796.44434" + y="888.05188" + id="text4028-8" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + id="tspan4030-2" + x="796.44434" + y="888.05188" + style="font-size:20px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;font-family:Arial;-inkscape-font-specification:Arial">NUT</tspan><tspan + sodipodi:role="line" + x="796.44434" + y="908.65186" + id="tspan4032-0" + style="font-size:20px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;font-family:Arial;-inkscape-font-specification:Arial">(RTE)</tspan></text> + <path + inkscape:connector-curvature="0" + id="130-3" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 264.42725,533.68597 508.66341,0" /> + <path + inkscape:connector-curvature="0" + id="131-2" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 519.81423,513.99534 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-7" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 775.11271,514.27808 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <rect + style="fill:#ffffff;fill-opacity:1;stroke:none" + id="rect4380" + width="57.578693" + height="29.294424" + x="358.14981" + y="499.76208" + rx="23.942125" + ry="23.942125" /> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="362.21854" + y="519.86017" + id="text4028-9" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="362.21854" + y="519.86017" + id="tspan4032-5" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">Flow 0</tspan></text> + <path + inkscape:connector-curvature="0" + id="131-2-1" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 509.92292,533.82453 10,5 c -1.6,-3.15 -1.6,-6.85 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-0" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 263.67299,533.64775 10,5 c -1.6,-3.15 -1.6,-6.85 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-7" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 764.48137,533.82453 10,5 c -1.6,-3.15 -1.6,-6.85 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="130-31" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 264.7107,569.61938 508.6634,0" /> + <path + inkscape:connector-curvature="0" + id="131-1" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 272.93943,569.74182 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <path + inkscape:connector-curvature="0" + id="130-3-3" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 263.76633,589.07357 508.66337,0" /> + <path + inkscape:connector-curvature="0" + id="131-2-5" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 519.1533,569.38294 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-7-7" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 774.4518,569.66568 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <rect + style="fill:#ffffff;fill-opacity:1;stroke:none" + id="rect4380-6" + width="57.578693" + height="29.294424" + x="357.48889" + y="555.14972" + rx="23.942125" + ry="23.942125" /> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="361.55762" + y="575.2478" + id="text4028-9-8" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="361.55762" + y="575.2478" + id="tspan4032-5-5" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">Flow 1</tspan></text> + <path + inkscape:connector-curvature="0" + id="131-2-1-8" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 509.262,589.21214 10,5 c -1.6,-3.15001 -1.6,-6.85001 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-0-7" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 263.01207,589.03535 10,5 c -1.6,-3.15 -1.6,-6.85 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-7-9" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 765.94172,588.50503 10,5 c -1.6,-3.15001 -1.6,-6.85001 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="130-34" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 266.17109,647.92558 508.66341,0" /> + <path + inkscape:connector-curvature="0" + id="131-9" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 274.39982,648.04802 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <path + inkscape:connector-curvature="0" + id="130-3-5" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 265.22672,667.37977 508.66338,0" /> + <path + inkscape:connector-curvature="0" + id="131-2-11" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 520.6137,647.68914 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-7-6" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 775.9122,647.97188 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <rect + style="fill:#ffffff;fill-opacity:1;stroke:none" + id="rect4380-8" + width="57.578693" + height="29.294424" + x="358.94928" + y="633.45587" + rx="23.942125" + ry="23.942125" /> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="363.01801" + y="653.55396" + id="text4028-9-1" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="363.01801" + y="653.55396" + id="tspan4032-5-1" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">Flow 2</tspan></text> + <path + inkscape:connector-curvature="0" + id="131-2-1-2" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 510.7224,667.51834 10,5 c -1.6,-3.15001 -1.6,-6.85001 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-0-9" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 264.47246,667.34155 10,5 c -1.6,-3.15 -1.6,-6.85 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-7-96" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 765.2808,667.51834 10,5 c -1.6,-3.15001 -1.6,-6.85001 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="130-31-4" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 265.51017,703.31318 508.66343,0" /> + <path + inkscape:connector-curvature="0" + id="131-1-4" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 273.7389,703.43562 -10,5 c 1.6,-3.14999 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <path + inkscape:connector-curvature="0" + id="130-3-3-9" + style="fill:none;stroke:#000000;stroke-width:2.08607888px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + d="m 264.5658,722.76737 508.6634,0" /> + <path + inkscape:connector-curvature="0" + id="131-2-5-9" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 519.9528,703.07674 -10,5 c 1.6,-3.14999 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-7-7-2" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 775.2513,703.35948 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" /> + <rect + style="fill:#ffffff;fill-opacity:1;stroke:none" + id="rect4380-6-5" + width="57.578693" + height="29.294424" + x="358.28836" + y="688.84351" + rx="23.942125" + ry="23.942125" /> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="362.35709" + y="708.94159" + id="text4028-9-8-2" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="362.35709" + y="708.94159" + id="tspan4032-5-5-5" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">Flow 3</tspan></text> + <path + inkscape:connector-curvature="0" + id="131-2-1-8-3" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 510.0615,722.90594 10,5 c -1.6,-3.15001 -1.6,-6.85001 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-0-7-9" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 263.81154,722.72915 10,5 c -1.6,-3.15 -1.6,-6.85 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="131-2-1-7-9-7" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 764.6199,722.90594 10,5 c -1.6,-3.15001 -1.6,-6.85001 0,-10 l 0,0 -10,5 z" /> + <path + inkscape:connector-curvature="0" + id="130-38" + style="fill:none;stroke:#000000;stroke-width:1.95051968;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.90103936, 3.90103936;stroke-dashoffset:0" + d="m 776.77917,533.43352 18.93346,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-5" + style="fill:none;stroke:#000000;stroke-width:1.95051968;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.90103936, 3.90103936;stroke-dashoffset:0" + d="m 776.27809,570.08799 18.93346,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-4" + style="fill:none;stroke:#000000;stroke-width:2.03578472;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.07156944, 4.07156944;stroke-dashoffset:0" + d="m 795.76128,534.45831 0,35.11152" /> + <path + inkscape:connector-curvature="0" + id="130-38-51" + style="fill:none;stroke:#000000;stroke-width:1.95051968;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.90103936, 3.90103936;stroke-dashoffset:0" + d="m 776.36881,667.40751 18.93347,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-5-0" + style="fill:none;stroke:#000000;stroke-width:1.95051968;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.90103936, 3.90103936;stroke-dashoffset:0" + d="m 775.86773,704.06198 18.93347,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-4-3" + style="fill:none;stroke:#000000;stroke-width:2.03578472;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.07156944, 4.07156944;stroke-dashoffset:0" + d="m 795.35093,668.4323 0,35.11152" /> + <path + inkscape:connector-curvature="0" + id="130-38-7" + style="fill:none;stroke:#000000;stroke-width:2.03061938;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.06123936, 4.06123936;stroke-dashoffset:0" + d="m 776.04259,515.37859 52.63775,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-7-7" + style="fill:none;stroke:#000000;stroke-width:2.03061938;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.06123936, 4.06123936;stroke-dashoffset:0" + d="m 776.70517,587.76566 52.63774,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-7-7-1" + style="fill:none;stroke:#000000;stroke-width:2.03595257;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.07190574, 4.07190574;stroke-dashoffset:0" + d="m 829.46781,518.11584 0,67.90517" /> + <path + inkscape:connector-curvature="0" + id="130-38-7-3" + style="fill:none;stroke:#000000;stroke-width:2.03061938;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.06123936, 4.06123936;stroke-dashoffset:0" + d="m 775.40198,648.44575 52.63775,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-7-7-5" + style="fill:none;stroke:#000000;stroke-width:2.03061938;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.06123936, 4.06123936;stroke-dashoffset:0" + d="m 776.06456,720.83282 52.63774,0" /> + <path + inkscape:connector-curvature="0" + id="130-38-7-7-1-0" + style="fill:none;stroke:#000000;stroke-width:2.03595257;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.07190574, 4.07190574;stroke-dashoffset:0" + d="m 828.8272,651.183 0,67.90517" /> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="249.25908" + y="517.75714" + id="text4028-9-14" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="249.25908" + y="517.75714" + id="tspan4032-5-19" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">0</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="248.77629" + y="591.84143" + id="text4028-9-14-8" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="248.77629" + y="591.84143" + id="tspan4032-5-19-8" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">1</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="249.80574" + y="651.65784" + id="text4028-9-14-1" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="249.80574" + y="651.65784" + id="tspan4032-5-19-6" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">2</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="249.32295" + y="727.74213" + id="text4028-9-14-8-8" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="249.32295" + y="727.74213" + id="tspan4032-5-19-8-6" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">3</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="843.30432" + y="519.68823" + id="text4028-9-14-9" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="843.30432" + y="519.68823" + id="tspan4032-5-19-0" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">0</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="842.82153" + y="589.77252" + id="text4028-9-14-8-3" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="842.82153" + y="589.77252" + id="tspan4032-5-19-8-8" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">1</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="843.85095" + y="671.58893" + id="text4028-9-14-1-2" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="843.85095" + y="671.58893" + id="tspan4032-5-19-6-8" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">2</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="843.36816" + y="725.67316" + id="text4028-9-14-8-8-6" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="843.36816" + y="725.67316" + id="tspan4032-5-19-8-6-3" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial">3</tspan></text> + <text + xml:space="preserve" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Arial;-inkscape-font-specification:Arial" + x="514.69012" + y="789.7041" + id="text4028-1" + sodipodi:linespacing="103%"><tspan + sodipodi:role="line" + x="514.69012" + y="789.7041" + id="tspan4032-3" + style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:102.99999714%;writing-mode:lr-tb;text-anchor:start;font-family:Arial;-inkscape-font-specification:Arial">...</tspan></text> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_encrypt_flow.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_encrypt_flow.svg new file mode 100644 index 000000000..492c3df09 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_encrypt_flow.svg @@ -0,0 +1,194 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export l2fwd-crypto-encrypt-flow.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + width="10.3779in" height="2.38075in" viewBox="0 0 747.207 171.414" xml:space="preserve" color-interpolation-filters="sRGB" + class="st15"> + <style type="text/css"> + <![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#5b9bd5;fill-opacity:0.25;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.25} + .st3 {fill:#ffc000;stroke:#40709c;stroke-width:0.75} + .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em} + .st5 {font-size:1em} + .st6 {fill:#4672c4;stroke:#40709c;stroke-width:0.75} + .st7 {fill:#538135;stroke:#40709c;stroke-width:0.75} + .st8 {marker-end:url(#mrkr4-58);stroke:#41719c;stroke-dasharray:3,3;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st9 {fill:#41719c;fill-opacity:1;stroke:#41719c;stroke-opacity:1;stroke-width:0.28409090909091} + .st10 {marker-end:url(#mrkr4-58);stroke:#41719c;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st11 {fill:none;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} + .st12 {stroke:#c7c8c8;stroke-width:0.25} + .st13 {fill:none;stroke:none;stroke-width:0.25} + .st14 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em} + .st15 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <defs id="Markers"> + <g id="lend4"> + <path d="M 2 1 L 0 0 L 2 -1 L 2 1 " style="stroke:none"/> + </g> + <marker id="mrkr4-58" class="st9" refX="-7.04" orient="auto" markerUnits="strokeWidth" overflow="visible"> + <use xlink:href="#lend4" transform="scale(-3.52,-3.52) "/> + </marker> + </defs> + <defs id="Filters"> + <filter id="filter_2"> + <feGaussianBlur stdDeviation="2"/> + </filter> + </defs> + <g> + <title>Page-1</title> + <g id="shape101-1" transform="translate(3.73674,-5.34781)"> + <title>Circle.53</title> + <desc>RX P0 Q0</desc> + <g id="shadow101-2" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st3"/> + <text x="21.35" y="141.75" class="st4">RX <tspan x="14.52" dy="1.2em" class="st5">P</tspan>0 Q0</text> </g> + <g id="shape102-8" transform="translate(101.797,-5.34781)"> + <title>Circle.56</title> + <desc>RX</desc> + <g id="shadow102-9" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st6"/> + <text x="21.35" y="147.75" class="st4">RX</text> </g> + <g id="shape103-14" transform="translate(395.977,-5.34781)"> + <title>Circle.57</title> + <desc>CRYPTO DEQ</desc> + <g id="shadow103-15" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st7"/> + <text x="10.52" y="141.75" class="st4">CRYPTO <tspan x="17.78" dy="1.2em" class="st5">DEQ</tspan></text> </g> + <g id="shape104-21" transform="translate(297.917,-5.34781)"> + <title>Circle.58</title> + <desc>CRYPTO ENQ</desc> + <g id="shadow104-22" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st7"/> + <text x="10.52" y="141.75" class="st4">CRYPTO <tspan x="17.63" dy="1.2em" class="st5">ENQ</tspan></text> </g> + <g id="shape105-28" transform="translate(690.158,-5.34781)"> + <title>Circle.73</title> + <desc>TX P0 Q0</desc> + <g id="shadow105-29" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st3"/> + <text x="21.63" y="141.75" class="st4">TX <tspan x="14.52" dy="1.2em" class="st5">P</tspan>0 Q0</text> </g> + <g id="shape106-35" transform="translate(494.037,-5.34781)"> + <title>Circle.74</title> + <desc>MAC</desc> + <g id="shadow106-36" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st6"/> + <text x="16.83" y="147.75" class="st4">MAC</text> </g> + <g id="shape107-41" transform="translate(199.857,-5.34781)"> + <title>Circle.61</title> + <desc>PAD</desc> + <g id="shadow107-42" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st6"/> + <text x="18.11" y="147.75" class="st4">PAD</text> </g> + <g id="shape108-47" transform="translate(592.097,-5.34781)"> + <title>Circle.62</title> + <desc>TX</desc> + <g id="shadow108-48" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st2"/> + </g> + <path d="M0 144.75 A26.6643 26.6643 0 0 1 53.33 144.75 A26.6643 26.6643 0 1 1 0 144.75 Z" class="st6"/> + <text x="21.63" y="147.75" class="st4">TX</text> </g> + <g id="shape109-53" transform="translate(57.0653,-24.9255)"> + <title>Dynamic connector.63</title> + <path d="M0 164.33 L37.69 164.33" class="st8"/> + </g> + <g id="shape110-59" transform="translate(155.125,-24.9255)"> + <title>Dynamic connector.65</title> + <path d="M0 164.33 L37.69 164.33" class="st10"/> + </g> + <g id="shape111-64" transform="translate(253.186,-24.9255)"> + <title>Dynamic connector.66</title> + <path d="M0 164.33 L37.69 164.33" class="st10"/> + </g> + <g id="shape112-69" transform="translate(351.246,-24.9255)"> + <title>Dynamic connector.67</title> + <path d="M0 164.33 L37.69 164.33" class="st8"/> + </g> + <g id="shape113-74" transform="translate(449.306,-24.9255)"> + <title>Dynamic connector.68</title> + <path d="M0 164.33 L37.69 164.33" class="st10"/> + </g> + <g id="shape114-79" transform="translate(547.366,-24.9255)"> + <title>Dynamic connector.69</title> + <path d="M0 164.33 L37.69 164.33" class="st10"/> + </g> + <g id="shape115-84" transform="translate(645.426,-24.9255)"> + <title>Dynamic connector.70</title> + <path d="M0 164.33 L37.69 164.33" class="st8"/> + </g> + <g id="shape116-89" transform="translate(174.599,68.9848) rotate(90)"> + <title>Left Brace</title> + <g id="shadow116-90" transform="matrix(1,0,0,1,1.97279,-0.345598)" class="st1"> + <path d="M28.35 171.41 A24.4921 16.4101 0 0 1 14.17 167.43 L14.17 95.44 L0 95.44 L14.17 95.44 L14.17 23.46 A24.4921 + 16.4101 0 0 1 28.35 19.47" class="st11"/> + </g> + <path d="M28.35 171.41 A24.4921 16.4101 0 0 1 14.17 167.43 L14.17 95.44 L0 95.44 L14.17 95.44 L14.17 23.46 A24.4921 16.4101 + 0 0 1 28.35 19.47" class="st12"/> + </g> + <g id="shape117-97" transform="translate(371.271,68.9848) rotate(90)"> + <title>Left Brace.74</title> + <g id="shadow117-98" transform="matrix(1,0,0,1,1.97279,-0.345598)" class="st1"> + <path d="M28.35 171.41 A23.1398 15.504 0 0 1 14.17 163.51 L14.17 95.44 L0 95.44 L14.17 95.44 L14.17 27.38 A23.1398 + 15.504 0 0 1 28.35 19.47" class="st11"/> + </g> + <path d="M28.35 171.41 A23.1398 15.504 0 0 1 14.17 163.51 L14.17 95.44 L0 95.44 L14.17 95.44 L14.17 27.38 A23.1398 15.504 + 0 0 1 28.35 19.47" class="st12"/> + </g> + <g id="shape118-105" transform="translate(212.048,-117.835)"> + <title>Sheet.118</title> + <desc>Stage 2: Pad packets and enqueue crypto operations</desc> + <rect x="0" y="118.085" width="127.559" height="53.3286" class="st13"/> + <text x="13.85" y="133.95" class="st14">Stage 2: Pad packets <tspan x="14.65" dy="1.2em" class="st5">and enqueue crypto </tspan><tspan + x="37.46" dy="1.2em" class="st5">operations</tspan></text> </g> + <g id="shape119-110" transform="translate(15.3756,-117.835)"> + <title>Sheet.119</title> + <desc>Stage 1: Read packets from port</desc> + <rect x="0" y="118.085" width="127.559" height="53.3286" class="st13"/> + <text x="10.71" y="141.15" class="st14">Stage 1: Read packets <tspan x="40.13" dy="1.2em" class="st5">from port</tspan></text> </g> + <g id="shape120-114" transform="translate(567.943,68.9848) rotate(90)"> + <title>Left Brace.78</title> + <g id="shadow120-115" transform="matrix(1,0,0,1,1.97279,-0.345598)" class="st1"> + <path d="M28.35 171.41 A37.9502 8.92454 0 0 1 14.17 166.08 L14.17 144.75 L0 144.75 L14.17 144.75 L14.17 123.42 A37.9502 + 8.92454 0 0 1 28.35 118.09" class="st11"/> + </g> + <path d="M28.35 171.41 A37.9502 8.92454 0 0 1 14.17 166.08 L14.17 144.75 L0 144.75 L14.17 144.75 L14.17 123.42 A37.9502 + 8.92454 0 0 1 28.35 118.09" class="st12"/> + </g> + <g id="shape121-122" transform="translate(371.106,-117.835)"> + <title>Sheet.121</title> + <desc>Stage 3: Dequeue processed crypto operations</desc> + <rect x="0" y="118.085" width="99.248" height="53.3286" class="st13"/> + <text x="6.71" y="133.95" class="st14">Stage 3: Dequeue <tspan x="7.83" dy="1.2em" class="st5">processed crypto </tspan><tspan + x="23.31" dy="1.2em" class="st5">operations</tspan></text> </g> + <g id="shape122-127" transform="translate(666.003,71.9952) rotate(90)"> + <title>Left Brace.80</title> + <g id="shadow122-128" transform="matrix(1,0,0,1,1.97279,-0.345598)" class="st1"> + <path d="M28.35 171.41 A22.2255 24.1253 0 0 1 14.17 161.02 L14.17 48.34 L0 48.34 L14.17 48.34 L14.17 -64.35 A22.2255 + 24.1253 0 0 1 28.35 -74.74" class="st11"/> + </g> + <path d="M28.35 171.41 A22.2255 24.1253 0 0 1 14.17 161.02 L14.17 48.34 L0 48.34 L14.17 48.34 L14.17 -64.35 A22.2255 + 24.1253 0 0 1 28.35 -74.74" class="st12"/> + </g> + <g id="shape123-135" transform="translate(553.887,-111.814)"> + <title>Sheet.123</title> + <desc>Stage 4: Modify Packet MAC header and transmit</desc> + <rect x="0" y="118.085" width="127.559" height="53.3286" class="st13"/> + <text x="8.01" y="133.95" class="st14">Stage 4: Modify Packet <tspan x="22.85" dy="1.2em" class="st5">MAC header and </tspan><tspan + x="43.12" dy="1.2em" class="st5">transmit </tspan></text> </g> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_virtenv_benchmark_setup.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_virtenv_benchmark_setup.png Binary files differnew file mode 100644 index 000000000..317df8aa9 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_virtenv_benchmark_setup.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_vm2vm.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_vm2vm.svg new file mode 100644 index 000000000..b84dcb274 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/l2_fwd_vm2vm.svg @@ -0,0 +1,311 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:osb="http://www.openswatchbook.org/uri/2009/osb" + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="554.46204" + height="443.63278" + viewBox="0 0 554.46204 443.63279" + id="svg3917" + version="1.1" + inkscape:version="0.91 r13725" + sodipodi:docname="l2_fwd_vm2vm.svg"> + <defs + id="defs3919"> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker8020" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path8022" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker7177" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path7179" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker6025" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + id="path6027" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path5351" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5348" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <inkscape:path-effect + effect="powerstroke" + id="path-effect4780" + is_visible="true" + offset_points="0,0.5" + sort_points="true" + interpolator_type="Linear" + interpolator_beta="0.2" + start_linecap_type="zerowidth" + linejoin_type="round" + miter_limit="4" + end_linecap_type="zerowidth" + cusp_linecap_type="round" /> + <linearGradient + id="linearGradient4729" + osb:paint="solid"> + <stop + style="stop-color:#000000;stop-opacity:1;" + offset="0" + id="stop4731" /> + </linearGradient> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-5" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5351-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + inkscape:connector-curvature="0" + id="path5351-2" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5351-2-2" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1" + inkscape:cx="323.29803" + inkscape:cy="27.634604" + inkscape:document-units="px" + inkscape:current-layer="layer1" + showgrid="false" + inkscape:snap-nodes="false" + inkscape:snap-bbox="true" + inkscape:window-width="1276" + inkscape:window-height="1400" + inkscape:window-x="1280" + inkscape:window-y="38" + inkscape:window-maximized="0" + units="px" + fit-margin-top="5" + fit-margin-left="5" + fit-margin-right="5" + fit-margin-bottom="5" /> + <metadata + id="metadata3922"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-0.56091356,-0.34416246)"> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:2.10537624;stroke-opacity:1" + id="rect4727" + width="542.35669" + height="431.5274" + x="6.6136017" + y="6.3968506" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:30.53249741px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="237.30467" + y="33.252548" + id="text4735" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4737" + x="237.30467" + y="33.252548">Host</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-opacity:1" + id="rect4739" + width="207.08128" + height="202.03053" + x="38.385803" + y="45.240112" /> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-opacity:1" + id="rect4739-3" + width="207.08128" + height="202.03053" + x="301.53052" + y="44.22995" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:19.96650314px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="101.13004" + y="63.706543" + id="text4756" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4758" + x="101.13004" + y="63.706543">Guest1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:19.96650314px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="369.73492" + y="63.619873" + id="text4756-6" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4758-7" + x="369.73492" + y="63.619873">Guest2</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect5336" + width="477.80215" + height="85.862968" + x="39.39595" + y="316.97116" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:23.81648636px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="237.96404" + y="398.79352" + id="text5338" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan5340" + x="237.96404" + y="398.79352">L2FWD</tspan></text> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.9760201px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend)" + d="m 120.20815,247.27063 0,68.32236" + id="path5342" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.9760201px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-5)" + d="m 382.84782,246.56645 0,68.32236" + id="path5342-5" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.9760201px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-6)" + d="m 162.63455,316.66519 0,-68.32236" + id="path5342-9" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.9760201px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-6-1)" + d="m 423.25391,315.65504 0,-68.32236" + id="path5342-9-7" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.60951841;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:4.82855511, 1.60951837;stroke-dashoffset:0;stroke-opacity:1" + d="m 119.48645,319.66266 0,47.47156 303.479,0 0,-51.26929" + id="path10412" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.1137104;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3.3411313, 1.11371043;stroke-dashoffset:0;stroke-opacity:1" + d="m 162.67537,318.28501 0,31.19206 221.14177,0 0,-33.68743" + id="path10412-0" + inkscape:connector-curvature="0" /> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/load_bal_app_arch.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/load_bal_app_arch.png Binary files differnew file mode 100644 index 000000000..848e48baf --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/load_bal_app_arch.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/overlay_networking.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/overlay_networking.svg new file mode 100644 index 000000000..caf1c264e --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/overlay_networking.svg @@ -0,0 +1,1847 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Generated by Microsoft Visio, SVG Export overlay_networking.svg Page-1 --> + +<svg + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:xlink="http://www.w3.org/1999/xlink" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="7.4165835in" + height="4.6068053in" + viewBox="0 0 533.99401 331.68998" + xml:space="preserve" + class="st29" + id="svg2" + version="1.1" + inkscape:version="0.91 r13725" + sodipodi:docname="overlay_networking.svg" + style="font-size:12px;overflow:visible;color-interpolation-filters:sRGB;fill:none;fill-rule:evenodd;stroke-linecap:square;stroke-miterlimit:3"><metadata + id="metadata498"><rdf:RDF><cc:Work + rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /><dc:title></dc:title></cc:Work></rdf:RDF></metadata><sodipodi:namedview + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1" + objecttolerance="10" + gridtolerance="10" + guidetolerance="10" + inkscape:pageopacity="0" + inkscape:pageshadow="2" + inkscape:window-width="885" + inkscape:window-height="480" + id="namedview496" + showgrid="false" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" + inkscape:zoom="0.23838384" + inkscape:cx="340.46875" + inkscape:cy="137.425" + inkscape:window-x="0" + inkscape:window-y="0" + inkscape:window-maximized="0" + inkscape:current-layer="svg2" /><v:documentProperties + v:langID="1033" + v:viewMarkup="false"><v:userDefs><v:ud + v:nameU="msvSubprocessMaster" + v:prompt="" + v:val="VT4(Rectangle)" /><v:ud + v:nameU="msvNoAutoConnect" + v:val="VT0(1):26" /><v:ud + v:nameU="msvConvertTheme" /></v:userDefs></v:documentProperties><style + type="text/css" + id="style4"><![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#b2b2b2;fill-opacity:0.5;stroke:#b2b2b2;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5} + .st3 {fill:#d8d8d8;fill-opacity:0.7;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st4 {fill:#000000;font-family:Calibri;font-size:1.16666em} + .st5 {fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5} + .st6 {fill:#6b9bc7;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st7 {fill:#000000;font-family:Calibri;font-size:0.833336em} + .st8 {fill:#f6d5b9;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st9 {fill:#000000;font-family:Calibri;font-size:0.75em} + .st10 {fill:url(#grad0-21);stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st11 {fill:#000000;font-family:Calibri;font-size:1.00001em} + .st12 {fill:#00b0f0;fill-opacity:0.8;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st13 {fill:#00b0f0;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st14 {fill:#7030a0;fill-opacity:0.6;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st15 {fill:url(#grad0-121)} + .st16 {stroke:#0070c0;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st17 {fill:url(#grad0-128)} + .st18 {stroke:#d26d19;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st19 {stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5;stroke-width:1.5} + .st20 {stroke:#d26d19;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st21 {stroke:#0070c0;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st22 {stroke:#7030a0;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st23 {stroke:#43365a;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st24 {stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st25 {stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st26 {fill:url(#grad0-236)} + .st27 {stroke:#7030a0;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st28 {fill:#8fa350;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.5} + .st29 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]></style><defs + id="Patterns_And_Gradients"><linearGradient + id="grad0-21" + x1="-0.13807119" + y1="655.71232" + x2="175.84111" + y2="655.71232" + gradientTransform="scale(1.0499298,0.95244464)" + gradientUnits="userSpaceOnUse"><stop + offset="0" + stop-color="#f0f0f0" + stop-opacity="1" + id="stop8" /><stop + offset="1" + stop-color="#ffffff" + stop-opacity="1" + id="stop10" /></linearGradient><linearGradient + id="grad0-121" + x1="-0.15931553" + y1="5138.0002" + x2="31.958082" + y2="5138.0002" + gradientTransform="scale(6.5279263,0.15318801)" + gradientUnits="userSpaceOnUse"><stop + offset="0" + stop-color="#f0f0f0" + stop-opacity="1" + id="stop13" /><stop + offset="1" + stop-color="#0070c0" + stop-opacity="1" + id="stop15" /></linearGradient><linearGradient + id="grad0-128" + x1="0" + y1="0" + x2="1" + y2="0" + gradientTransform="matrix(0,-1,1,0,0,1)"><stop + offset="0" + stop-color="#f0f0f0" + stop-opacity="1" + id="stop18" /><stop + offset="1" + stop-color="#d26d19" + stop-opacity="1" + id="stop20" /></linearGradient><linearGradient + id="grad0-236" + x1="-0.19200153" + y1="4918.6482" + x2="31.120248" + y2="4918.6482" + gradientTransform="scale(6.2499501,0.16000128)" + gradientUnits="userSpaceOnUse"><stop + offset="0" + stop-color="#f0f0f0" + stop-opacity="1" + id="stop23" /><stop + offset="1" + stop-color="#7030a0" + stop-opacity="1" + id="stop25" /></linearGradient><linearGradient + inkscape:collect="always" + xlink:href="#grad0-128" + id="linearGradient4626" + gradientTransform="scale(7.734852,0.12928496)" + x1="0" + y1="6087.7926" + x2="38.210169" + y2="6087.7926" + gradientUnits="userSpaceOnUse" /><linearGradient + inkscape:collect="always" + xlink:href="#grad0-128" + id="linearGradient4628" + gradientTransform="scale(8.1000686,0.12345574)" + x1="0" + y1="6380.262" + x2="34.992296" + y2="6380.262" + gradientUnits="userSpaceOnUse" /></defs><g + v:mID="0" + v:index="1" + v:groupContext="foregroundPage" + id="g27" + transform="translate(-33.625,-174.25)"><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /><v:ud + v:nameU="msvThemeOrder" + v:val="VT0(0):26" /></v:userDefs><title + id="title29">Page-1</title><v:pageProperties + v:drawingScale="1" + v:pageScale="1" + v:drawingUnits="0" + v:shadowOffsetX="9" + v:shadowOffsetY="-9" /><v:layer + v:name="Flowchart" + v:index="0" /><v:layer + v:name="Connector" + v:index="1" /><g + id="shape3-1" + v:mID="3" + v:groupContext="shape" + transform="translate(85.5,-493.875)"><title + id="title32">Rounded rectangle</title><desc + id="desc34">Host</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="50.5382" + cy="730.125" + width="101.08" + height="123.75" /><g + id="shadow3-2" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 92.08,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-105.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -83.08,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 83.08,0 z" + class="st2" + id="path37" + inkscape:connector-curvature="0" + style="fill:#b2b2b2;fill-opacity:0.5;stroke:#b2b2b2;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 92.08,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-105.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -83.08,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 83.08,0 z" + class="st3" + id="path39" + inkscape:connector-curvature="0" + style="fill:#d8d8d8;fill-opacity:0.7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="37.400002" + y="784.71997" + class="st4" + v:langID="2052" + id="text41" + style="font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar />Host</text> +</g><g + id="shape10-6" + v:mID="10" + v:groupContext="shape" + transform="translate(33.75,-569.7)"><title + id="title44">Rounded rectangle.10</title><desc + id="desc46">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="19.4595" + cy="779.85" + width="38.92" + height="24.3" /><g + id="shadow10-7" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 29.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.3 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,767.7 a 9.00007,9.00007 -180 0 0 -9,9 l 0,6.3 a 9.00007,9.00007 -180 0 0 9,9 l 20.92,0 z" + class="st5" + id="path49" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 29.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.3 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,767.7 a 9.00007,9.00007 -180 0 0 -9,9 l 0,6.3 a 9.00007,9.00007 -180 0 0 9,9 l 20.92,0 z" + class="st6" + id="path51" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="12.35" + y="782.84998" + class="st7" + v:langID="2052" + id="text53" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape11-11" + v:mID="11" + v:groupContext="shape" + transform="translate(93.171,-566.775)"><title + id="title56">Rounded rectangle.11</title><desc + id="desc58">VNID 100</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="27.6645" + cy="783.9" + width="55.33" + height="16.2" /><g + id="shadow11-12" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 47.23,792 a 8.10006,8.10006 -180 0 0 8.1,-8.1 8.10006,8.10006 -180 0 0 -8.1,-8.1 l -39.13,0 A 8.10006,8.10006 -180 0 0 0,783.9 8.10006,8.10006 -180 0 0 8.1,792 l 39.13,0 z" + class="st5" + id="path61" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 47.23,792 a 8.10006,8.10006 -180 0 0 8.1,-8.1 8.10006,8.10006 -180 0 0 -8.1,-8.1 l -39.13,0 A 8.10006,8.10006 -180 0 0 0,783.9 8.10006,8.10006 -180 0 0 8.1,792 l 39.13,0 z" + class="st8" + id="path63" + inkscape:connector-curvature="0" + style="fill:#f6d5b9;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="10.44" + y="786.59998" + class="st9" + v:langID="2052" + id="text65" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 100</text> +</g><g + id="shape14-16" + v:mID="14" + v:groupContext="shape" + v:layerMember="0" + transform="translate(204.576,-381.375)"><title + id="title68">Object</title><desc + id="desc70">IP Network</desc><v:custProps><v:cp + v:nameU="Cost" + v:lbl="Cost" + v:type="7" + v:format="@" + v:langID="1033" /><v:cp + v:nameU="Duration" + v:lbl="Duration" + v:type="2" + v:langID="1033" /><v:cp + v:nameU="Resources" + v:lbl="Resources" + v:langID="1033" /></v:custProps><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(1,1,1,1)" + v:tabSpace="42.5197" /><v:textRect + cx="92.25" + cy="708.331" + width="184.5" + height="167.337" /><g + id="shadow14-17" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 26.52,755.72 a 21.2239,24.0427 -180 0 0 30.57,20.35 29.3964,33.3006 -180 0 0 53.17,-7.53 8.89171,10.0726 -180 0 0 9.59,10.87 33.8326,38.3261 -180 0 0 32.49,-15.89 21.0233,23.8155 -180 0 0 22.9,-15.06 36.2371,30.7434 -123.25 0 0 -13.86,-32.32 124.285,106.223 -59.84 0 0 23.08,-31.84 46.9314,46.8568 -180 0 0 -24.31,-42.9 28.0325,27.988 -180 0 0 -53.62,13.39 24.2803,21.6216 -26.25 0 0 -40.25,1.24 34.8986,32.2655 -147.51 0 0 -43.54,45.6 27.2819,27.2385 -180 0 0 3.78,54.09 z" + class="st5" + id="path73" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 26.52,755.72 a 21.2239,24.0427 -180 0 0 30.57,20.35 29.3964,33.3006 -180 0 0 53.17,-7.53 8.89171,10.0726 -180 0 0 9.59,10.87 33.8326,38.3261 -180 0 0 32.49,-15.89 21.0233,23.8155 -180 0 0 22.9,-15.06 36.2371,30.7434 -123.25 0 0 -13.86,-32.32 124.285,106.223 -59.84 0 0 23.08,-31.84 46.9314,46.8568 -180 0 0 -24.31,-42.9 28.0325,27.988 -180 0 0 -53.62,13.39 24.2803,21.6216 -26.25 0 0 -40.25,1.24 34.8986,32.2655 -147.51 0 0 -43.54,45.6 27.2819,27.2385 -180 0 0 3.78,54.09 z" + class="st10" + id="path75" + inkscape:connector-curvature="0" + style="fill:url(#grad0-21);stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="65.139999" + y="711.92999" + class="st11" + v:langID="2052" + id="text77" + style="font-size:12.00012016px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />IP Network</text> +</g><g + id="shape1-23" + v:mID="1" + v:groupContext="shape" + transform="translate(33.75,-521.1)"><title + id="title80">Rounded rectangle.1</title><desc + id="desc82">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="19.4595" + cy="779.85" + width="38.92" + height="24.3" /><g + id="shadow1-24" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 29.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.3 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,767.7 a 9.00007,9.00007 -180 0 0 -9,9 l 0,6.3 a 9.00007,9.00007 -180 0 0 9,9 l 20.92,0 z" + class="st5" + id="path85" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 29.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.3 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,767.7 a 9.00007,9.00007 -180 0 0 -9,9 l 0,6.3 a 9.00007,9.00007 -180 0 0 9,9 l 20.92,0 z" + class="st6" + id="path87" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="12.35" + y="782.84998" + class="st7" + v:langID="2052" + id="text89" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape22-28" + v:mID="22" + v:groupContext="shape" + transform="translate(93.171,-530.325)"><title + id="title92">Rounded rectangle.22</title><desc + id="desc94">VNID 101</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="27.6645" + cy="783.9" + width="55.33" + height="16.2" /><g + id="shadow22-29" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 47.23,792 a 8.10006,8.10006 -180 0 0 8.1,-8.1 8.10006,8.10006 -180 0 0 -8.1,-8.1 l -39.13,0 A 8.10006,8.10006 -180 0 0 0,783.9 8.10006,8.10006 -180 0 0 8.1,792 l 39.13,0 z" + class="st5" + id="path97" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 47.23,792 a 8.10006,8.10006 -180 0 0 8.1,-8.1 8.10006,8.10006 -180 0 0 -8.1,-8.1 l -39.13,0 A 8.10006,8.10006 -180 0 0 0,783.9 8.10006,8.10006 -180 0 0 8.1,792 l 39.13,0 z" + class="st12" + id="path99" + inkscape:connector-curvature="0" + style="fill:#00b0f0;fill-opacity:0.8;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="10.44" + y="786.59998" + class="st9" + v:langID="2052" + id="text101" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 101</text> +</g><g + id="shape23-33" + v:mID="23" + v:groupContext="shape" + transform="translate(85.5,-288)"><title + id="title104">Rounded rectangle.23</title><desc + id="desc106">Host</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="51.75" + cy="733.5" + width="103.5" + height="117" /><g + id="shadow23-34" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 94.5,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-99 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,675 a 9.00007,9.00007 -180 0 0 -9,9 l 0,99 a 9.00007,9.00007 -180 0 0 9,9 l 85.5,0 z" + class="st2" + id="path109" + inkscape:connector-curvature="0" + style="fill:#b2b2b2;fill-opacity:0.5;stroke:#b2b2b2;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 94.5,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-99 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,675 a 9.00007,9.00007 -180 0 0 -9,9 l 0,99 a 9.00007,9.00007 -180 0 0 9,9 l 85.5,0 z" + class="st3" + id="path111" + inkscape:connector-curvature="0" + style="fill:#d8d8d8;fill-opacity:0.7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="38.610001" + y="788.09998" + class="st4" + v:langID="2052" + id="text113" + style="font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar />Host</text> +</g><g + id="shape24-38" + v:mID="24" + v:groupContext="shape" + transform="translate(33.75,-362.1)"><title + id="title116">Rounded rectangle.24</title><desc + id="desc118">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="20.25" + cy="780.3" + width="40.51" + height="23.4" /><g + id="shadow24-39" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 31.5,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-5.4 a 9.00007,9.00007 -180 0 0 -9,-9 l -22.5,0 a 9.00007,9.00007 -180 0 0 -9,9 l 0,5.4 a 9.00007,9.00007 -180 0 0 9,9 l 22.5,0 z" + class="st5" + id="path121" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 31.5,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-5.4 a 9.00007,9.00007 -180 0 0 -9,-9 l -22.5,0 a 9.00007,9.00007 -180 0 0 -9,9 l 0,5.4 a 9.00007,9.00007 -180 0 0 9,9 l 22.5,0 z" + class="st6" + id="path123" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="13.14" + y="783.29999" + class="st7" + v:langID="2052" + id="text125" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape25-43" + v:mID="25" + v:groupContext="shape" + transform="translate(95.4166,-358.2)"><title + id="title128">Rounded rectangle.25</title><desc + id="desc130">VNID 100</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="28.7917" + cy="784.2" + width="57.59" + height="15.6" /><g + id="shadow25-44" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 49.78,792 a 7.80006,7.80006 -180 0 0 7.8,-7.8 7.80006,7.80006 -180 0 0 -7.8,-7.8 l -41.98,0 A 7.80006,7.80006 -180 0 0 0,784.2 7.80006,7.80006 -180 0 0 7.8,792 l 41.98,0 z" + class="st5" + id="path133" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 49.78,792 a 7.80006,7.80006 -180 0 0 7.8,-7.8 7.80006,7.80006 -180 0 0 -7.8,-7.8 l -41.98,0 A 7.80006,7.80006 -180 0 0 0,784.2 7.80006,7.80006 -180 0 0 7.8,792 l 41.98,0 z" + class="st8" + id="path135" + inkscape:connector-curvature="0" + style="fill:#f6d5b9;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="11.57" + y="786.90002" + class="st9" + v:langID="2052" + id="text137" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 100</text> +</g><g + id="shape31-48" + v:mID="31" + v:groupContext="shape" + transform="translate(33.75,-315.3)"><title + id="title140">Rounded rectangle.31</title><desc + id="desc142">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="20.6757" + cy="780.3" + width="41.36" + height="23.4" /><g + id="shadow31-49" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 32.35,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-5.4 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,768.6 a 9.00007,9.00007 -180 0 0 -9,9 l 0,5.4 a 9.00007,9.00007 -180 0 0 9,9 l 23.35,0 z" + class="st5" + id="path145" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 32.35,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-5.4 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,768.6 a 9.00007,9.00007 -180 0 0 -9,9 l 0,5.4 a 9.00007,9.00007 -180 0 0 9,9 l 23.35,0 z" + class="st6" + id="path147" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="13.56" + y="783.29999" + class="st7" + v:langID="2052" + id="text149" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape32-53" + v:mID="32" + v:groupContext="shape" + transform="translate(95.4166,-323.1)"><title + id="title152">Rounded rectangle.32</title><desc + id="desc154">VNID 101</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="28.7917" + cy="784.2" + width="57.59" + height="15.6" /><g + id="shadow32-54" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 49.78,792 a 7.80006,7.80006 -180 0 0 7.8,-7.8 7.80006,7.80006 -180 0 0 -7.8,-7.8 l -41.98,0 A 7.80006,7.80006 -180 0 0 0,784.2 7.80006,7.80006 -180 0 0 7.8,792 l 41.98,0 z" + class="st5" + id="path157" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 49.78,792 a 7.80006,7.80006 -180 0 0 7.8,-7.8 7.80006,7.80006 -180 0 0 -7.8,-7.8 l -41.98,0 A 7.80006,7.80006 -180 0 0 0,784.2 7.80006,7.80006 -180 0 0 7.8,792 l 41.98,0 z" + class="st13" + id="path159" + inkscape:connector-curvature="0" + style="fill:#00b0f0;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="11.57" + y="786.90002" + class="st9" + v:langID="2052" + id="text161" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 101</text> +</g><g + id="shape34-58" + v:mID="34" + v:groupContext="shape" + transform="translate(407.076,-491.625)"><title + id="title164">Rounded rectangle.34</title><desc + id="desc166">Host</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="48.4618" + cy="730.125" + width="96.93" + height="123.75" /><g + id="shadow34-59" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 87.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-105.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -78.92,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 78.92,0 z" + class="st2" + id="path169" + inkscape:connector-curvature="0" + style="fill:#b2b2b2;fill-opacity:0.5;stroke:#b2b2b2;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 87.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-105.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -78.92,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 78.92,0 z" + class="st3" + id="path171" + inkscape:connector-curvature="0" + style="fill:#d8d8d8;fill-opacity:0.7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="35.330002" + y="784.71997" + class="st4" + v:langID="2052" + id="text173" + style="font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar />Host</text> +</g><g + id="shape35-63" + v:mID="35" + v:groupContext="shape" + transform="translate(520.049,-580.313)"><title + id="title176">Rounded rectangle.35</title><desc + id="desc178">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="18.8889" + cy="779.625" + width="37.78" + height="24.75" /><g + id="shadow35-64" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 28.78,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -19.78,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 19.78,0 z" + class="st5" + id="path181" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 28.78,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -19.78,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 19.78,0 z" + class="st6" + id="path183" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="11.78" + y="782.63" + class="st7" + v:langID="2052" + id="text185" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape36-68" + v:mID="36" + v:groupContext="shape" + transform="translate(441.076,-582.375)"><title + id="title188">Rounded rectangle.36</title><desc + id="desc190">VNID 100</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="26.4444" + cy="783.75" + width="52.89" + height="16.5" /><g + id="shadow36-69" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 44.64,792 a 8.25006,8.25006 -180 0 0 8.25,-8.25 8.25006,8.25006 -180 0 0 -8.25,-8.25 l -36.39,0 A 8.25006,8.25006 -180 0 0 0,783.75 8.25006,8.25006 -180 0 0 8.25,792 l 36.39,0 z" + class="st5" + id="path193" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 44.64,792 a 8.25006,8.25006 -180 0 0 8.25,-8.25 8.25006,8.25006 -180 0 0 -8.25,-8.25 l -36.39,0 A 8.25006,8.25006 -180 0 0 0,783.75 8.25006,8.25006 -180 0 0 8.25,792 l 36.39,0 z" + class="st8" + id="path195" + inkscape:connector-curvature="0" + style="fill:#f6d5b9;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="9.2200003" + y="786.45001" + class="st9" + v:langID="2052" + id="text197" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 100</text> +</g><g + id="shape37-73" + v:mID="37" + v:groupContext="shape" + transform="translate(520.049,-541.125)"><title + id="title200">Rounded rectangle.37</title><desc + id="desc202">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="18.8889" + cy="779.625" + width="37.78" + height="24.75" /><g + id="shadow37-74" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 28.78,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -19.78,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 19.78,0 z" + class="st5" + id="path205" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 28.78,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -19.78,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 19.78,0 z" + class="st6" + id="path207" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="11.78" + y="782.63" + class="st7" + v:langID="2052" + id="text209" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape38-78" + v:mID="38" + v:groupContext="shape" + transform="translate(441.076,-551.438)"><title + id="title212">Rounded rectangle.38</title><desc + id="desc214">VNID 101</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="26.4444" + cy="783.75" + width="52.89" + height="16.5" /><g + id="shadow38-79" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 44.64,792 a 8.25006,8.25006 -180 0 0 8.25,-8.25 8.25006,8.25006 -180 0 0 -8.25,-8.25 l -36.39,0 A 8.25006,8.25006 -180 0 0 0,783.75 8.25006,8.25006 -180 0 0 8.25,792 l 36.39,0 z" + class="st5" + id="path217" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 44.64,792 a 8.25006,8.25006 -180 0 0 8.25,-8.25 8.25006,8.25006 -180 0 0 -8.25,-8.25 l -36.39,0 A 8.25006,8.25006 -180 0 0 0,783.75 8.25006,8.25006 -180 0 0 8.25,792 l 36.39,0 z" + class="st12" + id="path219" + inkscape:connector-curvature="0" + style="fill:#00b0f0;fill-opacity:0.8;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="9.2200003" + y="786.45001" + class="st9" + v:langID="2052" + id="text221" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 101</text> +</g><g + id="shape40-83" + v:mID="40" + v:groupContext="shape" + transform="translate(398.076,-288)"><title + id="title224">Rounded rectangle.40</title><desc + id="desc226">Host</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="52.9618" + cy="727.875" + width="105.93" + height="128.25" /><g + id="shadow40-84" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 96.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-110.25 a 9.00007,9.00007 -180 0 0 -9,-9 l -87.92,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 87.92,0 z" + class="st2" + id="path229" + inkscape:connector-curvature="0" + style="fill:#b2b2b2;fill-opacity:0.5;stroke:#b2b2b2;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 96.92,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-110.25 a 9.00007,9.00007 -180 0 0 -9,-9 l -87.92,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 87.92,0 z" + class="st3" + id="path231" + inkscape:connector-curvature="0" + style="fill:#d8d8d8;fill-opacity:0.7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="39.830002" + y="782.46997" + class="st4" + v:langID="2052" + id="text233" + style="font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar /><v:newlineChar />Host</text> +</g><g + id="shape41-88" + v:mID="41" + v:groupContext="shape" + transform="translate(522,-369.225)"><title + id="title236">Rounded rectangle.41</title><desc + id="desc238">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="21.1806" + cy="779.175" + width="42.37" + height="25.65" /><g + id="shadow41-89" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 33.36,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-7.65 a 9.00007,9.00007 -180 0 0 -9,-9 l -24.36,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 24.36,0 z" + class="st5" + id="path241" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 33.36,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-7.65 a 9.00007,9.00007 -180 0 0 -9,-9 l -24.36,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 24.36,0 z" + class="st6" + id="path243" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="14.07" + y="782.16998" + class="st7" + v:langID="2052" + id="text245" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape42-93" + v:mID="42" + v:groupContext="shape" + transform="translate(436.201,-369)"><title + id="title248">Rounded rectangle.42</title><desc + id="desc250">VNID 100</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="29.6528" + cy="783.45" + width="59.31" + height="17.1" /><g + id="shadow42-94" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 50.76,792 a 8.55007,8.55007 -180 0 0 8.55,-8.55 8.55007,8.55007 -180 0 0 -8.55,-8.55 l -42.21,0 A 8.55007,8.55007 -180 0 0 0,783.45 8.55007,8.55007 -180 0 0 8.55,792 l 42.21,0 z" + class="st5" + id="path253" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 50.76,792 a 8.55007,8.55007 -180 0 0 8.55,-8.55 8.55007,8.55007 -180 0 0 -8.55,-8.55 l -42.21,0 A 8.55007,8.55007 -180 0 0 0,783.45 8.55007,8.55007 -180 0 0 8.55,792 l 42.21,0 z" + class="st8" + id="path255" + inkscape:connector-curvature="0" + style="fill:#f6d5b9;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="12.43" + y="786.15002" + class="st9" + v:langID="2052" + id="text257" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 100</text> +</g><g + id="shape43-98" + v:mID="43" + v:groupContext="shape" + transform="translate(523.319,-317.925)"><title + id="title260">Rounded rectangle.43</title><desc + id="desc262">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="21.1806" + cy="779.175" + width="42.37" + height="25.65" /><g + id="shadow43-99" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 33.36,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-7.65 a 9.00007,9.00007 -180 0 0 -9,-9 l -24.36,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 24.36,0 z" + class="st5" + id="path265" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 33.36,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-7.65 a 9.00007,9.00007 -180 0 0 -9,-9 l -24.36,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 24.36,0 z" + class="st6" + id="path267" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="14.07" + y="782.16998" + class="st7" + v:langID="2052" + id="text269" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape44-103" + v:mID="44" + v:groupContext="shape" + transform="translate(436.201,-324.9)"><title + id="title272">Rounded rectangle.44</title><desc + id="desc274">VNID 102</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="29.6528" + cy="783.45" + width="59.31" + height="17.1" /><g + id="shadow44-104" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 50.76,792 a 8.55007,8.55007 -180 0 0 8.55,-8.55 8.55007,8.55007 -180 0 0 -8.55,-8.55 l -42.21,0 A 8.55007,8.55007 -180 0 0 0,783.45 8.55007,8.55007 -180 0 0 8.55,792 l 42.21,0 z" + class="st5" + id="path277" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 50.76,792 a 8.55007,8.55007 -180 0 0 8.55,-8.55 8.55007,8.55007 -180 0 0 -8.55,-8.55 l -42.21,0 A 8.55007,8.55007 -180 0 0 0,783.45 8.55007,8.55007 -180 0 0 8.55,792 l 42.21,0 z" + class="st14" + id="path279" + inkscape:connector-curvature="0" + style="fill:#7030a0;fill-opacity:0.6;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="12.43" + y="786.15002" + class="st9" + v:langID="2052" + id="text281" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 102</text> +</g><g + id="shape46-108" + v:mID="46" + v:groupContext="shape" + transform="translate(441.076,-518.438)"><title + id="title284">Rounded rectangle.46</title><desc + id="desc286">VNID 102</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="26.4444" + cy="783.75" + width="52.89" + height="16.5" /><g + id="shadow46-109" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 44.64,792 a 8.25006,8.25006 -180 0 0 8.25,-8.25 8.25006,8.25006 -180 0 0 -8.25,-8.25 l -36.39,0 A 8.25006,8.25006 -180 0 0 0,783.75 8.25006,8.25006 -180 0 0 8.25,792 l 36.39,0 z" + class="st5" + id="path289" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 44.64,792 a 8.25006,8.25006 -180 0 0 8.25,-8.25 8.25006,8.25006 -180 0 0 -8.25,-8.25 l -36.39,0 A 8.25006,8.25006 -180 0 0 0,783.75 8.25006,8.25006 -180 0 0 8.25,792 l 36.39,0 z" + class="st14" + id="path291" + inkscape:connector-curvature="0" + style="fill:#7030a0;fill-opacity:0.6;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="9.2200003" + y="786.45001" + class="st9" + v:langID="2052" + id="text293" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VNID 102</text> +</g><g + id="shape47-113" + v:mID="47" + v:groupContext="shape" + transform="translate(520.049,-501.937)"><title + id="title296">Rounded rectangle.47</title><desc + id="desc298">VM</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="18.8889" + cy="779.625" + width="37.78" + height="24.75" /><g + id="shadow47-114" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 28.78,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -19.78,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 19.78,0 z" + class="st5" + id="path301" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 28.78,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-6.75 a 9.00007,9.00007 -180 0 0 -9,-9 l -19.78,0 a 9.00007,9.00007 -180 0 0 -9,9 L 0,783 a 9.00007,9.00007 -180 0 0 9,9 l 19.78,0 z" + class="st6" + id="path303" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="11.78" + y="782.63" + class="st7" + v:langID="2052" + id="text305" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM</text> +</g><g + id="shape48-118" + v:mID="48" + v:groupContext="shape" + transform="matrix(0.02167878,0.99976499,-0.99976499,0.02167878,940.314,236.405)"><title + id="title308">Arced arrow</title><v:userDefs><v:ud + v:nameU="ControlX" + v:val="VT0(0.125):0" /><v:ud + v:nameU="Scale" + v:val="VT0(1):26" /><v:ud + v:nameU="AntiScale" + v:val="VT0(1):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="ControlX" + v:prompt="" + v:val="VT0(0.0625):1" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><path + d="M 1.13,787.65 3.31,788.21 0,792 l -1.04,-4.92 2.17,0.57 z m 205.31,0 -2.18,0.56 3.31,3.79 1.05,-4.92 -2.18,0.57 z" + class="st15" + id="path310" + inkscape:connector-curvature="0" + style="fill:url(#grad0-121)" /><path + d="m 1.13,787.65 a 106.681,106.681 0 0 1 205.31,0" + class="st16" + id="path312" + inkscape:connector-curvature="0" + style="stroke:#0070c0;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="M 1.13,787.65 3.31,788.21 0,792 l -1.04,-4.92 2.17,0.57" + class="st16" + id="path314" + inkscape:connector-curvature="0" + style="stroke:#0070c0;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="m 206.44,787.65 -2.18,0.56 3.31,3.79 1.05,-4.92 -2.18,0.57" + class="st16" + id="path316" + inkscape:connector-curvature="0" + style="stroke:#0070c0;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape49-125" + v:mID="49" + v:groupContext="shape" + transform="matrix(-0.99814331,0.06090917,-0.06090917,-0.99814331,489.312,991.905)"><title + id="title319">Arced arrow.49</title><v:userDefs><v:ud + v:nameU="ControlX" + v:val="VT0(0.125):0" /><v:ud + v:nameU="Scale" + v:val="VT0(1):26" /><v:ud + v:nameU="AntiScale" + v:val="VT0(1):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="ControlX" + v:prompt="" + v:val="VT0(0.0625):1" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><path + d="M 2.72,788.42 4.51,789.78 0,792 l 0.93,-4.94 1.79,1.36 z m 290.11,0 -1.8,1.36 4.52,2.22 -0.93,-4.94 -1.79,1.36 z" + class="st17" + id="path321" + inkscape:connector-curvature="0" + style="fill:url(#linearGradient4626)" /><path + d="m 2.72,788.42 a 183.886,183.886 0 0 1 290.11,0" + class="st18" + id="path323" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="M 2.72,788.42 4.51,789.78 0,792 l 0.93,-4.94 1.79,1.36" + class="st18" + id="path325" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="m 292.83,788.42 -1.8,1.36 4.52,2.22 -0.93,-4.94 -1.79,1.36" + class="st18" + id="path327" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape50-132" + v:mID="50" + v:groupContext="shape" + transform="matrix(0.99916938,-0.04074983,0.04074983,0.99916938,120.726,-365.342)"><title + id="title330">Arced arrow.50</title><v:userDefs><v:ud + v:nameU="ControlX" + v:val="VT0(0.125):0" /><v:ud + v:nameU="Scale" + v:val="VT0(1):26" /><v:ud + v:nameU="AntiScale" + v:val="VT0(1):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="ControlX" + v:prompt="" + v:val="VT0(0.0625):1" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><path + d="m 3.79,789.57 1.21,1.9 -5,0.53 2.58,-4.32 1.21,1.89 z m 275.86,0 -1.22,1.9 5.01,0.53 -2.58,-4.32 -1.21,1.89 z" + class="st17" + id="path332" + inkscape:connector-curvature="0" + style="fill:url(#linearGradient4628)" /><path + d="m 3.79,789.57 a 259.249,259.249 0 0 1 275.86,0" + class="st18" + id="path334" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="m 3.79,789.57 1.21,1.9 -5,0.53 2.58,-4.32 1.21,1.89" + class="st18" + id="path336" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="m 279.65,789.57 -1.22,1.9 5.01,0.53 -2.58,-4.32 -1.21,1.89" + class="st18" + id="path338" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape4-138" + v:mID="4" + v:groupContext="shape" + transform="matrix(0.93831753,0.34577481,-0.34577481,0.93831753,348.104,-324.947)"><title + id="title341">Sheet.4</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow4-139" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.84909,0.853261)" + class="st1" + style="visibility:visible"><path + d="m 0,792 22.56,0" + class="st19" + id="path344" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 22.56,0" + class="st20" + id="path346" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape5-145" + v:mID="5" + v:groupContext="shape" + transform="matrix(0.98206713,-0.18853156,0.18853156,0.98206713,-74.2157,-312.797)"><title + id="title349">Sheet.5</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow5-146" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.14269,1.68566)" + class="st1" + style="visibility:visible"><path + d="m 0,792 20.69,0" + class="st19" + id="path352" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 20.69,0" + class="st21" + id="path354" + inkscape:connector-curvature="0" + style="stroke:#0070c0;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape6-152" + v:mID="6" + v:groupContext="shape" + transform="matrix(0.94671224,0.32208064,-0.32208064,0.94671224,327.757,-539.646)"><title + id="title357">Sheet.6</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow6-153" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.82706,0.89947)" + class="st1" + style="visibility:visible"><path + d="m 0,792 21.66,0" + class="st19" + id="path360" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 21.66,0" + class="st20" + id="path362" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape7-159" + v:mID="7" + v:groupContext="shape" + transform="matrix(0.96958947,-0.24473714,0.24473714,0.96958947,-121.163,-509.165)"><title + id="title365">Sheet.7</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow7-160" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.04379,1.74863)" + class="st1" + style="visibility:visible"><path + d="m 0,792 21.15,0" + class="st19" + id="path368" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 21.15,0" + class="st21" + id="path370" + inkscape:connector-curvature="0" + style="stroke:#0070c0;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape8-166" + v:mID="8" + v:groupContext="shape" + transform="matrix(0.99688828,-0.07882744,0.07882744,0.99688828,431.534,-588.161)"><title + id="title373">Sheet.8</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow8-167" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.32201,1.54903)" + class="st1" + style="visibility:visible"><path + d="m 0,792 26.16,0" + class="st19" + id="path376" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 26.16,0" + class="st20" + id="path378" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape13-173" + v:mID="13" + v:groupContext="shape" + transform="matrix(0.97299749,0.2308157,-0.2308157,0.97299749,676.771,-538.302)"><title + id="title381">Sheet.13</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow13-174" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.73349,1.06874)" + class="st1" + style="visibility:visible"><path + d="m 0,792 26.81,0" + class="st19" + id="path384" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 26.81,0" + class="st21" + id="path386" + inkscape:connector-curvature="0" + style="stroke:#0070c0;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape15-180" + v:mID="15" + v:groupContext="shape" + transform="matrix(0.90347299,0.42864501,-0.42864501,0.90347299,833.452,-450.238)"><title + id="title389">Sheet.15</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow15-181" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.91825,0.683753)" + class="st1" + style="visibility:visible"><path + d="m 0,792 28.87,0" + class="st19" + id="path392" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 28.87,0" + class="st22" + id="path394" + inkscape:connector-curvature="0" + style="stroke:#7030a0;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape16-187" + v:mID="16" + v:groupContext="shape" + transform="matrix(0.98587931,-0.16745741,0.16745741,0.98587931,362.881,-366.366)"><title + id="title397">Sheet.16</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow16-188" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.17853,1.6608)" + class="st1" + style="visibility:visible"><path + d="m 0,792 26.87,0" + class="st19" + id="path400" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 26.87,0" + class="st20" + id="path402" + inkscape:connector-curvature="0" + style="stroke:#d26d19;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape17-194" + v:mID="17" + v:groupContext="shape" + transform="matrix(0.99532092,0.09662438,-0.09662438,0.99532092,572.033,-329.744)"><title + id="title405">Sheet.17</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow17-195" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.5724,1.29412)" + class="st1" + style="visibility:visible"><path + d="m 0,792 27.94,0" + class="st19" + id="path408" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 27.94,0" + class="st23" + id="path410" + inkscape:connector-curvature="0" + style="stroke:#43365a;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape18-201" + v:mID="18" + v:groupContext="shape" + transform="matrix(0.56466455,-0.82532051,0.82532051,0.56466455,-464.654,-1.71391)"><title + id="title413">Sheet.18</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow18-202" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(-0.375346,2.00158)" + class="st1" + style="visibility:visible"><path + d="m 0,792 71.36,0" + class="st19" + id="path416" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 71.36,0" + class="st24" + id="path418" + inkscape:connector-curvature="0" + style="stroke:#404040;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape19-208" + v:mID="19" + v:groupContext="shape" + transform="matrix(0.46159689,0.8870898,-0.8870898,0.46159689,1074.07,13.0882)"><title + id="title421">Sheet.19</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow19-209" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.94211,-0.612711)" + class="st1" + style="visibility:visible"><path + d="m 0,792 57.58,0" + class="st19" + id="path424" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 57.58,0" + class="st24" + id="path426" + inkscape:connector-curvature="0" + style="stroke:#404040;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape20-215" + v:mID="20" + v:groupContext="shape" + transform="matrix(0.57372226,0.81904992,-0.81904992,0.57372226,835.264,-218.138)"><title + id="title429">Sheet.20</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow20-216" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(2.00559,-0.353273)" + class="st1" + style="visibility:visible"><path + d="m 0,792 62.72,0" + class="st19" + id="path432" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 62.72,0" + class="st25" + id="path434" + inkscape:connector-curvature="0" + style="stroke:#000000;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape21-222" + v:mID="21" + v:groupContext="shape" + transform="matrix(-0.68364325,0.72981635,-0.72981635,-0.68364325,985.087,779.949)"><title + id="title437">Sheet.21</title><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow21-223" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(0.0664758,-2.03538)" + class="st1" + style="visibility:visible"><path + d="m 0,792 43.41,0" + class="st19" + id="path440" + inkscape:connector-curvature="0" + style="stroke:#cdcdcd;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,792 43.41,0" + class="st24" + id="path442" + inkscape:connector-curvature="0" + style="stroke:#404040;stroke-width:1.5;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape68-229" + v:mID="68" + v:groupContext="shape" + transform="matrix(-0.78308598,0.62191345,-0.62191345,-0.78308598,933.635,852.514)"><title + id="title445">Arced arrow.68</title><v:userDefs><v:ud + v:nameU="ControlX" + v:val="VT0(0.125):0" /><v:ud + v:nameU="Scale" + v:val="VT0(1):26" /><v:ud + v:nameU="AntiScale" + v:val="VT0(1):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="ControlX" + v:prompt="" + v:val="VT0(0):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><path + d="m 0,792 a 329.322,329.322 0 0 1 367.87,0" + class="st16" + id="path447" + inkscape:connector-curvature="0" + style="stroke:#0070c0;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape70-233" + v:mID="70" + v:groupContext="shape" + transform="matrix(0.02522082,-0.9996819,0.9996819,0.02522082,-355.547,438.576)"><title + id="title450">Arced arrow.70</title><v:userDefs><v:ud + v:nameU="ControlX" + v:val="VT0(0.125):0" /><v:ud + v:nameU="Scale" + v:val="VT0(1):26" /><v:ud + v:nameU="AntiScale" + v:val="VT0(1):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="ControlX" + v:prompt="" + v:val="VT0(0.06400492):1" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><path + d="M 1.04,787.51 3.29,788.03 0,792 l -1.2,-5.01 2.24,0.52 z m 191.22,0 -2.25,0.52 3.29,3.97 1.2,-5.01 -2.24,0.52 z" + class="st26" + id="path452" + inkscape:connector-curvature="0" + style="fill:url(#grad0-236)" /><path + d="m 1.04,787.51 a 98.7059,98.7059 0 0 1 191.22,0" + class="st27" + id="path454" + inkscape:connector-curvature="0" + style="stroke:#7030a0;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="M 1.04,787.51 3.29,788.03 0,792 l -1.2,-5.01 2.24,0.52" + class="st27" + id="path456" + inkscape:connector-curvature="0" + style="stroke:#7030a0;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /><path + d="m 192.26,787.51 -2.25,0.52 3.29,3.97 1.2,-5.01 -2.24,0.52" + class="st27" + id="path458" + inkscape:connector-curvature="0" + style="stroke:#7030a0;stroke-width:1;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape84-240" + v:mID="84" + v:groupContext="shape" + transform="translate(157.5,-510.75)"><title + id="title461">Sheet.84</title><desc + id="desc463">TEP</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="11.25" + cy="753.75" + width="22.5" + height="76.5" /><rect + x="0" + y="715.5" + width="22.5" + height="76.5" + class="st28" + id="rect465" + style="fill:#8fa350;stroke:#404040;stroke-width:0.5;stroke-linecap:round;stroke-linejoin:round" /><text + x="4.54" + y="756.45001" + class="st9" + v:langID="2052" + id="text467" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />TEP</text> +</g><g + id="shape86-243" + v:mID="86" + v:groupContext="shape" + transform="translate(162,-315)"><title + id="title470">Sheet.86</title><desc + id="desc472">TEP</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="11.8125" + cy="753.75" + width="23.63" + height="76.5" /><rect + x="0" + y="715.5" + width="23.625" + height="76.5" + class="st28" + id="rect474" + style="fill:#8fa350;stroke:#404040;stroke-width:0.5;stroke-linecap:round;stroke-linejoin:round" /><text + x="5.0999999" + y="756.45001" + class="st9" + v:langID="2052" + id="text476" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />TEP</text> +</g><g + id="shape87-246" + v:mID="87" + v:groupContext="shape" + transform="translate(405,-324)"><title + id="title479">Sheet.87</title><desc + id="desc481">TEP</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="11.8125" + cy="753.75" + width="23.63" + height="76.5" /><rect + x="0" + y="715.5" + width="23.625" + height="76.5" + class="st28" + id="rect483" + style="fill:#8fa350;stroke:#404040;stroke-width:0.5;stroke-linecap:round;stroke-linejoin:round" /><text + x="5.0999999" + y="756.45001" + class="st9" + v:langID="2052" + id="text485" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />TEP</text> +</g><g + id="shape88-249" + v:mID="88" + v:groupContext="shape" + transform="translate(411.75,-513)"><title + id="title488">Sheet.88</title><desc + id="desc490">TEP</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="11.25" + cy="754.875" + width="22.5" + height="74.25" /><rect + x="0" + y="717.75" + width="22.5" + height="74.25" + class="st28" + id="rect492" + style="fill:#8fa350;stroke:#404040;stroke-width:0.5;stroke-linecap:round;stroke-linejoin:round" /><text + x="4.54" + y="757.58002" + class="st9" + v:langID="2052" + id="text494" + style="font-size:9px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />TEP</text> +</g></g></svg>
\ No newline at end of file diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/performance_thread_1.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/performance_thread_1.svg new file mode 100644 index 000000000..db01d7c24 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/performance_thread_1.svg @@ -0,0 +1,799 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="449.57141" + height="187.34319" + viewBox="0 0 449.57143 187.34319" + id="svg2" + version="1.1" + inkscape:version="0.48.3.1 r9886" + sodipodi:docname="performance_thread_1.svg" + inkscape:export-filename="C:\Users\tkulasex\Documents\L-threads\model-v2.png" + inkscape:export-xdpi="90" + inkscape:export-ydpi="90"> + <defs + id="defs4"> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker11487" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11489" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker11285" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11287" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker11107" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11109" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10757" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10759" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10421" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10423" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10273" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10275" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9983" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path9985" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9853" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path9855" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4248-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker4992-4" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path4994-2" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mend-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4254-1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker4992-4-0" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path4994-2-9" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6-8" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4248-0-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5952-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path5954-4" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5952-2-1" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path5954-4-2" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker6881-5" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path6883-0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4-2" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3-0-4" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4-2-4" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4-6" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10119-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend" + inkscape:collect="always"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10121-6" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker11487-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11489-6" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10585" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path10587" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10273-9" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10275-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10421-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10423-1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-2" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-5" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10119" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend" + inkscape:collect="always"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10121" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10923" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend" + inkscape:collect="always"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10925" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10757-4" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10759-3" + inkscape:connector-curvature="0" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1.4" + inkscape:cx="138.23152" + inkscape:cy="-30.946457" + inkscape:document-units="px" + inkscape:current-layer="g4142-7" + showgrid="false" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" + inkscape:window-width="1920" + inkscape:window-height="1148" + inkscape:window-x="0" + inkscape:window-y="0" + inkscape:window-maximized="1" + width="744.09px" /> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-40.428564,-78.569476)"> + <g + transform="translate(7.9156519e-7,106.78572)" + id="g4142-7"> + <g + transform="translate(162.14285,0.35714094)" + id="g4177-1"> + <g + transform="translate(-160.49999,-56.592401)" + id="g4142-55-1"> + <rect + y="43.076488" + x="39.285713" + height="65" + width="38.57143" + id="rect4136-65-2" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-4-8" + y="62.447506" + x="-95.515633" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="62.447506" + x="-95.515633" + id="tspan4140-2-4" + sodipodi:role="line">Port 1</tspan></text> + </g> + <rect + y="93.269798" + x="-121.21429" + height="65" + width="38.57143" + id="rect4136-8-3-7" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 1.00000002;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-8-7-3" + y="-98.052498" + x="-145.70891" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="-98.052498" + x="-145.70891" + id="tspan4140-5-8-3" + sodipodi:role="line">Port 2</tspan></text> + <g + transform="translate(-158.35713,1.6218895)" + id="g4177-7-6"> + <rect + y="1.2907723" + x="132.85715" + height="46.42857" + width="94.285713" + id="rect4171-1-9" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + sodipodi:linespacing="125%" + id="text4173-0-0" + y="29.147915" + x="146.42856" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="29.147915" + x="146.42856" + id="tspan4175-6-1" + sodipodi:role="line">rx-thread</tspan></text> + </g> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="86.642853" + y="78.626976" + id="text5627-0-5" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan5629-8-6" + x="86.642853" + y="78.626976">rings</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10757)" + d="m -83.357144,17.912679 56.42858,4.28571" + id="path4239-3-5" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10923)" + d="m -82.808124,125.71821 53.57145,-9.28573" + id="path4239-0-3-6" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 2.00000004;stroke-dashoffset:0;marker-end:url(#marker10119)" + d="m 68.78571,29.341249 62.5,28.21429" + id="path5457-1-2" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <g + transform="translate(-161.92858,95.100119)" + id="g4177-7-6-7"> + <rect + y="1.2907723" + x="132.85715" + height="46.42857" + width="94.285713" + id="rect4171-1-9-8" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + sodipodi:linespacing="125%" + id="text4173-0-0-6" + y="29.147915" + x="146.42856" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="29.147915" + x="146.42856" + id="tspan4175-6-1-8" + sodipodi:role="line">rx-thread</tspan></text> + </g> + <g + transform="translate(249.5,-71.149881)" + id="g4142-5-1-2"> + <rect + y="43.076488" + x="39.285713" + height="65" + width="38.57143" + id="rect4136-6-5-3" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-3-3-5" + y="62.447506" + x="-95.515633" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="62.447506" + x="-95.515633" + id="tspan4140-7-3-5" + sodipodi:role="line">Port 1</tspan></text> + </g> + <rect + y="74.426659" + x="288.07141" + height="65" + width="38.57143" + id="rect4136-8-4-7-7" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 1.00000002;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-8-2-5-8" + y="311.23318" + x="-126.86578" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="311.23318" + x="-126.86578" + id="tspan4140-5-4-9-6" + sodipodi:role="line">Port 2</tspan></text> + <g + id="g5905-4" + transform="translate(-1.2142913,-215.16774)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3" + x="146.42856" + y="278.34433">tx-thread</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10431)" + d="M 226.28573,52.462339 287.7143,2.8194795" + id="path4984-4-07" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10421)" + d="m 227.09388,122.75669 60.35714,9.64286" + id="path4984-1-6-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <g + id="g5905-6-0" + transform="translate(0.21427875,-156.1499)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-9-1" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-0-0" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-7-8" + x="146.42856" + y="278.34433">tx-thread</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10273)" + d="m 227.19687,67.801919 58.92857,41.071411" + id="path4984-4-0-4" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10585)" + d="M 227.30382,110.24508 286.94667,24.530799" + id="path4984-4-0-0-7" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 2.00000004;stroke-dashoffset:0;marker-end:url(#marker11487)" + d="m 66.28572,118.8909 65.71429,-2.14285" + id="path5457-1-2-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <g + id="g5905-4-6" + transform="translate(-3.5000113,-277.43173)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0-7" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6-8" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3-5" + x="146.42856" + y="278.34433">tx-thread</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 2.00000004;stroke-dashoffset:0;marker-end:url(#marker10119-2)" + d="M 68.35772,16.118199 127.64343,-6.3818105" + id="path5457-1-2-2" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10431-3)" + d="m 224.52079,-13.531251 64.28571,2.14286" + id="path4984-4-07-4" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10431-3-0)" + d="M 224.17025,2.1505695 287.02739,87.864849" + id="path4984-4-07-4-7" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + </g> + </g> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/performance_thread_2.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/performance_thread_2.svg new file mode 100644 index 000000000..48cf83383 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/performance_thread_2.svg @@ -0,0 +1,865 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="449.57141" + height="187.34319" + viewBox="0 0 449.57143 187.34319" + id="svg2" + version="1.1" + inkscape:version="0.48.3.1 r9886" + sodipodi:docname="performance_thread_2.svg" + inkscape:export-filename="C:\Users\tkulasex\Documents\L-threads\model-v2.png" + inkscape:export-xdpi="90" + inkscape:export-ydpi="90"> + <defs + id="defs4"> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker11487" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11489" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker11285" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11287" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker11107" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11109" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10757" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10759" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10421" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10423" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10273" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10275" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9983" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path9985" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9853" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path9855" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4248-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker4992-4" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path4994-2" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mend-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4254-1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker4992-4-0" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path4994-2-9" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6-8" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4248-0-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5952-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path5954-4" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5952-2-1" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path5954-4-2" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker6881-5" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + inkscape:connector-curvature="0" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path6883-0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4-2" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3-0-4" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4-2-4" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-3-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-4-6" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10119-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend" + inkscape:collect="always"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10121-6" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker11487-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path11489-6" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10585" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path10587" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10273-9" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10275-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10421-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10423-1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker10431-2" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10433-5" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10119" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend" + inkscape:collect="always"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10121" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10923" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend" + inkscape:collect="always"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10925" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10757-4" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + id="path10759-3" + inkscape:connector-curvature="0" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1.4" + inkscape:cx="138.23152" + inkscape:cy="-30.946457" + inkscape:document-units="px" + inkscape:current-layer="g4177-1" + showgrid="false" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" + inkscape:window-width="1920" + inkscape:window-height="1148" + inkscape:window-x="0" + inkscape:window-y="0" + inkscape:window-maximized="1" + width="744.09px" /> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-40.428564,-78.569476)"> + <g + transform="translate(7.9156519e-7,106.78572)" + id="g4142-7"> + <g + transform="translate(162.14285,0.35714094)" + id="g4177-1"> + <g + transform="translate(-160.49999,-56.592401)" + id="g4142-55-1"> + <rect + y="43.076488" + x="39.285713" + height="65" + width="38.57143" + id="rect4136-65-2" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-4-8" + y="62.447506" + x="-95.515633" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="62.447506" + x="-95.515633" + id="tspan4140-2-4" + sodipodi:role="line">Port 1</tspan></text> + </g> + <rect + y="93.269798" + x="-121.21429" + height="65" + width="38.57143" + id="rect4136-8-3-7" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 1.00000002;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-8-7-3" + y="-98.052498" + x="-145.70891" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="-98.052498" + x="-145.70891" + id="tspan4140-5-8-3" + sodipodi:role="line">Port 2</tspan></text> + <g + transform="translate(-158.35713,1.6218895)" + id="g4177-7-6"> + <rect + y="1.2907723" + x="132.85715" + height="46.42857" + width="94.285713" + id="rect4171-1-9" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + sodipodi:linespacing="125%" + id="text4173-0-0" + y="29.147915" + x="146.42856" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="29.147915" + x="146.42856" + id="tspan4175-6-1" + sodipodi:role="line">rx-thread</tspan></text> + </g> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="86.642853" + y="78.626976" + id="text5627-0-5" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan5629-8-6" + x="86.642853" + y="78.626976">rings</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10757)" + d="m -83.357144,17.912679 56.42858,4.28571" + id="path4239-3-5" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10923)" + d="m -82.808124,125.71821 53.57145,-9.28573" + id="path4239-0-3-6" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 2.00000004;stroke-dashoffset:0;marker-end:url(#marker10119)" + d="m 68.78571,29.341249 62.5,28.21429" + id="path5457-1-2" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <g + transform="translate(-161.92858,95.100119)" + id="g4177-7-6-7"> + <rect + y="1.2907723" + x="132.85715" + height="46.42857" + width="94.285713" + id="rect4171-1-9-8" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + sodipodi:linespacing="125%" + id="text4173-0-0-6" + y="29.147915" + x="146.42856" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="29.147915" + x="146.42856" + id="tspan4175-6-1-8" + sodipodi:role="line">rx-thread</tspan></text> + </g> + <g + transform="translate(249.5,-71.149881)" + id="g4142-5-1-2"> + <rect + y="43.076488" + x="39.285713" + height="65" + width="38.57143" + id="rect4136-6-5-3" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-3-3-5" + y="62.447506" + x="-95.515633" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="62.447506" + x="-95.515633" + id="tspan4140-7-3-5" + sodipodi:role="line">Port 1</tspan></text> + </g> + <rect + y="74.426659" + x="288.07141" + height="65" + width="38.57143" + id="rect4136-8-4-7-7" + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 1.00000002;stroke-dashoffset:0" /> + <text + transform="matrix(0,-1,1,0,0,0)" + sodipodi:linespacing="125%" + id="text4138-8-2-5-8" + y="311.23318" + x="-126.86578" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + y="311.23318" + x="-126.86578" + id="tspan4140-5-4-9-6" + sodipodi:role="line">Port 2</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10431)" + d="M 226.28573,52.462339 287.7143,2.8194795" + id="path4984-4-07" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10421)" + d="m 227.09388,122.75669 60.35714,9.64286" + id="path4984-1-6-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10273)" + d="m 227.19687,67.801919 58.92857,41.071411" + id="path4984-4-0-4" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10585)" + d="M 228.01811,113.10222 287.66096,27.387942" + id="path4984-4-0-0-7" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 2.00000004;stroke-dashoffset:0;marker-end:url(#marker11487)" + d="m 66.28572,118.8909 65.71429,-2.14285" + id="path5457-1-2-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <g + id="g5905-4-6" + transform="matrix(1,0,0,0.48279909,-0.64286832,-142.16523)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0-7" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6-8" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3-5" + x="146.42856" + y="278.34433">tx-thread</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.00000008, 2.00000004;stroke-dashoffset:0;marker-end:url(#marker10119-2)" + d="M 68.35772,16.118199 127.64343,-6.3818105" + id="path5457-1-2-2" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10431-3)" + d="m 224.52079,-13.531251 64.28571,2.14286" + id="path4984-4-07-4" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker10431-3-0)" + d="M 224.17025,2.1505695 287.02739,87.864849" + id="path4984-4-07-4-7" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <g + id="g5905-4-6-5" + transform="matrix(1,0,0,0.45244466,-0.99999222,-110.73112)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0-7-6" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6-8-7" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3-5-0" + x="146.42856" + y="278.34433">tx-drain</tspan></text> + </g> + <g + id="g5905-4-6-2" + transform="matrix(1,0,0,0.48279909,1.3158755,-80.292458)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0-7-8" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6-8-0" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3-5-6" + x="146.42856" + y="278.34433">tx-thread</tspan></text> + </g> + <g + id="g5905-4-6-5-9" + transform="matrix(1,0,0,0.45244466,0.95875552,-48.858358)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0-7-6-6" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6-8-7-4" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3-5-0-0" + x="146.42856" + y="278.34433">tx-drain</tspan></text> + </g> + <g + id="g5905-4-6-6" + transform="matrix(1,0,0,0.48279909,1.315876,-24.578174)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0-7-3" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6-8-78" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3-5-9" + x="146.42856" + y="278.34433">tx-thread</tspan></text> + </g> + <g + id="g5905-4-6-5-0" + transform="matrix(1,0,0,0.45244466,0.958756,6.8559263)"> + <rect + style="fill:none;stroke:#000000;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4, 1;stroke-dashoffset:0" + id="rect4171-9-0-0-7-6-0" + width="94.285713" + height="46.42857" + x="132.85715" + y="250.48721" /> + <text + xml:space="preserve" + style="font-size:15px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + x="146.42856" + y="278.34433" + id="text4173-9-2-6-8-7-0" + sodipodi:linespacing="125%"><tspan + sodipodi:role="line" + id="tspan4175-0-7-3-5-0-3" + x="146.42856" + y="278.34433">tx-drain</tspan></text> + </g> + </g> + </g> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/pipeline_overview.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/pipeline_overview.png Binary files differnew file mode 100644 index 000000000..218203671 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/pipeline_overview.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/ptpclient.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ptpclient.svg new file mode 100644 index 000000000..fd78ef839 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ptpclient.svg @@ -0,0 +1,528 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="88.134132mm" + height="96.380394mm" + viewBox="0 0 312.28629 341.50533" + id="svg2" + version="1.1" + inkscape:version="0.91 r13725" + sodipodi:docname="ptpclient.svg"> + <defs + id="defs4"> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4256" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6,-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4247" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(1.1,0,0,1.1,1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker6746" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Lend"> + <path + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#ff00ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path6748" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0" + refX="0" + id="marker6526" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path6528" + style="fill:#ff00ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker6276" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path6278" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#ff00ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker6128" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DotM"> + <path + transform="matrix(0.4,0,0,0.4,2.96,0.4)" + style="fill:#ff00ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="m -2.5,-1 c 0,2.76 -2.24,5 -5,5 -2.76,0 -5,-2.24 -5,-5 0,-2.76 2.24,-5 5,-5 2.76,0 5,2.24 5,5 z" + id="path6130" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker6022" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Lend" + inkscape:collect="always"> + <path + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#ff00ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path6024" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5922" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DotM"> + <path + transform="matrix(0.4,0,0,0.4,2.96,0.4)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="m -2.5,-1 c 0,2.76 -2.24,5 -5,5 -2.76,0 -5,-2.24 -5,-5 0,-2.76 2.24,-5 5,-5 2.76,0 5,2.24 5,5 z" + id="path5924" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Lend" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path4250" + style="fill:#0000ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="DotM" + orient="auto" + refY="0" + refX="0" + id="marker5676" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path5678" + d="m -2.5,-1 c 0,2.76 -2.24,5 -5,5 -2.76,0 -5,-2.24 -5,-5 0,-2.76 2.24,-5 5,-5 2.76,0 5,2.24 5,5 z" + style="fill:#0000ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.4,0,0,0.4,2.96,0.4)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="marker5600" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5602" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker5510" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DotM" + inkscape:collect="always"> + <path + transform="matrix(0.4,0,0,0.4,2.96,0.4)" + style="fill:#ff00ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + d="m -2.5,-1 c 0,2.76 -2.24,5 -5,5 -2.76,0 -5,-2.24 -5,-5 0,-2.76 2.24,-5 5,-5 2.76,0 5,2.24 5,5 z" + id="path5512" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker4651" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4653" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker4605" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4607" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4232" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="DotM" + orient="auto" + refY="0" + refX="0" + id="DotM" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path4293" + d="m -2.5,-1 c 0,2.76 -2.24,5 -5,5 -2.76,0 -5,-2.24 -5,-5 0,-2.76 2.24,-5 5,-5 2.76,0 5,2.24 5,5 z" + style="fill:#ff00ff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.4,0,0,0.4,2.96,0.4)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4229" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <inkscape:path-effect + effect="spiro" + id="path-effect3402" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect3398" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect3394" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect3378" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect3374" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect3370" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect3366" + is_visible="true" /> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1.06" + inkscape:cx="246.81474" + inkscape:cy="136.37428" + inkscape:document-units="px" + inkscape:current-layer="layer1" + showgrid="false" + showguides="false" + inkscape:window-width="1920" + inkscape:window-height="1053" + inkscape:window-x="94" + inkscape:window-y="69" + inkscape:window-maximized="1" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" /> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-33.920555,-581.05015)"> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.30149066;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + d="m 64.094156,600.92012 c 0,49.42495 0,321.26225 0,321.26225" + id="path3388" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.47424495;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + d="m 291.64401,600.59617 0,321.95932" + id="path3390" + inkscape:connector-curvature="0" /> + <path + style="fill:#ff00ff;fill-rule:evenodd;stroke:#000000;stroke-width:1.28789771;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:2;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#DotM);marker-end:url(#marker6746)" + d="M 64.094156,623.68474 289.7829,665.38811" + id="path4223" + inkscape:connector-curvature="0" /> + <path + style="fill:#ff00ff;fill-rule:evenodd;stroke:#000000;stroke-width:1.28789771;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#marker5510);marker-end:url(#marker6526)" + d="M 63.199944,683.08006 290.44233,725.29448" + id="path5274" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="295.22058" + y="665.10974" + id="text5424" + sodipodi:linespacing="100%"><tspan + sodipodi:role="line" + id="tspan5426" + x="295.22058" + y="665.10974">T2</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#800080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="242.44649" + y="657.08087" + id="text5428" + sodipodi:linespacing="100%" + transform="matrix(0.98289795,0.18415108,-0.18415108,0.98289795,0,0)"><tspan + sodipodi:role="line" + id="tspan5430" + x="242.44649" + y="657.08087" + style="font-size:17.5px">FOLLOW UP:T1</tspan></text> + <path + style="fill:#0000ff;fill-rule:evenodd;stroke:#000000;stroke-width:1.28789771;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#marker5676);marker-end:url(#Arrow2Lend)" + d="M 291.5001,774.36878 64.400801,816.4401" + id="path5586" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="-44.967201" + y="809.64362" + id="text5898" + sodipodi:linespacing="100%" + transform="matrix(0.98291896,-0.18403889,0.18403889,0.98291896,0,0)"><tspan + sodipodi:role="line" + id="tspan5900" + x="-44.967201" + y="809.64362" + style="font-size:17.5px">DELAY REQUEST</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="297.08099" + y="774.7981" + id="text5902" + sodipodi:linespacing="100%"><tspan + sodipodi:role="line" + id="tspan5904" + x="297.08099" + y="774.7981">T3</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#800080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="35.20845" + y="817.29871" + id="text5906" + sodipodi:linespacing="100%"><tspan + sodipodi:role="line" + id="tspan5908" + x="35.20845" + y="817.29871">T4</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#800080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="33.920555" + y="623.68475" + id="text5910" + sodipodi:linespacing="100%"><tspan + sodipodi:role="line" + id="tspan5912" + x="33.920555" + y="623.68475">T1</tspan></text> + <path + style="fill:#ff00ff;fill-rule:evenodd;stroke:#000000;stroke-width:1.28789771;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#marker6128);marker-end:url(#marker6022)" + d="M 63.971502,838.62056 290.6415,881.55049" + id="path5914" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#800080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="223.59686" + y="811.73932" + id="text6858" + sodipodi:linespacing="100%" + transform="matrix(0.98367106,0.17997568,-0.17997568,0.98367106,0,0)"><tspan + sodipodi:role="line" + id="tspan6860" + x="223.59686" + y="811.73932" + style="font-size:17.5px">DELAY RESPONSE:T4</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.39868927px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="698.62549" + y="-320.85892" + id="text6862" + sodipodi:linespacing="100%" + transform="matrix(0.0147127,0.99989176,-0.99989176,0.0147127,0,0)"><tspan + sodipodi:role="line" + id="tspan6864" + x="698.62549" + y="-320.85892">time</tspan></text> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.1464963;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend)" + d="m 337.72657,748.13938 0,66.11208" + id="path6866" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13.42257881px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#800080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="38.764343" + y="590.47479" + id="text7094" + sodipodi:linespacing="100%"><tspan + sodipodi:role="line" + id="tspan7096" + x="38.764343" + y="590.47479">master</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:47.51625061px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="571.69812" + y="61.796165" + id="text7098" + sodipodi:linespacing="100%"><tspan + sodipodi:role="line" + id="tspan7100" + x="571.69812" + y="61.796165" /></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:13.42257881px;line-height:100%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="271.23392" + y="593.71478" + id="text7102" + sodipodi:linespacing="100%"><tspan + sodipodi:role="line" + id="tspan7104" + x="271.23392" + y="593.71478">slave</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:20.3917141px;line-height:125%;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Normal';text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#800080;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="244.40062" + y="602.10443" + id="text4317" + sodipodi:linespacing="125%" + transform="matrix(0.98605316,0.16643065,-0.16643065,0.98605316,0,0)"><tspan + sodipodi:role="line" + id="tspan4319" + x="244.40062" + y="602.10443" + style="font-size:17.5px">SYNC</tspan></text> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/qos_sched_app_arch.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/qos_sched_app_arch.png Binary files differnew file mode 100644 index 000000000..63294367f --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/qos_sched_app_arch.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/quickassist_block_diagram.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/quickassist_block_diagram.png Binary files differnew file mode 100644 index 000000000..61043eae0 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/quickassist_block_diagram.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/ring_pipeline_perf_setup.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ring_pipeline_perf_setup.png Binary files differnew file mode 100644 index 000000000..d8820c081 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/ring_pipeline_perf_setup.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/server_node_efd.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/server_node_efd.svg new file mode 100644 index 000000000..9aee30bc2 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/server_node_efd.svg @@ -0,0 +1,1254 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export efd_i6.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="8.2496in" height="5.89673in" + viewBox="0 0 593.971 424.565" xml:space="preserve" color-interpolation-filters="sRGB" class="st27"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"> + <v:userDefs> + <v:ud v:nameU="msvSubprocessMaster" v:prompt="" v:val="VT4(Rectangle)"/> + <v:ud v:nameU="msvNoAutoConnect" v:val="VT0(1):26"/> + </v:userDefs> + </v:documentProperties> + + <style type="text/css"> + <![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} + .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25} + .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em} + .st5 {fill:#feffff;font-family:Calibri;font-size:0.75em} + .st6 {fill:none;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} + .st7 {fill:none;stroke:#2e75b5;stroke-width:2.25} + .st8 {fill:#305497;stroke:#2e75b5;stroke-width:1} + .st9 {fill:#feffff;font-family:Calibri;font-size:0.833336em;font-weight:bold} + .st10 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2)} + .st11 {fill:#5b9bd5} + .st12 {stroke:#c7c8c8;stroke-width:0.25} + .st13 {fill:#acccea;stroke:#c7c8c8;stroke-width:0.25} + .st14 {fill:#feffff;font-family:Calibri;font-size:1.00001em;font-weight:bold} + .st15 {fill:#ed7d31;stroke:#c7c8c8;stroke-width:0.25} + .st16 {fill:#deebf6;stroke:#c7c8c8;stroke-width:0.25} + .st17 {marker-end:url(#mrkr5-212);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st18 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.28409090909091} + .st19 {fill:none;stroke:#2e75b5;stroke-width:1} + .st20 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em} + .st21 {fill:none;stroke:none;stroke-width:0.25} + .st22 {font-size:1em} + .st23 {fill:#ffffff} + .st24 {stroke:#5b9bd5;stroke-dasharray:1.5,3;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st25 {marker-end:url(#mrkr5-444);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5} + .st26 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.37313432835821} + .st27 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <defs id="Markers"> + <g id="lend5"> + <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/> + </g> + <marker id="mrkr5-212" class="st18" v:arrowType="5" v:arrowSize="2" v:setback="5.8" refX="-5.8" orient="auto" + markerUnits="strokeWidth" overflow="visible"> + <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/> + </marker> + <marker id="mrkr5-444" class="st26" v:arrowType="5" v:arrowSize="2" v:setback="4.69" refX="-4.69" orient="auto" + markerUnits="strokeWidth" overflow="visible"> + <use xlink:href="#lend5" transform="scale(-2.68,-2.68) "/> + </marker> + </defs> + <defs id="Filters"> + <filter id="filter_2"> + <feGaussianBlur stdDeviation="2"/> + </filter> + </defs> + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <v:userDefs> + <v:ud v:nameU="msvThemeOrder" v:val="VT0(0):26"/> + </v:userDefs> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(319.501,-335.688)"> + <title>Rectangle.58</title> + <desc>Key 1</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow3-2" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="4.74" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key 1</text> </g> + <g id="shape4-7" v:mID="4" v:groupContext="shape" transform="translate(353.251,-335.688)"> + <title>Rectangle.59</title> + <desc>Action 1</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow4-8" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.62" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action 1</text> </g> + <g id="shape5-13" v:mID="5" v:groupContext="shape" transform="translate(400.501,-335.688)"> + <title>Rectangle.60</title> + <desc>Key 2</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow5-14" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="4.74" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key 2</text> </g> + <g id="shape6-19" v:mID="6" v:groupContext="shape" transform="translate(434.251,-335.688)"> + <title>Rectangle.61</title> + <desc>Action 2</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow6-20" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.62" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action 2</text> </g> + <g id="shape7-25" v:mID="7" v:groupContext="shape" transform="translate(481.501,-335.688)"> + <title>Rectangle.62</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow7-26" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape8-30" v:mID="8" v:groupContext="shape" transform="translate(515.251,-335.688)"> + <title>Rectangle.63</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow8-31" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape9-35" v:mID="9" v:groupContext="shape" transform="translate(319.501,-313.188)"> + <title>Rectangle.64</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow9-36" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape10-40" v:mID="10" v:groupContext="shape" transform="translate(353.251,-313.188)"> + <title>Rectangle.65</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow10-41" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape11-45" v:mID="11" v:groupContext="shape" transform="translate(400.501,-313.188)"> + <title>Rectangle.66</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow11-46" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape12-50" v:mID="12" v:groupContext="shape" transform="translate(434.251,-313.188)"> + <title>Rectangle.67</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow12-51" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape13-55" v:mID="13" v:groupContext="shape" transform="translate(481.501,-313.188)"> + <title>Rectangle.68</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow13-56" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape14-60" v:mID="14" v:groupContext="shape" transform="translate(515.251,-313.188)"> + <title>Rectangle.69</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow14-61" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape15-65" v:mID="15" v:groupContext="shape" transform="translate(319.501,-277.188)"> + <title>Rectangle.70</title> + <desc>Key x</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow15-66" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.11" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key x</text> </g> + <g id="shape16-71" v:mID="16" v:groupContext="shape" transform="translate(353.251,-277.188)"> + <title>Rectangle.71</title> + <desc>Action x</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow16-72" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.99" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action x</text> </g> + <g id="shape17-77" v:mID="17" v:groupContext="shape" transform="translate(400.501,-277.188)"> + <title>Rectangle.72</title> + <desc>Key y</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow17-78" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.01" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key y</text> </g> + <g id="shape18-83" v:mID="18" v:groupContext="shape" transform="translate(434.251,-277.188)"> + <title>Rectangle.73</title> + <desc>Action y</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow18-84" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.89" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action y</text> </g> + <g id="shape19-89" v:mID="19" v:groupContext="shape" transform="translate(481.501,-277.188)"> + <title>Rectangle.74</title> + <desc>Key z</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow19-90" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.3" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key z</text> </g> + <g id="shape20-95" v:mID="20" v:groupContext="shape" transform="translate(515.251,-277.188)"> + <title>Rectangle.75</title> + <desc>Action z</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow20-96" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="5.18" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action z</text> </g> + <g id="shape21-101" v:mID="21" v:groupContext="shape" transform="translate(319.501,-240.687)"> + <title>Rectangle.76</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow21-102" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape22-106" v:mID="22" v:groupContext="shape" transform="translate(353.251,-240.687)"> + <title>Rectangle.77</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow22-107" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape23-111" v:mID="23" v:groupContext="shape" transform="translate(400.501,-240.687)"> + <title>Rectangle.78</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow23-112" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape24-116" v:mID="24" v:groupContext="shape" transform="translate(434.251,-240.687)"> + <title>Rectangle.79</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow24-117" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape25-121" v:mID="25" v:groupContext="shape" transform="translate(481.501,-240.687)"> + <title>Rectangle.80</title> + <desc>Key N</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow25-122" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.21" y="418.26" class="st5" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key N</text> </g> + <g id="shape26-127" v:mID="26" v:groupContext="shape" transform="translate(515.251,-240.687)"> + <title>Rectangle.81</title> + <desc>Action N</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow26-128" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="5.67" y="418.26" class="st5" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action N</text> </g> + <g id="shape27-133" v:mID="27" v:groupContext="shape" transform="translate(317.251,-231.687)"> + <title>Rectangle.82</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow27-134" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="289.065" width="245.25" height="135.5" class="st6"/> + </g> + <rect x="0" y="289.065" width="245.25" height="135.5" class="st7"/> + </g> + <g id="shape28-138" v:mID="28" v:groupContext="shape" transform="translate(328.501,-362.688)"> + <title>Sheet.28</title> + <desc>Local Table for N Specific Flows Serviced at Node 1</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="110.423" cy="418.94" width="220.85" height="11.25"/> + <rect x="0" y="413.315" width="220.846" height="11.25" class="st8"/> + <text x="5.77" y="421.94" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Local Table for N Specific Flows Serviced at Node 1</text> </g> + <g id="group34-141" transform="translate(66.0294,-165.569)" v:mID="34" v:groupContext="group"> + <v:custProps> + <v:cp v:nameU="AssetNumber" v:lbl="Asset Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SerialNumber" v:lbl="Serial Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Location" v:lbl="Location" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Building" v:lbl="Building" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Room" v:lbl="Room" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Manufacturer" v:lbl="Manufacturer" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductNumber" v:lbl="Product Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="PartNumber" v:lbl="Part Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductDescription" v:lbl="Product Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkName" v:lbl="Network Name" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="IPAddress" v:lbl="IP Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SubnetMask" v:lbl="Subnet Mask" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="AdminInterface" v:lbl="Administrative Interface" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NumberofPorts" v:lbl="Number of Ports" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="CommunityString" v:lbl="Community String" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkDescription" v:lbl="Network Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="MACAddress" v:lbl="MAC Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ShapeClass" v:lbl="ShapeClass" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Equipment)"/> + <v:cp v:nameU="ShapeType" v:lbl="ShapeType" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Device)"/> + <v:cp v:nameU="SubShapeType" v:lbl="SubShapeType" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Load balancer)"/> + </v:custProps> + <v:userDefs> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="ShapeClass" v:prompt="" v:val="VT0(5):26"/> + <v:ud v:nameU="SolSH" v:prompt="" v:val="VT15({BF0433D9-CD73-4EB5-8390-8653BE590246}):41"/> + <v:ud v:nameU="visLegendShape" v:prompt="" v:val="VT0(2):26"/> + </v:userDefs> + <title>Load balancer</title> + <g id="shape35-142" v:mID="35" v:groupContext="shape" transform="translate(0,-7.33146)"> + <title>Sheet.35</title> + <g id="shadow35-143" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M54 367.23 L18 367.23 L0 377.86 L0 424.56 L72 424.56 L72 377.86 L54 367.23 Z" class="st10"/> + <path d="M0 377.86 L72 377.86" class="st6"/> + <path d="M54 367.23 L18 367.23 L0 377.86 L0 424.56 L72 424.56 L72 377.86 L54 367.23" class="st6"/> + </g> + <path d="M54 367.23 L18 367.23 L0 377.86 L0 424.56 L72 424.56 L72 377.86 L54 367.23 Z" class="st11"/> + <path d="M0 377.86 L72 377.86" class="st12"/> + <path d="M54 367.23 L18 367.23 L0 377.86 L0 424.56 L72 424.56 L72 377.86 L54 367.23" class="st12"/> + </g> + <g id="shape36-152" v:mID="36" v:groupContext="shape" transform="translate(8.03054,-12.9324)"> + <title>Sheet.36</title> + <g id="shadow36-153" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M45.34 421.81 L41.2 422.66 L44.12 424.56 L49.68 423.16 L48.75 417.51 L45.8 415.59 L46.69 419.68 L36.97 + 413.34 L35.6 415.45 L45.34 421.81 ZM50.83 405.36 L39.2 405.36 L39.2 407.88 L50.8 407.88 L47.82 410.83 + L51.34 410.83 L55.21 406.61 L51.32 402.39 L47.83 402.39 L50.83 405.36 ZM46.49 392.01 L36.75 398.37 + L38.13 400.48 L47.84 394.14 L46.96 398.23 L49.91 396.31 L50.84 390.66 L45.28 389.26 L42.36 391.16 + L46.49 392.01 ZM27.71 397.16 C22.66 397.16 18.58 401.25 18.58 406.29 C18.58 411.33 22.66 415.42 + 27.71 415.42 C32.75 415.42 36.84 411.33 36.84 406.29 C36.84 401.25 32.75 397.16 27.71 397.16 ZM27.71 + 400.04 C31.15 400.04 33.96 402.84 33.96 406.29 C33.96 409.74 31.15 412.54 27.71 412.54 C24.26 412.54 + 21.46 409.74 21.46 406.29 C21.46 402.84 24.26 400.04 27.71 400.04 ZM11.64 405.04 L0 405.04 L0 407.56 + L11.6 407.56 L8.62 410.51 L12.14 410.51 L16.01 406.29 L12.12 402.07 L8.64 402.07 L11.64 405.04 Z" + class="st2"/> + </g> + <path d="M45.34 421.81 L41.2 422.66 L44.12 424.56 L49.68 423.16 L48.75 417.51 L45.8 415.59 L46.69 419.68 L36.97 413.34 + L35.6 415.45 L45.34 421.81 ZM50.83 405.36 L39.2 405.36 L39.2 407.88 L50.8 407.88 L47.82 410.83 L51.34 + 410.83 L55.21 406.61 L51.32 402.39 L47.83 402.39 L50.83 405.36 ZM46.49 392.01 L36.75 398.37 L38.13 400.48 + L47.84 394.14 L46.96 398.23 L49.91 396.31 L50.84 390.66 L45.28 389.26 L42.36 391.16 L46.49 392.01 ZM27.71 + 397.16 C22.66 397.16 18.58 401.25 18.58 406.29 C18.58 411.33 22.66 415.42 27.71 415.42 C32.75 415.42 + 36.84 411.33 36.84 406.29 C36.84 401.25 32.75 397.16 27.71 397.16 ZM27.71 400.04 C31.15 400.04 33.96 + 402.84 33.96 406.29 C33.96 409.74 31.15 412.54 27.71 412.54 C24.26 412.54 21.46 409.74 21.46 406.29 + C21.46 402.84 24.26 400.04 27.71 400.04 ZM11.64 405.04 L0 405.04 L0 407.56 L11.6 407.56 L8.62 410.51 + L12.14 410.51 L16.01 406.29 L12.12 402.07 L8.64 402.07 L11.64 405.04 Z" class="st13"/> + </g> + </g> + <g id="shape37-157" v:mID="37" v:groupContext="shape" transform="translate(21.0294,-45.4375)"> + <title>Rectangle</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow37-158" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="336.433" width="135" height="88.1315" class="st2"/> + </g> + <rect x="0" y="336.433" width="135" height="88.1315" class="st3"/> + </g> + <g id="shape38-162" v:mID="38" v:groupContext="shape" transform="translate(34.693,-126.438)"> + <title>Sheet.38</title> + <desc>EFD Table</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="49.3364" cy="415.565" width="98.68" height="18"/> + <rect x="0" y="406.565" width="98.6728" height="18" class="st8"/> + <text x="24.87" y="419.17" class="st14" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>EFD Table</text> </g> + <g id="shape39-165" v:mID="39" v:groupContext="shape" transform="translate(30.0294,-99.4375)"> + <title>Rectangle.39</title> + <desc>Group_id</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="26.9182" cy="415.565" width="53.84" height="18"/> + <g id="shadow39-166" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="53.8364" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="53.8364" height="18" class="st15"/> + <text x="7.87" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Group_id</text> </g> + <g id="shape40-171" v:mID="40" v:groupContext="shape" transform="translate(93.193,-99.4375)"> + <title>Rectangle.40</title> + <desc>Hash index</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="26.9182" cy="415.565" width="53.84" height="18"/> + <g id="shadow40-172" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="53.8364" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="53.8364" height="18" class="st15"/> + <text x="4.64" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Hash index</text> </g> + <g id="shape41-177" v:mID="41" v:groupContext="shape" transform="translate(30.193,-82.4275)"> + <title>Rectangle.41</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow41-178" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st2"/> + </g> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st16"/> + </g> + <g id="shape42-182" v:mID="42" v:groupContext="shape" transform="translate(30.193,-66.8125)"> + <title>Rectangle.42</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow42-183" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st2"/> + </g> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st16"/> + </g> + <g id="shape43-187" v:mID="43" v:groupContext="shape" transform="translate(30.1112,-52.1875)"> + <title>Rectangle.43</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow43-188" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st2"/> + </g> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st16"/> + </g> + <g id="shape44-192" v:mID="44" v:groupContext="shape" transform="translate(93.0294,-81.4375)"> + <title>Rectangle.44</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow44-193" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st2"/> + </g> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st16"/> + </g> + <g id="shape45-197" v:mID="45" v:groupContext="shape" transform="translate(93.193,-66.8125)"> + <title>Rectangle.45</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow45-198" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st2"/> + </g> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st16"/> + </g> + <g id="shape46-202" v:mID="46" v:groupContext="shape" transform="translate(93.193,-52.1875)"> + <title>Rectangle.46</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow46-203" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st2"/> + </g> + <rect x="0" y="413.315" width="53.8364" height="11.25" class="st16"/> + </g> + <g id="shape47-207" v:mID="47" v:groupContext="shape" transform="translate(374.924,544.022) rotate(135)"> + <title>Sheet.47</title> + <path d="M-0 417.75 A40.674 18.0151 -156.2 0 0 40.24 422.15 L40.49 421.89" class="st17"/> + </g> + <g id="shape48-213" v:mID="48" v:groupContext="shape" transform="translate(21.0294,-19)"> + <title>Sheet.48</title> + <desc>Supports X*N Flows</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="67.5" cy="415.565" width="135" height="18"/> + <rect x="0" y="406.565" width="135" height="18" class="st19"/> + <text x="19.05" y="419.17" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Supports X*N Flows</text> </g> + <g id="shape49-216" v:mID="49" v:groupContext="shape" transform="translate(48.0294,-229.938)"> + <title>Sheet.49</title> + <desc>Frontend Server or Load Balancer</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="54" cy="400.94" width="108" height="47.25"/> + <rect x="0" y="377.315" width="108" height="47.25" class="st21"/> + <text x="14.56" y="397.34" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Frontend Server<v:newlineChar/><tspan + x="13.16" dy="1.2em" class="st22">or Load Balancer </tspan> </text> </g> + <g id="group51-220" transform="translate(223.876,-310.938)" v:mID="51" v:groupContext="group"> + <v:custProps> + <v:cp v:nameU="AssetNumber" v:lbl="Asset Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SerialNumber" v:lbl="Serial Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Location" v:lbl="Location" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Building" v:lbl="Building" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Room" v:lbl="Room" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Manufacturer" v:lbl="Manufacturer" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductNumber" v:lbl="Product Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="PartNumber" v:lbl="Part Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductDescription" v:lbl="Product Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkName" v:lbl="Network Name" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="IPAddress" v:lbl="IP Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SubnetMask" v:lbl="Subnet Mask" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="AdminInterface" v:lbl="Administrative Interface" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NumberofPorts" v:lbl="Number of Ports" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="CommunityString" v:lbl="Community String" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkDescription" v:lbl="Network Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="MACAddress" v:lbl="MAC Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="CPU" v:lbl="CPU" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Memory" v:lbl="Memory" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="OperatingSystem" v:lbl="Operating System" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="HardDriveSize" v:lbl="Hard Drive Capacity" v:prompt="" v:type="0" v:format="" + v:sortKey="Workstation" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Department" v:lbl="Department" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ShapeClass" v:lbl="ShapeClass" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Equipment)"/> + <v:cp v:nameU="ShapeType" v:lbl="ShapeType" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Server)"/> + <v:cp v:nameU="BelongsTo" v:lbl="Belongs To" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0"/> + </v:custProps> + <v:userDefs> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="ShapeClass" v:prompt="" v:val="VT0(5):26"/> + <v:ud v:nameU="SolSH" v:prompt="" v:val="VT15({BF0433D9-CD73-4EB5-8390-8653BE590246}):41"/> + <v:ud v:nameU="visLegendShape" v:prompt="" v:val="VT0(2):26"/> + </v:userDefs> + <title>Server</title> + <g id="shape52-221" v:mID="52" v:groupContext="shape" transform="translate(13.0183,0)"> + <title>Sheet.52</title> + <g id="shadow52-222" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="352.565" width="45.9634" height="72" class="st2"/> + </g> + <rect x="0" y="352.565" width="45.9634" height="72" class="st3"/> + </g> + <g id="shape53-226" v:mID="53" v:groupContext="shape" transform="translate(47.371,-30.7354)"> + <title>Sheet.53</title> + <g id="shadow53-227" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <ellipse cx="2.77848" cy="421.786" rx="2.77848" ry="2.77848" class="st2"/> + </g> + <ellipse cx="2.77848" cy="421.786" rx="2.77848" ry="2.77848" class="st13"/> + </g> + <g id="shape54-231" v:mID="54" v:groupContext="shape" transform="translate(30.51,-11.8022)"> + <title>Sheet.54</title> + <v:userDefs> + <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(1)"/> + <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/> + </v:userDefs> + <g id="shadow54-232" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M-0 424.56 L22.42 424.56 L22.42 422.76 L-0 422.76 L-0 424.56 ZM-0 419.11 L22.42 419.11 L22.42 417.31 + L-0 417.31 L-0 419.11 ZM-0 413.65 L22.42 413.65 L22.42 411.84 L-0 411.84 L-0 413.65 Z" + class="st10"/> + </g> + <path d="M-0 424.56 L22.42 424.56 L22.42 422.76 L-0 422.76 L-0 424.56 ZM-0 419.11 L22.42 419.11 L22.42 417.31 L-0 + 417.31 L-0 419.11 ZM-0 413.65 L22.42 413.65 L22.42 411.84 L-0 411.84 L-0 413.65 Z" class="st23"/> + </g> + </g> + <g id="shape59-239" v:mID="59" v:groupContext="shape" transform="translate(277.876,-373.938)"> + <title>Sheet.59</title> + <path d="M-0 424.56 A111.108 53.2538 42.31 0 1 93.83 421.21 L94.14 421.41" class="st17"/> + </g> + <g id="shape60-244" v:mID="60" v:groupContext="shape" transform="translate(205.876,-283.938)"> + <title>Sheet.60</title> + <desc>Backend Server 1</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="54" cy="408.124" width="108" height="32.8815"/> + <rect x="0" y="391.683" width="108" height="32.8815" class="st21"/> + <text x="11.93" y="411.72" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Backend Server 1</text> </g> + <g id="group61-247" transform="translate(223.876,-207.438)" v:mID="61" v:groupContext="group"> + <v:custProps> + <v:cp v:nameU="AssetNumber" v:lbl="Asset Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SerialNumber" v:lbl="Serial Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Location" v:lbl="Location" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Building" v:lbl="Building" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Room" v:lbl="Room" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Manufacturer" v:lbl="Manufacturer" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductNumber" v:lbl="Product Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="PartNumber" v:lbl="Part Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductDescription" v:lbl="Product Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkName" v:lbl="Network Name" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="IPAddress" v:lbl="IP Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SubnetMask" v:lbl="Subnet Mask" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="AdminInterface" v:lbl="Administrative Interface" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NumberofPorts" v:lbl="Number of Ports" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="CommunityString" v:lbl="Community String" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkDescription" v:lbl="Network Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="MACAddress" v:lbl="MAC Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="CPU" v:lbl="CPU" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Memory" v:lbl="Memory" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="OperatingSystem" v:lbl="Operating System" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="HardDriveSize" v:lbl="Hard Drive Capacity" v:prompt="" v:type="0" v:format="" + v:sortKey="Workstation" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Department" v:lbl="Department" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ShapeClass" v:lbl="ShapeClass" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Equipment)"/> + <v:cp v:nameU="ShapeType" v:lbl="ShapeType" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Server)"/> + <v:cp v:nameU="BelongsTo" v:lbl="Belongs To" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0"/> + </v:custProps> + <v:userDefs> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="ShapeClass" v:prompt="" v:val="VT0(5):26"/> + <v:ud v:nameU="SolSH" v:prompt="" v:val="VT15({BF0433D9-CD73-4EB5-8390-8653BE590246}):41"/> + <v:ud v:nameU="visLegendShape" v:prompt="" v:val="VT0(2):26"/> + </v:userDefs> + <title>Server.61</title> + <g id="shape62-248" v:mID="62" v:groupContext="shape" transform="translate(13.0183,0)"> + <title>Sheet.62</title> + <g id="shadow62-249" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="352.565" width="45.9634" height="72" class="st2"/> + </g> + <rect x="0" y="352.565" width="45.9634" height="72" class="st3"/> + </g> + <g id="shape63-253" v:mID="63" v:groupContext="shape" transform="translate(47.371,-30.7354)"> + <title>Sheet.63</title> + <g id="shadow63-254" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <ellipse cx="2.77848" cy="421.786" rx="2.77848" ry="2.77848" class="st2"/> + </g> + <ellipse cx="2.77848" cy="421.786" rx="2.77848" ry="2.77848" class="st13"/> + </g> + <g id="shape64-258" v:mID="64" v:groupContext="shape" transform="translate(30.51,-11.8022)"> + <title>Sheet.64</title> + <v:userDefs> + <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(1)"/> + <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/> + </v:userDefs> + <g id="shadow64-259" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M-0 424.56 L22.42 424.56 L22.42 422.76 L-0 422.76 L-0 424.56 ZM-0 419.11 L22.42 419.11 L22.42 417.31 + L-0 417.31 L-0 419.11 ZM-0 413.65 L22.42 413.65 L22.42 411.84 L-0 411.84 L-0 413.65 Z" + class="st10"/> + </g> + <path d="M-0 424.56 L22.42 424.56 L22.42 422.76 L-0 422.76 L-0 424.56 ZM-0 419.11 L22.42 419.11 L22.42 417.31 L-0 + 417.31 L-0 419.11 ZM-0 413.65 L22.42 413.65 L22.42 411.84 L-0 411.84 L-0 413.65 Z" class="st23"/> + </g> + </g> + <g id="shape65-266" v:mID="65" v:groupContext="shape" transform="translate(205.876,-180.437)"> + <title>Sheet.65</title> + <desc>Backend Server 2</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="54" cy="408.124" width="108" height="32.8815"/> + <rect x="0" y="391.683" width="108" height="32.8815" class="st21"/> + <text x="11.93" y="411.72" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Backend Server 2</text> </g> + <g id="group66-269" transform="translate(219.029,-58.9375)" v:mID="66" v:groupContext="group"> + <v:custProps> + <v:cp v:nameU="AssetNumber" v:lbl="Asset Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SerialNumber" v:lbl="Serial Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Location" v:lbl="Location" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Building" v:lbl="Building" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Room" v:lbl="Room" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Manufacturer" v:lbl="Manufacturer" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductNumber" v:lbl="Product Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="PartNumber" v:lbl="Part Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ProductDescription" v:lbl="Product Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkName" v:lbl="Network Name" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="IPAddress" v:lbl="IP Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="SubnetMask" v:lbl="Subnet Mask" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="AdminInterface" v:lbl="Administrative Interface" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NumberofPorts" v:lbl="Number of Ports" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="CommunityString" v:lbl="Community String" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="NetworkDescription" v:lbl="Network Description" v:prompt="" v:type="0" v:format="" + v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="MACAddress" v:lbl="MAC Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="CPU" v:lbl="CPU" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Memory" v:lbl="Memory" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="OperatingSystem" v:lbl="Operating System" v:prompt="" v:type="0" v:format="" v:sortKey="Workstation" + v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="HardDriveSize" v:lbl="Hard Drive Capacity" v:prompt="" v:type="0" v:format="" + v:sortKey="Workstation" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="Department" v:lbl="Department" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="false" + v:ask="false" v:langID="1033" v:cal="0"/> + <v:cp v:nameU="ShapeClass" v:lbl="ShapeClass" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Equipment)"/> + <v:cp v:nameU="ShapeType" v:lbl="ShapeType" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Server)"/> + <v:cp v:nameU="BelongsTo" v:lbl="Belongs To" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true" + v:ask="false" v:langID="1033" v:cal="0"/> + </v:custProps> + <v:userDefs> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="ShapeClass" v:prompt="" v:val="VT0(5):26"/> + <v:ud v:nameU="SolSH" v:prompt="" v:val="VT15({BF0433D9-CD73-4EB5-8390-8653BE590246}):41"/> + <v:ud v:nameU="visLegendShape" v:prompt="" v:val="VT0(2):26"/> + </v:userDefs> + <title>Server.66</title> + <g id="shape67-270" v:mID="67" v:groupContext="shape" transform="translate(13.0183,0)"> + <title>Sheet.67</title> + <g id="shadow67-271" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="352.565" width="45.9634" height="72" class="st2"/> + </g> + <rect x="0" y="352.565" width="45.9634" height="72" class="st3"/> + </g> + <g id="shape68-275" v:mID="68" v:groupContext="shape" transform="translate(47.371,-30.7354)"> + <title>Sheet.68</title> + <g id="shadow68-276" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <ellipse cx="2.77848" cy="421.786" rx="2.77848" ry="2.77848" class="st2"/> + </g> + <ellipse cx="2.77848" cy="421.786" rx="2.77848" ry="2.77848" class="st13"/> + </g> + <g id="shape69-280" v:mID="69" v:groupContext="shape" transform="translate(30.51,-11.8022)"> + <title>Sheet.69</title> + <v:userDefs> + <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(1)"/> + <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/> + </v:userDefs> + <g id="shadow69-281" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M-0 424.56 L22.42 424.56 L22.42 422.76 L-0 422.76 L-0 424.56 ZM-0 419.11 L22.42 419.11 L22.42 417.31 + L-0 417.31 L-0 419.11 ZM-0 413.65 L22.42 413.65 L22.42 411.84 L-0 411.84 L-0 413.65 Z" + class="st10"/> + </g> + <path d="M-0 424.56 L22.42 424.56 L22.42 422.76 L-0 422.76 L-0 424.56 ZM-0 419.11 L22.42 419.11 L22.42 417.31 L-0 + 417.31 L-0 419.11 ZM-0 413.65 L22.42 413.65 L22.42 411.84 L-0 411.84 L-0 413.65 Z" class="st23"/> + </g> + </g> + <g id="shape70-288" v:mID="70" v:groupContext="shape" transform="translate(201.029,-26.056)"> + <title>Sheet.70</title> + <desc>Backend Server X</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="54" cy="408.124" width="108" height="32.8815"/> + <rect x="0" y="391.683" width="108" height="32.8815" class="st21"/> + <text x="11.86" y="411.72" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Backend Server X</text> </g> + <g id="shape71-291" v:mID="71" v:groupContext="shape" transform="translate(684.44,239.627) rotate(90)"> + <title>Sheet.71</title> + <path d="M0 424.56 L45 424.56" class="st24"/> + </g> + <g id="shape72-294" v:mID="72" v:groupContext="shape" transform="translate(6.85967,-22.443) rotate(-38.1076)"> + <title>Sheet.72</title> + <path d="M-0 424.56 A96.1331 44.4001 55.03 0 1 68.24 420.56 L68.51 420.79" class="st17"/> + </g> + <g id="shape73-299" v:mID="73" v:groupContext="shape" transform="translate(328.501,-135.937)"> + <title>Rectangle.73</title> + <desc>Key 1</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow73-300" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="4.74" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key 1</text> </g> + <g id="shape74-305" v:mID="74" v:groupContext="shape" transform="translate(362.251,-135.937)"> + <title>Rectangle.74</title> + <desc>Action 1</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow74-306" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.62" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action 1</text> </g> + <g id="shape75-311" v:mID="75" v:groupContext="shape" transform="translate(409.501,-135.937)"> + <title>Rectangle.75</title> + <desc>Key 2</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow75-312" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="4.74" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key 2</text> </g> + <g id="shape76-317" v:mID="76" v:groupContext="shape" transform="translate(443.251,-135.937)"> + <title>Rectangle.76</title> + <desc>Action 2</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow76-318" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.62" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action 2</text> </g> + <g id="shape77-323" v:mID="77" v:groupContext="shape" transform="translate(490.501,-135.937)"> + <title>Rectangle.77</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow77-324" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape78-328" v:mID="78" v:groupContext="shape" transform="translate(524.251,-135.937)"> + <title>Rectangle.78</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow78-329" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape79-333" v:mID="79" v:groupContext="shape" transform="translate(328.501,-113.437)"> + <title>Rectangle.79</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow79-334" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape80-338" v:mID="80" v:groupContext="shape" transform="translate(362.251,-113.437)"> + <title>Rectangle.80</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow80-339" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape81-343" v:mID="81" v:groupContext="shape" transform="translate(409.501,-113.437)"> + <title>Rectangle.81</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow81-344" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape82-348" v:mID="82" v:groupContext="shape" transform="translate(443.251,-113.437)"> + <title>Rectangle.82</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow82-349" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape83-353" v:mID="83" v:groupContext="shape" transform="translate(490.501,-113.437)"> + <title>Rectangle.83</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow83-354" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape84-358" v:mID="84" v:groupContext="shape" transform="translate(524.251,-113.437)"> + <title>Rectangle.84</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow84-359" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape85-363" v:mID="85" v:groupContext="shape" transform="translate(328.501,-77.4375)"> + <title>Rectangle.85</title> + <desc>Key x</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow85-364" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.11" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key x</text> </g> + <g id="shape86-369" v:mID="86" v:groupContext="shape" transform="translate(362.251,-77.4375)"> + <title>Rectangle.86</title> + <desc>Action x</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow86-370" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.99" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action x</text> </g> + <g id="shape87-375" v:mID="87" v:groupContext="shape" transform="translate(409.501,-77.4375)"> + <title>Rectangle.87</title> + <desc>Key y</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow87-376" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.01" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key y</text> </g> + <g id="shape88-381" v:mID="88" v:groupContext="shape" transform="translate(443.251,-77.4375)"> + <title>Rectangle.88</title> + <desc>Action y</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow88-382" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="4.89" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action y</text> </g> + <g id="shape89-387" v:mID="89" v:groupContext="shape" transform="translate(490.501,-77.4375)"> + <title>Rectangle.89</title> + <desc>Key z</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow89-388" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.3" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key z</text> </g> + <g id="shape90-393" v:mID="90" v:groupContext="shape" transform="translate(524.251,-77.4375)"> + <title>Rectangle.90</title> + <desc>Action z</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow90-394" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="5.18" y="418.56" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action z</text> </g> + <g id="shape91-399" v:mID="91" v:groupContext="shape" transform="translate(328.501,-40.9375)"> + <title>Rectangle.91</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow91-400" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape92-404" v:mID="92" v:groupContext="shape" transform="translate(362.251,-40.9375)"> + <title>Rectangle.92</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow92-405" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape93-409" v:mID="93" v:groupContext="shape" transform="translate(409.501,-40.9375)"> + <title>Rectangle.93</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow93-410" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + </g> + <g id="shape94-414" v:mID="94" v:groupContext="shape" transform="translate(443.251,-40.9375)"> + <title>Rectangle.94</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow94-415" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + </g> + <g id="shape95-419" v:mID="95" v:groupContext="shape" transform="translate(490.501,-40.9375)"> + <title>Rectangle.95</title> + <desc>Key N</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="15.75" cy="415.565" width="31.5" height="18"/> + <g id="shadow95-420" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="31.5" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="31.5" height="18" class="st3"/> + <text x="5.21" y="418.26" class="st5" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Key N</text> </g> + <g id="shape96-425" v:mID="96" v:groupContext="shape" transform="translate(524.251,-40.9375)"> + <title>Rectangle.96</title> + <desc>Action N</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="21.375" cy="415.565" width="42.75" height="18"/> + <g id="shadow96-426" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="406.565" width="42.75" height="18" class="st2"/> + </g> + <rect x="0" y="406.565" width="42.75" height="18" class="st3"/> + <text x="5.67" y="418.26" class="st5" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Action N</text> </g> + <g id="shape97-431" v:mID="97" v:groupContext="shape" transform="translate(326.251,-31.9375)"> + <title>Rectangle.97</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <g id="shadow97-432" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="289.065" width="245.25" height="135.5" class="st6"/> + </g> + <rect x="0" y="289.065" width="245.25" height="135.5" class="st7"/> + </g> + <g id="shape98-436" v:mID="98" v:groupContext="shape" transform="translate(337.501,-162.938)"> + <title>Sheet.98</title> + <desc>Local Table for N Specific Flows Serviced at Node X</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="110.423" cy="418.94" width="220.85" height="11.25"/> + <rect x="0" y="413.315" width="220.846" height="11.25" class="st8"/> + <text x="5.55" y="421.94" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Local Table for N Specific Flows Serviced at Node X</text> </g> + <g id="shape99-439" v:mID="99" v:groupContext="shape" transform="translate(-204.342,-29.4449) rotate(-53.7462)"> + <title>Sheet.99</title> + <path d="M0 424.56 L160.37 424.56" class="st25"/> + </g> + <g id="shape100-445" v:mID="100" v:groupContext="shape" transform="translate(-37.6568,-164.882) rotate(-24.444)"> + <title>Sheet.100</title> + <path d="M0 424.56 L101.71 424.56" class="st25"/> + </g> + <g id="shape101-450" v:mID="101" v:groupContext="shape" transform="translate(464.049,-50.8578) rotate(50.099)"> + <title>Sheet.101</title> + <path d="M0 424.56 L139.8 424.56" class="st25"/> + </g> + <g id="shape102-455" v:mID="102" v:groupContext="shape" transform="translate(372.376,-207.438)"> + <title>Sheet.102</title> + <desc>Supports N Flows</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="67.5" cy="415.565" width="135" height="18"/> + <rect x="0" y="406.565" width="135" height="18" class="st19"/> + <text x="25.15" y="419.17" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Supports N Flows</text> </g> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/sym_multi_proc_app.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/sym_multi_proc_app.png Binary files differnew file mode 100644 index 000000000..4aaebccdb --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/sym_multi_proc_app.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/tep_termination_arch.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/tep_termination_arch.svg new file mode 100644 index 000000000..54f1655cd --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/tep_termination_arch.svg @@ -0,0 +1,1400 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Generated by Microsoft Visio, SVG Export VXLAN.svg Page-1 --> + +<svg + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:xlink="http://www.w3.org/1999/xlink" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="4.9898949in" + height="6.1537709in" + viewBox="0 0 359.27243 443.07151" + xml:space="preserve" + class="st24" + id="svg2" + version="1.1" + inkscape:version="0.91 r13725" + sodipodi:docname="tep_termination_arch.svg" + style="font-size:12px;overflow:visible;color-interpolation-filters:sRGB;fill:none;fill-rule:evenodd;stroke-linecap:square;stroke-miterlimit:3"><metadata + id="metadata389"><rdf:RDF><cc:Work + rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /><dc:title></dc:title></cc:Work></rdf:RDF></metadata><sodipodi:namedview + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1" + objecttolerance="10" + gridtolerance="10" + guidetolerance="10" + inkscape:pageopacity="0" + inkscape:pageshadow="2" + inkscape:window-width="1875" + inkscape:window-height="1056" + id="namedview387" + showgrid="false" + inkscape:zoom="0.23838384" + inkscape:cx="236.40625" + inkscape:cy="204.93313" + inkscape:window-x="-2" + inkscape:window-y="-3" + inkscape:window-maximized="1" + inkscape:current-layer="svg2" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" /><v:documentProperties + v:langID="1033" + v:viewMarkup="false"><v:userDefs><v:ud + v:nameU="msvSubprocessMaster" + v:prompt="" + v:val="VT4(Rectangle)" /><v:ud + v:nameU="msvNoAutoConnect" + v:val="VT0(1):26" /><v:ud + v:nameU="msvConvertTheme" /></v:userDefs></v:documentProperties><style + type="text/css" + id="style4"><![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5} + .st3 {fill:#d1dab4;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st4 {fill:#6b9bc7;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st5 {fill:#000000;font-family:Calibri;font-size:0.916672em;font-weight:bold} + .st6 {fill:#92d050;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st7 {fill:#000000;font-family:Calibri;font-size:0.833336em} + .st8 {font-size:1em} + .st9 {fill:#3e75a6;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st10 {fill:#d8d8d8;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.24} + .st11 {fill:#404040;fill-opacity:1;stroke:#404040;stroke-opacity:1;stroke-width:0.08695652173913} + .st12 {fill:none;stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st13 {fill:#000000;font-family:Calibri;font-size:1.00001em;font-weight:bold} + .st14 {fill:#000000;font-family:Calibri;font-size:1.00001em} + .st15 {fill:#000000;font-family:Calibri;font-size:1.16666em;font-weight:bold} + .st16 {fill:url(#grad0-72);stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st17 {fill:url(#grad0-83);stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st18 {fill:#f2c096;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st19 {fill:#e1ebf3;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st20 {fill:#000000;font-family:Calibri;font-size:1.16666em} + .st21 {font-size:0.857157em} + .st22 {fill:#d26d19;stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75} + .st23 {fill:#a6c3dd;stroke:#404040;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25} + .st24 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]></style><defs + id="Patterns_And_Gradients"><linearGradient + id="grad0-72" + x1="0" + y1="0" + x2="1" + y2="0" + gradientTransform="matrix(0,-1,1,0,0,1)"><stop + offset="0" + stop-color="#efaf7a" + stop-opacity="1" + id="stop8" /><stop + offset="1" + stop-color="#f2bf96" + stop-opacity="1" + id="stop10" /></linearGradient><linearGradient + id="grad0-83" + x1="-0.024014981" + y1="1433.2565" + x2="86.829642" + y2="1433.2565" + gradientTransform="scale(1.9190145,0.52110079)" + gradientUnits="userSpaceOnUse"><stop + offset="0" + stop-color="#f0f0f0" + stop-opacity="1" + id="stop13" /><stop + offset="1" + stop-color="#ffffff" + stop-opacity="1" + id="stop15" /></linearGradient><linearGradient + inkscape:collect="always" + xlink:href="#grad0-72" + id="linearGradient4536" + gradientTransform="scale(5.2221044,0.19149368)" + x1="-0.023935719" + y1="4088.255" + x2="48.280345" + y2="4088.255" + gradientUnits="userSpaceOnUse" /><linearGradient + inkscape:collect="always" + xlink:href="#grad0-72" + id="linearGradient4538" + gradientTransform="scale(5.2221044,0.19149368)" + x1="-0.023935719" + y1="4088.255" + x2="48.280345" + y2="4088.255" + gradientUnits="userSpaceOnUse" /></defs><defs + id="Markers"><g + id="lend13"><path + d="M 3,1 0,0 3,-1 3,1" + style="stroke:none" + id="path19" + inkscape:connector-curvature="0" /></g><marker + id="mrkr13-28" + class="st11" + v:arrowType="13" + v:arrowSize="2" + v:setback="34.5" + refX="-34.5" + orient="auto" + markerUnits="strokeWidth" + overflow="visible" + style="overflow:visible;fill:#404040;fill-opacity:1;stroke:#404040;stroke-width:0.08695652;stroke-opacity:1"><use + xlink:href="#lend13" + transform="scale(-11.5,-11.5)" + id="use22" + x="0" + y="0" + width="100%" + height="100%" /></marker></defs><g + v:mID="0" + v:index="1" + v:groupContext="foregroundPage" + id="g24" + transform="translate(-116.875,-116.875)"><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /><v:ud + v:nameU="msvThemeOrder" + v:val="VT0(0):26" /></v:userDefs><title + id="title26">Page-1</title><v:pageProperties + v:drawingScale="1" + v:pageScale="1" + v:drawingUnits="0" + v:shadowOffsetX="9" + v:shadowOffsetY="-9" /><v:layer + v:name="Flowchart" + v:index="0" /><g + id="shape33-1" + v:mID="33" + v:groupContext="shape" + transform="translate(117,-432)"><title + id="title29">Rectangle.18</title><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow33-2" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><rect + x="0" + y="666" + width="342" + height="126" + class="st2" + id="rect32" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><rect + x="0" + y="666" + width="342" + height="126" + class="st3" + id="rect34" + style="fill:#d1dab4;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape10-5" + v:mID="10" + v:groupContext="shape" + transform="translate(216,-522)"><title + id="title37">Rounded rectangle.10</title><desc + id="desc39">Userspace vhost Interface</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="74.25" + cy="778.5" + width="148.5" + height="27" /><g + id="shadow10-6" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 139.5,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-9 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,765 a 9.00007,9.00007 -180 0 0 -9,9 l 0,9 a 9.00007,9.00007 -180 0 0 9,9 l 130.5,0 z" + class="st2" + id="path42" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 139.5,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-9 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,765 a 9.00007,9.00007 -180 0 0 -9,9 l 0,9 a 9.00007,9.00007 -180 0 0 9,9 l 130.5,0 z" + class="st4" + id="path44" + inkscape:connector-curvature="0" + style="fill:#6b9bc7;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="15.14" + y="781.79999" + class="st5" + v:langID="2052" + id="text46" + style="font-weight:bold;font-size:11.0000639px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />Userspace vhost Interface </text> +</g><g + id="shape12-10" + v:mID="12" + v:groupContext="shape" + transform="translate(355.5,-481.5)"><title + id="title49">Rounded rectangle.12</title><desc + id="desc51">Encapsulation</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="45" + cy="785.25" + width="90" + height="13.5" /><g + id="shadow12-11" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="M 83.25,792 A 6.75005,6.75005 -180 0 0 90,785.25 6.75005,6.75005 -180 0 0 83.25,778.5 l -76.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 76.5,0 z" + class="st2" + id="path54" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="M 83.25,792 A 6.75005,6.75005 -180 0 0 90,785.25 6.75005,6.75005 -180 0 0 83.25,778.5 l -76.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 76.5,0 z" + class="st6" + id="path56" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="16.58" + y="788.25" + class="st7" + v:langID="1033" + id="text58" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />Encapsulation</text> +</g><g + id="shape13-15" + v:mID="13" + v:groupContext="shape" + transform="translate(153,-483.75)"><title + id="title61">Rounded rectangle.13</title><desc + id="desc63">Decapsulation</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="36" + cy="785.25" + width="72" + height="13.5" /><g + id="shadow13-16" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="M 65.25,792 A 6.75005,6.75005 -180 0 0 72,785.25 6.75005,6.75005 -180 0 0 65.25,778.5 l -58.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 58.5,0 z" + class="st2" + id="path66" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="M 65.25,792 A 6.75005,6.75005 -180 0 0 72,785.25 6.75005,6.75005 -180 0 0 65.25,778.5 l -58.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 58.5,0 z" + class="st6" + id="path68" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="8.2200003" + y="788.25" + class="st7" + v:langID="1033" + id="text70" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />D<tspan + class="st8" + v:langID="2052" + id="tspan72" + style="font-size:10.00003242px">ecap</tspan>sulation</text> +</g><g + id="shape35-21" + v:mID="35" + v:groupContext="shape" + transform="translate(117,-324)"><title + id="title75">Rectangle.18.35</title><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow35-22" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><rect + x="0" + y="697.5" + width="342" + height="94.5" + class="st2" + id="rect78" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><rect + x="0" + y="697.5" + width="342" + height="94.5" + class="st9" + id="rect80" + style="fill:#3e75a6;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape15-25" + v:mID="15" + v:groupContext="shape" + transform="translate(259.654,-263.25)"><title + id="title83">Data store</title><v:userDefs><v:ud + v:nameU="CH" + v:val="VT0(5):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /></v:userDefs><path + d="m 0,600.13 a 28.3465,10.6299 0 0 1 56.69,0 28.3465,10.6299 0 0 1 -56.69,0 z m 0,0 0,181.24 a 28.3465,10.6299 -180 1 0 56.69,0 l 0,-181.24 a 28.3465,10.6299 0 0 1 -56.69,0 z" + class="st10" + id="path85" + inkscape:connector-curvature="0" + style="fill:#d8d8d8;stroke:#404040;stroke-width:0.23999999;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape16-29" + v:mID="16" + v:groupContext="shape" + transform="translate(324,-391.5)"><title + id="title88">Rounded rectangle.16</title><desc + id="desc90">Inner IP/L4 checksum offload</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="67.5" + cy="785.25" + width="135" + height="13.5" /><g + id="shadow16-30" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 128.25,792 a 6.75005,6.75005 -180 0 0 6.75,-6.75 6.75005,6.75005 -180 0 0 -6.75,-6.75 l -121.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 121.5,0 z" + class="st2" + id="path93" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 128.25,792 a 6.75005,6.75005 -180 0 0 6.75,-6.75 6.75005,6.75005 -180 0 0 -6.75,-6.75 l -121.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 121.5,0 z" + class="st6" + id="path95" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="8.2700005" + y="788.25" + class="st7" + v:langID="2052" + id="text97" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />Inner IP/L4 checksum offload</text> +</g><g + id="shape17-34" + v:mID="17" + v:groupContext="shape" + transform="translate(144,-337.5)"><title + id="title100">Rounded rectangle.17</title><desc + id="desc102">UDP port configuration</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="54" + cy="785.25" + width="108" + height="13.5" /><g + id="shadow17-35" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 101.25,792 a 6.75005,6.75005 -180 0 0 6.75,-6.75 6.75005,6.75005 -180 0 0 -6.75,-6.75 l -94.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 94.5,0 z" + class="st2" + id="path105" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 101.25,792 a 6.75005,6.75005 -180 0 0 6.75,-6.75 6.75005,6.75005 -180 0 0 -6.75,-6.75 l -94.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 94.5,0 z" + class="st6" + id="path107" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="6.9299998" + y="788.25" + class="st7" + v:langID="1033" + id="text109" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />UDP port configuration</text> +</g><g + id="shape18-39" + v:mID="18" + v:groupContext="shape" + transform="translate(145.346,-357.75)"><title + id="title112">Rounded rectangle.18</title><desc + id="desc114">Inner IP/L4 csum verify</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="53.3268" + cy="785.25" + width="106.66" + height="13.5" /><g + id="shadow18-40" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 99.9,792 a 6.75005,6.75005 -180 0 0 6.75,-6.75 6.75005,6.75005 -180 0 0 -6.75,-6.75 l -93.15,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 93.15,0 z" + class="st2" + id="path117" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 99.9,792 a 6.75005,6.75005 -180 0 0 6.75,-6.75 6.75005,6.75005 -180 0 0 -6.75,-6.75 l -93.15,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 93.15,0 z" + class="st6" + id="path119" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="6.6599998" + y="788.25" + class="st7" + v:langID="2052" + id="text121" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />Inner IP/L4<tspan + class="st8" + v:langID="1033" + id="tspan123" + style="font-size:10.00003242px" /><tspan + class="st8" + v:langID="1033" + id="tspan125" + style="font-size:10.00003242px">csum</tspan> verify</text> +</g><g + id="shape19-46" + v:mID="19" + v:groupContext="shape" + transform="translate(162,-378)"><title + id="title128">Rounded rectangle.19</title><desc + id="desc130">Tunnel Filter</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="45" + cy="785.25" + width="90" + height="13.5" /><g + id="shadow19-47" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="M 83.25,792 A 6.75005,6.75005 -180 0 0 90,785.25 6.75005,6.75005 -180 0 0 83.25,778.5 l -76.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 76.5,0 z" + class="st2" + id="path133" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="M 83.25,792 A 6.75005,6.75005 -180 0 0 90,785.25 6.75005,6.75005 -180 0 0 83.25,778.5 l -76.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 76.5,0 z" + class="st6" + id="path135" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="18.290001" + y="788.25" + class="st7" + v:langID="1033" + id="text137" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />Tunnel <tspan + class="st8" + v:langID="2052" + id="tspan139" + style="font-size:10.00003242px" />F<tspan + class="st8" + v:langID="2052" + id="tspan141" + style="font-size:10.00003242px">ilter</tspan></text> +</g><g + id="shape20-53" + v:mID="20" + v:groupContext="shape" + transform="translate(162,-398.25)"><title + id="title144">Rounded rectangle.20</title><desc + id="desc146">Packet Type</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="45" + cy="785.25" + width="90" + height="13.5" /><g + id="shadow20-54" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="M 83.25,792 A 6.75005,6.75005 -180 0 0 90,785.25 6.75005,6.75005 -180 0 0 83.25,778.5 l -76.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 76.5,0 z" + class="st2" + id="path149" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="M 83.25,792 A 6.75005,6.75005 -180 0 0 90,785.25 6.75005,6.75005 -180 0 0 83.25,778.5 l -76.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 76.5,0 z" + class="st6" + id="path151" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="20.530001" + y="788.25" + class="st7" + v:langID="1033" + id="text153" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />Packet Type</text> +</g><g + id="shape27-58" + v:mID="27" + v:groupContext="shape" + transform="translate(326.25,-441)"><title + id="title156">Sheet.27</title><desc + id="desc158">DPDK Application</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="54" + cy="783" + width="108" + height="18" /><rect + x="0" + y="774" + width="108" + height="18" + class="st12" + id="rect160" + style="fill:none;stroke:none;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="10.11" + y="786.59998" + class="st13" + v:langID="2052" + id="text162" + style="font-weight:bold;font-size:12.00012016px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />DPDK Application</text> +</g><g + id="shape26-61" + v:mID="26" + v:groupContext="shape" + transform="translate(324,-279)"><title + id="title165">Sheet.26</title><desc + id="desc167">VXLAN Tunnel</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="45" + cy="778.5" + width="90" + height="27" /><rect + x="0" + y="765" + width="90" + height="27" + class="st12" + id="rect169" + style="fill:none;stroke:none;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="10.51" + y="782.09998" + class="st14" + v:langID="2052" + id="text171" + style="font-size:12.00012016px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VXLAN Tunnel</text> +</g><g + id="shape28-64" + v:mID="28" + v:groupContext="shape" + transform="translate(342,-333)"><title + id="title174">Sheet.28</title><desc + id="desc176">NIC</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="45" + cy="783" + width="90" + height="18" /><rect + x="0" + y="774" + width="90" + height="18" + class="st12" + id="rect178" + style="fill:none;stroke:none;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="36.400002" + y="787.20001" + class="st15" + v:langID="2052" + id="text180" + style="font-weight:bold;font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />NIC</text> +</g><g + id="shape29-67" + v:mID="29" + v:groupContext="shape" + transform="matrix(0,1,1,0,-517.5,270)"><title + id="title183">45 degree single</title><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow29-68" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 0,787.5 4.5,-4.5 0,2.97 247.5,0 0,3.06 -247.5,0 0,2.97 -4.5,-4.5 z" + class="st2" + id="path186" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,787.5 4.5,-4.5 0,2.97 247.5,0 0,3.06 -247.5,0 0,2.97 -4.5,-4.5 z" + class="st16" + id="path188" + inkscape:connector-curvature="0" + style="fill:url(#linearGradient4536);stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape30-73" + v:mID="30" + v:groupContext="shape" + transform="matrix(0,-1,-1,0,1093.5,522)"><title + id="title191">45 degree single.30</title><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow30-74" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(-1.44,-1.44)" + class="st1" + style="visibility:visible"><path + d="m 0,787.5 4.5,-4.5 0,2.97 247.5,0 0,3.06 -247.5,0 0,2.97 -4.5,-4.5 z" + class="st2" + id="path194" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 0,787.5 4.5,-4.5 0,2.97 247.5,0 0,3.06 -247.5,0 0,2.97 -4.5,-4.5 z" + class="st16" + id="path196" + inkscape:connector-curvature="0" + style="fill:url(#linearGradient4538);stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape14-78" + v:mID="14" + v:groupContext="shape" + v:layerMember="0" + transform="translate(207,-234)"><title + id="title199">Object</title><desc + id="desc201">WAN/LAN</desc><v:custProps><v:cp + v:nameU="Cost" + v:lbl="Cost" + v:type="7" + v:format="@" + v:langID="1033" /><v:cp + v:nameU="Duration" + v:lbl="Duration" + v:type="2" + v:langID="1033" /><v:cp + v:nameU="Resources" + v:lbl="Resources" + v:langID="1033" /></v:custProps><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(1,1,1,1)" + v:tabSpace="42.5197" /><v:textRect + cx="83.25" + cy="769.5" + width="166.5" + height="45" /><g + id="shadow14-79" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 23.93,782.24 a 19.1532,6.46551 -180 0 0 27.59,5.48 26.5285,8.95514 -180 0 0 47.98,-2.03 8.02422,2.70871 -180 0 0 8.66,2.93 30.5319,10.3066 -180 0 0 29.32,-4.28 18.9723,6.40442 -180 0 0 20.66,-4.05 29.3596,9.20857 -176.98 0 0 -12.51,-8.69 100.331,31.9328 -2.78 0 0 20.84,-8.56 42.3527,12.6007 -180 0 0 -21.95,-11.54 25.2977,7.52648 -180 0 0 -48.39,3.6 21.4707,5.9338 -1.58 0 0 -36.32,0.33 30.834,8.86245 -178.72 0 0 -39.29,12.27 24.6202,7.32494 -180 0 0 3.41,14.54 z" + class="st2" + id="path204" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 23.93,782.24 a 19.1532,6.46551 -180 0 0 27.59,5.48 26.5285,8.95514 -180 0 0 47.98,-2.03 8.02422,2.70871 -180 0 0 8.66,2.93 30.5319,10.3066 -180 0 0 29.32,-4.28 18.9723,6.40442 -180 0 0 20.66,-4.05 29.3596,9.20857 -176.98 0 0 -12.51,-8.69 100.331,31.9328 -2.78 0 0 20.84,-8.56 42.3527,12.6007 -180 0 0 -21.95,-11.54 25.2977,7.52648 -180 0 0 -48.39,3.6 21.4707,5.9338 -1.58 0 0 -36.32,0.33 30.834,8.86245 -178.72 0 0 -39.29,12.27 24.6202,7.32494 -180 0 0 3.41,14.54 z" + class="st17" + id="path206" + inkscape:connector-curvature="0" + style="fill:url(#grad0-83);stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="53.23" + y="773.70001" + class="st15" + v:langID="2052" + id="text208" + style="font-weight:bold;font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />WAN/LAN</text> +</g><g + id="shape11-85" + v:mID="11" + v:groupContext="shape" + transform="translate(236.25,-472.5)"><title + id="title211">Rounded rectangle.11</title><desc + id="desc213">TEP</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="54" + cy="774" + width="108" + height="36" /><g + id="shadow11-86" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 99,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-18 a 9.00007,9.00007 -180 0 0 -9,-9 l -90,0 a 9.00007,9.00007 -180 0 0 -9,9 l 0,18 a 9.00007,9.00007 -180 0 0 9,9 l 90,0 z" + class="st2" + id="path216" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 99,792 a 9.00007,9.00007 -180 0 0 9,-9 l 0,-18 a 9.00007,9.00007 -180 0 0 -9,-9 l -90,0 a 9.00007,9.00007 -180 0 0 -9,9 l 0,18 a 9.00007,9.00007 -180 0 0 9,9 l 90,0 z" + class="st18" + id="path218" + inkscape:connector-curvature="0" + style="fill:#f2c096;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="44.91" + y="777.59998" + class="st13" + v:langID="2052" + id="text220" + style="font-weight:bold;font-size:12.00012016px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />TEP</text> +</g><g + id="shape36-90" + v:mID="36" + v:groupContext="shape" + transform="translate(117,-571.5)"><title + id="title223">Rectangle.18.36</title><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><g + id="shadow36-91" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><rect + x="0" + y="688.5" + width="342" + height="103.5" + class="st2" + id="rect226" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><rect + x="0" + y="688.5" + width="342" + height="103.5" + class="st19" + id="rect228" + style="fill:#e1ebf3;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape5-94" + v:mID="5" + v:groupContext="shape" + transform="translate(306,-594)"><title + id="title231">Rounded rectangle.5</title><desc + id="desc233">VM-2 (VNID:200)</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="61.3125" + cy="765" + width="122.63" + height="54" /><g + id="shadow5-95" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 113.62,792 a 9.00007,9.00007 -180 0 0 9.01,-9 l -0.01,-36 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,738 a 9.00007,9.00007 -180 0 0 -9,9 l 0,36 a 9.00007,9.00007 -180 0 0 9,9 l 104.62,0 z" + class="st2" + id="path236" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 113.62,792 a 9.00007,9.00007 -180 0 0 9.01,-9 l -0.01,-36 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,738 a 9.00007,9.00007 -180 0 0 -9,9 l 0,36 a 9.00007,9.00007 -180 0 0 9,9 l 104.62,0 z" + class="st9" + id="path238" + inkscape:connector-curvature="0" + style="fill:#3e75a6;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="16.93" + y="769.20001" + class="st20" + v:langID="2052" + id="text240" + style="font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM<tspan + class="st8" + v:langID="1033" + id="tspan242" + style="font-size:13.99991989px">-</tspan>2 (<tspan + class="st21" + id="tspan244" + style="font-size:12.00012875px">VNID</tspan><tspan + class="st21" + id="tspan246" + style="font-size:12.00012875px">:</tspan><tspan + class="st21" + id="tspan248" + style="font-size:12.00012875px">200</tspan><tspan + class="st21" + id="tspan250" + style="font-size:12.00012875px">)</tspan></text> +</g><g + id="shape3-104" + v:mID="3" + v:groupContext="shape" + transform="translate(135,-594)"><title + id="title253">Rounded rectangle</title><desc + id="desc255">VM-1 (VNID: 100)</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="61.3125" + cy="765" + width="122.63" + height="54" /><g + id="shadow3-105" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="m 113.62,792 a 9.00007,9.00007 -180 0 0 9.01,-9 l -0.01,-36 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,738 a 9.00007,9.00007 -180 0 0 -9,9 l 0,36 a 9.00007,9.00007 -180 0 0 9,9 l 104.62,0 z" + class="st2" + id="path258" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="m 113.62,792 a 9.00007,9.00007 -180 0 0 9.01,-9 l -0.01,-36 a 9.00007,9.00007 -180 0 0 -9,-9 L 9,738 a 9.00007,9.00007 -180 0 0 -9,9 l 0,36 a 9.00007,9.00007 -180 0 0 9,9 l 104.62,0 z" + class="st9" + id="path260" + inkscape:connector-curvature="0" + style="fill:#3e75a6;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="15.58" + y="769.20001" + class="st20" + v:langID="2052" + id="text262" + style="font-size:13.99991989px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />VM<tspan + class="st8" + v:langID="1033" + id="tspan264" + style="font-size:13.99991989px">-</tspan>1 (<tspan + class="st21" + id="tspan266" + style="font-size:12.00012875px">VNID</tspan><tspan + class="st21" + id="tspan268" + style="font-size:12.00012875px">: </tspan><tspan + class="st21" + id="tspan270" + style="font-size:12.00012875px">100</tspan><tspan + class="st21" + id="tspan272" + style="font-size:12.00012875px">)</tspan></text> +</g><g + id="shape23-114" + v:mID="23" + v:groupContext="shape" + transform="matrix(0.8626982,0.50571911,-0.50571911,0.8626982,624.813,-477.284)"><title + id="title275">Process boundary.23</title><v:userDefs><v:ud + v:nameU="CH" + v:val="VT0(5):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /></v:userDefs><rect + x="0" + y="789.16498" + width="75.634804" + height="2.83465" + class="st22" + id="rect277" + style="fill:#d26d19;stroke:none;stroke-width:0.75;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape22-116" + v:mID="22" + v:groupContext="shape" + transform="matrix(0.95221525,0.30542775,-0.30542775,0.95221525,412.466,-548.055)"><title + id="title280">Process boundary</title><v:userDefs><v:ud + v:nameU="CH" + v:val="VT0(5):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /></v:userDefs><rect + x="0" + y="789.16498" + width="125.234" + height="2.83465" + class="st22" + id="rect282" + style="fill:#d26d19;stroke:none;stroke-width:0.75;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape24-118" + v:mID="24" + v:groupContext="shape" + transform="matrix(-0.93547487,0.35339321,-0.35339321,-0.93547487,670.892,944.318)"><title + id="title285">Process boundary.24</title><v:userDefs><v:ud + v:nameU="CH" + v:val="VT0(5):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /></v:userDefs><rect + x="0" + y="789.16498" + width="108.234" + height="2.83465" + class="st22" + id="rect287" + style="fill:#d26d19;stroke:none;stroke-width:0.75;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape25-120" + v:mID="25" + v:groupContext="shape" + transform="matrix(-0.78441927,0.62023093,-0.62023093,-0.78441927,828.973,824.895)"><title + id="title290">Process boundary.25</title><v:userDefs><v:ud + v:nameU="CH" + v:val="VT0(5):26" /><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /></v:userDefs><rect + x="0" + y="789.16498" + width="61.670101" + height="2.83465" + class="st22" + id="rect292" + style="fill:#d26d19;stroke:none;stroke-width:0.75;stroke-linecap:round;stroke-linejoin:round" /></g><g + id="shape4-122" + v:mID="4" + v:groupContext="shape" + transform="translate(148.5,-587.25)"><title + id="title295">Rectangle</title><desc + id="desc297">vport-0</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="22.5" + cy="784.125" + width="45" + height="15.75" /><g + id="shadow4-123" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><rect + x="0" + y="776.25" + width="45" + height="15.75" + class="st2" + id="rect300" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><rect + x="0" + y="776.25" + width="45" + height="15.75" + class="st23" + id="rect302" + style="fill:#a6c3dd;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="7.5" + y="787.13" + class="st7" + v:langID="1033" + id="text304" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />v<tspan + class="st8" + v:langID="2052" + id="tspan306" + style="font-size:10.00003242px">port</tspan><tspan + class="st8" + v:langID="2052" + id="tspan308" + style="font-size:10.00003242px">-</tspan><tspan + class="st8" + v:langID="2052" + id="tspan310" + style="font-size:10.00003242px">0</tspan></text> +</g><g + id="shape2-130" + v:mID="2" + v:groupContext="shape" + transform="translate(202.5,-587.25)"><title + id="title313">Rectangle.2</title><desc + id="desc315">vport-1</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="22.5" + cy="784.125" + width="45" + height="15.75" /><g + id="shadow2-131" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><rect + x="0" + y="776.25" + width="45" + height="15.75" + class="st2" + id="rect318" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><rect + x="0" + y="776.25" + width="45" + height="15.75" + class="st23" + id="rect320" + style="fill:#a6c3dd;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="7.5" + y="787.13" + class="st7" + v:langID="1033" + id="text322" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />v<tspan + class="st8" + v:langID="2052" + id="tspan324" + style="font-size:10.00003242px">port</tspan><tspan + class="st8" + v:langID="2052" + id="tspan326" + style="font-size:10.00003242px">-</tspan><tspan + class="st8" + v:langID="2052" + id="tspan328" + style="font-size:10.00003242px">1</tspan></text> +</g><g + id="shape6-138" + v:mID="6" + v:groupContext="shape" + transform="translate(317.25,-587.25)"><title + id="title331">Rectangle.6</title><desc + id="desc333">vport-0</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="21.375" + cy="784.125" + width="42.75" + height="15.75" /><g + id="shadow6-139" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><rect + x="0" + y="776.25" + width="42.75" + height="15.75" + class="st2" + id="rect336" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><rect + x="0" + y="776.25" + width="42.75" + height="15.75" + class="st23" + id="rect338" + style="fill:#a6c3dd;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="6.3699999" + y="787.13" + class="st7" + v:langID="1033" + id="text340" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />v<tspan + class="st8" + v:langID="2052" + id="tspan342" + style="font-size:10.00003242px">port</tspan><tspan + class="st8" + v:langID="2052" + id="tspan344" + style="font-size:10.00003242px">-</tspan><tspan + class="st8" + v:langID="2052" + id="tspan346" + style="font-size:10.00003242px">0</tspan></text> +</g><g + id="shape7-146" + v:mID="7" + v:groupContext="shape" + transform="translate(369,-587.25)"><title + id="title349">Rectangle.7</title><desc + id="desc351">vport-1</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="22.5" + cy="784.125" + width="45" + height="15.75" /><g + id="shadow7-147" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><rect + x="0" + y="776.25" + width="45" + height="15.75" + class="st2" + id="rect354" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><rect + x="0" + y="776.25" + width="45" + height="15.75" + class="st23" + id="rect356" + style="fill:#a6c3dd;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="7.5" + y="787.13" + class="st7" + v:langID="1033" + id="text358" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />v<tspan + class="st8" + v:langID="2052" + id="tspan360" + style="font-size:10.00003242px">port</tspan><tspan + class="st8" + v:langID="2052" + id="tspan362" + style="font-size:10.00003242px">-</tspan><tspan + class="st8" + v:langID="2052" + id="tspan364" + style="font-size:10.00003242px">1</tspan></text> +</g><g + id="shape37-154" + v:mID="37" + v:groupContext="shape" + transform="translate(234,-648)"><title + id="title367">Sheet.37</title><desc + id="desc369">Guest</desc><v:userDefs><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="54" + cy="783" + width="108" + height="18" /><rect + x="0" + y="774" + width="108" + height="18" + class="st12" + id="rect371" + style="fill:none;stroke:none;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="39.459999" + y="786.59998" + class="st13" + v:langID="1033" + id="text373" + style="font-weight:bold;font-size:12.00012016px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />Guest</text> +</g><g + id="shape38-157" + v:mID="38" + v:groupContext="shape" + transform="translate(324,-357.75)"><title + id="title376">Rounded rectangle.38</title><desc + id="desc378">TSO offload</desc><v:userDefs><v:ud + v:nameU="visVersion" + v:val="VT0(14):26" /><v:ud + v:nameU="msvThemeColors" + v:val="VT0(36):26" /><v:ud + v:nameU="msvThemeEffects" + v:val="VT0(16):26" /></v:userDefs><v:textBlock + v:margins="rect(4,4,4,4)" /><v:textRect + cx="36" + cy="785.25" + width="72" + height="13.5" /><g + id="shadow38-158" + v:groupContext="shadow" + v:shadowOffsetX="1.44" + v:shadowOffsetY="-1.44" + v:shadowType="1" + transform="translate(1.44,1.44)" + class="st1" + style="visibility:visible"><path + d="M 65.25,792 A 6.75005,6.75005 -180 0 0 72,785.25 6.75005,6.75005 -180 0 0 65.25,778.5 l -58.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 58.5,0 z" + class="st2" + id="path381" + inkscape:connector-curvature="0" + style="fill:#cdcdcd;fill-opacity:0.5;stroke:#cdcdcd;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.5" /></g><path + d="M 65.25,792 A 6.75005,6.75005 -180 0 0 72,785.25 6.75005,6.75005 -180 0 0 65.25,778.5 l -58.5,0 A 6.75005,6.75005 -180 0 0 0,785.25 6.75005,6.75005 -180 0 0 6.75,792 l 58.5,0 z" + class="st6" + id="path383" + inkscape:connector-curvature="0" + style="fill:#92d050;stroke:#404040;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round" /><text + x="12.33" + y="788.25" + class="st7" + v:langID="2052" + id="text385" + style="font-size:10.00003242px;font-family:Calibri;fill:#000000"><v:paragraph + v:horizAlign="1" /><v:tabList />TSO offload</text> +</g></g></svg>
\ No newline at end of file diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/test_pipeline_app.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/test_pipeline_app.png Binary files differnew file mode 100644 index 000000000..d322a9e46 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/test_pipeline_app.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/threads_pipelines.png b/src/spdk/dpdk/doc/guides/sample_app_ug/img/threads_pipelines.png Binary files differnew file mode 100644 index 000000000..27b5e86d2 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/threads_pipelines.png diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/vm_power_mgr_highlevel.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/vm_power_mgr_highlevel.svg new file mode 100644 index 000000000..cb26d6c78 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/vm_power_mgr_highlevel.svg @@ -0,0 +1,1345 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> + +<!-- Copyright(c) 2014 Intel Corporation --> + +<svg + xmlns:osb="http://www.openswatchbook.org/uri/2009/osb" + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="135mm" + height="135mm" + viewBox="0 0 135 135" + version="1.1" + id="svg8" + inkscape:version="0.92.3 (2405546, 2018-03-11)" + sodipodi:docname="vm_power_mgr_highlevel.svg"> + <defs + id="defs2"> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5459" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker12961" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path12959" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker12763" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path12761" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker12573" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path12571" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0" + refX="0" + id="marker12395" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path12393" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <linearGradient + id="linearGradient12389" + osb:paint="solid"> + <stop + style="stop-color:#d07f7d;stop-opacity:1;" + offset="0" + id="stop12387" /> + </linearGradient> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker6022" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path6020" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="Arrow2Send" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5468" + style="fill:#bdb6b6;fill-opacity:1;fill-rule:evenodd;stroke:#bdb6b6;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5462" + style="fill:#bdb6b6;fill-opacity:1;fill-rule:evenodd;stroke:#bdb6b6;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + style="overflow:visible" + id="DistanceStart" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DistanceStart" + inkscape:isstock="true"> + <g + id="g2300" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-opacity:1"> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.14999998;stroke-linecap:square;stroke-opacity:1" + d="M 0,0 H 2" + id="path2306" + inkscape:connector-curvature="0" /> + <path + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-opacity:1" + d="M 0,0 13,4 9,0 13,-4 Z" + id="path2302" + inkscape:connector-curvature="0" /> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1;stroke-linecap:square;stroke-opacity:1" + d="M 0,-4 V 40" + id="path2304" + inkscape:connector-curvature="0" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0" + refX="0" + id="marker5751" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5749" + style="fill:#bdb6b6;fill-opacity:1;fill-rule:evenodd;stroke:#bdb6b6;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5438" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#bdb6b6;fill-opacity:1;fill-rule:evenodd;stroke:#bdb6b6;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Lend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5456" + style="fill:#bdb6b6;fill-opacity:1;fill-rule:evenodd;stroke:#bdb6b6;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + style="overflow:visible" + id="DistanceStart-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DistanceStart" + inkscape:isstock="true"> + <g + id="g2300-9" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-opacity:1"> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.14999998;stroke-linecap:square;stroke-opacity:1" + d="M 0,0 H 2" + id="path2306-7" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-opacity:1" + d="M 0,0 13,4 9,0 13,-4 Z" + id="path2302-1" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1;stroke-linecap:square;stroke-opacity:1" + d="M 0,-4 V 40" + id="path2304-4" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker6022-2" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6020-9" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + style="overflow:visible" + id="DistanceStart-8" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DistanceStart" + inkscape:isstock="true"> + <g + id="g2300-8" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-opacity:1"> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.14999998;stroke-linecap:square;stroke-opacity:1" + d="M 0,0 H 2" + id="path2306-1" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-opacity:1" + d="M 0,0 13,4 9,0 13,-4 Z" + id="path2302-6" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1;stroke-linecap:square;stroke-opacity:1" + d="M 0,-4 V 40" + id="path2304-7" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker6022-8" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6020-3" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + style="overflow:visible" + id="DistanceStart-9" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DistanceStart" + inkscape:isstock="true"> + <g + id="g2300-6" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-opacity:1"> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.14999998;stroke-linecap:square;stroke-opacity:1" + d="M 0,0 H 2" + id="path2306-12" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-opacity:1" + d="M 0,0 13,4 9,0 13,-4 Z" + id="path2302-9" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1;stroke-linecap:square;stroke-opacity:1" + d="M 0,-4 V 40" + id="path2304-2" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker6022-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6020-4" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + style="overflow:visible" + id="DistanceStart-80" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DistanceStart" + inkscape:isstock="true"> + <g + id="g2300-7" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-opacity:1"> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.14999998;stroke-linecap:square;stroke-opacity:1" + d="M 0,0 H 2" + id="path2306-8" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-opacity:1" + d="M 0,0 13,4 9,0 13,-4 Z" + id="path2302-0" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1;stroke-linecap:square;stroke-opacity:1" + d="M 0,-4 V 40" + id="path2304-75" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker6022-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6020-8" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + style="overflow:visible" + id="DistanceStart-89" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DistanceStart" + inkscape:isstock="true"> + <g + id="g2300-0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-opacity:1"> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.14999998;stroke-linecap:square;stroke-opacity:1" + d="M 0,0 H 2" + id="path2306-9" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-opacity:1" + d="M 0,0 13,4 9,0 13,-4 Z" + id="path2302-2" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1;stroke-linecap:square;stroke-opacity:1" + d="M 0,-4 V 40" + id="path2304-5" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker6022-38" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6020-1" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + <marker + style="overflow:visible" + id="DistanceStart-8-1" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="DistanceStart" + inkscape:isstock="true"> + <g + id="g2300-8-1" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-opacity:1"> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.14999998;stroke-linecap:square;stroke-opacity:1" + d="M 0,0 H 2" + id="path2306-1-0" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-opacity:1" + d="M 0,0 13,4 9,0 13,-4 Z" + id="path2302-6-2" /> + <path + inkscape:connector-curvature="0" + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1;stroke-linecap:square;stroke-opacity:1" + d="M 0,-4 V 40" + id="path2304-7-3" /> + </g> + </marker> + <marker + inkscape:stockid="Arrow2Send" + orient="auto" + refY="0" + refX="0" + id="marker6022-8-5" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6020-3-7" + style="fill:#666b72;fill-opacity:1;fill-rule:evenodd;stroke:#666b72;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-0.3,0,0,-0.3,0.69,0)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1.4" + inkscape:cx="424.83465" + inkscape:cy="254.38985" + inkscape:document-units="mm" + inkscape:current-layer="layer2" + showgrid="false" + inkscape:window-width="1920" + inkscape:window-height="1017" + inkscape:window-x="-8" + inkscape:window-y="-8" + inkscape:window-maximized="1" /> + <metadata + id="metadata5"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:groupmode="layer" + id="layer2" + inkscape:label="Drawing" + transform="translate(0,-162)"> + <rect + style="opacity:1;fill:#b3ca83;fill-opacity:1;stroke:#000000;stroke-width:0.37057942;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4791" + width="33.990398" + height="45.425076" + x="85.676407" + y="164.63306" /> + <rect + style="opacity:1;fill:#b3ca83;fill-opacity:1;stroke:#000000;stroke-width:0.35639697;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4530" + width="65.05616" + height="45.42617" + x="15.335129" + y="164.63112" /> + <rect + style="opacity:0.97000002;fill:#a9c373;fill-opacity:1;stroke:#000000;stroke-width:0.22279529;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4530-7-8" + width="127.68393" + height="26.879263" + x="3.8373337" + y="267.58688" /> + <rect + style="opacity:1;fill:#e6f3f7;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4532" + width="62.87516" + height="7.3101764" + x="16.408901" + y="165.59154" /> + <rect + style="opacity:1;fill:#a8c270;fill-opacity:1;stroke:#000000;stroke-width:0.32159171;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4532-2" + width="62.906345" + height="6.0718322" + x="16.393307" + y="172.88612" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="35.209579" + y="169.98332" + id="text4551"><tspan + sodipodi:role="line" + id="tspan4549" + x="35.209579" + y="169.98332" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">DPDK Application</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="36.921368" + y="176.82596" + id="text4551-6"><tspan + sodipodi:role="line" + id="tspan4549-8" + x="36.921368" + y="176.82596" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">librte_power(vm)</tspan></text> + <rect + style="opacity:1;fill:#f9b278;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571" + width="13.510916" + height="10.574122" + x="16.399574" + y="178.96422" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="22.99836" + y="182.19067" + id="text4551-6-1"><tspan + sodipodi:role="line" + x="22.99836" + y="182.19067" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605">lcore</tspan><tspan + sodipodi:role="line" + x="22.99836" + y="185.01289" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4609">channel</tspan><tspan + sodipodi:role="line" + x="22.99836" + y="187.83511" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593">0</tspan></text> + <flowRoot + xml:space="preserve" + id="flowRoot4595" + style="font-style:normal;font-weight:normal;font-size:10.66666698px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none"><flowRegion + id="flowRegion4597" + style="font-size:10.66666698px"><rect + id="rect4599" + width="7.0710678" + height="66.670067" + x="116.67262" + y="238.13113" + style="font-size:10.66666698px" /></flowRegion><flowPara + id="flowPara4601" /></flowRoot> <rect + style="opacity:1;fill:#f9b278;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-6" + width="13.510917" + height="10.574122" + x="33.050274" + y="178.97897" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="39.649059" + y="182.20543" + id="text4551-6-1-2"><tspan + sodipodi:role="line" + x="39.649059" + y="182.20543" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-8">lcore</tspan><tspan + sodipodi:role="line" + x="39.649059" + y="185.02765" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4609-3">channel</tspan><tspan + sodipodi:role="line" + x="39.649059" + y="187.84987" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-5">1</tspan></text> + <rect + style="opacity:1;fill:#f9b278;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-7" + width="13.510917" + height="10.574122" + x="49.888256" + y="178.97897" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="56.487041" + y="182.20544" + id="text4551-6-1-0"><tspan + sodipodi:role="line" + x="56.487041" + y="182.20544" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-7">lcore</tspan><tspan + sodipodi:role="line" + x="56.487041" + y="185.02766" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4609-9">channel</tspan><tspan + sodipodi:role="line" + x="56.487041" + y="187.84988" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-0">2</tspan></text> + <rect + style="opacity:1;fill:#f9b278;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-75" + width="13.510917" + height="10.574122" + x="65.788734" + y="178.95795" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="72.38752" + y="182.18442" + id="text4551-6-1-5"><tspan + sodipodi:role="line" + x="72.38752" + y="182.18442" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-2">lcore</tspan><tspan + sodipodi:role="line" + x="72.38752" + y="185.00664" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4609-6">channel</tspan><tspan + sodipodi:role="line" + x="72.38752" + y="187.82886" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-02">3</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="47.749531" + y="195.34956" + id="text4551-6-1-3-2"><tspan + sodipodi:role="line" + x="47.749531" + y="195.34956" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:'Arial Bold';text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-6-6">VM 0</tspan></text> + <rect + style="opacity:1;fill:#e6f3f7;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4793" + width="31.844494" + height="7.3705354" + x="86.692558" + y="165.5556" /> + <rect + style="opacity:1;fill:#a8c270;fill-opacity:1;stroke:#000000;stroke-width:0.32056567;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4793-5" + width="31.876707" + height="6.0798306" + x="86.67646" + y="172.91003" /> + <rect + style="opacity:1;fill:#f9b278;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-75-0" + width="13.510918" + height="10.574122" + x="86.67646" + y="178.98987" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="93.275238" + y="182.21632" + id="text4551-6-1-5-9"><tspan + sodipodi:role="line" + x="93.275238" + y="182.21632" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-2-2">lcore</tspan><tspan + sodipodi:role="line" + x="93.275238" + y="185.03854" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4609-6-6">channel</tspan><tspan + sodipodi:role="line" + x="93.275238" + y="187.86076" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-02-9">0</tspan></text> + <rect + style="opacity:1;fill:#f9b278;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-75-40" + width="13.510918" + height="10.574122" + x="105.04224" + y="178.98987" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="111.64102" + y="182.21632" + id="text4551-6-1-5-98"><tspan + sodipodi:role="line" + x="111.64102" + y="182.21632" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-2-1">lcore</tspan><tspan + sodipodi:role="line" + x="111.64102" + y="185.03854" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4609-6-8">channel</tspan><tspan + sodipodi:role="line" + x="111.64102" + y="187.86076" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-02-4">1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="103.07201" + y="195.55463" + id="text4551-6-1-3-2-6"><tspan + sodipodi:role="line" + x="103.07201" + y="195.55463" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:'Arial Bold';text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-6-6-6">VM 1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="91.442757" + y="176.86929" + id="text4551-6-6"><tspan + sodipodi:role="line" + id="tspan4549-8-7" + x="91.442757" + y="176.86929" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">librte_power(vm)</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="91.124741" + y="170.44301" + id="text4551-2"><tspan + sodipodi:role="line" + id="tspan4549-0" + x="91.124741" + y="170.44301" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">DPDK Application</tspan></text> + <rect + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5309" + width="14.835566" + height="11.339286" + x="4.6515064" + y="274.38187" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="7.2982635" + y="281.1879" + id="text5313"><tspan + sodipodi:role="line" + id="tspan5311" + x="7.2982635" + y="281.1879" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Core 0</tspan></text> + <rect + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5309-9" + width="14.835566" + height="11.339286" + x="20.545406" + y="274.38187" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="23.192165" + y="281.1879" + id="text5313-3"><tspan + sodipodi:role="line" + id="tspan5311-3" + x="23.192165" + y="281.1879" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Core 1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="68.209221" + y="291.73907" + id="text4551-6-1-3-2-3"><tspan + sodipodi:role="line" + x="68.209221" + y="291.73907" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:'Arial Bold';text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-6-6-9">Host</tspan></text> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.62289333;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-mid:url(#DistanceStart);marker-end:url(#marker6022)" + d="m 39.535285,203.58895 4.503022,68.50018" + id="path5433" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <rect + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5309-9-2" + width="14.835566" + height="11.339286" + x="36.439308" + y="274.38187" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="39.086063" + y="281.1879" + id="text5313-3-7"><tspan + sodipodi:role="line" + id="tspan5311-3-5" + x="39.086063" + y="281.1879" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Core 2</tspan></text> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.5908072;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-mid:url(#DistanceStart-2);marker-end:url(#marker6022-2)" + d="m 55.916812,205.90598 4.293524,66.48448" + id="path5433-9" + inkscape:connector-curvature="0" /> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.61559618;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-mid:url(#DistanceStart-8);marker-end:url(#marker6022-8)" + d="m 71.933822,205.59072 4.043782,66.57817" + id="path5433-5" + inkscape:connector-curvature="0" /> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.64176393;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-mid:url(#DistanceStart-9);marker-end:url(#marker6022-0)" + d="m 22.002638,205.5264 70.610773,67.15007" + id="path5433-6" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.58748102;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-mid:url(#DistanceStart-80);marker-end:url(#marker6022-3)" + d="M 92.495303,205.27884 123.88617,272.4858" + id="path5433-7" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#666b72;fill-opacity:1;stroke:#666b72;stroke-width:1.72912371;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-mid:url(#DistanceStart-89);marker-end:url(#marker6022-38)" + d="m 112.28597,207.28466 -4.3989,65.03043" + id="path5433-2" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <rect + style="opacity:0.9;fill:#d07f7d;fill-opacity:0.96396401;stroke:#000000;stroke-width:0.2724604;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4530-7" + width="127.62878" + height="40.216084" + x="3.8649101" + y="227.3748" /> + <rect + style="opacity:0.97000002;fill:#aac475;fill-opacity:1;stroke:#000000;stroke-width:0.35901672;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5115" + width="50.111969" + height="36.47134" + x="5.1204758" + y="229.27213" /> + <rect + style="opacity:0.97000002;fill:#f8a25f;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5117" + width="45.451633" + height="5.4806519" + x="7.2379589" + y="230.65007" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="9.1533337" + y="234.43925" + id="text4551-6-8"><tspan + sodipodi:role="line" + id="tspan4549-8-0" + x="9.1533337" + y="234.43925" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Endpoint Monitor (lcore channels)</tspan></text> + <rect + style="opacity:0.97000002;fill:#a4c3e8;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5117-9" + width="45.451633" + height="5.4806519" + x="7.2379532" + y="236.2211" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="17.619999" + y="239.7863" + id="text4551-6-8-8"><tspan + sodipodi:role="line" + id="tspan4549-8-0-2" + x="17.619999" + y="239.7863" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Channel Manager</tspan></text> + <rect + style="opacity:0.97000002;fill:#d99896;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5117-1" + width="45.451633" + height="5.4806519" + x="7.2379456" + y="241.70174" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="17.090824" + y="245.26694" + id="text4551-6-8-1"><tspan + sodipodi:role="line" + id="tspan4549-8-0-1" + x="17.090824" + y="245.26694" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">librte_power (Host)</tspan></text> + <rect + style="opacity:0.97000002;fill:#a3c3e9;fill-opacity:0.98039216;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5117-4" + width="45.451633" + height="5.4806519" + x="7.2379494" + y="247.1824" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="19.207497" + y="250.7476" + id="text4551-6-8-13"><tspan + sodipodi:role="line" + id="tspan4549-8-0-9" + x="19.207497" + y="250.7476" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">VM Power CLI</tspan></text> + <rect + style="opacity:0.97000002;fill:#f9ae72;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5117-0" + width="45.451633" + height="5.4806519" + x="81.805901" + y="230.65007" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="99.596291" + y="234.21524" + id="text4551-6-8-4"><tspan + sodipodi:role="line" + id="tspan4549-8-0-97" + x="99.596291" + y="234.21524" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">QEMU</tspan></text> + <rect + style="opacity:0.97000002;fill:#dfdfdf;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5117-0-8" + width="45.451633" + height="5.4806519" + x="81.805908" + y="236.13072" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="100.65464" + y="239.69588" + id="text4551-6-8-4-0"><tspan + sodipodi:role="line" + id="tspan4549-8-0-97-5" + x="100.65464" + y="239.69588" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">libvirt</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="29.925892" + y="260.64609" + id="text4551-6-1-3-2-9"><tspan + sodipodi:role="line" + x="29.925892" + y="260.64609" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:'Arial Bold';text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-6-6-0">VM Power Monitor Application</tspan></text> + <rect + style="opacity:1;fill:#d07f7d;fill-opacity:0.91372549;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5117-0-7" + width="53.336025" + height="9.3962946" + x="73.564217" + y="244.95467" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="75.594803" + y="248.72948" + id="text4551-6-8-4-4"><tspan + sodipodi:role="line" + id="tspan4549-8-0-97-0" + x="75.594803" + y="248.72948" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Linux "userspace" power governor</tspan><tspan + sodipodi:role="line" + x="75.594803" + y="252.28992" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332" + id="tspan5289">/sys/devices/system/cpu/cpuN/cpufreq/</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="68.648186" + y="265.05075" + id="text4551-6-1-3-2-8"><tspan + sodipodi:role="line" + x="68.648186" + y="265.05075" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:'Arial Bold';text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-6-6-7">OS/Hypervisor</tspan></text> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-65" + width="13.510917" + height="10.574122" + x="16.394726" + y="198.42061" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="22.993511" + y="202.70541" + id="text4551-6-1-3"><tspan + sodipodi:role="line" + x="22.993511" + y="202.70541" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-6">Virtual</tspan><tspan + sodipodi:role="line" + x="22.993511" + y="205.52763" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-6">Core 0</tspan></text> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-6-9" + width="13.510918" + height="10.574122" + x="33.045425" + y="198.43536" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="39.644207" + y="202.72017" + id="text4551-6-1-2-3"><tspan + sodipodi:role="line" + x="39.644207" + y="202.72017" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-8-0">Virtual</tspan><tspan + sodipodi:role="line" + x="39.644207" + y="205.54239" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-5-9">Core 1</tspan></text> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-7-1" + width="13.510918" + height="10.574122" + x="49.883404" + y="198.43536" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="56.482189" + y="202.72017" + id="text4551-6-1-0-8"><tspan + sodipodi:role="line" + x="56.482189" + y="202.72017" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-7-1">Virtual</tspan><tspan + sodipodi:role="line" + x="56.482189" + y="205.54239" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-0-7">Core 2</tspan></text> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-75-4" + width="13.510918" + height="10.574122" + x="65.783882" + y="198.41434" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="72.382668" + y="202.69914" + id="text4551-6-1-5-6"><tspan + sodipodi:role="line" + x="72.382668" + y="202.69914" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-2-8">Virtual</tspan><tspan + sodipodi:role="line" + x="72.382668" + y="205.52136" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-02-1">Core 3</tspan></text> + <rect + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5309-9-7" + width="14.835566" + height="11.339286" + x="52.33321" + y="274.38187" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="54.979961" + y="281.1879" + id="text5313-3-0"><tspan + sodipodi:role="line" + id="tspan5311-3-55" + x="54.979961" + y="281.1879" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Core 3</tspan></text> + <rect + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5309-9-8" + width="14.835566" + height="11.339286" + x="68.227104" + y="274.38187" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="70.873863" + y="281.1879" + id="text5313-3-3"><tspan + sodipodi:role="line" + id="tspan5311-3-9" + x="70.873863" + y="281.1879" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Core 4</tspan></text> + <rect + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect5309-9-5" + width="14.835566" + height="11.339286" + x="84.120987" + y="274.38187" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="86.767769" + y="281.1879" + id="text5313-3-05"><tspan + sodipodi:role="line" + id="tspan5311-3-2" + x="86.767769" + y="281.1879" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332">Core 5</tspan></text> + <rect + y="274.38187" + x="100.01488" + height="11.339286" + width="14.835566" + id="rect5309-9-4" + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" /> + <text + id="text5313-3-5" + y="281.1879" + x="102.66164" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332" + y="281.1879" + x="102.66164" + id="tspan5311-3-50" + sodipodi:role="line">Core 6</tspan></text> + <rect + y="274.38187" + x="115.90883" + height="11.339286" + width="14.835566" + id="rect5309-9-6" + style="opacity:0.97000002;fill:#b2c5dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" /> + <text + id="text5313-3-03" + y="281.1879" + x="118.55559" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;font-family:Arial;-inkscape-font-specification:Arial;stroke-width:0.26458332" + y="281.1879" + x="118.55559" + id="tspan5311-3-90" + sodipodi:role="line">Core 7</tspan></text> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-6-9-8" + width="13.510919" + height="10.574122" + x="105.02563" + y="198.43262" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="111.62439" + y="202.71742" + id="text4551-6-1-2-3-5"><tspan + sodipodi:role="line" + x="111.62439" + y="202.71742" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-8-0-6">Virtual</tspan><tspan + sodipodi:role="line" + x="111.62439" + y="205.53964" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-5-9-0">Core 1</tspan></text> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.36531466;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4571-65-0" + width="13.510918" + height="10.574122" + x="86.711884" + y="198.41786" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:2.82222223px;line-height:0;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="94.028839" + y="202.70267" + id="text4551-6-1-3-3"><tspan + sodipodi:role="line" + x="94.028839" + y="202.70267" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4605-6-1">Virtual</tspan><tspan + sodipodi:role="line" + x="94.028839" + y="205.52489" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.82222223px;line-height:1;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan4593-6-3">Core 0</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.28939158px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker12763)" + d="m 52.951491,233.29052 h 28.50186" + id="path12353" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.28786668;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart);marker-end:url(#marker12573)" + d="M 53.218755,238.7333 H 81.453341" + id="path12353-4" + inkscape:connector-curvature="0" /> + <rect + style="opacity:1;fill:#fffdfd;fill-opacity:1;stroke:none;stroke-width:0.30100071;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect12385" + width="21.548515" + height="3.6533842" + x="56.826885" + y="236.97552" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="57.317425" + y="239.42633" + id="text12372"><tspan + sodipodi:role="line" + id="tspan12370" + x="57.317425" + y="239.42633" + style="font-size:2.11666656px;stroke-width:0.26458332">Map vCPU to pCPU</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.21566178px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker12961)" + d="M 52.689586,244.25695 H 63.60363 v 5.05835 h 9.755338" + id="path12951" + inkscape:connector-curvature="0" /> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/vm_power_mgr_vm_request_seq.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/vm_power_mgr_vm_request_seq.svg new file mode 100644 index 000000000..0c6d49f0c --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/vm_power_mgr_vm_request_seq.svg @@ -0,0 +1,700 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2014 Intel Corporation --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:xlink="http://www.w3.org/1999/xlink" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="190mm" + height="105mm" + viewBox="0 0 190 105" + version="1.1" + id="svg13567" + inkscape:version="0.92.4 (5da689c313, 2019-01-14)" + sodipodi:docname="VM_Req_to_Scale_Freq.svg"> + <defs + id="defs13561"> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker2723" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path2721" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path899" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path902" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path881" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.4,0,0,0.4,4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker1748" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Mend"> + <path + transform="matrix(-0.4,0,0,-0.4,-4,0)" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + id="path1746" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path884" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Lend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path896" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(-1.1,0,0,-1.1,-1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0" + refX="0" + id="marker1565-4" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path1563-2" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.4,0,0,-0.4,-4,0)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend-5" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path902-8" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path899-2" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend-5-8" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path902-8-5" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path899-2-6" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path899-9" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path902-2" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1.4" + inkscape:cx="422.48941" + inkscape:cy="233.91334" + inkscape:document-units="mm" + inkscape:current-layer="layer3" + showgrid="false" + inkscape:window-width="1920" + inkscape:window-height="1137" + inkscape:window-x="1912" + inkscape:window-y="-8" + inkscape:window-maximized="1" /> + <metadata + id="metadata13564"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:groupmode="layer" + id="layer3" + inkscape:label="Drawing" + transform="translate(0,-192)"> + <rect + style="opacity:1;fill:#ffffff;fill-opacity:1;stroke:#000010;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:0.35277779, 0.35277779;stroke-dashoffset:0;stroke-opacity:1" + id="rect6314" + width="54.923889" + height="77.909065" + x="47.091534" + y="211.93466" /> + <rect + style="opacity:0.50199998;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581" + width="23.118814" + height="8.4189901" + x="2.0566161" + y="194.62044" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1.25;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="13.409275" + y="197.77512" + id="text13585"><tspan + sodipodi:role="line" + x="13.409275" + y="197.77512" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13587">librte_power</tspan><tspan + sodipodi:role="line" + x="13.409275" + y="201.30289" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591">(VM)</tspan></text> + <rect + style="opacity:0.50199998;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.35277778;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-3" + width="23.118814" + height="8.4189901" + x="26.728348" + y="194.62044" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1.25;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="38.081005" + y="197.77512" + id="text13585-4"><tspan + sodipodi:role="line" + x="38.081005" + y="197.77512" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13587-4">guest_channel</tspan><tspan + sodipodi:role="line" + x="38.081005" + y="201.30289" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-7">(VM)</tspan></text> + <rect + style="opacity:0.50199998;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.36962408;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-3-9" + width="25.405504" + height="8.4103832" + x="61.705341" + y="194.62886" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1.25;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="74.146271" + y="197.77512" + id="text13585-4-3"><tspan + sodipodi:role="line" + x="74.146271" + y="197.77512" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13587-4-5">channel_monitor</tspan><tspan + sodipodi:role="line" + x="74.146271" + y="201.30289" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-7-5">(Host)</tspan></text> + <rect + style="opacity:0.50199998;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.3830055;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-3-9-9" + width="27.293171" + height="8.4058018" + x="98.781654" + y="194.63556" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1.25;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="112.44437" + y="197.77512" + id="text13585-4-3-8"><tspan + sodipodi:role="line" + x="112.44437" + y="197.77512" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13587-4-5-3">channel_manager</tspan><tspan + sodipodi:role="line" + x="112.44437" + y="201.30289" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-7-5-5">(Host)</tspan></text> + <rect + style="opacity:0.50199998;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.3830055;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-3-9-9-0" + width="27.293173" + height="8.4058018" + x="130.46378" + y="194.63556" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1.25;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="144.1265" + y="197.186" + id="text13585-4-3-8-4"><tspan + sodipodi:role="line" + x="144.1265" + y="197.186" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13587-4-5-3-0">power_manager</tspan><tspan + sodipodi:role="line" + x="144.1265" + y="200.71378" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-7-5-5-2">(Host)</tspan></text> + <rect + style="opacity:0.50199998;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.3830055;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-3-9-9-0-6" + width="27.293173" + height="8.4058018" + x="160.53162" + y="194.63556" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:2.82222223px;line-height:1.25;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="174.19434" + y="197.77512" + id="text13585-4-3-8-4-7"><tspan + sodipodi:role="line" + x="174.19434" + y="197.77512" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13587-4-5-3-0-8">librte_power</tspan><tspan + sodipodi:role="line" + x="174.19434" + y="201.30289" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:3.17499995px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-7-5-5-2-7">(Host)</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:1.05833333, 1.05833333;stroke-dashoffset:0;stroke-opacity:1" + d="m 13.816475,203.11478 v 91.27254" + id="path968" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:1.05833334, 1.05833334;stroke-dashoffset:0;stroke-opacity:1" + d="M 38.538907,203.44886 V 294.7214" + id="path968-7" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:1.05833335, 1.05833335;stroke-dashoffset:0;stroke-opacity:1" + d="m 74.219389,203.71614 v 91.27253" + id="path968-7-8" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:1.05833335, 1.05833335;stroke-dashoffset:0;stroke-opacity:1" + d="m 113.37436,203.44887 v 91.27254" + id="path968-7-5" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:1.05833335, 1.05833335;stroke-dashoffset:0;stroke-opacity:1" + d="m 144.6449,202.91433 v 91.27253" + id="path968-7-0" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:1.05833335, 1.05833335;stroke-dashoffset:0;stroke-opacity:1" + d="M 172.17366,203.44887 V 294.7214" + id="path968-7-6" + inkscape:connector-curvature="0" /> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.17638889;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5" + width="2.9174259" + height="6.2582946" + x="36.94656" + y="228.71709" /> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.17638889;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8" + width="2.9174256" + height="6.2582946" + x="12.285574" + y="228.71709" /> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.17638889;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-0" + width="2.545445" + height="69.763725" + x="72.813026" + y="217.14323" /> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.17638889;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-3" + width="2.849076" + height="12.337144" + x="112.08347" + y="247.05385" /> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.17638889;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-3-0" + width="2.849076" + height="12.337144" + x="143.354" + y="270.57895" /> + <rect + style="opacity:1;fill:#b3c6dc;fill-opacity:1;stroke:#000000;stroke-width:0.17638889;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-9" + width="2.9174259" + height="6.2582946" + x="170.84859" + y="276.69199" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="25.989178" + y="226.47469" + id="text13585-3"><tspan + sodipodi:role="line" + x="25.989178" + y="226.47469" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3">rte_power_freq_up()</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="24.740162" + y="233.53178" + id="text13585-3-3"><tspan + sodipodi:role="line" + x="24.740162" + y="233.53178" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-0">status</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:2;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="56.144329" + y="214.84552" + id="text13585-3-1"><tspan + sodipodi:role="line" + x="56.486771" + y="214.84552" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-7">Loop: for each </tspan><tspan + sodipodi:role="line" + x="56.144329" + y="217.66774" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan1137">epoll event</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="91.464294" + y="233.68802" + id="text13585-3-4-9"><tspan + sodipodi:role="line" + x="91.464294" + y="233.68802" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-6-1">process_request</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="91.556564" + y="245.283" + id="text13585-3-4-9-4"><tspan + sodipodi:role="line" + x="91.556564" + y="245.283" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-6-1-0">get_pcpu_mask()</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="90.904503" + y="257.57739" + id="text13585-3-4-9-2"><tspan + sodipodi:role="line" + x="90.904503" + y="257.57739" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-6-1-05">pcpu_mask</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="158.28529" + y="281.20514" + id="text13585-3-3-2-1"><tspan + sodipodi:role="line" + x="158.28529" + y="281.20514" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-0-1-8">status</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="159.21713" + y="274.58026" + id="text13585-3-4-9-4-5"><tspan + sodipodi:role="line" + x="159.21713" + y="274.58026" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-6-1-0-5">rte_power_freq_up()</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow1Mend)" + d="m 75.021197,234.51894 c 6.293309,0.0388 13.971298,0.23602 14.165285,3.60814 0.04205,3.38442 -8.987218,3.32283 -13.497112,3.47451" + id="path873" + inkscape:connector-curvature="0" + sodipodi:nodetypes="ccc" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend)" + d="M 15.203,228.85074 H 36.946558" + id="path1561" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:0.70555558, 0.70555558;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#Arrow2Mstart)" + d="M 15.203001,234.9754 H 36.946559" + id="path1561-3" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker2723)" + d="M 39.724979,234.9754 H 72.577131" + id="path1561-0" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend-5)" + d="m 75.241702,246.95037 36.841758,0.10349" + id="path1561-5" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:0.70555559, 0.70555559;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0)" + d="M 75.372816,259.391 H 112.08346" + id="path1561-3-9" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend-5-8)" + d="M 75.241704,270.47547 143.354,270.57896" + id="path1561-5-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:0.7055556, 0.7055556;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0-0)" + d="M 75.372824,282.9161 H 143.354" + id="path1561-3-9-8" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend-3)" + d="m 146.29869,276.69198 h 24.54989" + id="path1561-4" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:0.70555559, 0.70555559;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#Arrow2Mstart-3)" + d="m 146.43232,282.81664 h 24.41626" + id="path1561-3-7" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <rect + style="opacity:1;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:0.12931521;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-4" + width="2.9174259" + height="3.3636653" + x="111.91566" + y="265.54959" /> + <rect + style="opacity:1;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:0.12931521;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-4-8" + width="2.9174259" + height="3.3636653" + x="45.63282" + y="230.29805" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="56.678867" + y="232.61894" + id="text13585-3-4"><tspan + sodipodi:role="line" + x="56.678867" + y="232.61894" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-6">guest_channel_send_msg()</tspan></text> + <rect + style="opacity:1;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:0.12931521;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-4-9" + width="2.9174259" + height="3.3636653" + x="100.28944" + y="265.70587" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="110.37933" + y="268.28806" + id="text13585-3-4-9-2-6"><tspan + sodipodi:role="line" + x="110.37933" + y="268.28806" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-6-1-05-4">scale_freq_up(pcpu_mask)</tspan></text> + <rect + style="opacity:1;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:0.12931521;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect13581-8-5-4-99" + width="2.9174259" + height="3.3636653" + x="111.78201" + y="278.54019" /> + <text + xml:space="preserve" + style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:1.41111112px;line-height:1.5;font-family:Arial;-inkscape-font-specification:'Arial Bold';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="113.21278" + y="280.93787" + id="text13585-3-3-2"><tspan + sodipodi:role="line" + x="113.21278" + y="280.93787" + style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:2.46944451px;line-height:0px;font-family:Arial;-inkscape-font-specification:Arial;text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan13591-3-0-1">status</tspan></text> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/img/vmdq_dcb_example.svg b/src/spdk/dpdk/doc/guides/sample_app_ug/img/vmdq_dcb_example.svg new file mode 100644 index 000000000..4d2882ab0 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/img/vmdq_dcb_example.svg @@ -0,0 +1,764 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2011 Intel Corporation --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="854.83661" + height="504.1741" + id="svg2" + version="1.1"> + <defs + id="defs4" /> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + showgrid="false" + fit-margin-top="0.1" + fit-margin-left="0.1" + fit-margin-right="0.1" + fit-margin-bottom="0.1" + units="cm" /> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title /> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-102.91177,-430.74991)"> + <g + id="1" + transform="translate(105.20508,428.83072)"> + <path + d="m 208.15,255.15 0,2.3 c 0,0.65 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.5 -1.15,-1.15 l 0,-2.3 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 z m 0,6.9 0,2.35 c 0,0.6 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.55 -1.15,-1.15 l 0,-2.35 c 0,-0.6 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.55 1.2,1.15 z m 0,6.95 0,2.3 c 0,0.6 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.55 -1.15,-1.15 l 0,-2.3 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 z m 0,6.9 0,2.3 c 0,0.65 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.5 -1.15,-1.15 l 0,-2.3 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 z m 0,6.9 0,1.15 c 0,0.65 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.5 -1.15,-1.15 l 0,-1.15 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 z" + style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.80000001px;stroke-linecap:butt;stroke-linejoin:bevel;stroke-opacity:1;stroke-dasharray:none" + id="2" + inkscape:connector-curvature="0" /> + <path + d="m 55.3,197.6 0,143.9 123.45,0 0,-143.9 -123.45,0 z" + style="fill:#facca7;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="3" + inkscape:connector-curvature="0" /> + <path + d="m 55.3,341.5 123.45,0 0,-143.9 -123.45,0 0,143.9 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="4" + inkscape:connector-curvature="0" /> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="264" + x="72" + xml:space="preserve" + id="5">NIC Port </text> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="264" + x="151.2" + xml:space="preserve" + id="6">0</text> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="287.20001" + x="106.4" + xml:space="preserve" + id="7">RX</text> + <path + d="m 672.6,197.6 0,143.9 123.5,0 0,-143.9 -123.5,0 z" + style="fill:#facca7;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="8" + inkscape:connector-curvature="0" /> + <path + d="m 672.6,341.5 123.5,0 0,-143.9 -123.5,0 0,143.9 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="9" + inkscape:connector-curvature="0" /> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="264" + x="689.59998" + xml:space="preserve" + id="10">NIC Port </text> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="264" + x="768.79999" + xml:space="preserve" + id="11">0 </text> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="287.20001" + x="722.40002" + xml:space="preserve" + id="12">TX</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="14.4" + x="251.2" + xml:space="preserve" + id="13">HW RX Queue </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="14.4" + x="339.20001" + xml:space="preserve" + id="14">0</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="30.4" + x="252" + xml:space="preserve" + id="15">(</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="30.4" + x="256" + xml:space="preserve" + id="16">VLAN </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="30.4" + x="293.60001" + xml:space="preserve" + id="17">0</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="30.4" + x="300.79999" + xml:space="preserve" + id="18">, </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="30.4" + x="307.20001" + xml:space="preserve" + id="19">Prio </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="30.4" + x="333.60001" + xml:space="preserve" + id="20">0</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="30.4" + x="340.79999" + xml:space="preserve" + id="21">)</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="89.599998" + x="530.40002" + xml:space="preserve" + id="22">HW TX Queue </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="89.599998" + x="616.79999" + xml:space="preserve" + id="23">0</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="406.39999" + x="530.40002" + xml:space="preserve" + id="24">HW TX Queue </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="406.39999" + x="616.79999" + xml:space="preserve" + id="25">15</text> + <path + d="m 390.4,63.3 0,95.9 123.45,0 0,-95.9 -123.45,0 z" + style="fill:#729fdc;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="26" + inkscape:connector-curvature="0" /> + <path + d="m 390.4,159.2 123.45,0 0,-95.9 -123.45,0 0,95.9 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="27" + inkscape:connector-curvature="0" /> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="105.6" + x="398.39999" + xml:space="preserve" + id="28">User Thread </text> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="128.8" + x="446.39999" + xml:space="preserve" + id="29">0</text> + <path + d="m 513.85,111.25 45.4,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="30" + inkscape:connector-curvature="0" /> + <path + d="m 566.8,111.25 -10,5 c 1.55,-3.15 1.55,-6.85 0,-10 l 10,5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="31" + inkscape:connector-curvature="0" /> + <path + d="m 310.45,88.85 0,2.3 c 0,0.65 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.5 -1.15,-1.15 l 0,-2.3 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 z m 0,6.9 0,2.35 c 0,0.6 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.55 -1.15,-1.15 l 0,-2.35 c 0,-0.6 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.55 1.2,1.15 z m 0,6.95 0,2.15 c 0,0.65 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.5 -1.15,-1.15 l 0,-2.15 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 z" + style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.80000001px;stroke-linecap:butt;stroke-linejoin:bevel;stroke-opacity:1;stroke-dasharray:none" + id="32" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,133.65 20.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="33" + inkscape:connector-curvature="0" /> + <path + d="m 390.4,133.65 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="34" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,133.65 0,31.95 -24.7,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="35" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,85.65 20.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="36" + inkscape:connector-curvature="0" /> + <path + d="m 390.4,85.65 -10,-5 c 1.6,3.15 1.6,6.85 0,10 l 0,0 10,-5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="37" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,85.65 0,-31.95 -24.7,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="38" + inkscape:connector-curvature="0" /> + <path + d="m 644.4,245.55 20.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="39" + inkscape:connector-curvature="0" /> + <path + d="m 672.6,245.55 -9.95,-5 c 1.55,3.15 1.55,6.85 0,10 l 9.95,-5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="40" + inkscape:connector-curvature="0" /> + <path + d="m 644.4,245.55 0,-134.3 -24.7,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="41" + inkscape:connector-curvature="0" /> + <path + d="m 270.85,165.6 -23.3,0 0,79.95 -68.8,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="42" + inkscape:connector-curvature="0" /> + <path + d="m 269.6,160.6 15,5 -15,5 0,-10 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="43" + inkscape:connector-curvature="0" /> + <path + d="m 178.75,220 26.45,0 0,-166.3 65.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="44" + inkscape:connector-curvature="0" /> + <path + d="m 269.6,48.7 15,5 -15,4.95 0,-9.95 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="45" + inkscape:connector-curvature="0" /> + <path + d="m 390.4,379.9 0,95.95 123.45,0 0,-95.95 -123.45,0 z" + style="fill:#729fdc;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="46" + inkscape:connector-curvature="0" /> + <path + d="m 390.4,379.9 123.45,0 0,95.95 -123.45,0 0,-95.95 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="47" + inkscape:connector-curvature="0" /> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="422.39999" + x="398.39999" + xml:space="preserve" + id="48">User Thread </text> + <text + style="font-size:19.20000076px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="444.79999" + x="441.60001" + xml:space="preserve" + id="49">15</text> + <path + d="m 513.85,427.85 45.4,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="50" + inkscape:connector-curvature="0" /> + <path + d="m 566.8,427.85 -10,-5 c 1.55,3.15 1.55,6.85 0,10 l 10,-5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="51" + inkscape:connector-curvature="0" /> + <path + d="m 308.1,421.45 0,-2.3 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 l 0,2.3 c 0,0.65 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.5 -1.15,-1.15 z m 0,-6.9 0,-2.3 c 0,-0.65 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.5 1.2,1.15 l 0,2.3 c 0,0.65 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.5 -1.15,-1.15 z m 0,-6.9 0,-2.2 c 0,-0.6 0.55,-1.15 1.15,-1.15 0.65,0 1.2,0.55 1.2,1.15 l 0,2.2 c 0,0.6 -0.55,1.15 -1.2,1.15 -0.6,0 -1.15,-0.55 -1.15,-1.15 z" + style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.80000001px;stroke-linecap:butt;stroke-linejoin:bevel;stroke-opacity:1;stroke-dasharray:none" + id="52" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,405.45 20.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="53" + inkscape:connector-curvature="0" /> + <path + d="m 390.4,405.45 -10,-4.95 c 1.6,3.1 1.6,6.8 0,9.95 l 0,0 10,-5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="54" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,405.45 0,-31.95 -24.7,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="55" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,453.45 20.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="56" + inkscape:connector-curvature="0" /> + <path + d="m 390.4,453.45 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="57" + inkscape:connector-curvature="0" /> + <path + d="m 362.2,453.45 0,31.95 -24.7,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="58" + inkscape:connector-curvature="0" /> + <path + d="m 644.4,293.55 20.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="59" + inkscape:connector-curvature="0" /> + <path + d="m 672.6,293.55 -9.95,5 c 1.55,-3.15 1.55,-6.85 0,-10 l 9.95,5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="60" + inkscape:connector-curvature="0" /> + <path + d="m 644.4,293.55 0,134.3 -24.7,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="61" + inkscape:connector-curvature="0" /> + <path + d="m 270.85,373.5 -23.3,0 0,-79.95 -68.8,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="62" + inkscape:connector-curvature="0" /> + <path + d="m 269.6,378.5 15,-5 -15,-5 0,10 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="63" + inkscape:connector-curvature="0" /> + <path + d="m 178.75,319.1 26.45,0 0,166.3 65.65,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="64" + inkscape:connector-curvature="0" /> + <path + d="m 269.6,490.4 15,-5 -15,-4.95 0,9.95 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="65" + inkscape:connector-curvature="0" /> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="126.4" + x="251.2" + xml:space="preserve" + id="66">HW RX Queue </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="126.4" + x="339.20001" + xml:space="preserve" + id="67">7</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="141.60001" + x="252" + xml:space="preserve" + id="68">(</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="141.60001" + x="256" + xml:space="preserve" + id="69">VLAN </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="141.60001" + x="293.60001" + xml:space="preserve" + id="70">0</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="141.60001" + x="300.79999" + xml:space="preserve" + id="71">, </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="141.60001" + x="307.20001" + xml:space="preserve" + id="72">Prio </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="141.60001" + x="333.60001" + xml:space="preserve" + id="73">7</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="141.60001" + x="340.79999" + xml:space="preserve" + id="74">)</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="334.39999" + x="264.79999" + xml:space="preserve" + id="75">HW RX Queue </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="334.39999" + x="353.60001" + xml:space="preserve" + id="76">120</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="349.60001" + x="269.60001" + xml:space="preserve" + id="77">(</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="349.60001" + x="273.60001" + xml:space="preserve" + id="78">VLAN </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="349.60001" + x="311.20001" + xml:space="preserve" + id="79">15</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="349.60001" + x="324.79999" + xml:space="preserve" + id="80">, </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="349.60001" + x="332" + xml:space="preserve" + id="81">Prio </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="349.60001" + x="358.39999" + xml:space="preserve" + id="82">0</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="349.60001" + x="365.60001" + xml:space="preserve" + id="83">)</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="446.39999" + x="240" + xml:space="preserve" + id="84">HW RX Queue </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="446.39999" + x="328.79999" + xml:space="preserve" + id="85">127</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="461.60001" + x="244.8" + xml:space="preserve" + id="86">(</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="461.60001" + x="249.60001" + xml:space="preserve" + id="87">VLAN </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="461.60001" + x="286.39999" + xml:space="preserve" + id="88">15</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="461.60001" + x="300.79999" + xml:space="preserve" + id="89">, </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="461.60001" + x="307.20001" + xml:space="preserve" + id="90">Prio </text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="461.60001" + x="333.60001" + xml:space="preserve" + id="91">7</text> + <text + style="font-size:12.80000019px;font-style:italic;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Arial" + y="461.60001" + x="340.79999" + xml:space="preserve" + id="92">)</text> + <path + d="m 652.6,255.15 0,2.3 c 0,0.65 -0.5,1.15 -1.15,1.15 -0.65,0 -1.15,-0.5 -1.15,-1.15 l 0,-2.3 c 0,-0.65 0.5,-1.15 1.15,-1.15 0.65,0 1.15,0.5 1.15,1.15 z m 0,6.9 0,2.35 c 0,0.6 -0.5,1.15 -1.15,1.15 -0.65,0 -1.15,-0.55 -1.15,-1.15 l 0,-2.35 c 0,-0.6 0.5,-1.15 1.15,-1.15 0.65,0 1.15,0.55 1.15,1.15 z m 0,6.95 0,2.3 c 0,0.6 -0.5,1.15 -1.15,1.15 -0.65,0 -1.15,-0.55 -1.15,-1.15 l 0,-2.3 c 0,-0.65 0.5,-1.15 1.15,-1.15 0.65,0 1.15,0.5 1.15,1.15 z m 0,6.9 0,2.3 c 0,0.65 -0.5,1.15 -1.15,1.15 -0.65,0 -1.15,-0.5 -1.15,-1.15 l 0,-2.3 c 0,-0.65 0.5,-1.15 1.15,-1.15 0.65,0 1.15,0.5 1.15,1.15 z m 0,6.9 0,1.15 c 0,0.65 -0.5,1.15 -1.15,1.15 -0.65,0 -1.15,-0.5 -1.15,-1.15 l 0,-1.15 c 0,-0.65 0.5,-1.15 1.15,-1.15 0.65,0 1.15,0.5 1.15,1.15 z" + style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.80000001px;stroke-linecap:butt;stroke-linejoin:bevel;stroke-opacity:1;stroke-dasharray:none" + id="93" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,37.7 0,31.95 17.65,0 0,-31.95 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="94" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,69.65 17.65,0 0,-31.95 -17.65,0 0,31.95 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="95" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,37.7 0,31.95 17.65,0 0,-31.95 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="96" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,69.65 17.65,0 0,-31.95 -17.65,0 0,31.95 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="97" + inkscape:connector-curvature="0" /> + <path + d="m 284.6,69.65 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="98" + inkscape:connector-curvature="0" /> + <path + d="m 285.45,37.7 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="99" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,149.6 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="100" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,181.6 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="101" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,149.6 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="102" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,181.6 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="103" + inkscape:connector-curvature="0" /> + <path + d="m 284.6,181.6 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="104" + inkscape:connector-curvature="0" /> + <path + d="m 285.45,149.6 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="105" + inkscape:connector-curvature="0" /> + <path + d="m 584.4,95.25 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="106" + inkscape:connector-curvature="0" /> + <path + d="m 584.4,127.25 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="107" + inkscape:connector-curvature="0" /> + <path + d="m 602.05,95.25 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="108" + inkscape:connector-curvature="0" /> + <path + d="m 602.05,127.25 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="109" + inkscape:connector-curvature="0" /> + <path + d="m 566.8,127.25 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="110" + inkscape:connector-curvature="0" /> + <path + d="m 567.65,95.25 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="111" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,357.5 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="112" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,389.5 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="113" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,357.5 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="114" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,389.5 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="115" + inkscape:connector-curvature="0" /> + <path + d="m 284.6,389.5 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="116" + inkscape:connector-curvature="0" /> + <path + d="m 285.45,357.5 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="117" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,469.45 0,31.95 17.65,0 0,-31.95 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="118" + inkscape:connector-curvature="0" /> + <path + d="m 302.2,501.4 17.65,0 0,-31.95 -17.65,0 0,31.95 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="119" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,469.45 0,31.95 17.65,0 0,-31.95 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="120" + inkscape:connector-curvature="0" /> + <path + d="m 319.85,501.4 17.65,0 0,-31.95 -17.65,0 0,31.95 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="121" + inkscape:connector-curvature="0" /> + <path + d="m 284.6,501.4 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="122" + inkscape:connector-curvature="0" /> + <path + d="m 285.45,469.45 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="123" + inkscape:connector-curvature="0" /> + <path + d="m 584.4,411.85 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="124" + inkscape:connector-curvature="0" /> + <path + d="m 584.4,443.85 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="125" + inkscape:connector-curvature="0" /> + <path + d="m 602.05,411.85 0,32 17.65,0 0,-32 -17.65,0 z" + style="fill:#f28226;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="126" + inkscape:connector-curvature="0" /> + <path + d="m 602.05,443.85 17.65,0 0,-32 -17.65,0 0,32 z" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="127" + inkscape:connector-curvature="0" /> + <path + d="m 566.8,443.85 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="128" + inkscape:connector-curvature="0" /> + <path + d="m 567.65,411.85 16.75,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="129" + inkscape:connector-curvature="0" /> + <path + d="m 2.4,268.6 45.35,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="130" + inkscape:connector-curvature="0" /> + <path + d="m 55.3,268.6 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="131" + inkscape:connector-curvature="0" /> + <path + d="m 796.1,268.6 45.35,0" + style="fill:none;stroke:#000000;stroke-width:2.29999995px;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" + id="132" + inkscape:connector-curvature="0" /> + <path + d="m 849,268.6 -10,5 c 1.6,-3.15 1.6,-6.85 0,-10 l 0,0 10,5 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none" + id="133" + inkscape:connector-curvature="0" /> + </g> + </g> +</svg> diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/index.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/index.rst new file mode 100644 index 000000000..affa9c574 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/index.rst @@ -0,0 +1,60 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2015 Intel Corporation. + +Sample Applications User Guides +=============================== + +.. toctree:: + :maxdepth: 2 + :numbered: + + intro + compiling + cmd_line + ethtool + hello_world + skeleton + rxtx_callbacks + flow_classify + flow_filtering + ip_frag + ipv4_multicast + ip_reassembly + kernel_nic_interface + keep_alive + ioat + l2_forward_crypto + l2_forward_job_stats + l2_forward_real_virtual + l2_forward_event + l2_forward_cat + l3_forward + l3_forward_graph + l3_forward_power_man + l3_forward_access_ctrl + link_status_intr + server_node_efd + service_cores + multi_process + qos_metering + qos_scheduler + timer + packet_ordering + vmdq_dcb_forwarding + vmdq_forwarding + vhost + vhost_blk + vhost_crypto + vdpa + ip_pipeline + test_pipeline + eventdev_pipeline + dist_app + vm_power_management + tep_termination + ptpclient + performance_thread + fips_validation + ipsec_secgw + bbdev_app + ntb diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/intro.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/intro.rst new file mode 100644 index 000000000..8ff223b16 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/intro.rst @@ -0,0 +1,96 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2017 Intel Corporation. + +Introduction to the DPDK Sample Applications +============================================ + +The DPDK Sample Applications are small standalone applications which +demonstrate various features of DPDK. They can be considered as a cookbook of +DPDK features. Users interested in getting started with DPDK can take the +applications, try out the features, and then extend them to fit their needs. + + +Running Sample Applications +--------------------------- + +Some sample applications may have their own command-line parameters described in +their respective guides, however all of them also share the same EAL parameters. +Please refer to :doc:`EAL parameters (Linux) <../linux_gsg/linux_eal_parameters>` +or :doc:`EAL parameters (FreeBSD) <../freebsd_gsg/freebsd_eal_parameters>` for +a list of available EAL command-line options. + + +The DPDK Sample Applications +---------------------------- + +There are many sample applications available in the examples directory of DPDK. +These examples range from simple to reasonably complex but most are designed +to demonstrate one particular feature of DPDK. Some of the more interesting +examples are highlighted below. + + +* :doc:`Hello World<hello_world>`: As with most introductions to a + programming framework a good place to start is with the Hello World + application. The Hello World example sets up the DPDK Environment Abstraction + Layer (EAL), and prints a simple "Hello World" message to each of the DPDK + enabled cores. This application doesn't do any packet forwarding but it is a + good way to test if the DPDK environment is compiled and set up properly. + +* :doc:`Basic Forwarding/Skeleton Application<skeleton>`: The Basic + Forwarding/Skeleton contains the minimum amount of code required to enable + basic packet forwarding with DPDK. This allows you to test if your network + interfaces are working with DPDK. + +* :doc:`Network Layer 2 forwarding<l2_forward_real_virtual>`: The Network Layer 2 + forwarding, or ``l2fwd`` application does forwarding based on Ethernet MAC + addresses like a simple switch. + +* :doc:`Network Layer 2 forwarding<l2_forward_event>`: The Network Layer 2 + forwarding, or ``l2fwd-event`` application does forwarding based on Ethernet MAC + addresses like a simple switch. It demonstrates usage of poll and event mode + IO mechanism under a single application. + +* :doc:`Network Layer 3 forwarding<l3_forward>`: The Network Layer3 + forwarding, or ``l3fwd`` application does forwarding based on Internet + Protocol, IPv4 or IPv6 like a simple router. + +* :doc:`Network Layer 3 forwarding Graph<l3_forward_graph>`: The Network Layer3 + forwarding Graph, or ``l3fwd_graph`` application does forwarding based on IPv4 + like a simple router with DPDK Graph framework. + +* :doc:`Hardware packet copying<ioat>`: The Hardware packet copying, + or ``ioatfwd`` application demonstrates how to use IOAT rawdev driver for + copying packets between two threads. + +* :doc:`Packet Distributor<dist_app>`: The Packet Distributor + demonstrates how to distribute packets arriving on an Rx port to different + cores for processing and transmission. + +* :doc:`Multi-Process Application<multi_process>`: The + multi-process application shows how two DPDK processes can work together using + queues and memory pools to share information. + +* :doc:`RX/TX callbacks Application<rxtx_callbacks>`: The RX/TX + callbacks sample application is a packet forwarding application that + demonstrates the use of user defined callbacks on received and transmitted + packets. The application calculates the latency of a packet between RX + (packet arrival) and TX (packet transmission) by adding callbacks to the RX + and TX packet processing functions. + +* :doc:`IPsec Security Gateway<ipsec_secgw>`: The IPsec Security + Gateway application is minimal example of something closer to a real world + example. This is also a good example of an application using the DPDK + Cryptodev framework. + +* :doc:`Precision Time Protocol (PTP) client<ptpclient>`: The PTP + client is another minimal implementation of a real world application. + In this case the application is a PTP client that communicates with a PTP + master clock to synchronize time on a Network Interface Card (NIC) using the + IEEE1588 protocol. + +* :doc:`Quality of Service (QoS) Scheduler<qos_scheduler>`: The QoS + Scheduler application demonstrates the use of DPDK to provide QoS scheduling. + +There are many more examples shown in the following chapters. Each of the +documented sample applications show how to compile, configure and run the +application as well as explaining the main functionality of the code. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ioat.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ioat.rst new file mode 100644 index 000000000..bab7654b8 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ioat.rst @@ -0,0 +1,581 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +.. include:: <isonum.txt> + +Packet copying using Intel\ |reg| QuickData Technology +====================================================== + +Overview +-------- + +This sample is intended as a demonstration of the basic components of a DPDK +forwarding application and example of how to use IOAT driver API to make +packets copies. + +Also while forwarding, the MAC addresses are affected as follows: + +* The source MAC address is replaced by the TX port MAC address + +* The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID + +This application can be used to compare performance of using software packet +copy with copy done using a DMA device for different sizes of packets. +The example will print out statistics each second. The stats shows +received/send packets and packets dropped or failed to copy. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ioat`` sub-directory. + + +Running the Application +----------------------- + +In order to run the hardware copy application, the copying device +needs to be bound to user-space IO driver. + +Refer to the "IOAT Rawdev Driver" chapter in the "Rawdev Drivers" document +for information on using the driver. + +The application requires a number of command line options: + +.. code-block:: console + + ./build/ioatfwd [EAL options] -- [-p MASK] [-q NQ] [-s RS] [-c <sw|hw>] + [--[no-]mac-updating] + +where, + +* p MASK: A hexadecimal bitmask of the ports to configure (default is all) + +* q NQ: Number of Rx queues used per port equivalent to CBDMA channels + per port (default is 1) + +* c CT: Performed packet copy type: software (sw) or hardware using + DMA (hw) (default is hw) + +* s RS: Size of IOAT rawdev ring for hardware copy mode or rte_ring for + software copy mode (default is 2048) + +* --[no-]mac-updating: Whether MAC address of packets should be changed + or not (default is mac-updating) + +The application can be launched in various configurations depending on +provided parameters. The app can use up to 2 lcores: one of them receives +incoming traffic and makes a copy of each packet. The second lcore then +updates MAC address and sends the copy. If one lcore per port is used, +both operations are done sequentially. For each configuration an additional +lcore is needed since the master lcore does not handle traffic but is +responsible for configuration, statistics printing and safe shutdown of +all ports and devices. + +The application can use a maximum of 8 ports. + +To run the application in a Linux environment with 3 lcores (the master lcore, +plus two forwarding cores), a single port (port 0), software copying and MAC +updating issue the command: + +.. code-block:: console + + $ ./build/ioatfwd -l 0-2 -n 2 -- -p 0x1 --mac-updating -c sw + +To run the application in a Linux environment with 2 lcores (the master lcore, +plus one forwarding core), 2 ports (ports 0 and 1), hardware copying and no MAC +updating issue the command: + +.. code-block:: console + + $ ./build/ioatfwd -l 0-1 -n 1 -- -p 0x3 --no-mac-updating -c hw + +Refer to the *DPDK Getting Started Guide* for general information on +running applications and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide an explanation of the main components of the +code. + +All DPDK library functions used in the sample code are prefixed with +``rte_`` and are explained in detail in the *DPDK API Documentation*. + + +The Main Function +~~~~~~~~~~~~~~~~~ + +The ``main()`` function performs the initialization and calls the execution +threads for each lcore. + +The first task is to initialize the Environment Abstraction Layer (EAL). +The ``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` +function. The value returned is the number of parsed arguments: + +.. code-block:: c + + /* init EAL */ + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); + + +The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers) +used by the application: + +.. code-block:: c + + nb_mbufs = RTE_MAX(rte_eth_dev_count_avail() * (nb_rxd + nb_txd + + MAX_PKT_BURST + rte_lcore_count() * MEMPOOL_CACHE_SIZE), + MIN_POOL_SIZE); + + /* Create the mbuf pool */ + ioat_pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", nb_mbufs, + MEMPOOL_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, + rte_socket_id()); + if (ioat_pktmbuf_pool == NULL) + rte_exit(EXIT_FAILURE, "Cannot init mbuf pool\n"); + +Mbufs are the packet buffer structure used by DPDK. They are explained in +detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*. + +The ``main()`` function also initializes the ports: + +.. code-block:: c + + /* Initialise each port */ + RTE_ETH_FOREACH_DEV(portid) { + port_init(portid, ioat_pktmbuf_pool); + } + +Each port is configured using ``port_init()`` function. The Ethernet +ports are configured with local settings using the ``rte_eth_dev_configure()`` +function and the ``port_conf`` struct. The RSS is enabled so that +multiple Rx queues could be used for packet receiving and copying by +multiple CBDMA channels per port: + +.. code-block:: c + + /* configuring port to use RSS for multiple RX queues */ + static const struct rte_eth_conf port_conf = { + .rxmode = { + .mq_mode = ETH_MQ_RX_RSS, + .max_rx_pkt_len = RTE_ETHER_MAX_LEN + }, + .rx_adv_conf = { + .rss_conf = { + .rss_key = NULL, + .rss_hf = ETH_RSS_PROTO_MASK, + } + } + }; + +For this example the ports are set up with the number of Rx queues provided +with -q option and 1 Tx queue using the ``rte_eth_rx_queue_setup()`` +and ``rte_eth_tx_queue_setup()`` functions. + +The Ethernet port is then started: + +.. code-block:: c + + ret = rte_eth_dev_start(portid); + if (ret < 0) + rte_exit(EXIT_FAILURE, "rte_eth_dev_start:err=%d, port=%u\n", + ret, portid); + + +Finally the Rx port is set in promiscuous mode: + +.. code-block:: c + + rte_eth_promiscuous_enable(portid); + + +After that each port application assigns resources needed. + +.. code-block:: c + + check_link_status(ioat_enabled_port_mask); + + if (!cfg.nb_ports) { + rte_exit(EXIT_FAILURE, + "All available ports are disabled. Please set portmask.\n"); + } + + /* Check if there is enough lcores for all ports. */ + cfg.nb_lcores = rte_lcore_count() - 1; + if (cfg.nb_lcores < 1) + rte_exit(EXIT_FAILURE, + "There should be at least one slave lcore.\n"); + + ret = 0; + + if (copy_mode == COPY_MODE_IOAT_NUM) { + assign_rawdevs(); + } else /* copy_mode == COPY_MODE_SW_NUM */ { + assign_rings(); + } + +Depending on mode set (whether copy should be done by software or by hardware) +special structures are assigned to each port. If software copy was chosen, +application have to assign ring structures for packet exchanging between lcores +assigned to ports. + +.. code-block:: c + + static void + assign_rings(void) + { + uint32_t i; + + for (i = 0; i < cfg.nb_ports; i++) { + char ring_name[20]; + + snprintf(ring_name, 20, "rx_to_tx_ring_%u", i); + /* Create ring for inter core communication */ + cfg.ports[i].rx_to_tx_ring = rte_ring_create( + ring_name, ring_size, + rte_socket_id(), RING_F_SP_ENQ); + + if (cfg.ports[i].rx_to_tx_ring == NULL) + rte_exit(EXIT_FAILURE, "%s\n", + rte_strerror(rte_errno)); + } + } + + +When using hardware copy each Rx queue of the port is assigned an +IOAT device (``assign_rawdevs()``) using IOAT Rawdev Driver API +functions: + +.. code-block:: c + + static void + assign_rawdevs(void) + { + uint16_t nb_rawdev = 0, rdev_id = 0; + uint32_t i, j; + + for (i = 0; i < cfg.nb_ports; i++) { + for (j = 0; j < cfg.ports[i].nb_queues; j++) { + struct rte_rawdev_info rdev_info = { 0 }; + + do { + if (rdev_id == rte_rawdev_count()) + goto end; + rte_rawdev_info_get(rdev_id++, &rdev_info); + } while (strcmp(rdev_info.driver_name, + IOAT_PMD_RAWDEV_NAME_STR) != 0); + + cfg.ports[i].ioat_ids[j] = rdev_id - 1; + configure_rawdev_queue(cfg.ports[i].ioat_ids[j]); + ++nb_rawdev; + } + } + end: + if (nb_rawdev < cfg.nb_ports * cfg.ports[0].nb_queues) + rte_exit(EXIT_FAILURE, + "Not enough IOAT rawdevs (%u) for all queues (%u).\n", + nb_rawdev, cfg.nb_ports * cfg.ports[0].nb_queues); + RTE_LOG(INFO, IOAT, "Number of used rawdevs: %u.\n", nb_rawdev); + } + + +The initialization of hardware device is done by ``rte_rawdev_configure()`` +function using ``rte_rawdev_info`` struct. After configuration the device is +started using ``rte_rawdev_start()`` function. Each of the above operations +is done in ``configure_rawdev_queue()``. + +.. code-block:: c + + static void + configure_rawdev_queue(uint32_t dev_id) + { + struct rte_ioat_rawdev_config dev_config = { .ring_size = ring_size }; + struct rte_rawdev_info info = { .dev_private = &dev_config }; + + if (rte_rawdev_configure(dev_id, &info) != 0) { + rte_exit(EXIT_FAILURE, + "Error with rte_rawdev_configure()\n"); + } + if (rte_rawdev_start(dev_id) != 0) { + rte_exit(EXIT_FAILURE, + "Error with rte_rawdev_start()\n"); + } + } + +If initialization is successful, memory for hardware device +statistics is allocated. + +Finally ``main()`` function starts all packet handling lcores and starts +printing stats in a loop on the master lcore. The application can be +interrupted and closed using ``Ctrl-C``. The master lcore waits for +all slave processes to finish, deallocates resources and exits. + +The processing lcores launching function are described below. + +The Lcores Launching Functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As described above, ``main()`` function invokes ``start_forwarding_cores()`` +function in order to start processing for each lcore: + +.. code-block:: c + + static void start_forwarding_cores(void) + { + uint32_t lcore_id = rte_lcore_id(); + + RTE_LOG(INFO, IOAT, "Entering %s on lcore %u\n", + __func__, rte_lcore_id()); + + if (cfg.nb_lcores == 1) { + lcore_id = rte_get_next_lcore(lcore_id, true, true); + rte_eal_remote_launch((lcore_function_t *)rxtx_main_loop, + NULL, lcore_id); + } else if (cfg.nb_lcores > 1) { + lcore_id = rte_get_next_lcore(lcore_id, true, true); + rte_eal_remote_launch((lcore_function_t *)rx_main_loop, + NULL, lcore_id); + + lcore_id = rte_get_next_lcore(lcore_id, true, true); + rte_eal_remote_launch((lcore_function_t *)tx_main_loop, NULL, + lcore_id); + } + } + +The function launches Rx/Tx processing functions on configured lcores +using ``rte_eal_remote_launch()``. The configured ports, their number +and number of assigned lcores are stored in user-defined +``rxtx_transmission_config`` struct: + +.. code-block:: c + + struct rxtx_transmission_config { + struct rxtx_port_config ports[RTE_MAX_ETHPORTS]; + uint16_t nb_ports; + uint16_t nb_lcores; + }; + +The structure is initialized in 'main()' function with the values +corresponding to ports and lcores configuration provided by the user. + +The Lcores Processing Functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For receiving packets on each port, the ``ioat_rx_port()`` function is used. +The function receives packets on each configured Rx queue. Depending on the +mode the user chose, it will enqueue packets to IOAT rawdev channels and +then invoke copy process (hardware copy), or perform software copy of each +packet using ``pktmbuf_sw_copy()`` function and enqueue them to an rte_ring: + +.. code-block:: c + + /* Receive packets on one port and enqueue to IOAT rawdev or rte_ring. */ + static void + ioat_rx_port(struct rxtx_port_config *rx_config) + { + uint32_t nb_rx, nb_enq, i, j; + struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; + for (i = 0; i < rx_config->nb_queues; i++) { + + nb_rx = rte_eth_rx_burst(rx_config->rxtx_port, i, + pkts_burst, MAX_PKT_BURST); + + if (nb_rx == 0) + continue; + + port_statistics.rx[rx_config->rxtx_port] += nb_rx; + + if (copy_mode == COPY_MODE_IOAT_NUM) { + /* Perform packet hardware copy */ + nb_enq = ioat_enqueue_packets(pkts_burst, + nb_rx, rx_config->ioat_ids[i]); + if (nb_enq > 0) + rte_ioat_do_copies(rx_config->ioat_ids[i]); + } else { + /* Perform packet software copy, free source packets */ + int ret; + struct rte_mbuf *pkts_burst_copy[MAX_PKT_BURST]; + + ret = rte_mempool_get_bulk(ioat_pktmbuf_pool, + (void *)pkts_burst_copy, nb_rx); + + if (unlikely(ret < 0)) + rte_exit(EXIT_FAILURE, + "Unable to allocate memory.\n"); + + for (j = 0; j < nb_rx; j++) + pktmbuf_sw_copy(pkts_burst[j], + pkts_burst_copy[j]); + + rte_mempool_put_bulk(ioat_pktmbuf_pool, + (void *)pkts_burst, nb_rx); + + nb_enq = rte_ring_enqueue_burst( + rx_config->rx_to_tx_ring, + (void *)pkts_burst_copy, nb_rx, NULL); + + /* Free any not enqueued packets. */ + rte_mempool_put_bulk(ioat_pktmbuf_pool, + (void *)&pkts_burst_copy[nb_enq], + nb_rx - nb_enq); + } + + port_statistics.copy_dropped[rx_config->rxtx_port] += + (nb_rx - nb_enq); + } + } + +The packets are received in burst mode using ``rte_eth_rx_burst()`` +function. When using hardware copy mode the packets are enqueued in +copying device's buffer using ``ioat_enqueue_packets()`` which calls +``rte_ioat_enqueue_copy()``. When all received packets are in the +buffer the copy operations are started by calling ``rte_ioat_do_copies()``. +Function ``rte_ioat_enqueue_copy()`` operates on physical address of +the packet. Structure ``rte_mbuf`` contains only physical address to +start of the data buffer (``buf_iova``). Thus the address is adjusted +by ``addr_offset`` value in order to get the address of ``rearm_data`` +member of ``rte_mbuf``. That way both the packet data and metadata can +be copied in a single operation. This method can be used because the mbufs +are direct mbufs allocated by the apps. If another app uses external buffers, +or indirect mbufs, then multiple copy operations must be used. + +.. code-block:: c + + static uint32_t + ioat_enqueue_packets(struct rte_mbuf **pkts, + uint32_t nb_rx, uint16_t dev_id) + { + int ret; + uint32_t i; + struct rte_mbuf *pkts_copy[MAX_PKT_BURST]; + + const uint64_t addr_offset = RTE_PTR_DIFF(pkts[0]->buf_addr, + &pkts[0]->rearm_data); + + ret = rte_mempool_get_bulk(ioat_pktmbuf_pool, + (void *)pkts_copy, nb_rx); + + if (unlikely(ret < 0)) + rte_exit(EXIT_FAILURE, "Unable to allocate memory.\n"); + + for (i = 0; i < nb_rx; i++) { + /* Perform data copy */ + ret = rte_ioat_enqueue_copy(dev_id, + pkts[i]->buf_iova + - addr_offset, + pkts_copy[i]->buf_iova + - addr_offset, + rte_pktmbuf_data_len(pkts[i]) + + addr_offset, + (uintptr_t)pkts[i], + (uintptr_t)pkts_copy[i], + 0 /* nofence */); + + if (ret != 1) + break; + } + + ret = i; + /* Free any not enqueued packets. */ + rte_mempool_put_bulk(ioat_pktmbuf_pool, (void *)&pkts[i], nb_rx - i); + rte_mempool_put_bulk(ioat_pktmbuf_pool, (void *)&pkts_copy[i], + nb_rx - i); + + return ret; + } + + +All completed copies are processed by ``ioat_tx_port()`` function. When using +hardware copy mode the function invokes ``rte_ioat_completed_copies()`` +on each assigned IOAT channel to gather copied packets. If software copy +mode is used the function dequeues copied packets from the rte_ring. Then each +packet MAC address is changed if it was enabled. After that copies are sent +in burst mode using `` rte_eth_tx_burst()``. + + +.. code-block:: c + + /* Transmit packets from IOAT rawdev/rte_ring for one port. */ + static void + ioat_tx_port(struct rxtx_port_config *tx_config) + { + uint32_t i, j, nb_dq = 0; + struct rte_mbuf *mbufs_src[MAX_PKT_BURST]; + struct rte_mbuf *mbufs_dst[MAX_PKT_BURST]; + + for (i = 0; i < tx_config->nb_queues; i++) { + if (copy_mode == COPY_MODE_IOAT_NUM) { + /* Deque the mbufs from IOAT device. */ + nb_dq = rte_ioat_completed_copies( + tx_config->ioat_ids[i], MAX_PKT_BURST, + (void *)mbufs_src, (void *)mbufs_dst); + } else { + /* Deque the mbufs from rx_to_tx_ring. */ + nb_dq = rte_ring_dequeue_burst( + tx_config->rx_to_tx_ring, (void *)mbufs_dst, + MAX_PKT_BURST, NULL); + } + + if (nb_dq == 0) + return; + + if (copy_mode == COPY_MODE_IOAT_NUM) + rte_mempool_put_bulk(ioat_pktmbuf_pool, + (void *)mbufs_src, nb_dq); + + /* Update macs if enabled */ + if (mac_updating) { + for (j = 0; j < nb_dq; j++) + update_mac_addrs(mbufs_dst[j], + tx_config->rxtx_port); + } + + const uint16_t nb_tx = rte_eth_tx_burst( + tx_config->rxtx_port, 0, + (void *)mbufs_dst, nb_dq); + + port_statistics.tx[tx_config->rxtx_port] += nb_tx; + + /* Free any unsent packets. */ + if (unlikely(nb_tx < nb_dq)) + rte_mempool_put_bulk(ioat_pktmbuf_pool, + (void *)&mbufs_dst[nb_tx], + nb_dq - nb_tx); + } + } + +The Packet Copying Functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In order to perform packet copy there is a user-defined function +``pktmbuf_sw_copy()`` used. It copies a whole packet by copying +metadata from source packet to new mbuf, and then copying a data +chunk of source packet. Both memory copies are done using +``rte_memcpy()``: + +.. code-block:: c + + static inline void + pktmbuf_sw_copy(struct rte_mbuf *src, struct rte_mbuf *dst) + { + /* Copy packet metadata */ + rte_memcpy(&dst->rearm_data, + &src->rearm_data, + offsetof(struct rte_mbuf, cacheline1) + - offsetof(struct rte_mbuf, rearm_data)); + + /* Copy packet data */ + rte_memcpy(rte_pktmbuf_mtod(dst, char *), + rte_pktmbuf_mtod(src, char *), src->data_len); + } + +The metadata in this example is copied from ``rearm_data`` member of +``rte_mbuf`` struct up to ``cacheline1``. + +In order to understand why software packet copying is done as shown +above please refer to the "Mbuf Library" section of the +*DPDK Programmer's Guide*. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ip_frag.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ip_frag.rst new file mode 100644 index 000000000..afeaff363 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ip_frag.rst @@ -0,0 +1,140 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +IP Fragmentation Sample Application +=================================== + +The IPv4 Fragmentation application is a simple example of packet processing +using the Data Plane Development Kit (DPDK). +The application does L3 forwarding with IPv4 and IPv6 packet fragmentation. + +Overview +-------- + +The application demonstrates the use of zero-copy buffers for packet fragmentation. +The initialization and run-time paths are very similar to those of the :doc:`l2_forward_real_virtual`. +This guide highlights the differences between the two applications. + +There are three key differences from the L2 Forwarding sample application: + +* The first difference is that the IP Fragmentation sample application makes use of indirect buffers. + +* The second difference is that the forwarding decision is taken + based on information read from the input packet's IP header. + +* The third difference is that the application differentiates between + IP and non-IP traffic by means of offload flags. + +The Longest Prefix Match (LPM for IPv4, LPM6 for IPv6) table is used to store/lookup an outgoing port number, +associated with that IP address. +Any unmatched packets are forwarded to the originating port. + +By default, input frame sizes up to 9.5 KB are supported. +Before forwarding, the input IP packet is fragmented to fit into the "standard" Ethernet* v2 MTU (1500 bytes). + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ip_fragmentation`` sub-directory. + +Running the Application +----------------------- + +The LPM object is created and loaded with the pre-configured entries read from +global l3fwd_ipv4_route_array and l3fwd_ipv6_route_array tables. +For each input packet, the packet forwarding decision +(that is, the identification of the output interface for the packet) is taken as a result of LPM lookup. +If the IP packet size is greater than default output MTU, +then the input packet is fragmented and several fragments are sent via the output interface. + +Application usage: + +.. code-block:: console + + ./build/ip_fragmentation [EAL options] -- -p PORTMASK [-q NQ] + +where: + +* -p PORTMASK is a hexadecimal bitmask of ports to configure + +* -q NQ is the number of queue (=ports) per lcore (the default is 1) + +To run the example in linux environment with 2 lcores (2,4) over 2 ports(0,2) with 1 RX queue per lcore: + +.. code-block:: console + + ./build/ip_fragmentation -l 2,4 -n 3 -- -p 5 + EAL: coremask set to 14 + EAL: Detected lcore 0 on socket 0 + EAL: Detected lcore 1 on socket 1 + EAL: Detected lcore 2 on socket 0 + EAL: Detected lcore 3 on socket 1 + EAL: Detected lcore 4 on socket 0 + ... + + Initializing port 0 on lcore 2... Address:00:1B:21:76:FA:2C, rxq=0 txq=2,0 txq=4,1 + done: Link Up - speed 10000 Mbps - full-duplex + Skipping disabled port 1 + Initializing port 2 on lcore 4... Address:00:1B:21:5C:FF:54, rxq=0 txq=2,0 txq=4,1 + done: Link Up - speed 10000 Mbps - full-duplex + Skipping disabled port 3IP_FRAG: Socket 0: adding route 100.10.0.0/16 (port 0) + IP_FRAG: Socket 0: adding route 100.20.0.0/16 (port 1) + ... + IP_FRAG: Socket 0: adding route 0101:0101:0101:0101:0101:0101:0101:0101/48 (port 0) + IP_FRAG: Socket 0: adding route 0201:0101:0101:0101:0101:0101:0101:0101/48 (port 1) + ... + IP_FRAG: entering main loop on lcore 4 + IP_FRAG: -- lcoreid=4 portid=2 + IP_FRAG: entering main loop on lcore 2 + IP_FRAG: -- lcoreid=2 portid=0 + +To run the example in linux environment with 1 lcore (4) over 2 ports(0,2) with 2 RX queues per lcore: + +.. code-block:: console + + ./build/ip_fragmentation -l 4 -n 3 -- -p 5 -q 2 + +To test the application, flows should be set up in the flow generator that match the values in the +l3fwd_ipv4_route_array and/or l3fwd_ipv6_route_array table. + +The default l3fwd_ipv4_route_array table is: + +.. code-block:: c + + struct l3fwd_ipv4_route l3fwd_ipv4_route_array[] = { + {RTE_IPV4(100, 10, 0, 0), 16, 0}, + {RTE_IPV4(100, 20, 0, 0), 16, 1}, + {RTE_IPV4(100, 30, 0, 0), 16, 2}, + {RTE_IPV4(100, 40, 0, 0), 16, 3}, + {RTE_IPV4(100, 50, 0, 0), 16, 4}, + {RTE_IPV4(100, 60, 0, 0), 16, 5}, + {RTE_IPV4(100, 70, 0, 0), 16, 6}, + {RTE_IPV4(100, 80, 0, 0), 16, 7}, + }; + +The default l3fwd_ipv6_route_array table is: + +.. code-block:: c + + struct l3fwd_ipv6_route l3fwd_ipv6_route_array[] = { + {{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 0}, + {{2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 1}, + {{3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 2}, + {{4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 3}, + {{5, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 4}, + {{6, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 5}, + {{7, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 6}, + {{8, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 7}, + }; + +For example, for the input IPv4 packet with destination address: 100.10.1.1 and packet length 9198 bytes, +seven IPv4 packets will be sent out from port #0 to the destination address 100.10.1.1: +six of those packets will have length 1500 bytes and one packet will have length 318 bytes. +IP Fragmentation sample application provides basic NUMA support +in that all the memory structures are allocated on all sockets that have active lcores on them. + + +Refer to the *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ip_pipeline.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ip_pipeline.rst new file mode 100644 index 000000000..56014be17 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ip_pipeline.rst @@ -0,0 +1,561 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015-2018 Intel Corporation. + +Internet Protocol (IP) Pipeline Application +=========================================== + +Application overview +-------------------- + +The *Internet Protocol (IP) Pipeline* application is intended to be a vehicle for rapid development of packet processing +applications on multi-core CPUs. + +Following OpenFlow and P4 design principles, the application can be used to create functional blocks called pipelines out +of input/output ports, tables and actions in a modular way. Multiple pipelines can be inter-connected through packet queues +to create complete applications (super-pipelines). + +The pipelines are mapped to application threads, with each pipeline executed by a single thread and each thread able to run +one or several pipelines. The possibilities of creating pipelines out of ports, tables and actions, connecting multiple +pipelines together and mapping the pipelines to execution threads are endless, therefore this application can be seen as +a true application generator. + +Pipelines are created and managed through Command Line Interface (CLI): + + * Any standard TCP client (e.g. telnet, netcat, custom script, etc) is typically able to connect to the application, send + commands through the network and wait for the response before pushing the next command. + + * All the application objects are created and managed through CLI commands: + * 'Primitive' objects used to create pipeline ports: memory pools, links (i.e. network interfaces), SW queues, traffic managers, etc. + * Action profiles: used to define the actions to be executed by pipeline input/output ports and tables. + * Pipeline components: input/output ports, tables, pipelines, mapping of pipelines to execution threads. + +Running the application +----------------------- + +The application startup command line is:: + + ip_pipeline [EAL_ARGS] -- [-s SCRIPT_FILE] [-h HOST] [-p PORT] + +The application startup arguments are: + +``-s SCRIPT_FILE`` + + * Optional: Yes + + * Default: Not present + + * Argument: Path to the CLI script file to be run at application startup. + No CLI script file will run at startup if this argument is not present. + +``-h HOST`` + + * Optional: Yes + + * Default: ``0.0.0.0`` + + * Argument: IP Address of the host running ip pipeline application to be used by + remote TCP based client (telnet, netcat, etc.) for connection. + +``-p PORT`` + + * Optional: Yes + + * Default: ``8086`` + + * Argument: TCP port number at which the ip pipeline is running. + This port number should be used by remote TCP client (such as telnet, netcat, etc.) to connect to host application. + +Refer to *DPDK Getting Started Guide* for general information on running applications and the Environment Abstraction Layer (EAL) options. + +The following is an example command to run ip pipeline application configured for layer 2 forwarding: + +.. code-block:: console + + $ ./build/ip_pipeline -c 0x3 -- -s examples/route_ecmp.cli + +The application should start successfully and display as follows: + +.. code-block:: console + + EAL: Detected 40 lcore(s) + EAL: Detected 2 NUMA nodes + EAL: Multi-process socket /var/run/.rte_unix + EAL: Probing VFIO support... + EAL: PCI device 0000:02:00.0 on NUMA socket 0 + EAL: probe driver: 8086:10fb net_ixgbe + ... + +To run remote client (e.g. telnet) to communicate with the ip pipeline application: + +.. code-block:: console + + $ telnet 127.0.0.1 8086 + +When running a telnet client as above, command prompt is displayed: + +.. code-block:: console + + Trying 127.0.0.1... + Connected to 127.0.0.1. + Escape character is '^]'. + + Welcome to IP Pipeline! + + pipeline> + +Once application and telnet client start running, messages can be sent from client to application. +At any stage, telnet client can be terminated using the quit command. + + +Application stages +------------------ + +Initialization +~~~~~~~~~~~~~~ + +During this stage, EAL layer is initialised and application specific arguments are parsed. Furthermore, the data structures +(i.e. linked lists) for application objects are initialized. In case of any initialization error, an error message +is displayed and the application is terminated. + +.. _ip_pipeline_runtime: + +Run-time +~~~~~~~~ + +The master thread is creating and managing all the application objects based on CLI input. + +Each data plane thread runs one or several pipelines previously assigned to it in round-robin order. Each data plane thread +executes two tasks in time-sharing mode: + +1. *Packet processing task*: Process bursts of input packets read from the pipeline input ports. + +2. *Message handling task*: Periodically, the data plane thread pauses the packet processing task and polls for request + messages send by the master thread. Examples: add/remove pipeline to/from current data plane thread, add/delete rules + to/from given table of a specific pipeline owned by the current data plane thread, read statistics, etc. + +Examples +-------- + +.. _table_examples: + +.. tabularcolumns:: |p{3cm}|p{5cm}|p{4cm}|p{4cm}| + +.. table:: Pipeline examples provided with the application + + +-----------------------+----------------------+----------------+------------------------------------+ + | Name | Table(s) | Actions | Messages | + +=======================+======================+================+====================================+ + | L2fwd | Stub | Forward | 1. Mempool create | + | | | | 2. Link create | + | Note: Implemented | | | 3. Pipeline create | + | using pipeline with | | | 4. Pipeline port in/out | + | a simple pass-through | | | 5. Pipeline table | + | connection between | | | 6. Pipeline port in table | + | input and output | | | 7. Pipeline enable | + | ports. | | | 8. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | Flow classification | Exact match | Forward | 1. Mempool create | + | | | | 2. Link create | + | | * Key = byte array | | 3. Pipeline create | + | | (16 bytes) | | 4. Pipeline port in/out | + | | * Offset = 278 | | 5. Pipeline table | + | | * Table size = 64K | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add default | + | | | | 9. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | KNI | Stub | Forward | 1. Mempool create | + | | | | 2. Link create | + | | | | 3. Pipeline create | + | | | | 4. Pipeline port in/out | + | | | | 5. Pipeline table | + | | | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | Firewall | ACL | Allow/Drop | 1. Mempool create | + | | | | 2. Link create | + | | * Key = n-tuple | | 3. Pipeline create | + | | * Offset = 270 | | 4. Pipeline port in/out | + | | * Table size = 4K | | 5. Pipeline table | + | | | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add default | + | | | | 9. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | IP routing | LPM (IPv4) | Forward | 1. Mempool Create | + | | | | 2. Link create | + | | * Key = IP dest addr | | 3. Pipeline create | + | | * Offset = 286 | | 4. Pipeline port in/out | + | | * Table size = 4K | | 5. Pipeline table | + | | | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add default | + | | | | 9. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | Equal-cost multi-path | LPM (IPv4) | Forward, | 1. Mempool Create | + | routing (ECMP) | | load balance, | 2. Link create | + | | * Key = IP dest addr | encap ether | 3. Pipeline create | + | | * Offset = 286 | | 4. Pipeline port in/out | + | | * Table size = 4K | | 5. Pipeline table (LPM) | + | | | | 6. Pipeline table (Array) | + | | | | 7. Pipeline port in table (LPM) | + | | Array | | 8. Pipeline enable | + | | | | 9. Pipeline table rule add default | + | | * Key = Array index | | 10. Pipeline table rule add(LPM) | + | | * Offset = 256 | | 11. Pipeline table rule add(Array) | + | | * Size = 64K | | | + | | | | | + +-----------------------+----------------------+----------------+------------------------------------+ + +Command Line Interface (CLI) +---------------------------- + +Link +~~~~ + + Link configuration :: + + link <link_name> + dev <device_name>|port <port_id> + rxq <n_queues> <queue_size> <mempool_name> + txq <n_queues> <queue_size> promiscuous on | off + [rss <qid_0> ... <qid_n>] + + Note: The PCI device name must be specified in the Domain:Bus:Device.Function format. + + +Mempool +~~~~~~~ + + Mempool create :: + + mempool <mempool_name> buffer <buffer_size> + pool <pool_size> cache <cache_size> cpu <cpu_id> + + +Software queue +~~~~~~~~~~~~~~ + + Create software queue :: + + swq <swq_name> size <size> cpu <cpu_id> + + +Traffic manager +~~~~~~~~~~~~~~~ + + Add traffic manager subport profile :: + + tmgr subport profile + <tb_rate> <tb_size> + <tc0_rate> <tc1_rate> <tc2_rate> <tc3_rate> <tc4_rate> + <tc5_rate> <tc6_rate> <tc7_rate> <tc8_rate> + <tc9_rate> <tc10_rate> <tc11_rate> <tc12_rate> + <tc_period> + pps <n_pipes_per_subport> + qsize <qsize_tc0> <qsize_tc1> <qsize_tc2> + <qsize_tc3> <qsize_tc4> <qsize_tc5> <qsize_tc6> + <qsize_tc7> <qsize_tc8> <qsize_tc9> <qsize_tc10> + <qsize_tc11> <qsize_tc12> + + Add traffic manager pipe profile :: + + tmgr pipe profile + <tb_rate> <tb_size> + <tc0_rate> <tc1_rate> <tc2_rate> <tc3_rate> <tc4_rate> + <tc5_rate> <tc6_rate> <tc7_rate> <tc8_rate> + <tc9_rate> <tc10_rate> <tc11_rate> <tc12_rate> + <tc_period> + <tc_ov_weight> + <wrr_weight0..3> + + Create traffic manager port :: + + tmgr <tmgr_name> + rate <rate> + spp <n_subports_per_port> + fo <frame_overhead> + mtu <mtu> + cpu <cpu_id> + + Configure traffic manager subport :: + + tmgr <tmgr_name> + subport <subport_id> + profile <subport_profile_id> + + Configure traffic manager pipe :: + + tmgr <tmgr_name> + subport <subport_id> + pipe from <pipe_id_first> to <pipe_id_last> + profile <pipe_profile_id> + + +Tap +~~~ + + Create tap port :: + + tap <name> + + +Kni +~~~ + + Create kni port :: + + kni <kni_name> + link <link_name> + mempool <mempool_name> + [thread <thread_id>] + + +Cryptodev +~~~~~~~~~ + + Create cryptodev port :: + + cryptodev <cryptodev_name> + dev <DPDK Cryptodev PMD name> + queue <n_queues> <queue_size> + +Action profile +~~~~~~~~~~~~~~ + + Create action profile for pipeline input port :: + + port in action profile <profile_name> + [filter match | mismatch offset <key_offset> mask <key_mask> key <key_value> port <port_id>] + [balance offset <key_offset> mask <key_mask> port <port_id0> ... <port_id15>] + + Create action profile for the pipeline table :: + + table action profile <profile_name> + ipv4 | ipv6 + offset <ip_offset> + fwd + [balance offset <key_offset> mask <key_mask> outoffset <out_offset>] + [meter srtcm | trtcm + tc <n_tc> + stats none | pkts | bytes | both] + [tm spp <n_subports_per_port> pps <n_pipes_per_subport>] + [encap ether | vlan | qinq | mpls | pppoe] + [nat src | dst + proto udp | tcp] + [ttl drop | fwd + stats none | pkts] + [stats pkts | bytes | both] + [sym_crypto cryptodev <cryptodev_name> + mempool_create <mempool_name> mempool_init <mempool_name>] + [time] + + +Pipeline +~~~~~~~~ + +Create pipeline :: + + pipeline <pipeline_name> + period <timer_period_ms> + offset_port_id <offset_port_id> + cpu <cpu_id> + +Create pipeline input port :: + + pipeline <pipeline_name> port in + bsz <burst_size> + link <link_name> rxq <queue_id> + | swq <swq_name> + | tmgr <tmgr_name> + | tap <tap_name> mempool <mempool_name> mtu <mtu> + | kni <kni_name> + | source mempool <mempool_name> file <file_name> bpp <n_bytes_per_pkt> + [action <port_in_action_profile_name>] + [disabled] + +Create pipeline output port :: + + pipeline <pipeline_name> port out + bsz <burst_size> + link <link_name> txq <txq_id> + | swq <swq_name> + | tmgr <tmgr_name> + | tap <tap_name> + | kni <kni_name> + | sink [file <file_name> pkts <max_n_pkts>] + +Create pipeline table :: + + pipeline <pipeline_name> table + match + acl + ipv4 | ipv6 + offset <ip_header_offset> + size <n_rules> + | array + offset <key_offset> + size <n_keys> + | hash + ext | lru + key <key_size> + mask <key_mask> + offset <key_offset> + buckets <n_buckets> + size <n_keys> + | lpm + ipv4 | ipv6 + offset <ip_header_offset> + size <n_rules> + | stub + [action <table_action_profile_name>] + +Connect pipeline input port to table :: + + pipeline <pipeline_name> port in <port_id> table <table_id> + +Display statistics for specific pipeline input port, output port +or table :: + + pipeline <pipeline_name> port in <port_id> stats read [clear] + pipeline <pipeline_name> port out <port_id> stats read [clear] + pipeline <pipeline_name> table <table_id> stats read [clear] + +Enable given input port for specific pipeline instance :: + + pipeline <pipeline_name> port out <port_id> disable + +Disable given input port for specific pipeline instance :: + + pipeline <pipeline_name> port out <port_id> disable + +Add default rule to table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule add + match + default + action + fwd + drop + | port <port_id> + | meta + | table <table_id> + +Add rule to table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule add + + match + acl + priority <priority> + ipv4 | ipv6 <sa> <sa_depth> <da> <da_depth> + <sp0> <sp1> <dp0> <dp1> <proto> + | array <pos> + | hash + raw <key> + | ipv4_5tuple <sa> <da> <sp> <dp> <proto> + | ipv6_5tuple <sa> <da> <sp> <dp> <proto> + | ipv4_addr <addr> + | ipv6_addr <addr> + | qinq <svlan> <cvlan> + | lpm + ipv4 | ipv6 <addr> <depth> + + action + fwd + drop + | port <port_id> + | meta + | table <table_id> + [balance <out0> ... <out7>] + [meter + tc0 meter <meter_profile_id> policer g <pa> y <pa> r <pa> + [tc1 meter <meter_profile_id> policer g <pa> y <pa> r <pa> + tc2 meter <meter_profile_id> policer g <pa> y <pa> r <pa> + tc3 meter <meter_profile_id> policer g <pa> y <pa> r <pa>]] + [tm subport <subport_id> pipe <pipe_id>] + [encap + ether <da> <sa> + | vlan <da> <sa> <pcp> <dei> <vid> + | qinq <da> <sa> <pcp> <dei> <vid> <pcp> <dei> <vid> + | mpls unicast | multicast + <da> <sa> + label0 <label> <tc> <ttl> + [label1 <label> <tc> <ttl> + [label2 <label> <tc> <ttl> + [label3 <label> <tc> <ttl>]]] + | pppoe <da> <sa> <session_id>] + [nat ipv4 | ipv6 <addr> <port>] + [ttl dec | keep] + [stats] + [time] + [sym_crypto + encrypt | decrypt + type + | cipher + cipher_algo <algo> cipher_key <key> cipher_iv <iv> + | cipher_auth + cipher_algo <algo> cipher_key <key> cipher_iv <iv> + auth_algo <algo> auth_key <key> digest_size <size> + | aead + aead_algo <algo> aead_key <key> aead_iv <iv> aead_aad <aad> + digest_size <size> + data_offset <data_offset>] + + where: + <pa> ::= g | y | r | drop + +Add bulk rules to table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule add bulk <file_name> <n_rules> + + Where: + - file_name = path to file + - File line format = match <match> action <action> + +Delete table rule for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule delete + match <match> + +Delete default table rule for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule delete + match + default + +Add meter profile to the table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> meter profile <meter_profile_id> + add srtcm cir <cir> cbs <cbs> ebs <ebs> + | trtcm cir <cir> pir <pir> cbs <cbs> pbs <pbs> + +Delete meter profile from the table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> + meter profile <meter_profile_id> delete + + +Update the dscp table for meter or traffic manager action for specific +pipeline instance :: + + pipeline <pipeline_name> table <table_id> dscp <file_name> + + Where: + - file_name = path to file + - exactly 64 lines + - File line format = <tc_id> <tc_queue_id> <color>, with <color> as: g | y | r + + +Pipeline enable/disable +~~~~~~~~~~~~~~~~~~~~~~~ + + Enable given pipeline instance for specific data plane thread :: + + thread <thread_id> pipeline <pipeline_name> enable + + + Disable given pipeline instance for specific data plane thread :: + + thread <thread_id> pipeline <pipeline_name> disable diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ip_reassembly.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ip_reassembly.rst new file mode 100644 index 000000000..f34b9d005 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ip_reassembly.rst @@ -0,0 +1,238 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +IP Reassembly Sample Application +================================ + +The L3 Forwarding application is a simple example of packet processing using the DPDK. +The application performs L3 forwarding with reassembly for fragmented IPv4 and IPv6 packets. + +Overview +-------- + +The application demonstrates the use of the DPDK libraries to implement packet forwarding +with reassembly for IPv4 and IPv6 fragmented packets. +The initialization and run- time paths are very similar to those of the :doc:`l2_forward_real_virtual`. +The main difference from the L2 Forwarding sample application is that +it reassembles fragmented IPv4 and IPv6 packets before forwarding. +The maximum allowed size of reassembled packet is 9.5 KB. + +There are two key differences from the L2 Forwarding sample application: + +* The first difference is that the forwarding decision is taken based on information read from the input packet's IP header. + +* The second difference is that the application differentiates between IP and non-IP traffic by means of offload flags. + +The Longest Prefix Match (LPM for IPv4, LPM6 for IPv6) table is used to store/lookup an outgoing port number, +associated with that IPv4 address. Any unmatched packets are forwarded to the originating port. + + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ip_reassembly`` sub-directory. + + +Running the Application +----------------------- + +The application has a number of command line options: + +.. code-block:: console + + ./build/ip_reassembly [EAL options] -- -p PORTMASK [-q NQ] [--maxflows=FLOWS>] [--flowttl=TTL[(s|ms)]] + +where: + +* -p PORTMASK: Hexadecimal bitmask of ports to configure + +* -q NQ: Number of RX queues per lcore + +* --maxflows=FLOWS: determines maximum number of active fragmented flows (1-65535). Default value: 4096. + +* --flowttl=TTL[(s|ms)]: determines maximum Time To Live for fragmented packet. + If all fragments of the packet wouldn't appear within given time-out, + then they are considered as invalid and will be dropped. + Valid range is 1ms - 3600s. Default value: 1s. + +To run the example in linux environment with 2 lcores (2,4) over 2 ports(0,2) with 1 RX queue per lcore: + +.. code-block:: console + + ./build/ip_reassembly -l 2,4 -n 3 -- -p 5 + EAL: coremask set to 14 + EAL: Detected lcore 0 on socket 0 + EAL: Detected lcore 1 on socket 1 + EAL: Detected lcore 2 on socket 0 + EAL: Detected lcore 3 on socket 1 + EAL: Detected lcore 4 on socket 0 + ... + + Initializing port 0 on lcore 2... Address:00:1B:21:76:FA:2C, rxq=0 txq=2,0 txq=4,1 + done: Link Up - speed 10000 Mbps - full-duplex + Skipping disabled port 1 + Initializing port 2 on lcore 4... Address:00:1B:21:5C:FF:54, rxq=0 txq=2,0 txq=4,1 + done: Link Up - speed 10000 Mbps - full-duplex + Skipping disabled port 3IP_FRAG: Socket 0: adding route 100.10.0.0/16 (port 0) + IP_RSMBL: Socket 0: adding route 100.20.0.0/16 (port 1) + ... + + IP_RSMBL: Socket 0: adding route 0101:0101:0101:0101:0101:0101:0101:0101/48 (port 0) + IP_RSMBL: Socket 0: adding route 0201:0101:0101:0101:0101:0101:0101:0101/48 (port 1) + ... + + IP_RSMBL: entering main loop on lcore 4 + IP_RSMBL: -- lcoreid=4 portid=2 + IP_RSMBL: entering main loop on lcore 2 + IP_RSMBL: -- lcoreid=2 portid=0 + +To run the example in linux environment with 1 lcore (4) over 2 ports(0,2) with 2 RX queues per lcore: + +.. code-block:: console + + ./build/ip_reassembly -l 4 -n 3 -- -p 5 -q 2 + +To test the application, flows should be set up in the flow generator that match the values in the +l3fwd_ipv4_route_array and/or l3fwd_ipv6_route_array table. + +Please note that in order to test this application, +the traffic generator should be generating valid fragmented IP packets. +For IPv6, the only supported case is when no other extension headers other than +fragment extension header are present in the packet. + +The default l3fwd_ipv4_route_array table is: + +.. code-block:: c + + struct l3fwd_ipv4_route l3fwd_ipv4_route_array[] = { + {RTE_IPV4(100, 10, 0, 0), 16, 0}, + {RTE_IPV4(100, 20, 0, 0), 16, 1}, + {RTE_IPV4(100, 30, 0, 0), 16, 2}, + {RTE_IPV4(100, 40, 0, 0), 16, 3}, + {RTE_IPV4(100, 50, 0, 0), 16, 4}, + {RTE_IPV4(100, 60, 0, 0), 16, 5}, + {RTE_IPV4(100, 70, 0, 0), 16, 6}, + {RTE_IPV4(100, 80, 0, 0), 16, 7}, + }; + +The default l3fwd_ipv6_route_array table is: + +.. code-block:: c + + struct l3fwd_ipv6_route l3fwd_ipv6_route_array[] = { + {{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 0}, + {{2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 1}, + {{3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 2}, + {{4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 3}, + {{5, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 4}, + {{6, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 5}, + {{7, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 6}, + {{8, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 48, 7}, + }; + +For example, for the fragmented input IPv4 packet with destination address: 100.10.1.1, +a reassembled IPv4 packet be sent out from port #0 to the destination address 100.10.1.1 +once all the fragments are collected. + +Explanation +----------- + +The following sections provide some explanation of the sample application code. +As mentioned in the overview section, the initialization and run-time paths are very similar to those of the :doc:`l2_forward_real_virtual`. +The following sections describe aspects that are specific to the IP reassemble sample application. + +IPv4 Fragment Table Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This application uses the rte_ip_frag library. Please refer to Programmer's Guide for more detailed explanation of how to use this library. +Fragment table maintains information about already received fragments of the packet. +Each IP packet is uniquely identified by triple <Source IP address>, <Destination IP address>, <ID>. +To avoid lock contention, each RX queue has its own Fragment Table, +e.g. the application can't handle the situation when different fragments of the same packet arrive through different RX queues. +Each table entry can hold information about packet consisting of up to RTE_LIBRTE_IP_FRAG_MAX_FRAGS fragments. + +.. code-block:: c + + frag_cycles = (rte_get_tsc_hz() + MS_PER_S - 1) / MS_PER_S * max_flow_ttl; + + if ((qconf->frag_tbl[queue] = rte_ip_frag_tbl_create(max_flow_num, IPV4_FRAG_TBL_BUCKET_ENTRIES, max_flow_num, frag_cycles, socket)) == NULL) + { + RTE_LOG(ERR, IP_RSMBL, "ip_frag_tbl_create(%u) on " "lcore: %u for queue: %u failed\n", max_flow_num, lcore, queue); + return -1; + } + +Mempools Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +The reassembly application demands a lot of mbuf's to be allocated. +At any given time up to (2 \* max_flow_num \* RTE_LIBRTE_IP_FRAG_MAX_FRAGS \* <maximum number of mbufs per packet>) +can be stored inside Fragment Table waiting for remaining fragments. +To keep mempool size under reasonable limits and to avoid situation when one RX queue can starve other queues, +each RX queue uses its own mempool. + +.. code-block:: c + + nb_mbuf = RTE_MAX(max_flow_num, 2UL * MAX_PKT_BURST) * RTE_LIBRTE_IP_FRAG_MAX_FRAGS; + nb_mbuf *= (port_conf.rxmode.max_rx_pkt_len + BUF_SIZE - 1) / BUF_SIZE; + nb_mbuf *= 2; /* ipv4 and ipv6 */ + nb_mbuf += RTE_TEST_RX_DESC_DEFAULT + RTE_TEST_TX_DESC_DEFAULT; + nb_mbuf = RTE_MAX(nb_mbuf, (uint32_t)NB_MBUF); + + snprintf(buf, sizeof(buf), "mbuf_pool_%u_%u", lcore, queue); + + if ((rxq->pool = rte_mempool_create(buf, nb_mbuf, MBUF_SIZE, 0, sizeof(struct rte_pktmbuf_pool_private), rte_pktmbuf_pool_init, NULL, + rte_pktmbuf_init, NULL, socket, MEMPOOL_F_SP_PUT | MEMPOOL_F_SC_GET)) == NULL) { + + RTE_LOG(ERR, IP_RSMBL, "mempool_create(%s) failed", buf); + return -1; + } + +Packet Reassembly and Forwarding +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For each input packet, the packet forwarding operation is done by the l3fwd_simple_forward() function. +If the packet is an IPv4 or IPv6 fragment, then it calls rte_ipv4_reassemble_packet() for IPv4 packets, +or rte_ipv6_reassemble_packet() for IPv6 packets. +These functions either return a pointer to valid mbuf that contains reassembled packet, +or NULL (if the packet can't be reassembled for some reason). +Then l3fwd_simple_forward() continues with the code for the packet forwarding decision +(that is, the identification of the output interface for the packet) and +actual transmit of the packet. + +The rte_ipv4_reassemble_packet() or rte_ipv6_reassemble_packet() are responsible for: + +#. Searching the Fragment Table for entry with packet's <IP Source Address, IP Destination Address, Packet ID> + +#. If the entry is found, then check if that entry already timed-out. + If yes, then free all previously received fragments, + and remove information about them from the entry. + +#. If no entry with such key is found, then try to create a new one by one of two ways: + + #. Use as empty entry + + #. Delete a timed-out entry, free mbufs associated with it mbufs and store a new entry with specified key in it. + +#. Update the entry with new fragment information and check + if a packet can be reassembled (the packet's entry contains all fragments). + + #. If yes, then, reassemble the packet, mark table's entry as empty and return the reassembled mbuf to the caller. + + #. If no, then just return a NULL to the caller. + +If at any stage of packet processing a reassembly function encounters an error +(can't insert new entry into the Fragment table, or invalid/timed-out fragment), +then it will free all associated with the packet fragments, +mark the table entry as invalid and return NULL to the caller. + +Debug logging and Statistics Collection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The RTE_LIBRTE_IP_FRAG_TBL_STAT controls statistics collection for the IP Fragment Table. +This macro is disabled by default. +To make ip_reassembly print the statistics to the standard output, +the user must send either an USR1, INT or TERM signal to the process. +For all of these signals, the ip_reassembly process prints Fragment table statistics for each RX queue, +plus the INT and TERM will cause process termination as usual. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ipsec_secgw.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ipsec_secgw.rst new file mode 100644 index 000000000..81c5d4360 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ipsec_secgw.rst @@ -0,0 +1,958 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2016-2017 Intel Corporation. + Copyright (C) 2020 Marvell International Ltd. + +IPsec Security Gateway Sample Application +========================================= + +The IPsec Security Gateway application is an example of a "real world" +application using DPDK cryptodev framework. + +Overview +-------- + +The application demonstrates the implementation of a Security Gateway +(not IPsec compliant, see the Constraints section below) using DPDK based on RFC4301, +RFC4303, RFC3602 and RFC2404. + +Internet Key Exchange (IKE) is not implemented, so only manual setting of +Security Policies and Security Associations is supported. + +The Security Policies (SP) are implemented as ACL rules, the Security +Associations (SA) are stored in a table and the routing is implemented +using LPM. + +The application classifies the ports as *Protected* and *Unprotected*. +Thus, traffic received on an Unprotected or Protected port is consider +Inbound or Outbound respectively. + +The application also supports complete IPsec protocol offload to hardware +(Look aside crypto accelerator or using ethernet device). It also support +inline ipsec processing by the supported ethernet device during transmission. +These modes can be selected during the SA creation configuration. + +In case of complete protocol offload, the processing of headers(ESP and outer +IP header) is done by the hardware and the application does not need to +add/remove them during outbound/inbound processing. + +For inline offloaded outbound traffic, the application will not do the LPM +lookup for routing, as the port on which the packet has to be forwarded will be +part of the SA. Security parameters will be configured on that port only, and +sending the packet on other ports could result in unencrypted packets being +sent out. + +The Path for IPsec Inbound traffic is: + +* Read packets from the port. +* Classify packets between IPv4 and ESP. +* Perform Inbound SA lookup for ESP packets based on their SPI. +* Perform Verification/Decryption (Not needed in case of inline ipsec). +* Remove ESP and outer IP header (Not needed in case of protocol offload). +* Inbound SP check using ACL of decrypted packets and any other IPv4 packets. +* Routing. +* Write packet to port. + +The Path for the IPsec Outbound traffic is: + +* Read packets from the port. +* Perform Outbound SP check using ACL of all IPv4 traffic. +* Perform Outbound SA lookup for packets that need IPsec protection. +* Add ESP and outer IP header (Not needed in case protocol offload). +* Perform Encryption/Digest (Not needed in case of inline ipsec). +* Routing. +* Write packet to port. + +The application supports two modes of operation: poll mode and event mode. + +* In the poll mode a core receives packets from statically configured list + of eth ports and eth ports' queues. + +* In the event mode a core receives packets as events. After packet processing + is done core submits them back as events to an event device. This enables + multicore scaling and HW assisted scheduling by making use of the event device + capabilities. The event mode configuration is predefined. All packets reaching + given eth port will arrive at the same event queue. All event queues are mapped + to all event ports. This allows all cores to receive traffic from all ports. + Since the underlying event device might have varying capabilities, the worker + threads can be drafted differently to maximize performance. For example, if an + event device - eth device pair has Tx internal port, then application can call + rte_event_eth_tx_adapter_enqueue() instead of regular rte_event_enqueue_burst(). + So a thread which assumes that the device pair has internal port will not be the + right solution for another pair. The infrastructure added for the event mode aims + to help application to have multiple worker threads by maximizing performance from + every type of event device without affecting existing paths/use cases. The worker + to be used will be determined by the operating conditions and the underlying device + capabilities. **Currently the application provides non-burst, internal port worker + threads and supports inline protocol only.** It also provides infrastructure for + non-internal port however does not define any worker threads. + +Additionally the event mode introduces two submodes of processing packets: + +* Driver submode: This submode has bare minimum changes in the application to support + IPsec. There are no lookups, no routing done in the application. And for inline + protocol use case, the worker thread resembles l2fwd worker thread as the IPsec + processing is done entirely in HW. This mode can be used to benchmark the raw + performance of the HW. The driver submode is selected with --single-sa option + (used also by poll mode). When --single-sa option is used in conjution with event + mode then index passed to --single-sa is ignored. + +* App submode: This submode has all the features currently implemented with the + application (non librte_ipsec path). All the lookups, routing follows existing + methods and report numbers that can be compared against regular poll mode + benchmark numbers. + +Constraints +----------- + +* No IPv6 options headers. +* No AH mode. +* Supported algorithms: AES-CBC, AES-CTR, AES-GCM, 3DES-CBC, HMAC-SHA1 and NULL. +* Each SA must be handle by a unique lcore (*1 RX queue per port*). + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ipsec-secgw`` sub-directory. + +#. [Optional] Build the application for debugging: + This option adds some extra flags, disables compiler optimizations and + is verbose:: + + make DEBUG=1 + + +Running the Application +----------------------- + +The application has a number of command line options:: + + + ./build/ipsec-secgw [EAL options] -- + -p PORTMASK -P -u PORTMASK -j FRAMESIZE + -l -w REPLAY_WINOW_SIZE -e -a + -c SAD_CACHE_SIZE + -s NUMBER_OF_MBUFS_IN_PACKET_POOL + -f CONFIG_FILE_PATH + --config (port,queue,lcore)[,(port,queue,lcore)] + --single-sa SAIDX + --cryptodev_mask MASK + --transfer-mode MODE + --event-schedule-type TYPE + --rxoffload MASK + --txoffload MASK + --reassemble NUM + --mtu MTU + --frag-ttl FRAG_TTL_NS + +Where: + +* ``-p PORTMASK``: Hexadecimal bitmask of ports to configure. + +* ``-P``: *optional*. Sets all ports to promiscuous mode so that packets are + accepted regardless of the packet's Ethernet MAC destination address. + Without this option, only packets with the Ethernet MAC destination address + set to the Ethernet address of the port are accepted (default is enabled). + +* ``-u PORTMASK``: hexadecimal bitmask of unprotected ports + +* ``-j FRAMESIZE``: *optional*. data buffer size (in bytes), + in other words maximum data size for one segment. + Packets with length bigger then FRAMESIZE still can be received, + but will be segmented. + Default value: RTE_MBUF_DEFAULT_BUF_SIZE (2176) + Minimum value: RTE_MBUF_DEFAULT_BUF_SIZE (2176) + Maximum value: UINT16_MAX (65535). + +* ``-l``: enables code-path that uses librte_ipsec. + +* ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay window + size for each Security Association (available only with librte_ipsec + code path). + +* ``-e``: enables Security Association extended sequence number processing + (available only with librte_ipsec code path). + +* ``-a``: enables Security Association sequence number atomic behavior + (available only with librte_ipsec code path). + +* ``-c``: specifies the SAD cache size. Stores the most recent SA in a per + lcore cache. Cache represents flat array containing SA's indexed by SPI. + Zero value disables cache. + Default value: 128. + +* ``-s``: sets number of mbufs in packet pool, if not provided number of mbufs + will be calculated based on number of cores, eth ports and crypto queues. + +* ``-f CONFIG_FILE_PATH``: the full path of text-based file containing all + configuration items for running the application (See Configuration file + syntax section below). ``-f CONFIG_FILE_PATH`` **must** be specified. + **ONLY** the UNIX format configuration file is accepted. + +* ``--config (port,queue,lcore)[,(port,queue,lcore)]``: in poll mode determines + which queues from which ports are mapped to which cores. In event mode this + option is not used as packets are dynamically scheduled to cores by HW. + +* ``--single-sa SAIDX``: in poll mode use a single SA for outbound traffic, + bypassing the SP on both Inbound and Outbound. This option is meant for + debugging/performance purposes. In event mode selects driver submode, SA index + value is ignored. + +* ``--cryptodev_mask MASK``: hexadecimal bitmask of the crypto devices + to configure. + +* ``--transfer-mode MODE``: sets operating mode of the application + "poll" : packet transfer via polling (default) + "event" : Packet transfer via event device + +* ``--event-schedule-type TYPE``: queue schedule type, applies only when + --transfer-mode is set to event. + "ordered" : Ordered (default) + "atomic" : Atomic + "parallel" : Parallel + When --event-schedule-type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event + device will ensure the ordering. Ordering will be lost when tried in PARALLEL. + +* ``--rxoffload MASK``: RX HW offload capabilities to enable/use on this port + (bitmask of DEV_RX_OFFLOAD_* values). It is an optional parameter and + allows user to disable some of the RX HW offload capabilities. + By default all HW RX offloads are enabled. + +* ``--txoffload MASK``: TX HW offload capabilities to enable/use on this port + (bitmask of DEV_TX_OFFLOAD_* values). It is an optional parameter and + allows user to disable some of the TX HW offload capabilities. + By default all HW TX offloads are enabled. + +* ``--reassemble NUM``: max number of entries in reassemble fragment table. + Zero value disables reassembly functionality. + Default value: 0. + +* ``--mtu MTU``: MTU value (in bytes) on all attached ethernet ports. + Outgoing packets with length bigger then MTU will be fragmented. + Incoming packets with length bigger then MTU will be discarded. + Default value: 1500. + +* ``--frag-ttl FRAG_TTL_NS``: fragment lifetime (in nanoseconds). + If packet is not reassembled within this time, received fragments + will be discarded. Fragment lifetime should be decreased when + there is a high fragmented traffic loss in high bandwidth networks. + Should be lower for low number of reassembly buckets. + Valid values: from 1 ns to 10 s. Default value: 10000000 (10 s). + + +The mapping of lcores to port/queues is similar to other l3fwd applications. + +For example, given the following command line to run application in poll mode:: + + ./build/ipsec-secgw -l 20,21 -n 4 --socket-mem 0,2048 \ + --vdev "crypto_null" -- -p 0xf -P -u 0x3 \ + --config="(0,0,20),(1,0,20),(2,0,21),(3,0,21)" \ + -f /path/to/config_file --transfer-mode poll \ + +where each option means: + +* The ``-l`` option enables cores 20 and 21. + +* The ``-n`` option sets memory 4 channels. + +* The ``--socket-mem`` to use 2GB on socket 1. + +* The ``--vdev "crypto_null"`` option creates virtual NULL cryptodev PMD. + +* The ``-p`` option enables ports (detected) 0, 1, 2 and 3. + +* The ``-P`` option enables promiscuous mode. + +* The ``-u`` option sets ports 0 and 1 as unprotected, leaving 2 and 3 as protected. + +* The ``--config`` option enables one queue per port with the following mapping: + + +----------+-----------+-----------+---------------------------------------+ + | **Port** | **Queue** | **lcore** | **Description** | + | | | | | + +----------+-----------+-----------+---------------------------------------+ + | 0 | 0 | 20 | Map queue 0 from port 0 to lcore 20. | + | | | | | + +----------+-----------+-----------+---------------------------------------+ + | 1 | 0 | 20 | Map queue 0 from port 1 to lcore 20. | + | | | | | + +----------+-----------+-----------+---------------------------------------+ + | 2 | 0 | 21 | Map queue 0 from port 2 to lcore 21. | + | | | | | + +----------+-----------+-----------+---------------------------------------+ + | 3 | 0 | 21 | Map queue 0 from port 3 to lcore 21. | + | | | | | + +----------+-----------+-----------+---------------------------------------+ + +* The ``-f /path/to/config_file`` option enables the application read and + parse the configuration file specified, and configures the application + with a given set of SP, SA and Routing entries accordingly. The syntax of + the configuration file will be explained below in more detail. Please + **note** the parser only accepts UNIX format text file. Other formats + such as DOS/MAC format will cause a parse error. + +* The ``--transfer-mode`` option selects poll mode for processing packets. + +Similarly for example, given the following command line to run application in +event app mode:: + + ./build/ipsec-secgw -c 0x3 -- -P -p 0x3 -u 0x1 \ + -f /path/to/config_file --transfer-mode event \ + --event-schedule-type parallel \ + +where each option means: + +* The ``-c`` option selects cores 0 and 1 to run on. + +* The ``-P`` option enables promiscuous mode. + +* The ``-p`` option enables ports (detected) 0 and 1. + +* The ``-u`` option sets ports 0 as unprotected, leaving 1 as protected. + +* The ``-f /path/to/config_file`` option has the same behavior as in poll + mode example. + +* The ``--transfer-mode`` option selects event mode for processing packets. + +* The ``--event-schedule-type`` option selects parallel ordering of event queues. + + +Refer to the *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + +The application would do a best effort to "map" crypto devices to cores, with +hardware devices having priority. Basically, hardware devices if present would +be assigned to a core before software ones. +This means that if the application is using a single core and both hardware +and software crypto devices are detected, hardware devices will be used. + +A way to achieve the case where you want to force the use of virtual crypto +devices is to whitelist the Ethernet devices needed and therefore implicitly +blacklisting all hardware crypto devices. + +For example, something like the following command line: + +.. code-block:: console + + ./build/ipsec-secgw -l 20,21 -n 4 --socket-mem 0,2048 \ + -w 81:00.0 -w 81:00.1 -w 81:00.2 -w 81:00.3 \ + --vdev "crypto_aesni_mb" --vdev "crypto_null" \ + -- \ + -p 0xf -P -u 0x3 --config="(0,0,20),(1,0,20),(2,0,21),(3,0,21)" \ + -f sample.cfg + + +Configurations +-------------- + +The following sections provide the syntax of configurations to initialize +your SP, SA, Routing and Neighbour tables. +Configurations shall be specified in the configuration file to be passed to +the application. The file is then parsed by the application. The successful +parsing will result in the appropriate rules being applied to the tables +accordingly. + + +Configuration File Syntax +~~~~~~~~~~~~~~~~~~~~~~~~~ + +As mention in the overview, the Security Policies are ACL rules. +The application parsers the rules specified in the configuration file and +passes them to the ACL table, and replicates them per socket in use. + +Following are the configuration file syntax. + +General rule syntax +^^^^^^^^^^^^^^^^^^^ + +The parse treats one line in the configuration file as one configuration +item (unless the line concatenation symbol exists). Every configuration +item shall follow the syntax of either SP, SA, Routing or Neighbour +rules specified below. + +The configuration parser supports the following special symbols: + + * Comment symbol **#**. Any character from this symbol to the end of + line is treated as comment and will not be parsed. + + * Line concatenation symbol **\\**. This symbol shall be placed in the end + of the line to be concatenated to the line below. Multiple lines' + concatenation is supported. + + +SP rule syntax +^^^^^^^^^^^^^^ + +The SP rule syntax is shown as follows: + +.. code-block:: console + + sp <ip_ver> <dir> esp <action> <priority> <src_ip> <dst_ip> + <proto> <sport> <dport> + + +where each options means: + +``<ip_ver>`` + + * IP protocol version + + * Optional: No + + * Available options: + + * *ipv4*: IP protocol version 4 + * *ipv6*: IP protocol version 6 + +``<dir>`` + + * The traffic direction + + * Optional: No + + * Available options: + + * *in*: inbound traffic + * *out*: outbound traffic + +``<action>`` + + * IPsec action + + * Optional: No + + * Available options: + + * *protect <SA_idx>*: the specified traffic is protected by SA rule + with id SA_idx + * *bypass*: the specified traffic traffic is bypassed + * *discard*: the specified traffic is discarded + +``<priority>`` + + * Rule priority + + * Optional: Yes, default priority 0 will be used + + * Syntax: *pri <id>* + +``<src_ip>`` + + * The source IP address and mask + + * Optional: Yes, default address 0.0.0.0 and mask of 0 will be used + + * Syntax: + + * *src X.X.X.X/Y* for IPv4 + * *src XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX/Y* for IPv6 + +``<dst_ip>`` + + * The destination IP address and mask + + * Optional: Yes, default address 0.0.0.0 and mask of 0 will be used + + * Syntax: + + * *dst X.X.X.X/Y* for IPv4 + * *dst XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX/Y* for IPv6 + +``<proto>`` + + * The protocol start and end range + + * Optional: yes, default range of 0 to 0 will be used + + * Syntax: *proto X:Y* + +``<sport>`` + + * The source port start and end range + + * Optional: yes, default range of 0 to 0 will be used + + * Syntax: *sport X:Y* + +``<dport>`` + + * The destination port start and end range + + * Optional: yes, default range of 0 to 0 will be used + + * Syntax: *dport X:Y* + +Example SP rules: + +.. code-block:: console + + sp ipv4 out esp protect 105 pri 1 dst 192.168.115.0/24 sport 0:65535 \ + dport 0:65535 + + sp ipv6 in esp bypass pri 1 dst 0000:0000:0000:0000:5555:5555:\ + 0000:0000/96 sport 0:65535 dport 0:65535 + + +SA rule syntax +^^^^^^^^^^^^^^ + +The successfully parsed SA rules will be stored in an array table. + +The SA rule syntax is shown as follows: + +.. code-block:: console + + sa <dir> <spi> <cipher_algo> <cipher_key> <auth_algo> <auth_key> + <mode> <src_ip> <dst_ip> <action_type> <port_id> <fallback> + <flow-direction> <port_id> <queue_id> + +where each options means: + +``<dir>`` + + * The traffic direction + + * Optional: No + + * Available options: + + * *in*: inbound traffic + * *out*: outbound traffic + +``<spi>`` + + * The SPI number + + * Optional: No + + * Syntax: unsigned integer number + +``<cipher_algo>`` + + * Cipher algorithm + + * Optional: Yes, unless <aead_algo> is not used + + * Available options: + + * *null*: NULL algorithm + * *aes-128-cbc*: AES-CBC 128-bit algorithm + * *aes-192-cbc*: AES-CBC 192-bit algorithm + * *aes-256-cbc*: AES-CBC 256-bit algorithm + * *aes-128-ctr*: AES-CTR 128-bit algorithm + * *3des-cbc*: 3DES-CBC 192-bit algorithm + + * Syntax: *cipher_algo <your algorithm>* + +``<cipher_key>`` + + * Cipher key, NOT available when 'null' algorithm is used + + * Optional: Yes, unless <aead_algo> is not used. + Must be followed by <cipher_algo> option + + * Syntax: Hexadecimal bytes (0x0-0xFF) concatenate by colon symbol ':'. + The number of bytes should be as same as the specified cipher algorithm + key size. + + For example: *cipher_key A1:B2:C3:D4:A1:B2:C3:D4:A1:B2:C3:D4: + A1:B2:C3:D4* + +``<auth_algo>`` + + * Authentication algorithm + + * Optional: Yes, unless <aead_algo> is not used + + * Available options: + + * *null*: NULL algorithm + * *sha1-hmac*: HMAC SHA1 algorithm + +``<auth_key>`` + + * Authentication key, NOT available when 'null' or 'aes-128-gcm' algorithm + is used. + + * Optional: Yes, unless <aead_algo> is not used. + Must be followed by <auth_algo> option + + * Syntax: Hexadecimal bytes (0x0-0xFF) concatenate by colon symbol ':'. + The number of bytes should be as same as the specified authentication + algorithm key size. + + For example: *auth_key A1:B2:C3:D4:A1:B2:C3:D4:A1:B2:C3:D4:A1:B2:C3:D4: + A1:B2:C3:D4* + +``<aead_algo>`` + + * AEAD algorithm + + * Optional: Yes, unless <cipher_algo> and <auth_algo> are not used + + * Available options: + + * *aes-128-gcm*: AES-GCM 128-bit algorithm + * *aes-192-gcm*: AES-GCM 192-bit algorithm + * *aes-256-gcm*: AES-GCM 256-bit algorithm + + * Syntax: *cipher_algo <your algorithm>* + +``<aead_key>`` + + * Cipher key, NOT available when 'null' algorithm is used + + * Optional: Yes, unless <cipher_algo> and <auth_algo> are not used. + Must be followed by <aead_algo> option + + * Syntax: Hexadecimal bytes (0x0-0xFF) concatenate by colon symbol ':'. + Last 4 bytes of the provided key will be used as 'salt' and so, the + number of bytes should be same as the sum of specified AEAD algorithm + key size and salt size (4 bytes). + + For example: *aead_key A1:B2:C3:D4:A1:B2:C3:D4:A1:B2:C3:D4: + A1:B2:C3:D4:A1:B2:C3:D4* + +``<mode>`` + + * The operation mode + + * Optional: No + + * Available options: + + * *ipv4-tunnel*: Tunnel mode for IPv4 packets + * *ipv6-tunnel*: Tunnel mode for IPv6 packets + * *transport*: transport mode + + * Syntax: mode XXX + +``<src_ip>`` + + * The source IP address. This option is not available when + transport mode is used + + * Optional: Yes, default address 0.0.0.0 will be used + + * Syntax: + + * *src X.X.X.X* for IPv4 + * *src XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX* for IPv6 + +``<dst_ip>`` + + * The destination IP address. This option is not available when + transport mode is used + + * Optional: Yes, default address 0.0.0.0 will be used + + * Syntax: + + * *dst X.X.X.X* for IPv4 + * *dst XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX* for IPv6 + +``<type>`` + + * Action type to specify the security action. This option specify + the SA to be performed with look aside protocol offload to HW + accelerator or protocol offload on ethernet device or inline + crypto processing on the ethernet device during transmission. + + * Optional: Yes, default type *no-offload* + + * Available options: + + * *lookaside-protocol-offload*: look aside protocol offload to HW accelerator + * *inline-protocol-offload*: inline protocol offload on ethernet device + * *inline-crypto-offload*: inline crypto processing on ethernet device + * *no-offload*: no offloading to hardware + + ``<port_id>`` + + * Port/device ID of the ethernet/crypto accelerator for which the SA is + configured. For *inline-crypto-offload* and *inline-protocol-offload*, this + port will be used for routing. The routing table will not be referred in + this case. + + * Optional: No, if *type* is not *no-offload* + + * Syntax: + + * *port_id X* X is a valid device number in decimal + + ``<fallback>`` + + * Action type for ingress IPsec packets that inline processor failed to + process. Only a combination of *inline-crypto-offload* as a primary + session and *lookaside-none* as a fall-back session is supported at the + moment. + + If used in conjunction with IPsec window, its width needs be increased + due to different processing times of inline and lookaside modes which + results in packet reordering. + + * Optional: Yes. + + * Available options: + + * *lookaside-none*: use automatically chosen cryptodev to process packets + + * Syntax: + + * *fallback lookaside-none* + +``<flow-direction>`` + + * Option for redirecting a specific inbound ipsec flow of a port to a specific + queue of that port. + + * Optional: Yes. + + * Available options: + + * *port_id*: Port ID of the NIC for which the SA is configured. + * *queue_id*: Queue ID to which traffic should be redirected. + +Example SA rules: + +.. code-block:: console + + sa out 5 cipher_algo null auth_algo null mode ipv4-tunnel \ + src 172.16.1.5 dst 172.16.2.5 + + sa out 25 cipher_algo aes-128-cbc \ + cipher_key c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3 \ + auth_algo sha1-hmac \ + auth_key c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3:c3 \ + mode ipv6-tunnel \ + src 1111:1111:1111:1111:1111:1111:1111:5555 \ + dst 2222:2222:2222:2222:2222:2222:2222:5555 + + sa in 105 aead_algo aes-128-gcm \ + aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \ + mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 + + sa out 5 cipher_algo aes-128-cbc cipher_key 0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0 \ + auth_algo sha1-hmac auth_key 0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0 \ + mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \ + type lookaside-protocol-offload port_id 4 + + sa in 35 aead_algo aes-128-gcm \ + aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \ + mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \ + type inline-crypto-offload port_id 0 + + sa in 117 cipher_algo null auth_algo null mode ipv4-tunnel src 172.16.2.7 \ + dst 172.16.1.7 flow-direction 0 2 + +Routing rule syntax +^^^^^^^^^^^^^^^^^^^ + +The Routing rule syntax is shown as follows: + +.. code-block:: console + + rt <ip_ver> <src_ip> <dst_ip> <port> + + +where each options means: + +``<ip_ver>`` + + * IP protocol version + + * Optional: No + + * Available options: + + * *ipv4*: IP protocol version 4 + * *ipv6*: IP protocol version 6 + +``<src_ip>`` + + * The source IP address and mask + + * Optional: Yes, default address 0.0.0.0 and mask of 0 will be used + + * Syntax: + + * *src X.X.X.X/Y* for IPv4 + * *src XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX/Y* for IPv6 + +``<dst_ip>`` + + * The destination IP address and mask + + * Optional: Yes, default address 0.0.0.0 and mask of 0 will be used + + * Syntax: + + * *dst X.X.X.X/Y* for IPv4 + * *dst XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX/Y* for IPv6 + +``<port>`` + + * The traffic output port id + + * Optional: yes, default output port 0 will be used + + * Syntax: *port X* + +Example SP rules: + +.. code-block:: console + + rt ipv4 dst 172.16.1.5/32 port 0 + + rt ipv6 dst 1111:1111:1111:1111:1111:1111:1111:5555/116 port 0 + +Neighbour rule syntax +^^^^^^^^^^^^^^^^^^^^^ + +The Neighbour rule syntax is shown as follows: + +.. code-block:: console + + neigh <port> <dst_mac> + + +where each options means: + +``<port>`` + + * The output port id + + * Optional: No + + * Syntax: *port X* + +``<dst_mac>`` + + * The destination ethernet address to use for that port + + * Optional: No + + * Syntax: + + * XX:XX:XX:XX:XX:XX + +Example Neighbour rules: + +.. code-block:: console + + neigh port 0 DE:AD:BE:EF:01:02 + +Test directory +-------------- + +The test directory contains scripts for testing the various encryption +algorithms. + +The purpose of the scripts is to automate ipsec-secgw testing +using another system running linux as a DUT. + +The user must setup the following environment variables: + +* ``SGW_PATH``: path to the ipsec-secgw binary to test. + +* ``REMOTE_HOST``: IP address/hostname of the DUT. + +* ``REMOTE_IFACE``: interface name for the test-port on the DUT. + +* ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>') + +Also the user can optionally setup: + +* ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0) + +* ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified + appropriate vdevs will be created by the script + +Scripts can be used for multiple test scenarios. To check all available +options run: + +.. code-block:: console + + /bin/bash run_test.sh -h + +Note that most of the tests require the appropriate crypto PMD/device to be +available. + +Server configuration +~~~~~~~~~~~~~~~~~~~~ + +Two servers are required for the tests, SUT and DUT. + +Make sure the user from the SUT can ssh to the DUT without entering the password. +To enable this feature keys must be setup on the DUT. + +``ssh-keygen`` will make a private & public key pair on the SUT. + +``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the public +key to the DUT. It will ask for credentials so that it can upload the public key. + +The SUT and DUT are connected through at least 2 NIC ports. + +One NIC port is expected to be managed by linux on both machines and will be +used as a control path. + +The second NIC port (test-port) should be bound to DPDK on the SUT, and should +be managed by linux on the DUT. + +The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and +``tap vdev``. + +It then configures the local tap interface and the remote interface and IPsec +policies in the following way: + +Traffic going over the test-port in both directions has to be protected by IPsec. + +Traffic going over the TAP port in both directions does not have to be protected. + +i.e: + +DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS + +SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS + +It then tries to perform some data transfer using the scheme described above. + +Usage +~~~~~ + +In the ipsec-secgw/test directory run + +/bin/bash run_test.sh <options> <ipsec_mode> + +Available options: + +* ``-4`` Perform tests with use of IPv4. One or both [-46] options needs to be + selected. + +* ``-6`` Perform tests with use of IPv6. One or both [-46] options needs to be + selected. + +* ``-m`` Add IPSec tunnel mixed IP version tests - outer IP version different + than inner. Inner IP version will match selected option [-46]. + +* ``-i`` Run tests in inline mode. Regular tests will not be invoked. + +* ``-f`` Run tests for fallback mechanism. Regular tests will not be invoked. + +* ``-l`` Run tests in legacy mode only. It cannot be used with options [-fsc]. + On default library mode is used. + +* ``-s`` Run all tests with reassembly support. On default only tests for + fallback mechanism use reassembly support. + +* ``-c`` Run tests with use of cpu-crypto. For inline tests it will not be + applied. On default lookaside-none is used. + +* ``-p`` Perform packet validation tests. Option [-46] is not required. + +* ``-h`` Show usage. + +If <ipsec_mode> is specified, only tests for that mode will be invoked. For the +list of available modes please refer to run_test.sh.
\ No newline at end of file diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ipv4_multicast.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ipv4_multicast.rst new file mode 100644 index 000000000..8923a7f54 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ipv4_multicast.rst @@ -0,0 +1,325 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +IPv4 Multicast Sample Application +================================= + +The IPv4 Multicast application is a simple example of packet processing +using the Data Plane Development Kit (DPDK). +The application performs L3 multicasting. + +Overview +-------- + +The application demonstrates the use of zero-copy buffers for packet forwarding. +The initialization and run-time paths are very similar to those of the :doc:`l2_forward_real_virtual`. +This guide highlights the differences between the two applications. +There are two key differences from the L2 Forwarding sample application: + +* The IPv4 Multicast sample application makes use of indirect buffers. + +* The forwarding decision is taken based on information read from the input packet's IPv4 header. + +The lookup method is the Four-byte Key (FBK) hash-based method. +The lookup table is composed of pairs of destination IPv4 address (the FBK) +and a port mask associated with that IPv4 address. + +.. note:: + + The max port mask supported in the given hash table is 0xf, so only first + four ports can be supported. + If using non-consecutive ports, use the destination IPv4 address accordingly. + +For convenience and simplicity, this sample application does not take IANA-assigned multicast addresses into account, +but instead equates the last four bytes of the multicast group (that is, the last four bytes of the destination IP address) +with the mask of ports to multicast packets to. +Also, the application does not consider the Ethernet addresses; +it looks only at the IPv4 destination address for any given packet. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ipv4_multicast`` sub-directory. + +Running the Application +----------------------- + +The application has a number of command line options: + +.. code-block:: console + + ./build/ipv4_multicast [EAL options] -- -p PORTMASK [-q NQ] + +where, + +* -p PORTMASK: Hexadecimal bitmask of ports to configure + +* -q NQ: determines the number of queues per lcore + +.. note:: + + Unlike the basic L2/L3 Forwarding sample applications, + NUMA support is not provided in the IPv4 Multicast sample application. + +Typically, to run the IPv4 Multicast sample application, issue the following command (as root): + +.. code-block:: console + + ./build/ipv4_multicast -l 0-3 -n 3 -- -p 0x3 -q 1 + +In this command: + +* The -l option enables cores 0, 1, 2 and 3 + +* The -n option specifies 3 memory channels + +* The -p option enables ports 0 and 1 + +* The -q option assigns 1 queue to each lcore + +Refer to the *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. +As mentioned in the overview section, +the initialization and run-time paths are very similar to those of the :doc:`l2_forward_real_virtual`. +The following sections describe aspects that are specific to the IPv4 Multicast sample application. + +Memory Pool Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The IPv4 Multicast sample application uses three memory pools. +Two of the pools are for indirect buffers used for packet duplication purposes. +Memory pools for indirect buffers are initialized differently from the memory pool for direct buffers: + +.. code-block:: c + + packet_pool = rte_pktmbuf_pool_create("packet_pool", NB_PKT_MBUF, 32, + 0, PKT_MBUF_DATA_SIZE, rte_socket_id()); + header_pool = rte_pktmbuf_pool_create("header_pool", NB_HDR_MBUF, 32, + 0, HDR_MBUF_DATA_SIZE, rte_socket_id()); + clone_pool = rte_pktmbuf_pool_create("clone_pool", NB_CLONE_MBUF, 32, + 0, 0, rte_socket_id()); + +The reason for this is because indirect buffers are not supposed to hold any packet data and +therefore can be initialized with lower amount of reserved memory for each buffer. + +Hash Initialization +~~~~~~~~~~~~~~~~~~~ + +The hash object is created and loaded with the pre-configured entries read from a global array: + +.. code-block:: c + + static int + + init_mcast_hash(void) + { + uint32_t i; + mcast_hash_params.socket_id = rte_socket_id(); + + mcast_hash = rte_fbk_hash_create(&mcast_hash_params); + if (mcast_hash == NULL){ + return -1; + } + + for (i = 0; i < N_MCAST_GROUPS; i ++){ + if (rte_fbk_hash_add_key(mcast_hash, mcast_group_table[i].ip, mcast_group_table[i].port_mask) < 0) { + return -1; + } + } + return 0; + } + +Forwarding +~~~~~~~~~~ + +All forwarding is done inside the mcast_forward() function. +Firstly, the Ethernet* header is removed from the packet and the IPv4 address is extracted from the IPv4 header: + +.. code-block:: c + + /* Remove the Ethernet header from the input packet */ + + iphdr = (struct rte_ipv4_hdr *)rte_pktmbuf_adj(m, sizeof(struct rte_ether_hdr)); + RTE_ASSERT(iphdr != NULL); + dest_addr = rte_be_to_cpu_32(iphdr->dst_addr); + +Then, the packet is checked to see if it has a multicast destination address and +if the routing table has any ports assigned to the destination address: + +.. code-block:: c + + if (!RTE_IS_IPV4_MCAST(dest_addr) || + (hash = rte_fbk_hash_lookup(mcast_hash, dest_addr)) <= 0 || + (port_mask = hash & enabled_port_mask) == 0) { + rte_pktmbuf_free(m); + return; + } + +Then, the number of ports in the destination portmask is calculated with the help of the bitcnt() function: + +.. code-block:: c + + /* Get number of bits set. */ + + static inline uint32_t bitcnt(uint32_t v) + { + uint32_t n; + + for (n = 0; v != 0; v &= v - 1, n++) + ; + return n; + } + +This is done to determine which forwarding algorithm to use. +This is explained in more detail in the next section. + +Thereafter, a destination Ethernet address is constructed: + +.. code-block:: c + + /* construct destination Ethernet address */ + + dst_eth_addr = ETHER_ADDR_FOR_IPV4_MCAST(dest_addr); + +Since Ethernet addresses are also part of the multicast process, each outgoing packet carries the same destination Ethernet address. +The destination Ethernet address is constructed from the lower 23 bits of the multicast group OR-ed +with the Ethernet address 01:00:5e:00:00:00, as per RFC 1112: + +.. code-block:: c + + #define ETHER_ADDR_FOR_IPV4_MCAST(x) \ + (rte_cpu_to_be_64(0x01005e000000ULL | ((x) & 0x7fffff)) >> 16) + +Then, packets are dispatched to the destination ports according to the portmask associated with a multicast group: + +.. code-block:: c + + for (port = 0; use_clone != port_mask; port_mask >>= 1, port++) { + /* Prepare output packet and send it out. */ + + if ((port_mask & 1) != 0) { + if (likely ((mc = mcast_out_pkt(m, use_clone)) != NULL)) + mcast_send_pkt(mc, &dst_eth_addr.as_addr, qconf, port); + else if (use_clone == 0) + rte_pktmbuf_free(m); + } + } + +The actual packet transmission is done in the mcast_send_pkt() function: + +.. code-block:: c + + static inline void mcast_send_pkt(struct rte_mbuf *pkt, struct rte_ether_addr *dest_addr, struct lcore_queue_conf *qconf, uint16_t port) + { + struct rte_ether_hdr *ethdr; + uint16_t len; + + /* Construct Ethernet header. */ + + ethdr = (struct rte_ether_hdr *)rte_pktmbuf_prepend(pkt, (uint16_t) sizeof(*ethdr)); + + RTE_ASSERT(ethdr != NULL); + + rte_ether_addr_copy(dest_addr, ðdr->d_addr); + rte_ether_addr_copy(&ports_eth_addr[port], ðdr->s_addr); + ethdr->ether_type = rte_be_to_cpu_16(RTE_ETHER_TYPE_IPV4); + + /* Put new packet into the output queue */ + + len = qconf->tx_mbufs[port].len; + qconf->tx_mbufs[port].m_table[len] = pkt; + qconf->tx_mbufs[port].len = ++len; + + /* Transmit packets */ + + if (unlikely(MAX_PKT_BURST == len)) + send_burst(qconf, port); + } + +Buffer Cloning +~~~~~~~~~~~~~~ + +This is the most important part of the application since it demonstrates the use of zero- copy buffer cloning. +There are two approaches for creating the outgoing packet and although both are based on the data zero-copy idea, +there are some differences in the detail. + +The first approach creates a clone of the input packet, for example, +walk though all segments of the input packet and for each of segment, +create a new buffer and attach that new buffer to the segment +(refer to rte_pktmbuf_clone() in the rte_mbuf library for more details). +A new buffer is then allocated for the packet header and is prepended to the cloned buffer. + +The second approach does not make a clone, it just increments the reference counter for all input packet segment, +allocates a new buffer for the packet header and prepends it to the input packet. + +Basically, the first approach reuses only the input packet's data, but creates its own copy of packet's metadata. +The second approach reuses both input packet's data and metadata. + +The advantage of first approach is that each outgoing packet has its own copy of the metadata, +so we can safely modify the data pointer of the input packet. +That allows us to skip creation if the output packet is for the last destination port +and instead modify input packet's header in place. +For example, for N destination ports, we need to invoke mcast_out_pkt() (N-1) times. + +The advantage of the second approach is that there is less work to be done for each outgoing packet, +that is, the "clone" operation is skipped completely. +However, there is a price to pay. +The input packet's metadata must remain intact, so for N destination ports, +we need to invoke mcast_out_pkt() (N) times. + +Therefore, for a small number of outgoing ports (and segments in the input packet), +first approach is faster. +As the number of outgoing ports (and/or input segments) grows, the second approach becomes more preferable. + +Depending on the number of segments or the number of ports in the outgoing portmask, +either the first (with cloning) or the second (without cloning) approach is taken: + +.. code-block:: c + + use_clone = (port_num <= MCAST_CLONE_PORTS && m->pkt.nb_segs <= MCAST_CLONE_SEGS); + +It is the mcast_out_pkt() function that performs the packet duplication (either with or without actually cloning the buffers): + +.. code-block:: c + + static inline struct rte_mbuf *mcast_out_pkt(struct rte_mbuf *pkt, int use_clone) + { + struct rte_mbuf *hdr; + + /* Create new mbuf for the header. */ + + if (unlikely ((hdr = rte_pktmbuf_alloc(header_pool)) == NULL)) + return NULL; + + /* If requested, then make a new clone packet. */ + + if (use_clone != 0 && unlikely ((pkt = rte_pktmbuf_clone(pkt, clone_pool)) == NULL)) { + rte_pktmbuf_free(hdr); + return NULL; + } + + /* prepend new header */ + + hdr->pkt.next = pkt; + + /* update header's fields */ + + hdr->pkt.pkt_len = (uint16_t)(hdr->pkt.data_len + pkt->pkt.pkt_len); + hdr->pkt.nb_segs = pkt->pkt.nb_segs + 1; + + /* copy metadata from source packet */ + + hdr->pkt.in_port = pkt->pkt.in_port; + hdr->pkt.vlan_macip = pkt->pkt.vlan_macip; + hdr->pkt.hash = pkt->pkt.hash; + rte_mbuf_sanity_check(hdr, RTE_MBUF_PKT, 1); + + return hdr; + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/keep_alive.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/keep_alive.rst new file mode 100644 index 000000000..865ba69e5 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/keep_alive.rst @@ -0,0 +1,144 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015-2016 Intel Corporation. + +Keep Alive Sample Application +============================= + +The Keep Alive application is a simple example of a +heartbeat/watchdog for packet processing cores. It demonstrates how +to detect 'failed' DPDK cores and notify a fault management entity +of this failure. Its purpose is to ensure the failure of the core +does not result in a fault that is not detectable by a management +entity. + + +Overview +-------- + +The application demonstrates how to protect against 'silent outages' +on packet processing cores. A Keep Alive Monitor Agent Core (master) +monitors the state of packet processing cores (worker cores) by +dispatching pings at a regular time interval (default is 5ms) and +monitoring the state of the cores. Cores states are: Alive, MIA, Dead +or Buried. MIA indicates a missed ping, and Dead indicates two missed +pings within the specified time interval. When a core is Dead, a +callback function is invoked to restart the packet processing core; +A real life application might use this callback function to notify a +higher level fault management entity of the core failure in order to +take the appropriate corrective action. + +Note: Only the worker cores are monitored. A local (on the host) mechanism +or agent to supervise the Keep Alive Monitor Agent Core DPDK core is required +to detect its failure. + +Note: This application is based on the :doc:`l2_forward_real_virtual`. As +such, the initialization and run-time paths are very similar to those +of the L2 forwarding application. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l2fwd_keep_alive`` sub-directory. + +Running the Application +----------------------- + +The application has a number of command line options: + +.. code-block:: console + + ./build/l2fwd-keepalive [EAL options] \ + -- -p PORTMASK [-q NQ] [-K PERIOD] [-T PERIOD] + +where, + +* ``p PORTMASK``: A hexadecimal bitmask of the ports to configure + +* ``q NQ``: A number of queues (=ports) per lcore (default is 1) + +* ``K PERIOD``: Heartbeat check period in ms(5ms default; 86400 max) + +* ``T PERIOD``: statistics will be refreshed each PERIOD seconds (0 to + disable, 10 default, 86400 maximum). + +To run the application in linux environment with 4 lcores, 16 ports +8 RX queues per lcore and a ping interval of 10ms, issue the command: + +.. code-block:: console + + ./build/l2fwd-keepalive -l 0-3 -n 4 -- -q 8 -p ffff -K 10 + +Refer to the *DPDK Getting Started Guide* for general information on +running applications and the Environment Abstraction Layer (EAL) +options. + + +Explanation +----------- + +The following sections provide some explanation of the The +Keep-Alive/'Liveliness' conceptual scheme. As mentioned in the +overview section, the initialization and run-time paths are very +similar to those of the :doc:`l2_forward_real_virtual`. + +The Keep-Alive/'Liveliness' conceptual scheme: + +* A Keep- Alive Agent Runs every N Milliseconds. + +* DPDK Cores respond to the keep-alive agent. + +* If keep-alive agent detects time-outs, it notifies the + fault management entity through a callback function. + +The following sections provide some explanation of the code aspects +that are specific to the Keep Alive sample application. + +The keepalive functionality is initialized with a struct +rte_keepalive and the callback function to invoke in the +case of a timeout. + +.. code-block:: c + + rte_global_keepalive_info = rte_keepalive_create(&dead_core, NULL); + if (rte_global_keepalive_info == NULL) + rte_exit(EXIT_FAILURE, "keepalive_create() failed"); + +The function that issues the pings keepalive_dispatch_pings() +is configured to run every check_period milliseconds. + +.. code-block:: c + + if (rte_timer_reset(&hb_timer, + (check_period * rte_get_timer_hz()) / 1000, + PERIODICAL, + rte_lcore_id(), + &rte_keepalive_dispatch_pings, + rte_global_keepalive_info + ) != 0 ) + rte_exit(EXIT_FAILURE, "Keepalive setup failure.\n"); + +The rest of the initialization and run-time path follows +the same paths as the L2 forwarding application. The only +addition to the main processing loop is the mark alive +functionality and the example random failures. + +.. code-block:: c + + rte_keepalive_mark_alive(&rte_global_keepalive_info); + cur_tsc = rte_rdtsc(); + + /* Die randomly within 7 secs for demo purposes.. */ + if (cur_tsc - tsc_initial > tsc_lifetime) + break; + +The rte_keepalive_mark_alive function simply sets the core state to alive. + +.. code-block:: c + + static inline void + rte_keepalive_mark_alive(struct rte_keepalive *keepcfg) + { + keepcfg->live_data[rte_lcore_id()].core_state = RTE_KA_STATE_ALIVE; + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/kernel_nic_interface.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/kernel_nic_interface.rst new file mode 100644 index 000000000..aac4ebd8d --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/kernel_nic_interface.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Kernel NIC Interface Sample Application +======================================= + +The Kernel NIC Interface (KNI) is a DPDK control plane solution that +allows userspace applications to exchange packets with the kernel networking stack. +To accomplish this, DPDK userspace applications use an IOCTL call +to request the creation of a KNI virtual device in the Linux* kernel. +The IOCTL call provides interface information and the DPDK's physical address space, +which is re-mapped into the kernel address space by the KNI kernel loadable module +that saves the information to a virtual device context. +The DPDK creates FIFO queues for packet ingress and egress +to the kernel module for each device allocated. + +The KNI kernel loadable module is a standard net driver, +which upon receiving the IOCTL call access the DPDK's FIFO queue to +receive/transmit packets from/to the DPDK userspace application. +The FIFO queues contain pointers to data packets in the DPDK. This: + +* Provides a faster mechanism to interface with the kernel net stack and eliminates system calls + +* Facilitates the DPDK using standard Linux* userspace net tools (tshark, rsync, and so on) + +* Eliminate the copy_to_user and copy_from_user operations on packets. + +The Kernel NIC Interface sample application is a simple example that demonstrates the use +of the DPDK to create a path for packets to go through the Linux* kernel. +This is done by creating one or more kernel net devices for each of the DPDK ports. +The application allows the use of standard Linux tools (ethtool, iproute, tshark) with the DPDK ports and +also the exchange of packets between the DPDK application and the Linux* kernel. + +The Kernel NIC Interface sample application requires that the +KNI kernel module ``rte_kni`` be loaded into the kernel. See +:doc:`../prog_guide/kernel_nic_interface` for more information on loading +the ``rte_kni`` kernel module. + +Overview +-------- + +The Kernel NIC Interface sample application ``kni`` allocates one or more +KNI interfaces for each physical NIC port. For each physical NIC port, +``kni`` uses two DPDK threads in user space; one thread reads from the port and +writes to the corresponding KNI interfaces and the other thread reads from +the KNI interfaces and writes the data unmodified to the physical NIC port. + +It is recommended to configure one KNI interface for each physical NIC port. +The application can be configured with more than one KNI interface for +each physical NIC port for performance testing or it can work together with +VMDq support in future. + +The packet flow through the Kernel NIC Interface application is as shown +in the following figure. + +.. _figure_kernel_nic: + +.. figure:: img/kernel_nic.* + + Kernel NIC Application Packet Flow + +If link monitoring is enabled with the ``-m`` command line flag, one +additional pthread is launched which will check the link status of each +physical NIC port and will update the carrier status of the corresponding +KNI interface(s) to match the physical NIC port's state. This means that +the KNI interface(s) will be disabled automatically when the Ethernet link +goes down and enabled when the Ethernet link goes up. + +If link monitoring is enabled, the ``rte_kni`` kernel module should be loaded +such that the :ref:`default carrier state <kni_default_carrier_state>` is +set to *off*. This ensures that the KNI interface is only enabled *after* +the Ethernet link of the corresponding NIC port has reached the linkup state. + +If link monitoring is not enabled, the ``rte_kni`` kernel module should be +loaded with the :ref:`default carrier state <kni_default_carrier_state>` +set to *on*. This sets the carrier state of the KNI interfaces to *on* +when the KNI interfaces are enabled without regard to the actual link state +of the corresponding NIC port. This is useful for testing in loopback +mode where the NIC port may not be physically connected to anything. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``examples/kni`` sub-directory. + +.. note:: + + This application is intended as a linux only. + +Running the kni Example Application +----------------------------------- + +The ``kni`` example application requires a number of command line options: + +.. code-block:: console + + kni [EAL options] -- -p PORTMASK --config="(port,lcore_rx,lcore_tx[,lcore_kthread,...])[,(port,lcore_rx,lcore_tx[,lcore_kthread,...])]" [-P] [-m] + +Where: + +* ``-p PORTMASK``: + + Hexadecimal bitmask of ports to configure. + +* ``--config="(port,lcore_rx,lcore_tx[,lcore_kthread,...])[,(port,lcore_rx,lcore_tx[,lcore_kthread,...])]"``: + + Determines which lcores the Rx and Tx DPDK tasks, and (optionally) + the KNI kernel thread(s) are bound to for each physical port. + +* ``-P``: + + Optional flag to set all ports to promiscuous mode so that packets are + accepted regardless of the packet's Ethernet MAC destination address. + Without this option, only packets with the Ethernet MAC destination + address set to the Ethernet address of the port are accepted. + +* ``-m``: + + Optional flag to enable monitoring and updating of the Ethernet + carrier state. With this option set, a thread will be started which + will periodically check the Ethernet link status of the physical + Ethernet ports and set the carrier state of the corresponding KNI + network interface to match it. This means that the KNI interface will + be disabled automatically when the Ethernet link goes down and enabled + when the Ethernet link goes up. + +Refer to *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + +The ``-c coremask`` or ``-l corelist`` parameter of the EAL options must +include the lcores specified by ``lcore_rx`` and ``lcore_tx`` for each port, +but does not need to include lcores specified by ``lcore_kthread`` as those +cores are used to pin the kernel threads in the ``rte_kni`` kernel module. + +The ``--config`` parameter must include a set of +``(port,lcore_rx,lcore_tx,[lcore_kthread,...])`` values for each physical +port specified in the ``-p PORTMASK`` parameter. + +The optional ``lcore_kthread`` lcore ID parameter in ``--config`` can be +specified zero, one or more times for each physical port. + +If no lcore ID is specified for ``lcore_kthread``, one KNI interface will +be created for the physical port ``port`` and the KNI kernel thread(s) +will have no specific core affinity. + +If one or more lcore IDs are specified for ``lcore_kthread``, a KNI interface +will be created for each lcore ID specified, bound to the physical port +``port``. If the ``rte_kni`` kernel module is loaded in :ref:`multiple +kernel thread <kni_kernel_thread_mode>` mode, a kernel thread will be created +for each KNI interface and bound to the specified core. If the ``rte_kni`` +kernel module is loaded in :ref:`single kernel thread <kni_kernel_thread_mode>` +mode, only one kernel thread is started for all KNI interfaces. The kernel +thread will be bound to the first ``lcore_kthread`` lcore ID specified. + +Example Configurations +~~~~~~~~~~~~~~~~~~~~~~~ + +The following commands will first load the ``rte_kni`` kernel module in +:ref:`multiple kernel thread <kni_kernel_thread_mode>` mode. The ``kni`` +application is then started using two ports; Port 0 uses lcore 4 for the +Rx task, lcore 6 for the Tx task, and will create a single KNI interface +``vEth0_0`` with the kernel thread bound to lcore 8. Port 1 uses lcore +5 for the Rx task, lcore 7 for the Tx task, and will create a single KNI +interface ``vEth1_0`` with the kernel thread bound to lcore 9. + +.. code-block:: console + + # rmmod rte_kni + # insmod kmod/rte_kni.ko kthread_mode=multiple + # ./build/kni -l 4-7 -n 4 -- -P -p 0x3 -m --config="(0,4,6,8),(1,5,7,9)" + +The following example is identical, except an additional ``lcore_kthread`` +core is specified per physical port. In this case, ``kni`` will create +four KNI interfaces: ``vEth0_0``/``vEth0_1`` bound to physical port 0 and +``vEth1_0``/``vEth1_1`` bound to physical port 1. + +The kernel thread for each interface will be bound as follows: + + * ``vEth0_0`` - bound to lcore 8. + * ``vEth0_1`` - bound to lcore 10. + * ``vEth1_0`` - bound to lcore 9. + * ``vEth1_1`` - bound to lcore 11 + +.. code-block:: console + + # rmmod rte_kni + # insmod kmod/rte_kni.ko kthread_mode=multiple + # ./build/kni -l 4-7 -n 4 -- -P -p 0x3 -m --config="(0,4,6,8,10),(1,5,7,9,11)" + +The following example can be used to test the interface between the ``kni`` +test application and the ``rte_kni`` kernel module. In this example, +the ``rte_kni`` kernel module is loaded in :ref:`single kernel thread +mode <kni_kernel_thread_mode>`, :ref:`loopback mode <kni_loopback_mode>` +enabled, and the :ref:`default carrier state <kni_default_carrier_state>` +is set to *on* so that the corresponding physical NIC port does not have +to be connected in order to use the KNI interface. One KNI interface +``vEth0_0`` is created for port 0 and one KNI interface ``vEth1_0`` is +created for port 1. Since ``rte_kni`` is loaded in "single kernel thread" +mode, the one kernel thread is bound to lcore 8. + +Since the physical NIC ports are not being used, link monitoring can be +disabled by **not** specifying the ``-m`` flag to ``kni``: + +.. code-block:: console + + # rmmod rte_kni + # insmod kmod/rte_kni.ko lo_mode=lo_mode_fifo carrier=on + # ./build/kni -l 4-7 -n 4 -- -P -p 0x3 --config="(0,4,6,8),(1,5,7,9)" + +KNI Operations +-------------- + +Once the ``kni`` application is started, the user can use the normal +Linux commands to manage the KNI interfaces as if they were any other +Linux network interface. + +Enable KNI interface and assign an IP address: + +.. code-block:: console + + # ip addr add dev vEth0_0 192.168.0.1 + +Show KNI interface configuration and statistics: + +.. code-block:: console + + # ip -s -d addr show vEth0_0 + +The user can also check and reset the packet statistics inside the ``kni`` +application by sending the app the USR1 and USR2 signals: + +.. code-block:: console + + # Print statistics + # pkill -USR1 kni + + # Zero statistics + # pkill -USR2 kni + +Dump network traffic: + +.. code-block:: console + + # tshark -n -i vEth0_0 + +The normal Linux commands can also be used to change the MAC address and +MTU size used by the physical NIC which corresponds to the KNI interface. +However, if more than one KNI interface is configured for a physical port, +these commands will only work on the first KNI interface for that port. + +Change the MAC address: + +.. code-block:: console + + # ip link set dev vEth0_0 lladdr 0C:01:02:03:04:08 + +Change the MTU size: + +.. code-block:: console + + # ip link set dev vEth0_0 mtu 1450 + +Limited ethtool support: + +.. code-block:: console + + # ethtool -i vEth0_0 + +When the ``kni`` application is closed, all the KNI interfaces are deleted +from the Linux kernel. + +Explanation +----------- + +The following sections provide some explanation of code. + +Initialization +~~~~~~~~~~~~~~ + +Setup of mbuf pool, driver and queues is similar to the setup done in the :doc:`l2_forward_real_virtual`.. +In addition, one or more kernel NIC interfaces are allocated for each +of the configured ports according to the command line parameters. + +The code for allocating the kernel NIC interfaces for a specific port is +in the function ``kni_alloc``. + +The other step in the initialization process that is unique to this sample application +is the association of each port with lcores for RX, TX and kernel threads. + +* One lcore to read from the port and write to the associated one or more KNI devices + +* Another lcore to read from one or more KNI devices and write to the port + +* Other lcores for pinning the kernel threads on one by one + +This is done by using the ``kni_port_params_array[]`` array, which is indexed by the port ID. +The code is in the function ``parse_config``. + +Packet Forwarding +~~~~~~~~~~~~~~~~~ + +After the initialization steps are completed, the main_loop() function is run on each lcore. +This function first checks the lcore_id against the user provided lcore_rx and lcore_tx +to see if this lcore is reading from or writing to kernel NIC interfaces. + +For the case that reads from a NIC port and writes to the kernel NIC interfaces (``kni_ingress``), +the packet reception is the same as in L2 Forwarding sample application +(see :ref:`l2_fwd_app_rx_tx_packets`). +The packet transmission is done by sending mbufs into the kernel NIC interfaces by ``rte_kni_tx_burst()``. +The KNI library automatically frees the mbufs after the kernel successfully copied the mbufs. + +For the other case that reads from kernel NIC interfaces +and writes to a physical NIC port (``kni_egress``), +packets are retrieved by reading mbufs from kernel NIC interfaces by ``rte_kni_rx_burst()``. +The packet transmission is the same as in the L2 Forwarding sample application +(see :ref:`l2_fwd_app_rx_tx_packets`). diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_cat.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_cat.rst new file mode 100644 index 000000000..0a813200b --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_cat.rst @@ -0,0 +1,207 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2016 Intel Corporation. + +L2 Forwarding Sample Application with Cache Allocation Technology (CAT) +======================================================================= + +Basic Forwarding sample application is a simple *skeleton* example of +a forwarding application. It has been extended to make use of CAT via extended +command line options and linking against the libpqos library. + +It is intended as a demonstration of the basic components of a DPDK forwarding +application and use of the libpqos library to program CAT. +For more detailed implementations see the L2 and L3 forwarding +sample applications. + +CAT and Code Data Prioritization (CDP) features allow management of the CPU's +last level cache. CAT introduces classes of service (COS) that are essentially +bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to +one cache way in last level cache. +A CPU core is always assigned to one of the CAT classes. +By programming CPU core assignment and COS bitmasks, applications can be given +exclusive, shared, or mixed access to the CPU's last level cache. +CDP extends CAT so that there are two bitmasks per COS, +one for data and one for code. +The number of classes and number of valid bits in a COS bitmask is CPU model +specific and COS bitmasks need to be contiguous. Sample code calls this bitmask +``cbm`` or capacity bitmask. +By default, after reset, all CPU cores are assigned to COS 0 and all classes +are programmed to allow fill into all cache ways. +CDP is off by default. + +For more information about CAT please see: + +* https://github.com/01org/intel-cmt-cat + +White paper demonstrating example use case: + +* `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_ + +Compiling the Application +------------------------- +.. note:: + + Requires ``libpqos`` from Intel's + `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_ + hosted on GitHub repository. For installation notes, please see ``README`` file. + + GIT: + + * https://github.com/01org/intel-cmt-cat + + +#. To compile the application export the path to PQoS lib + and the DPDK source tree and go to the example directory: + + .. code-block:: console + + export PQOS_INSTALL_PATH=/path/to/libpqos + + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l2fwd-cat`` sub-directory. + + +Running the Application +----------------------- + +To run the example in a ``linux`` environment and enable CAT on cpus 0-2: + +.. code-block:: console + + ./build/l2fwd-cat -l 1 -n 4 -- --l3ca="0x3@(0-2)" + +or to enable CAT and CDP on cpus 1,3: + +.. code-block:: console + + ./build/l2fwd-cat -l 1 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)" + +If CDP is not supported it will fail with following error message: + +.. code-block:: console + + PQOS: CDP requested but not supported. + PQOS: Requested CAT configuration is not valid! + PQOS: Shutting down PQoS library... + EAL: Error - exiting with code: 1 + Cause: PQOS: L3CA init failed! + +The option to enable CAT is: + +* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``: + + where ``cbm`` stands for capacity bitmask and must be expressed in + hexadecimal form. + + ``common_cbm`` is a single mask, for a CDP enabled system, a group of two + masks (``code_cbm`` and ``data_cbm``) is used. + + ``(`` and ``)`` are necessary if it's a group. + + ``cpus`` could be a single digit/range or a group and must be expressed in + decimal form. + + ``(`` and ``)`` are necessary if it's a group. + + e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'`` + + * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6; + + * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3; + + * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways; + + * cpu 7 has exclusive access to all of its 4 ways; + + e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system + + * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and + data ways are not overlapping. + + +Refer to *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + + +To reset or list CAT configuration and control CDP please use ``pqos`` tool +from Intel's +`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_. + +To enabled or disable CDP: + +.. code-block:: console + + sudo ./pqos -S cdp-on + + sudo ./pqos -S cdp-off + +to reset CAT configuration: + +.. code-block:: console + + sudo ./pqos -R + +to list CAT config: + +.. code-block:: console + + sudo ./pqos -s + +For more info about ``pqos`` tool please see its man page or +`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_. + + +Explanation +----------- + +The following sections provide an explanation of the main components of the +code. + +All DPDK library functions used in the sample code are prefixed with ``rte_`` +and are explained in detail in the *DPDK API Documentation*. + + +The Main Function +~~~~~~~~~~~~~~~~~ + +The ``main()`` function performs the initialization and calls the execution +threads for each lcore. + +The first task is to initialize the Environment Abstraction Layer (EAL). The +``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` +function. The value returned is the number of parsed arguments: + +.. code-block:: c + + int ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); + +The next task is to initialize the PQoS library and configure CAT. The +``argc`` and ``argv`` arguments are provided to the ``cat_init()`` +function. The value returned is the number of parsed arguments: + +.. code-block:: c + + int ret = cat_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n"); + +``cat_init()`` is a wrapper function which parses the command, validates +the requested parameters and configures CAT accordingly. + +Parsing of command line arguments is done in ``parse_args(...)``. +libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is +queried for system CPU information and L3CA capabilities via +``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)`` +calls. When all capability and topology information is collected, the requested +CAT configuration is validated. A check is then performed (on per socket basis) +for a sufficient number of un-associated COS. COS are selected and +configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to +relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls. + +``atexit(...)`` is used to register ``cat_exit(...)`` to be called on +a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating +COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_crypto.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_crypto.rst new file mode 100644 index 000000000..962752f21 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_crypto.rst @@ -0,0 +1,484 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2016-2017 Intel Corporation. + +.. _l2_fwd_crypto_app: + +L2 Forwarding with Crypto Sample Application +============================================ + +The L2 Forwarding with Crypto (l2fwd-crypto) sample application is a simple example of packet processing using +the Data Plane Development Kit (DPDK), in conjunction with the Cryptodev library. + +Overview +-------- + +The L2 Forwarding with Crypto sample application performs a crypto operation (cipher/hash) +specified by the user from command line (or using the default values), +with a crypto device capable of doing that operation, +for each packet that is received on a RX_PORT and performs L2 forwarding. +The destination port is the adjacent port from the enabled portmask, that is, +if the first four ports are enabled (portmask 0xf), +ports 0 and 1 forward into each other, and ports 2 and 3 forward into each other. +Also, if MAC addresses updating is enabled, the MAC addresses are affected as follows: + +* The source MAC address is replaced by the TX_PORT MAC address + +* The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l2fwd-crypt`` sub-directory. + +Running the Application +----------------------- + +The application requires a number of command line options: + +.. code-block:: console + + ./build/l2fwd-crypto [EAL options] -- [-p PORTMASK] [-q NQ] [-s] [-T PERIOD] / + [--cdev_type HW/SW/ANY] [--chain HASH_CIPHER/CIPHER_HASH/CIPHER_ONLY/HASH_ONLY/AEAD] / + [--cipher_algo ALGO] [--cipher_op ENCRYPT/DECRYPT] [--cipher_key KEY] / + [--cipher_key_random_size SIZE] [--cipher_iv IV] [--cipher_iv_random_size SIZE] / + [--auth_algo ALGO] [--auth_op GENERATE/VERIFY] [--auth_key KEY] / + [--auth_key_random_size SIZE] [--auth_iv IV] [--auth_iv_random_size SIZE] / + [--aead_algo ALGO] [--aead_op ENCRYPT/DECRYPT] [--aead_key KEY] / + [--aead_key_random_size SIZE] [--aead_iv] [--aead_iv_random_size SIZE] / + [--aad AAD] [--aad_random_size SIZE] / + [--digest size SIZE] [--sessionless] [--cryptodev_mask MASK] / + [--mac-updating] [--no-mac-updating] + +where, + +* p PORTMASK: A hexadecimal bitmask of the ports to configure (default is all the ports) + +* q NQ: A number of queues (=ports) per lcore (default is 1) + +* s: manage all ports from single core + +* T PERIOD: statistics will be refreshed each PERIOD seconds + + (0 to disable, 10 default, 86400 maximum) + +* cdev_type: select preferred crypto device type: HW, SW or anything (ANY) + + (default is ANY) + +* chain: select the operation chaining to perform: Cipher->Hash (CIPHER_HASH), + + Hash->Cipher (HASH_CIPHER), Cipher (CIPHER_ONLY), Hash (HASH_ONLY) + + or AEAD (AEAD) + + (default is Cipher->Hash) + +* cipher_algo: select the ciphering algorithm (default is aes-cbc) + +* cipher_op: select the ciphering operation to perform: ENCRYPT or DECRYPT + + (default is ENCRYPT) + +* cipher_key: set the ciphering key to be used. Bytes has to be separated with ":" + +* cipher_key_random_size: set the size of the ciphering key, + + which will be generated randomly. + + Note that if --cipher_key is used, this will be ignored. + +* cipher_iv: set the cipher IV to be used. Bytes has to be separated with ":" + +* cipher_iv_random_size: set the size of the cipher IV, which will be generated randomly. + + Note that if --cipher_iv is used, this will be ignored. + +* auth_algo: select the authentication algorithm (default is sha1-hmac) + +* auth_op: select the authentication operation to perform: GENERATE or VERIFY + + (default is GENERATE) + +* auth_key: set the authentication key to be used. Bytes has to be separated with ":" + +* auth_key_random_size: set the size of the authentication key, + + which will be generated randomly. + + Note that if --auth_key is used, this will be ignored. + +* auth_iv: set the auth IV to be used. Bytes has to be separated with ":" + +* auth_iv_random_size: set the size of the auth IV, which will be generated randomly. + + Note that if --auth_iv is used, this will be ignored. + +* aead_algo: select the AEAD algorithm (default is aes-gcm) + +* aead_op: select the AEAD operation to perform: ENCRYPT or DECRYPT + + (default is ENCRYPT) + +* aead_key: set the AEAD key to be used. Bytes has to be separated with ":" + +* aead_key_random_size: set the size of the AEAD key, + + which will be generated randomly. + + Note that if --aead_key is used, this will be ignored. + +* aead_iv: set the AEAD IV to be used. Bytes has to be separated with ":" + +* aead_iv_random_size: set the size of the AEAD IV, which will be generated randomly. + + Note that if --aead_iv is used, this will be ignored. + +* aad: set the AAD to be used. Bytes has to be separated with ":" + +* aad_random_size: set the size of the AAD, which will be generated randomly. + + Note that if --aad is used, this will be ignored. + +* digest_size: set the size of the digest to be generated/verified. + +* sessionless: no crypto session will be created. + +* cryptodev_mask: A hexadecimal bitmask of the cryptodevs to be used by the + application. + + (default is all cryptodevs). + +* [no-]mac-updating: Enable or disable MAC addresses updating (enabled by default). + + +The application requires that crypto devices capable of performing +the specified crypto operation are available on application initialization. +This means that HW crypto device/s must be bound to a DPDK driver or +a SW crypto device/s (virtual crypto PMD) must be created (using --vdev). + +To run the application in linux environment with 2 lcores, 2 ports and 2 crypto devices, issue the command: + +.. code-block:: console + + $ ./build/l2fwd-crypto -l 0-1 -n 4 --vdev "crypto_aesni_mb0" \ + --vdev "crypto_aesni_mb1" -- -p 0x3 --chain CIPHER_HASH \ + --cipher_op ENCRYPT --cipher_algo aes-cbc \ + --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f \ + --auth_op GENERATE --auth_algo aes-xcbc-mac \ + --auth_key 10:11:12:13:14:15:16:17:18:19:1a:1b:1c:1d:1e:1f + +Refer to the *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +.. Note:: + + * The ``l2fwd-crypto`` sample application requires IPv4 packets for crypto operation. + + * If multiple Ethernet ports is passed, then equal number of crypto devices are to be passed. + + * All crypto devices shall use the same session. + +Explanation +----------- + +The L2 forward with Crypto application demonstrates the performance of a crypto operation +on a packet received on a RX PORT before forwarding it to a TX PORT. + +The following figure illustrates a sample flow of a packet in the application, +from reception until transmission. + +.. _figure_l2_fwd_encrypt_flow: + +.. figure:: img/l2_fwd_encrypt_flow.* + + Encryption flow Through the L2 Forwarding with Crypto Application + + +The following sections provide some explanation of the application. + +Crypto operation specification +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All the packets received in all the ports get transformed by the crypto device/s +(ciphering and/or authentication). +The crypto operation to be performed on the packet is parsed from the command line +(go to "Running the Application" section for all the options). + +If no parameter is passed, the default crypto operation is: + +* Encryption with AES-CBC with 128 bit key. + +* Authentication with SHA1-HMAC (generation). + +* Keys, IV and AAD are generated randomly. + +There are two methods to pass keys, IV and ADD from the command line: + +* Passing the full key, separated bytes by ":":: + + --cipher_key 00:11:22:33:44 + +* Passing the size, so key is generated randomly:: + + --cipher_key_random_size 16 + +**Note**: + If full key is passed (first method) and the size is passed as well (second method), + the latter will be ignored. + +Size of these keys are checked (regardless the method), before starting the app, +to make sure that it is supported by the crypto devices. + +Crypto device initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the encryption operation is defined, crypto devices are initialized. +The crypto devices must be either bound to a DPDK driver (if they are physical devices) +or created using the EAL option --vdev (if they are virtual devices), +when running the application. + +The initialize_cryptodevs() function performs the device initialization. +It iterates through the list of the available crypto devices and +check which ones are capable of performing the operation. +Each device has a set of capabilities associated with it, +which are stored in the device info structure, so the function checks if the operation +is within the structure of each device. + +The following code checks if the device supports the specified cipher algorithm +(similar for the authentication algorithm): + +.. code-block:: c + + /* Check if device supports cipher algo */ + i = 0; + opt_cipher_algo = options->cipher_xform.cipher.algo; + cap = &dev_info.capabilities[i]; + while (cap->op != RTE_CRYPTO_OP_TYPE_UNDEFINED) { + cap_cipher_algo = cap->sym.cipher.algo; + if (cap->sym.xform_type == + RTE_CRYPTO_SYM_XFORM_CIPHER) { + if (cap_cipher_algo == opt_cipher_algo) { + if (check_type(options, &dev_info) == 0) + break; + } + } + cap = &dev_info.capabilities[++i]; + } + +If a capable crypto device is found, key sizes are checked to see if they are supported +(cipher key and IV for the ciphering): + +.. code-block:: c + + /* + * Check if length of provided cipher key is supported + * by the algorithm chosen. + */ + if (options->ckey_param) { + if (check_supported_size( + options->cipher_xform.cipher.key.length, + cap->sym.cipher.key_size.min, + cap->sym.cipher.key_size.max, + cap->sym.cipher.key_size.increment) + != 0) { + printf("Unsupported cipher key length\n"); + return -1; + } + /* + * Check if length of the cipher key to be randomly generated + * is supported by the algorithm chosen. + */ + } else if (options->ckey_random_size != -1) { + if (check_supported_size(options->ckey_random_size, + cap->sym.cipher.key_size.min, + cap->sym.cipher.key_size.max, + cap->sym.cipher.key_size.increment) + != 0) { + printf("Unsupported cipher key length\n"); + return -1; + } + options->cipher_xform.cipher.key.length = + options->ckey_random_size; + /* No size provided, use minimum size. */ + } else + options->cipher_xform.cipher.key.length = + cap->sym.cipher.key_size.min; + +After all the checks, the device is configured and it is added to the +crypto device list. + +**Note**: + The number of crypto devices that supports the specified crypto operation + must be at least the number of ports to be used. + +Session creation +~~~~~~~~~~~~~~~~ + +The crypto operation has a crypto session associated to it, which contains +information such as the transform chain to perform (e.g. ciphering then hashing), +pointers to the keys, lengths... etc. + +This session is created and is later attached to the crypto operation: + +.. code-block:: c + + static struct rte_cryptodev_sym_session * + initialize_crypto_session(struct l2fwd_crypto_options *options, + uint8_t cdev_id) + { + struct rte_crypto_sym_xform *first_xform; + struct rte_cryptodev_sym_session *session; + uint8_t socket_id = rte_cryptodev_socket_id(cdev_id); + struct rte_mempool *sess_mp = session_pool_socket[socket_id]; + + + if (options->xform_chain == L2FWD_CRYPTO_AEAD) { + first_xform = &options->aead_xform; + } else if (options->xform_chain == L2FWD_CRYPTO_CIPHER_HASH) { + first_xform = &options->cipher_xform; + first_xform->next = &options->auth_xform; + } else if (options->xform_chain == L2FWD_CRYPTO_HASH_CIPHER) { + first_xform = &options->auth_xform; + first_xform->next = &options->cipher_xform; + } else if (options->xform_chain == L2FWD_CRYPTO_CIPHER_ONLY) { + first_xform = &options->cipher_xform; + } else { + first_xform = &options->auth_xform; + } + + session = rte_cryptodev_sym_session_create(sess_mp); + + if (session == NULL) + return NULL; + + if (rte_cryptodev_sym_session_init(cdev_id, session, + first_xform, sess_mp) < 0) + return NULL; + + return session; + } + + ... + + port_cparams[i].session = initialize_crypto_session(options, + port_cparams[i].dev_id); + +Crypto operation creation +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Given N packets received from a RX PORT, N crypto operations are allocated +and filled: + +.. code-block:: c + + if (nb_rx) { + /* + * If we can't allocate a crypto_ops, then drop + * the rest of the burst and dequeue and + * process the packets to free offload structs + */ + if (rte_crypto_op_bulk_alloc( + l2fwd_crypto_op_pool, + RTE_CRYPTO_OP_TYPE_SYMMETRIC, + ops_burst, nb_rx) != + nb_rx) { + for (j = 0; j < nb_rx; j++) + rte_pktmbuf_free(pkts_burst[i]); + + nb_rx = 0; + } + +After filling the crypto operation (including session attachment), +the mbuf which will be transformed is attached to it:: + + op->sym->m_src = m; + +Since no destination mbuf is set, the source mbuf will be overwritten +after the operation is done (in-place). + +Crypto operation enqueuing/dequeuing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the operation has been created, it has to be enqueued in one of the crypto devices. +Before doing so, for performance reasons, the operation stays in a buffer. +When the buffer has enough operations (MAX_PKT_BURST), they are enqueued in the device, +which will perform the operation at that moment: + +.. code-block:: c + + static int + l2fwd_crypto_enqueue(struct rte_crypto_op *op, + struct l2fwd_crypto_params *cparams) + { + unsigned lcore_id, len; + struct lcore_queue_conf *qconf; + + lcore_id = rte_lcore_id(); + + qconf = &lcore_queue_conf[lcore_id]; + len = qconf->op_buf[cparams->dev_id].len; + qconf->op_buf[cparams->dev_id].buffer[len] = op; + len++; + + /* enough ops to be sent */ + if (len == MAX_PKT_BURST) { + l2fwd_crypto_send_burst(qconf, MAX_PKT_BURST, cparams); + len = 0; + } + + qconf->op_buf[cparams->dev_id].len = len; + return 0; + } + + ... + + static int + l2fwd_crypto_send_burst(struct lcore_queue_conf *qconf, unsigned n, + struct l2fwd_crypto_params *cparams) + { + struct rte_crypto_op **op_buffer; + unsigned ret; + + op_buffer = (struct rte_crypto_op **) + qconf->op_buf[cparams->dev_id].buffer; + + ret = rte_cryptodev_enqueue_burst(cparams->dev_id, + cparams->qp_id, op_buffer, (uint16_t) n); + + crypto_statistics[cparams->dev_id].enqueued += ret; + if (unlikely(ret < n)) { + crypto_statistics[cparams->dev_id].errors += (n - ret); + do { + rte_pktmbuf_free(op_buffer[ret]->sym->m_src); + rte_crypto_op_free(op_buffer[ret]); + } while (++ret < n); + } + + return 0; + } + +After this, the operations are dequeued from the device, and the transformed mbuf +is extracted from the operation. Then, the operation is freed and the mbuf is +forwarded as it is done in the L2 forwarding application. + +.. code-block:: c + + /* Dequeue packets from Crypto device */ + do { + nb_rx = rte_cryptodev_dequeue_burst( + cparams->dev_id, cparams->qp_id, + ops_burst, MAX_PKT_BURST); + + crypto_statistics[cparams->dev_id].dequeued += + nb_rx; + + /* Forward crypto'd packets */ + for (j = 0; j < nb_rx; j++) { + m = ops_burst[j]->sym->m_src; + + rte_crypto_op_free(ops_burst[j]); + l2fwd_simple_forward(m, portid); + } + } while (nb_rx == MAX_PKT_BURST); diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_event.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_event.rst new file mode 100644 index 000000000..d536eee81 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_event.rst @@ -0,0 +1,692 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +.. _l2_fwd_event_app: + +L2 Forwarding Eventdev Sample Application +========================================= + +The L2 Forwarding eventdev sample application is a simple example of packet +processing using the Data Plane Development Kit (DPDK) to demonstrate usage of +poll and event mode packet I/O mechanism. + +Overview +-------- + +The L2 Forwarding eventdev sample application, performs L2 forwarding for each +packet that is received on an RX_PORT. The destination port is the adjacent port +from the enabled portmask, that is, if the first four ports are enabled (portmask=0x0f), +ports 1 and 2 forward into each other, and ports 3 and 4 forward into each other. +Also, if MAC addresses updating is enabled, the MAC addresses are affected as follows: + +* The source MAC address is replaced by the TX_PORT MAC address + +* The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID + +Application receives packets from RX_PORT using below mentioned methods: + +* Poll mode + +* Eventdev mode (default) + +This application can be used to benchmark performance using a traffic-generator, +as shown in the :numref:`figure_l2fwd_event_benchmark_setup`. + +.. _figure_l2fwd_event_benchmark_setup: + +.. figure:: img/l2_fwd_benchmark_setup.* + + Performance Benchmark Setup (Basic Environment) + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l2fwd-event`` sub-directory. + +Running the Application +----------------------- + +The application requires a number of command line options: + +.. code-block:: console + + ./build/l2fwd-event [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating --mode=MODE --eventq-sched=SCHED_MODE + +where, + +* p PORTMASK: A hexadecimal bitmask of the ports to configure + +* q NQ: A number of queues (=ports) per lcore (default is 1) + +* --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default). + +* --mode=MODE: Packet transfer mode for I/O, poll or eventdev. Eventdev by default. + +* --eventq-sched=SCHED_MODE: Event queue schedule mode, Ordered, Atomic or Parallel. Atomic by default. + +* --config: Configure forwarding port pair mapping. Alternate port pairs by default. + +Sample usage commands are given below to run the application into different mode: + +Poll mode with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address updating enabled, +issue the command: + +.. code-block:: console + + ./build/l2fwd-event -l 0-3 -n 4 -- -q 8 -p ffff --mode=poll + +Eventdev mode with 4 lcores, 16 ports , sched method ordered and MAC address updating enabled, +issue the command: + +.. code-block:: console + + ./build/l2fwd-event -l 0-3 -n 4 -- -p ffff --eventq-sched=ordered + +or + +.. code-block:: console + + ./build/l2fwd-event -l 0-3 -n 4 -- -q 8 -p ffff --mode=eventdev --eventq-sched=ordered + +Refer to the *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + +To run application with S/W scheduler, it uses following DPDK services: + +* Software scheduler +* Rx adapter service function +* Tx adapter service function + +Application needs service cores to run above mentioned services. Service cores +must be provided as EAL parameters along with the --vdev=event_sw0 to enable S/W +scheduler. Following is the sample command: + +.. code-block:: console + + ./build/l2fwd-event -l 0-7 -s 0-3 -n 4 --vdev event_sw0 -- -q 8 -p ffff --mode=eventdev --eventq-sched=ordered + +Explanation +----------- + +The following sections provide some explanation of the code. + +.. _l2_fwd_event_app_cmd_arguments: + +Command Line Arguments +~~~~~~~~~~~~~~~~~~~~~~ + +The L2 Forwarding eventdev sample application takes specific parameters, +in addition to Environment Abstraction Layer (EAL) arguments. +The preferred way to parse parameters is to use the getopt() function, +since it is part of a well-defined and portable library. + +The parsing of arguments is done in the **l2fwd_parse_args()** function for non +eventdev parameters and in **parse_eventdev_args()** for eventdev parameters. +The method of argument parsing is not described here. Refer to the +*glibc getopt(3)* man page for details. + +EAL arguments are parsed first, then application-specific arguments. +This is done at the beginning of the main() function and eventdev parameters +are parsed in eventdev_resource_setup() function during eventdev setup: + +.. code-block:: c + + /* init EAL */ + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_panic("Invalid EAL arguments\n"); + + argc -= ret; + argv += ret; + + /* parse application arguments (after the EAL ones) */ + + ret = l2fwd_parse_args(argc, argv); + if (ret < 0) + rte_panic("Invalid L2FWD arguments\n"); + . + . + . + + /* Parse eventdev command line options */ + ret = parse_eventdev_args(argc, argv); + if (ret < 0) + return ret; + + + + +.. _l2_fwd_event_app_mbuf_init: + +Mbuf Pool Initialization +~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the arguments are parsed, the mbuf pool is created. +The mbuf pool contains a set of mbuf objects that will be used by the driver +and the application to store network packet data: + +.. code-block:: c + + /* create the mbuf pool */ + + l2fwd_pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NB_MBUF, + MEMPOOL_CACHE_SIZE, 0, + RTE_MBUF_DEFAULT_BUF_SIZE, + rte_socket_id()); + if (l2fwd_pktmbuf_pool == NULL) + rte_panic("Cannot init mbuf pool\n"); + +The rte_mempool is a generic structure used to handle pools of objects. +In this case, it is necessary to create a pool that will be used by the driver. +The number of allocated pkt mbufs is NB_MBUF, with a data room size of +RTE_MBUF_DEFAULT_BUF_SIZE each. +A per-lcore cache of 32 mbufs is kept. +The memory is allocated in NUMA socket 0, +but it is possible to extend this code to allocate one mbuf pool per socket. + +The rte_pktmbuf_pool_create() function uses the default mbuf pool and mbuf +initializers, respectively rte_pktmbuf_pool_init() and rte_pktmbuf_init(). +An advanced application may want to use the mempool API to create the +mbuf pool with more control. + +.. _l2_fwd_event_app_drv_init: + +Driver Initialization +~~~~~~~~~~~~~~~~~~~~~ + +The main part of the code in the main() function relates to the initialization +of the driver. To fully understand this code, it is recommended to study the +chapters that related to the Poll Mode and Event mode Driver in the +*DPDK Programmer's Guide* - Rel 1.4 EAR and the *DPDK API Reference*. + +.. code-block:: c + + /* reset l2fwd_dst_ports */ + + for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) + l2fwd_dst_ports[portid] = 0; + + last_port = 0; + + /* + * Each logical core is assigned a dedicated TX queue on each port. + */ + + RTE_ETH_FOREACH_DEV(portid) { + /* skip ports that are not enabled */ + + if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) + continue; + + if (nb_ports_in_mask % 2) { + l2fwd_dst_ports[portid] = last_port; + l2fwd_dst_ports[last_port] = portid; + } + else + last_port = portid; + + nb_ports_in_mask++; + + rte_eth_dev_info_get((uint8_t) portid, &dev_info); + } + +The next step is to configure the RX and TX queues. For each port, there is only +one RX queue (only one lcore is able to poll a given port). The number of TX +queues depends on the number of available lcores. The rte_eth_dev_configure() +function is used to configure the number of queues for a port: + +.. code-block:: c + + ret = rte_eth_dev_configure((uint8_t)portid, 1, 1, &port_conf); + if (ret < 0) + rte_panic("Cannot configure device: err=%d, port=%u\n", + ret, portid); + +.. _l2_fwd_event_app_rx_init: + +RX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +The application uses one lcore to poll one or several ports, depending on the -q +option, which specifies the number of queues per lcore. + +For example, if the user specifies -q 4, the application is able to poll four +ports with one lcore. If there are 16 ports on the target (and if the portmask +argument is -p ffff ), the application will need four lcores to poll all the +ports. + +.. code-block:: c + + ret = rte_eth_rx_queue_setup((uint8_t) portid, 0, nb_rxd, SOCKET0, + &rx_conf, l2fwd_pktmbuf_pool); + if (ret < 0) + + rte_panic("rte_eth_rx_queue_setup: err=%d, port=%u\n", + ret, portid); + +The list of queues that must be polled for a given lcore is stored in a private +structure called struct lcore_queue_conf. + +.. code-block:: c + + struct lcore_queue_conf { + unsigned n_rx_port; + unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE]; + struct mbuf_table tx_mbufs[L2FWD_MAX_PORTS]; + } rte_cache_aligned; + + struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE]; + +The values n_rx_port and rx_port_list[] are used in the main packet processing +loop (see :ref:`l2_fwd_event_app_rx_tx_packets`). + +.. _l2_fwd_event_app_tx_init: + +TX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +Each lcore should be able to transmit on any port. For every port, a single TX +queue is initialized. + +.. code-block:: c + + /* init one TX queue on each port */ + + fflush(stdout); + + ret = rte_eth_tx_queue_setup((uint8_t) portid, 0, nb_txd, + rte_eth_dev_socket_id(portid), &tx_conf); + if (ret < 0) + rte_panic("rte_eth_tx_queue_setup:err=%d, port=%u\n", + ret, (unsigned) portid); + +To configure eventdev support, application setups following components: + +* Event dev +* Event queue +* Event Port +* Rx/Tx adapters +* Ethernet ports + +.. _l2_fwd_event_app_event_dev_init: + +Event device Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Application can use either H/W or S/W based event device scheduler +implementation and supports single instance of event device. It configures event +device as per below configuration + +.. code-block:: c + + struct rte_event_dev_config event_d_conf = { + .nb_event_queues = ethdev_count, /* Dedicated to each Ethernet port */ + .nb_event_ports = num_workers, /* Dedicated to each lcore */ + .nb_events_limit = 4096, + .nb_event_queue_flows = 1024, + .nb_event_port_dequeue_depth = 128, + .nb_event_port_enqueue_depth = 128 + }; + + ret = rte_event_dev_configure(event_d_id, &event_d_conf); + if (ret < 0) + rte_panic("Error in configuring event device\n"); + +In case of S/W scheduler, application runs eventdev scheduler service on service +core. Application retrieves service id and finds the best possible service core to +run S/W scheduler. + +.. code-block:: c + + rte_event_dev_info_get(evt_rsrc->event_d_id, &evdev_info); + if (evdev_info.event_dev_cap & RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED) { + ret = rte_event_dev_service_id_get(evt_rsrc->event_d_id, + &service_id); + if (ret != -ESRCH && ret != 0) + rte_panic("Error in starting eventdev service\n"); + l2fwd_event_service_enable(service_id); + } + +.. _l2_fwd_app_event_queue_init: + +Event queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Ethernet device is assigned a dedicated event queue which will be linked +to all available event ports i.e. each lcore can dequeue packets from any of the +Ethernet ports. + +.. code-block:: c + + struct rte_event_queue_conf event_q_conf = { + .nb_atomic_flows = 1024, + .nb_atomic_order_sequences = 1024, + .event_queue_cfg = 0, + .schedule_type = RTE_SCHED_TYPE_ATOMIC, + .priority = RTE_EVENT_DEV_PRIORITY_HIGHEST + }; + + /* User requested sched mode */ + event_q_conf.schedule_type = eventq_sched_mode; + for (event_q_id = 0; event_q_id < ethdev_count; event_q_id++) { + ret = rte_event_queue_setup(event_d_id, event_q_id, + &event_q_conf); + if (ret < 0) + rte_panic("Error in configuring event queue\n"); + } + +In case of S/W scheduler, an extra event queue is created which will be used for +Tx adapter service function for enqueue operation. + +.. _l2_fwd_app_event_port_init: + +Event port Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~ +Each worker thread is assigned a dedicated event port for enq/deq operations +to/from an event device. All event ports are linked with all available event +queues. + +.. code-block:: c + + struct rte_event_port_conf event_p_conf = { + .dequeue_depth = 32, + .enqueue_depth = 32, + .new_event_threshold = 4096 + }; + + for (event_p_id = 0; event_p_id < num_workers; event_p_id++) { + ret = rte_event_port_setup(event_d_id, event_p_id, + &event_p_conf); + if (ret < 0) + rte_panic("Error in configuring event port %d\n", event_p_id); + + ret = rte_event_port_link(event_d_id, event_p_id, NULL, + NULL, 0); + if (ret < 0) + rte_panic("Error in linking event port %d to queue\n", + event_p_id); + } + +In case of S/W scheduler, an extra event port is created by DPDK library which +is retrieved by the application and same will be used by Tx adapter service. + +.. code-block:: c + + ret = rte_event_eth_tx_adapter_event_port_get(tx_adptr_id, &tx_port_id); + if (ret) + rte_panic("Failed to get Tx adapter port id: %d\n", ret); + + ret = rte_event_port_link(event_d_id, tx_port_id, + &evt_rsrc.evq.event_q_id[ + evt_rsrc.evq.nb_queues - 1], + NULL, 1); + if (ret != 1) + rte_panic("Unable to link Tx adapter port to Tx queue:err=%d\n", + ret); + +.. _l2_fwd_event_app_adapter_init: + +Rx/Tx adapter Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Each Ethernet port is assigned a dedicated Rx/Tx adapter for H/W scheduler. Each +Ethernet port's Rx queues are connected to its respective event queue at +priority 0 via Rx adapter configuration and Ethernet port's tx queues are +connected via Tx adapter. + +.. code-block:: c + + RTE_ETH_FOREACH_DEV(port_id) { + if ((rsrc->enabled_port_mask & (1 << port_id)) == 0) + continue; + ret = rte_event_eth_rx_adapter_create(adapter_id, event_d_id, + &evt_rsrc->def_p_conf); + if (ret) + rte_panic("Failed to create rx adapter[%d]\n", + adapter_id); + + /* Configure user requested sched type*/ + eth_q_conf.ev.sched_type = rsrc->sched_type; + eth_q_conf.ev.queue_id = evt_rsrc->evq.event_q_id[q_id]; + ret = rte_event_eth_rx_adapter_queue_add(adapter_id, port_id, + -1, ð_q_conf); + if (ret) + rte_panic("Failed to add queues to Rx adapter\n"); + + ret = rte_event_eth_rx_adapter_start(adapter_id); + if (ret) + rte_panic("Rx adapter[%d] start Failed\n", adapter_id); + + evt_rsrc->rx_adptr.rx_adptr[adapter_id] = adapter_id; + adapter_id++; + if (q_id < evt_rsrc->evq.nb_queues) + q_id++; + } + + adapter_id = 0; + RTE_ETH_FOREACH_DEV(port_id) { + if ((rsrc->enabled_port_mask & (1 << port_id)) == 0) + continue; + ret = rte_event_eth_tx_adapter_create(adapter_id, event_d_id, + &evt_rsrc->def_p_conf); + if (ret) + rte_panic("Failed to create tx adapter[%d]\n", + adapter_id); + + ret = rte_event_eth_tx_adapter_queue_add(adapter_id, port_id, + -1); + if (ret) + rte_panic("Failed to add queues to Tx adapter\n"); + + ret = rte_event_eth_tx_adapter_start(adapter_id); + if (ret) + rte_panic("Tx adapter[%d] start Failed\n", adapter_id); + + evt_rsrc->tx_adptr.tx_adptr[adapter_id] = adapter_id; + adapter_id++; + } + +For S/W scheduler instead of dedicated adapters, common Rx/Tx adapters are +configured which will be shared among all the Ethernet ports. Also DPDK library +need service cores to run internal services for Rx/Tx adapters. Application gets +service id for Rx/Tx adapters and after successful setup it runs the services +on dedicated service cores. + +.. code-block:: c + + for (i = 0; i < evt_rsrc->rx_adptr.nb_rx_adptr; i++) { + ret = rte_event_eth_rx_adapter_caps_get(evt_rsrc->event_d_id, + evt_rsrc->rx_adptr.rx_adptr[i], &caps); + if (ret < 0) + rte_panic("Failed to get Rx adapter[%d] caps\n", + evt_rsrc->rx_adptr.rx_adptr[i]); + ret = rte_event_eth_rx_adapter_service_id_get( + evt_rsrc->event_d_id, + &service_id); + if (ret != -ESRCH && ret != 0) + rte_panic("Error in starting Rx adapter[%d] service\n", + evt_rsrc->rx_adptr.rx_adptr[i]); + l2fwd_event_service_enable(service_id); + } + + for (i = 0; i < evt_rsrc->tx_adptr.nb_tx_adptr; i++) { + ret = rte_event_eth_tx_adapter_caps_get(evt_rsrc->event_d_id, + evt_rsrc->tx_adptr.tx_adptr[i], &caps); + if (ret < 0) + rte_panic("Failed to get Rx adapter[%d] caps\n", + evt_rsrc->tx_adptr.tx_adptr[i]); + ret = rte_event_eth_tx_adapter_service_id_get( + evt_rsrc->event_d_id, + &service_id); + if (ret != -ESRCH && ret != 0) + rte_panic("Error in starting Rx adapter[%d] service\n", + evt_rsrc->tx_adptr.tx_adptr[i]); + l2fwd_event_service_enable(service_id); + } + +.. _l2_fwd_event_app_rx_tx_packets: + +Receive, Process and Transmit Packets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the **l2fwd_main_loop()** function, the main task is to read ingress packets from +the RX queues. This is done using the following code: + +.. code-block:: c + + /* + * Read packet from RX queues + */ + + for (i = 0; i < qconf->n_rx_port; i++) { + portid = qconf->rx_port_list[i]; + nb_rx = rte_eth_rx_burst((uint8_t) portid, 0, pkts_burst, + MAX_PKT_BURST); + + for (j = 0; j < nb_rx; j++) { + m = pkts_burst[j]; + rte_prefetch0(rte_pktmbuf_mtod(m, void *)); + l2fwd_simple_forward(m, portid); + } + } + +Packets are read in a burst of size MAX_PKT_BURST. The rte_eth_rx_burst() +function writes the mbuf pointers in a local table and returns the number of +available mbufs in the table. + +Then, each mbuf in the table is processed by the l2fwd_simple_forward() +function. The processing is very simple: process the TX port from the RX port, +then replace the source and destination MAC addresses if MAC addresses updating +is enabled. + +During the initialization process, a static array of destination ports +(l2fwd_dst_ports[]) is filled such that for each source port, a destination port +is assigned that is either the next or previous enabled port from the portmask. +If number of ports are odd in portmask then packet from last port will be +forwarded to first port i.e. if portmask=0x07, then forwarding will take place +like p0--->p1, p1--->p2, p2--->p0. + +Also to optimize enqueue operation, l2fwd_simple_forward() stores incoming mbufs +up to MAX_PKT_BURST. Once it reaches up to limit, all packets are transmitted to +destination ports. + +.. code-block:: c + + static void + l2fwd_simple_forward(struct rte_mbuf *m, uint32_t portid) + { + uint32_t dst_port; + int32_t sent; + struct rte_eth_dev_tx_buffer *buffer; + + dst_port = l2fwd_dst_ports[portid]; + + if (mac_updating) + l2fwd_mac_updating(m, dst_port); + + buffer = tx_buffer[dst_port]; + sent = rte_eth_tx_buffer(dst_port, 0, buffer, m); + if (sent) + port_statistics[dst_port].tx += sent; + } + +For this test application, the processing is exactly the same for all packets +arriving on the same RX port. Therefore, it would have been possible to call +the rte_eth_tx_buffer() function directly from the main loop to send all the +received packets on the same TX port, using the burst-oriented send function, +which is more efficient. + +However, in real-life applications (such as, L3 routing), +packet N is not necessarily forwarded on the same port as packet N-1. +The application is implemented to illustrate that, so the same approach can be +reused in a more complex application. + +To ensure that no packets remain in the tables, each lcore does a draining of TX +queue in its main loop. This technique introduces some latency when there are +not many packets to send, however it improves performance: + +.. code-block:: c + + cur_tsc = rte_rdtsc(); + + /* + * TX burst queue drain + */ + diff_tsc = cur_tsc - prev_tsc; + if (unlikely(diff_tsc > drain_tsc)) { + for (i = 0; i < qconf->n_rx_port; i++) { + portid = l2fwd_dst_ports[qconf->rx_port_list[i]]; + buffer = tx_buffer[portid]; + sent = rte_eth_tx_buffer_flush(portid, 0, + buffer); + if (sent) + port_statistics[portid].tx += sent; + } + + /* if timer is enabled */ + if (timer_period > 0) { + /* advance the timer */ + timer_tsc += diff_tsc; + + /* if timer has reached its timeout */ + if (unlikely(timer_tsc >= timer_period)) { + /* do this only on master core */ + if (lcore_id == rte_get_master_lcore()) { + print_stats(); + /* reset the timer */ + timer_tsc = 0; + } + } + } + + prev_tsc = cur_tsc; + } + +In the **l2fwd_event_loop()** function, the main task is to read ingress +packets from the event ports. This is done using the following code: + +.. code-block:: c + + /* Read packet from eventdev */ + nb_rx = rte_event_dequeue_burst(event_d_id, event_p_id, + events, deq_len, 0); + if (nb_rx == 0) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + mbuf[i] = events[i].mbuf; + rte_prefetch0(rte_pktmbuf_mtod(mbuf[i], void *)); + } + + +Before reading packets, deq_len is fetched to ensure correct allowed deq length +by the eventdev. +The rte_event_dequeue_burst() function writes the mbuf pointers in a local table +and returns the number of available mbufs in the table. + +Then, each mbuf in the table is processed by the l2fwd_eventdev_forward() +function. The processing is very simple: process the TX port from the RX port, +then replace the source and destination MAC addresses if MAC addresses updating +is enabled. + +During the initialization process, a static array of destination ports +(l2fwd_dst_ports[]) is filled such that for each source port, a destination port +is assigned that is either the next or previous enabled port from the portmask. +If number of ports are odd in portmask then packet from last port will be +forwarded to first port i.e. if portmask=0x07, then forwarding will take place +like p0--->p1, p1--->p2, p2--->p0. + +l2fwd_eventdev_forward() does not stores incoming mbufs. Packet will forwarded +be to destination ports via Tx adapter or generic event dev enqueue API +depending H/W or S/W scheduler is used. + +.. code-block:: c + + nb_tx = rte_event_eth_tx_adapter_enqueue(event_d_id, port_id, ev, + nb_rx); + while (nb_tx < nb_rx && !rsrc->force_quit) + nb_tx += rte_event_eth_tx_adapter_enqueue( + event_d_id, port_id, + ev + nb_tx, nb_rx - nb_tx); diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_job_stats.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_job_stats.rst new file mode 100644 index 000000000..8d0c38721 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_job_stats.rst @@ -0,0 +1,550 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2015 Intel Corporation. + +L2 Forwarding Sample Application (in Real and Virtualized Environments) with core load statistics. +================================================================================================== + +The L2 Forwarding sample application is a simple example of packet processing using +the Data Plane Development Kit (DPDK) which +also takes advantage of Single Root I/O Virtualization (SR-IOV) features in a virtualized environment. + +.. note:: + + This application is a variation of L2 Forwarding sample application. It demonstrate possible + scheme of job stats library usage therefore some parts of this document is identical with original + L2 forwarding application. + +Overview +-------- + +The L2 Forwarding sample application, which can operate in real and virtualized environments, +performs L2 forwarding for each packet that is received. +The destination port is the adjacent port from the enabled portmask, that is, +if the first four ports are enabled (portmask 0xf), +ports 1 and 2 forward into each other, and ports 3 and 4 forward into each other. +Also, the MAC addresses are affected as follows: + +* The source MAC address is replaced by the TX port MAC address + +* The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID + +This application can be used to benchmark performance using a traffic-generator, as shown in the :numref:`figure_l2_fwd_benchmark_setup_jobstats`. + +The application can also be used in a virtualized environment as shown in :numref:`figure_l2_fwd_virtenv_benchmark_setup_jobstats`. + +The L2 Forwarding application can also be used as a starting point for developing a new application based on the DPDK. + +.. _figure_l2_fwd_benchmark_setup_jobstats: + +.. figure:: img/l2_fwd_benchmark_setup.* + + Performance Benchmark Setup (Basic Environment) + +.. _figure_l2_fwd_virtenv_benchmark_setup_jobstats: + +.. figure:: img/l2_fwd_virtenv_benchmark_setup.* + + Performance Benchmark Setup (Virtualized Environment) + + +Virtual Function Setup Instructions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This application can use the virtual function available in the system and +therefore can be used in a virtual machine without passing through +the whole Network Device into a guest machine in a virtualized scenario. +The virtual functions can be enabled in the host machine or the hypervisor with the respective physical function driver. + +For example, in a Linux* host machine, it is possible to enable a virtual function using the following command: + +.. code-block:: console + + modprobe ixgbe max_vfs=2,2 + +This command enables two Virtual Functions on each of Physical Function of the NIC, +with two physical ports in the PCI configuration space. +It is important to note that enabled Virtual Function 0 and 2 would belong to Physical Function 0 +and Virtual Function 1 and 3 would belong to Physical Function 1, +in this case enabling a total of four Virtual Functions. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l2fwd-jobstats`` sub-directory. + +Running the Application +----------------------- + +The application requires a number of command line options: + +.. code-block:: console + + ./build/l2fwd-jobstats [EAL options] -- -p PORTMASK [-q NQ] [-l] + +where, + +* p PORTMASK: A hexadecimal bitmask of the ports to configure + +* q NQ: A number of queues (=ports) per lcore (default is 1) + +* l: Use locale thousands separator when formatting big numbers. + +To run the application in linux environment with 4 lcores, 16 ports, 8 RX queues per lcore and +thousands separator printing, issue the command: + +.. code-block:: console + + $ ./build/l2fwd-jobstats -l 0-3 -n 4 -- -q 8 -p ffff -l + +Refer to the *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. + +Command Line Arguments +~~~~~~~~~~~~~~~~~~~~~~ + +The L2 Forwarding sample application takes specific parameters, +in addition to Environment Abstraction Layer (EAL) arguments +(see `Running the Application`_). +The preferred way to parse parameters is to use the getopt() function, +since it is part of a well-defined and portable library. + +The parsing of arguments is done in the l2fwd_parse_args() function. +The method of argument parsing is not described here. +Refer to the *glibc getopt(3)* man page for details. + +EAL arguments are parsed first, then application-specific arguments. +This is done at the beginning of the main() function: + +.. code-block:: c + + /* init EAL */ + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); + + argc -= ret; + argv += ret; + + /* parse application arguments (after the EAL ones) */ + + ret = l2fwd_parse_args(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid L2FWD arguments\n"); + +Mbuf Pool Initialization +~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the arguments are parsed, the mbuf pool is created. +The mbuf pool contains a set of mbuf objects that will be used by the driver +and the application to store network packet data: + +.. code-block:: c + + /* create the mbuf pool */ + l2fwd_pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NB_MBUF, + MEMPOOL_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, + rte_socket_id()); + + if (l2fwd_pktmbuf_pool == NULL) + rte_exit(EXIT_FAILURE, "Cannot init mbuf pool\n"); + +The rte_mempool is a generic structure used to handle pools of objects. +In this case, it is necessary to create a pool that will be used by the driver. +The number of allocated pkt mbufs is NB_MBUF, with a data room size of +RTE_MBUF_DEFAULT_BUF_SIZE each. +A per-lcore cache of MEMPOOL_CACHE_SIZE mbufs is kept. +The memory is allocated in rte_socket_id() socket, +but it is possible to extend this code to allocate one mbuf pool per socket. + +The rte_pktmbuf_pool_create() function uses the default mbuf pool and mbuf +initializers, respectively rte_pktmbuf_pool_init() and rte_pktmbuf_init(). +An advanced application may want to use the mempool API to create the +mbuf pool with more control. + +Driver Initialization +~~~~~~~~~~~~~~~~~~~~~ + +The main part of the code in the main() function relates to the initialization of the driver. +To fully understand this code, it is recommended to study the chapters that related to the Poll Mode Driver +in the *DPDK Programmer's Guide* and the *DPDK API Reference*. + +.. code-block:: c + + /* reset l2fwd_dst_ports */ + + for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) + l2fwd_dst_ports[portid] = 0; + + last_port = 0; + + /* + * Each logical core is assigned a dedicated TX queue on each port. + */ + RTE_ETH_FOREACH_DEV(portid) { + /* skip ports that are not enabled */ + if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) + continue; + + if (nb_ports_in_mask % 2) { + l2fwd_dst_ports[portid] = last_port; + l2fwd_dst_ports[last_port] = portid; + } + else + last_port = portid; + + nb_ports_in_mask++; + + rte_eth_dev_info_get((uint8_t) portid, &dev_info); + } + +The next step is to configure the RX and TX queues. +For each port, there is only one RX queue (only one lcore is able to poll a given port). +The number of TX queues depends on the number of available lcores. +The rte_eth_dev_configure() function is used to configure the number of queues for a port: + +.. code-block:: c + + ret = rte_eth_dev_configure((uint8_t)portid, 1, 1, &port_conf); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Cannot configure device: " + "err=%d, port=%u\n", + ret, portid); + +RX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +The application uses one lcore to poll one or several ports, depending on the -q option, +which specifies the number of queues per lcore. + +For example, if the user specifies -q 4, the application is able to poll four ports with one lcore. +If there are 16 ports on the target (and if the portmask argument is -p ffff ), +the application will need four lcores to poll all the ports. + +.. code-block:: c + + ret = rte_eth_rx_queue_setup(portid, 0, nb_rxd, + rte_eth_dev_socket_id(portid), + NULL, + l2fwd_pktmbuf_pool); + + if (ret < 0) + rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup:err=%d, port=%u\n", + ret, (unsigned) portid); + +The list of queues that must be polled for a given lcore is stored in a private structure called struct lcore_queue_conf. + +.. code-block:: c + + struct lcore_queue_conf { + unsigned n_rx_port; + unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE]; + truct mbuf_table tx_mbufs[RTE_MAX_ETHPORTS]; + + struct rte_timer rx_timers[MAX_RX_QUEUE_PER_LCORE]; + struct rte_jobstats port_fwd_jobs[MAX_RX_QUEUE_PER_LCORE]; + + struct rte_timer flush_timer; + struct rte_jobstats flush_job; + struct rte_jobstats idle_job; + struct rte_jobstats_context jobs_context; + + rte_atomic16_t stats_read_pending; + rte_spinlock_t lock; + } __rte_cache_aligned; + +Values of struct lcore_queue_conf: + +* n_rx_port and rx_port_list[] are used in the main packet processing loop + (see Section `Receive, Process and Transmit Packets`_ later in this chapter). + +* rx_timers and flush_timer are used to ensure forced TX on low packet rate. + +* flush_job, idle_job and jobs_context are librte_jobstats objects used for managing l2fwd jobs. + +* stats_read_pending and lock are used during job stats read phase. + +TX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +Each lcore should be able to transmit on any port. For every port, a single TX queue is initialized. + +.. code-block:: c + + /* init one TX queue on each port */ + + fflush(stdout); + ret = rte_eth_tx_queue_setup(portid, 0, nb_txd, + rte_eth_dev_socket_id(portid), + NULL); + if (ret < 0) + rte_exit(EXIT_FAILURE, "rte_eth_tx_queue_setup:err=%d, port=%u\n", + ret, (unsigned) portid); + +Jobs statistics initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +There are several statistics objects available: + +* Flush job statistics + +.. code-block:: c + + rte_jobstats_init(&qconf->flush_job, "flush", drain_tsc, drain_tsc, + drain_tsc, 0); + + rte_timer_init(&qconf->flush_timer); + ret = rte_timer_reset(&qconf->flush_timer, drain_tsc, PERIODICAL, + lcore_id, &l2fwd_flush_job, NULL); + + if (ret < 0) { + rte_exit(1, "Failed to reset flush job timer for lcore %u: %s", + lcore_id, rte_strerror(-ret)); + } + +* Statistics per RX port + +.. code-block:: c + + rte_jobstats_init(job, name, 0, drain_tsc, 0, MAX_PKT_BURST); + rte_jobstats_set_update_period_function(job, l2fwd_job_update_cb); + + rte_timer_init(&qconf->rx_timers[i]); + ret = rte_timer_reset(&qconf->rx_timers[i], 0, PERIODICAL, lcore_id, + l2fwd_fwd_job, (void *)(uintptr_t)i); + + if (ret < 0) { + rte_exit(1, "Failed to reset lcore %u port %u job timer: %s", + lcore_id, qconf->rx_port_list[i], rte_strerror(-ret)); + } + +Following parameters are passed to rte_jobstats_init(): + +* 0 as minimal poll period + +* drain_tsc as maximum poll period + +* MAX_PKT_BURST as desired target value (RX burst size) + +Main loop +~~~~~~~~~ + +The forwarding path is reworked comparing to original L2 Forwarding application. +In the l2fwd_main_loop() function three loops are placed. + +.. code-block:: c + + for (;;) { + rte_spinlock_lock(&qconf->lock); + + do { + rte_jobstats_context_start(&qconf->jobs_context); + + /* Do the Idle job: + * - Read stats_read_pending flag + * - check if some real job need to be executed + */ + rte_jobstats_start(&qconf->jobs_context, &qconf->idle_job); + + do { + uint8_t i; + uint64_t now = rte_get_timer_cycles(); + + need_manage = qconf->flush_timer.expire < now; + /* Check if we was esked to give a stats. */ + stats_read_pending = + rte_atomic16_read(&qconf->stats_read_pending); + need_manage |= stats_read_pending; + + for (i = 0; i < qconf->n_rx_port && !need_manage; i++) + need_manage = qconf->rx_timers[i].expire < now; + + } while (!need_manage); + rte_jobstats_finish(&qconf->idle_job, qconf->idle_job.target); + + rte_timer_manage(); + rte_jobstats_context_finish(&qconf->jobs_context); + } while (likely(stats_read_pending == 0)); + + rte_spinlock_unlock(&qconf->lock); + rte_pause(); + } + +First infinite for loop is to minimize impact of stats reading. Lock is only locked/unlocked when asked. + +Second inner while loop do the whole jobs management. When any job is ready, the use rte_timer_manage() is used to call the job handler. +In this place functions l2fwd_fwd_job() and l2fwd_flush_job() are called when needed. +Then rte_jobstats_context_finish() is called to mark loop end - no other jobs are ready to execute. By this time stats are ready to be read +and if stats_read_pending is set, loop breaks allowing stats to be read. + +Third do-while loop is the idle job (idle stats counter). Its only purpose is monitoring if any job is ready or stats job read is pending +for this lcore. Statistics from this part of code is considered as the headroom available for additional processing. + +Receive, Process and Transmit Packets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The main task of l2fwd_fwd_job() function is to read ingress packets from the RX queue of particular port and forward it. +This is done using the following code: + +.. code-block:: c + + total_nb_rx = rte_eth_rx_burst((uint8_t) portid, 0, pkts_burst, + MAX_PKT_BURST); + + for (j = 0; j < total_nb_rx; j++) { + m = pkts_burst[j]; + rte_prefetch0(rte_pktmbuf_mtod(m, void *)); + l2fwd_simple_forward(m, portid); + } + +Packets are read in a burst of size MAX_PKT_BURST. +Then, each mbuf in the table is processed by the l2fwd_simple_forward() function. +The processing is very simple: process the TX port from the RX port, then replace the source and destination MAC addresses. + +The rte_eth_rx_burst() function writes the mbuf pointers in a local table and returns the number of available mbufs in the table. + +After first read second try is issued. + +.. code-block:: c + + if (total_nb_rx == MAX_PKT_BURST) { + const uint16_t nb_rx = rte_eth_rx_burst((uint8_t) portid, 0, pkts_burst, + MAX_PKT_BURST); + + total_nb_rx += nb_rx; + for (j = 0; j < nb_rx; j++) { + m = pkts_burst[j]; + rte_prefetch0(rte_pktmbuf_mtod(m, void *)); + l2fwd_simple_forward(m, portid); + } + } + +This second read is important to give job stats library a feedback how many packets was processed. + +.. code-block:: c + + /* Adjust period time in which we are running here. */ + if (rte_jobstats_finish(job, total_nb_rx) != 0) { + rte_timer_reset(&qconf->rx_timers[port_idx], job->period, PERIODICAL, + lcore_id, l2fwd_fwd_job, arg); + } + +To maximize performance exactly MAX_PKT_BURST is expected (the target value) to be read for each l2fwd_fwd_job() call. +If total_nb_rx is smaller than target value job->period will be increased. If it is greater the period will be decreased. + +.. note:: + + In the following code, one line for getting the output port requires some explanation. + +During the initialization process, a static array of destination ports (l2fwd_dst_ports[]) is filled such that for each source port, +a destination port is assigned that is either the next or previous enabled port from the portmask. +Naturally, the number of ports in the portmask must be even, otherwise, the application exits. + +.. code-block:: c + + static void + l2fwd_simple_forward(struct rte_mbuf *m, unsigned portid) + { + struct rte_ether_hdr *eth; + void *tmp; + unsigned dst_port; + + dst_port = l2fwd_dst_ports[portid]; + + eth = rte_pktmbuf_mtod(m, struct rte_ether_hdr *); + + /* 02:00:00:00:00:xx */ + + tmp = ð->d_addr.addr_bytes[0]; + + *((uint64_t *)tmp) = 0x000000000002 + ((uint64_t) dst_port << 40); + + /* src addr */ + + rte_ether_addr_copy(&l2fwd_ports_eth_addr[dst_port], ð->s_addr); + + l2fwd_send_packet(m, (uint8_t) dst_port); + } + +Then, the packet is sent using the l2fwd_send_packet (m, dst_port) function. +For this test application, the processing is exactly the same for all packets arriving on the same RX port. +Therefore, it would have been possible to call the l2fwd_send_burst() function directly from the main loop +to send all the received packets on the same TX port, +using the burst-oriented send function, which is more efficient. + +However, in real-life applications (such as, L3 routing), +packet N is not necessarily forwarded on the same port as packet N-1. +The application is implemented to illustrate that, so the same approach can be reused in a more complex application. + +The l2fwd_send_packet() function stores the packet in a per-lcore and per-txport table. +If the table is full, the whole packets table is transmitted using the l2fwd_send_burst() function: + +.. code-block:: c + + /* Send the packet on an output interface */ + + static int + l2fwd_send_packet(struct rte_mbuf *m, uint16_t port) + { + unsigned lcore_id, len; + struct lcore_queue_conf *qconf; + + lcore_id = rte_lcore_id(); + qconf = &lcore_queue_conf[lcore_id]; + len = qconf->tx_mbufs[port].len; + qconf->tx_mbufs[port].m_table[len] = m; + len++; + + /* enough pkts to be sent */ + + if (unlikely(len == MAX_PKT_BURST)) { + l2fwd_send_burst(qconf, MAX_PKT_BURST, port); + len = 0; + } + + qconf->tx_mbufs[port].len = len; return 0; + } + +To ensure that no packets remain in the tables, the flush job exists. The l2fwd_flush_job() +is called periodically to for each lcore draining TX queue of each port. +This technique introduces some latency when there are not many packets to send, +however it improves performance: + +.. code-block:: c + + static void + l2fwd_flush_job(__rte_unused struct rte_timer *timer, __rte_unused void *arg) + { + uint64_t now; + unsigned lcore_id; + struct lcore_queue_conf *qconf; + struct mbuf_table *m_table; + uint16_t portid; + + lcore_id = rte_lcore_id(); + qconf = &lcore_queue_conf[lcore_id]; + + rte_jobstats_start(&qconf->jobs_context, &qconf->flush_job); + + now = rte_get_timer_cycles(); + lcore_id = rte_lcore_id(); + qconf = &lcore_queue_conf[lcore_id]; + for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) { + m_table = &qconf->tx_mbufs[portid]; + if (m_table->len == 0 || m_table->next_flush_time <= now) + continue; + + l2fwd_send_burst(qconf, portid); + } + + + /* Pass target to indicate that this job is happy of time interval + * in which it was called. */ + rte_jobstats_finish(&qconf->flush_job, qconf->flush_job.target); + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_real_virtual.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_real_virtual.rst new file mode 100644 index 000000000..671d0c7c1 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l2_forward_real_virtual.rst @@ -0,0 +1,455 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +.. _l2_fwd_app_real_and_virtual: + +L2 Forwarding Sample Application (in Real and Virtualized Environments) +======================================================================= + +The L2 Forwarding sample application is a simple example of packet processing using +the Data Plane Development Kit (DPDK) which +also takes advantage of Single Root I/O Virtualization (SR-IOV) features in a virtualized environment. + +.. note:: + + Please note that previously a separate L2 Forwarding in Virtualized Environments sample application was used, + however, in later DPDK versions these sample applications have been merged. + +Overview +-------- + +The L2 Forwarding sample application, which can operate in real and virtualized environments, +performs L2 forwarding for each packet that is received on an RX_PORT. +The destination port is the adjacent port from the enabled portmask, that is, +if the first four ports are enabled (portmask 0xf), +ports 1 and 2 forward into each other, and ports 3 and 4 forward into each other. +Also, if MAC addresses updating is enabled, the MAC addresses are affected as follows: + +* The source MAC address is replaced by the TX_PORT MAC address + +* The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID + +This application can be used to benchmark performance using a traffic-generator, as shown in the :numref:`figure_l2_fwd_benchmark_setup`, +or in a virtualized environment as shown in :numref:`figure_l2_fwd_virtenv_benchmark_setup`. + +.. _figure_l2_fwd_benchmark_setup: + +.. figure:: img/l2_fwd_benchmark_setup.* + + Performance Benchmark Setup (Basic Environment) + +.. _figure_l2_fwd_virtenv_benchmark_setup: + +.. figure:: img/l2_fwd_virtenv_benchmark_setup.* + + Performance Benchmark Setup (Virtualized Environment) + +This application may be used for basic VM to VM communication as shown in :numref:`figure_l2_fwd_vm2vm`, +when MAC addresses updating is disabled. + +.. _figure_l2_fwd_vm2vm: + +.. figure:: img/l2_fwd_vm2vm.* + + Virtual Machine to Virtual Machine communication. + +The L2 Forwarding application can also be used as a starting point for developing a new application based on the DPDK. + +.. _l2_fwd_vf_setup: + +Virtual Function Setup Instructions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This application can use the virtual function available in the system and +therefore can be used in a virtual machine without passing through +the whole Network Device into a guest machine in a virtualized scenario. +The virtual functions can be enabled in the host machine or the hypervisor with the respective physical function driver. + +For example, in a Linux* host machine, it is possible to enable a virtual function using the following command: + +.. code-block:: console + + modprobe ixgbe max_vfs=2,2 + +This command enables two Virtual Functions on each of Physical Function of the NIC, +with two physical ports in the PCI configuration space. +It is important to note that enabled Virtual Function 0 and 2 would belong to Physical Function 0 +and Virtual Function 1 and 3 would belong to Physical Function 1, +in this case enabling a total of four Virtual Functions. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l2fwd`` sub-directory. + +Running the Application +----------------------- + +The application requires a number of command line options: + +.. code-block:: console + + ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating + +where, + +* p PORTMASK: A hexadecimal bitmask of the ports to configure + +* q NQ: A number of queues (=ports) per lcore (default is 1) + +* --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default). + +To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address +updating enabled, issue the command: + +.. code-block:: console + + $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff + +Refer to the *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. + +.. _l2_fwd_app_cmd_arguments: + +Command Line Arguments +~~~~~~~~~~~~~~~~~~~~~~ + +The L2 Forwarding sample application takes specific parameters, +in addition to Environment Abstraction Layer (EAL) arguments. +The preferred way to parse parameters is to use the getopt() function, +since it is part of a well-defined and portable library. + +The parsing of arguments is done in the l2fwd_parse_args() function. +The method of argument parsing is not described here. +Refer to the *glibc getopt(3)* man page for details. + +EAL arguments are parsed first, then application-specific arguments. +This is done at the beginning of the main() function: + +.. code-block:: c + + /* init EAL */ + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); + + argc -= ret; + argv += ret; + + /* parse application arguments (after the EAL ones) */ + + ret = l2fwd_parse_args(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid L2FWD arguments\n"); + +.. _l2_fwd_app_mbuf_init: + +Mbuf Pool Initialization +~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the arguments are parsed, the mbuf pool is created. +The mbuf pool contains a set of mbuf objects that will be used by the driver +and the application to store network packet data: + +.. code-block:: c + + /* create the mbuf pool */ + + l2fwd_pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NB_MBUF, + MEMPOOL_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, + rte_socket_id()); + + if (l2fwd_pktmbuf_pool == NULL) + rte_panic("Cannot init mbuf pool\n"); + +The rte_mempool is a generic structure used to handle pools of objects. +In this case, it is necessary to create a pool that will be used by the driver. +The number of allocated pkt mbufs is NB_MBUF, with a data room size of +RTE_MBUF_DEFAULT_BUF_SIZE each. +A per-lcore cache of 32 mbufs is kept. +The memory is allocated in NUMA socket 0, +but it is possible to extend this code to allocate one mbuf pool per socket. + +The rte_pktmbuf_pool_create() function uses the default mbuf pool and mbuf +initializers, respectively rte_pktmbuf_pool_init() and rte_pktmbuf_init(). +An advanced application may want to use the mempool API to create the +mbuf pool with more control. + +.. _l2_fwd_app_dvr_init: + +Driver Initialization +~~~~~~~~~~~~~~~~~~~~~ + +The main part of the code in the main() function relates to the initialization of the driver. +To fully understand this code, it is recommended to study the chapters that related to the Poll Mode Driver +in the *DPDK Programmer's Guide* - Rel 1.4 EAR and the *DPDK API Reference*. + +.. code-block:: c + + /* reset l2fwd_dst_ports */ + + for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) + l2fwd_dst_ports[portid] = 0; + + last_port = 0; + + /* + * Each logical core is assigned a dedicated TX queue on each port. + */ + + RTE_ETH_FOREACH_DEV(portid) { + /* skip ports that are not enabled */ + + if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) + continue; + + if (nb_ports_in_mask % 2) { + l2fwd_dst_ports[portid] = last_port; + l2fwd_dst_ports[last_port] = portid; + } + else + last_port = portid; + + nb_ports_in_mask++; + + rte_eth_dev_info_get((uint8_t) portid, &dev_info); + } + +The next step is to configure the RX and TX queues. +For each port, there is only one RX queue (only one lcore is able to poll a given port). +The number of TX queues depends on the number of available lcores. +The rte_eth_dev_configure() function is used to configure the number of queues for a port: + +.. code-block:: c + + ret = rte_eth_dev_configure((uint8_t)portid, 1, 1, &port_conf); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Cannot configure device: " + "err=%d, port=%u\n", + ret, portid); + +.. _l2_fwd_app_rx_init: + +RX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +The application uses one lcore to poll one or several ports, depending on the -q option, +which specifies the number of queues per lcore. + +For example, if the user specifies -q 4, the application is able to poll four ports with one lcore. +If there are 16 ports on the target (and if the portmask argument is -p ffff ), +the application will need four lcores to poll all the ports. + +.. code-block:: c + + ret = rte_eth_rx_queue_setup((uint8_t) portid, 0, nb_rxd, SOCKET0, &rx_conf, l2fwd_pktmbuf_pool); + if (ret < 0) + + rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup: " + "err=%d, port=%u\n", + ret, portid); + +The list of queues that must be polled for a given lcore is stored in a private structure called struct lcore_queue_conf. + +.. code-block:: c + + struct lcore_queue_conf { + unsigned n_rx_port; + unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE]; + struct mbuf_table tx_mbufs[L2FWD_MAX_PORTS]; + } rte_cache_aligned; + + struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE]; + +The values n_rx_port and rx_port_list[] are used in the main packet processing loop +(see :ref:`l2_fwd_app_rx_tx_packets`). + +.. _l2_fwd_app_tx_init: + +TX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +Each lcore should be able to transmit on any port. For every port, a single TX queue is initialized. + +.. code-block:: c + + /* init one TX queue on each port */ + + fflush(stdout); + + ret = rte_eth_tx_queue_setup((uint8_t) portid, 0, nb_txd, rte_eth_dev_socket_id(portid), &tx_conf); + if (ret < 0) + rte_exit(EXIT_FAILURE, "rte_eth_tx_queue_setup:err=%d, port=%u\n", ret, (unsigned) portid); + +The global configuration for TX queues is stored in a static structure: + +.. code-block:: c + + static const struct rte_eth_txconf tx_conf = { + .tx_thresh = { + .pthresh = TX_PTHRESH, + .hthresh = TX_HTHRESH, + .wthresh = TX_WTHRESH, + }, + .tx_free_thresh = RTE_TEST_TX_DESC_DEFAULT + 1, /* disable feature */ + }; + +.. _l2_fwd_app_rx_tx_packets: + +Receive, Process and Transmit Packets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the l2fwd_main_loop() function, the main task is to read ingress packets from the RX queues. +This is done using the following code: + +.. code-block:: c + + /* + * Read packet from RX queues + */ + + for (i = 0; i < qconf->n_rx_port; i++) { + portid = qconf->rx_port_list[i]; + nb_rx = rte_eth_rx_burst((uint8_t) portid, 0, pkts_burst, MAX_PKT_BURST); + + for (j = 0; j < nb_rx; j++) { + m = pkts_burst[j]; + rte_prefetch0[rte_pktmbuf_mtod(m, void *)); l2fwd_simple_forward(m, portid); + } + } + +Packets are read in a burst of size MAX_PKT_BURST. +The rte_eth_rx_burst() function writes the mbuf pointers in a local table and returns the number of available mbufs in the table. + +Then, each mbuf in the table is processed by the l2fwd_simple_forward() function. +The processing is very simple: process the TX port from the RX port, then replace the source and destination MAC addresses if MAC +addresses updating is enabled. + +.. note:: + + In the following code, one line for getting the output port requires some explanation. + +During the initialization process, a static array of destination ports (l2fwd_dst_ports[]) is filled such that for each source port, +a destination port is assigned that is either the next or previous enabled port from the portmask. +Naturally, the number of ports in the portmask must be even, otherwise, the application exits. + +.. code-block:: c + + static void + l2fwd_simple_forward(struct rte_mbuf *m, unsigned portid) + { + struct rte_ether_hdr *eth; + void *tmp; + unsigned dst_port; + + dst_port = l2fwd_dst_ports[portid]; + + eth = rte_pktmbuf_mtod(m, struct rte_ether_hdr *); + + /* 02:00:00:00:00:xx */ + + tmp = ð->d_addr.addr_bytes[0]; + + *((uint64_t *)tmp) = 0x000000000002 + ((uint64_t) dst_port << 40); + + /* src addr */ + + rte_ether_addr_copy(&l2fwd_ports_eth_addr[dst_port], ð->s_addr); + + l2fwd_send_packet(m, (uint8_t) dst_port); + } + +Then, the packet is sent using the l2fwd_send_packet (m, dst_port) function. +For this test application, the processing is exactly the same for all packets arriving on the same RX port. +Therefore, it would have been possible to call the l2fwd_send_burst() function directly from the main loop +to send all the received packets on the same TX port, +using the burst-oriented send function, which is more efficient. + +However, in real-life applications (such as, L3 routing), +packet N is not necessarily forwarded on the same port as packet N-1. +The application is implemented to illustrate that, so the same approach can be reused in a more complex application. + +The l2fwd_send_packet() function stores the packet in a per-lcore and per-txport table. +If the table is full, the whole packets table is transmitted using the l2fwd_send_burst() function: + +.. code-block:: c + + /* Send the packet on an output interface */ + + static int + l2fwd_send_packet(struct rte_mbuf *m, uint16_t port) + { + unsigned lcore_id, len; + struct lcore_queue_conf *qconf; + + lcore_id = rte_lcore_id(); + qconf = &lcore_queue_conf[lcore_id]; + len = qconf->tx_mbufs[port].len; + qconf->tx_mbufs[port].m_table[len] = m; + len++; + + /* enough pkts to be sent */ + + if (unlikely(len == MAX_PKT_BURST)) { + l2fwd_send_burst(qconf, MAX_PKT_BURST, port); + len = 0; + } + + qconf->tx_mbufs[port].len = len; return 0; + } + +To ensure that no packets remain in the tables, each lcore does a draining of TX queue in its main loop. +This technique introduces some latency when there are not many packets to send, +however it improves performance: + +.. code-block:: c + + cur_tsc = rte_rdtsc(); + + /* + * TX burst queue drain + */ + + diff_tsc = cur_tsc - prev_tsc; + + if (unlikely(diff_tsc > drain_tsc)) { + for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) { + if (qconf->tx_mbufs[portid].len == 0) + continue; + + l2fwd_send_burst(&lcore_queue_conf[lcore_id], qconf->tx_mbufs[portid].len, (uint8_t) portid); + + qconf->tx_mbufs[portid].len = 0; + } + + /* if timer is enabled */ + + if (timer_period > 0) { + /* advance the timer */ + + timer_tsc += diff_tsc; + + /* if timer has reached its timeout */ + + if (unlikely(timer_tsc >= (uint64_t) timer_period)) { + /* do this only on master core */ + + if (lcore_id == rte_get_master_lcore()) { + print_stats(); + + /* reset the timer */ + timer_tsc = 0; + } + } + } + + prev_tsc = cur_tsc; + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward.rst new file mode 100644 index 000000000..07c8d4493 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward.rst @@ -0,0 +1,384 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +L3 Forwarding Sample Application +================================ + +The L3 Forwarding application is a simple example of packet processing using +DPDK to demonstrate usage of poll and event mode packet I/O mechanism. +The application performs L3 forwarding. + +Overview +-------- + +The application demonstrates the use of the hash and LPM libraries in the DPDK +to implement packet forwarding using poll or event mode PMDs for packet I/O. +The initialization and run-time paths are very similar to those of the +:doc:`l2_forward_real_virtual` and :doc:`l2_forward_event`. +The main difference from the L2 Forwarding sample application is that optionally +packet can be Rx/Tx from/to eventdev instead of port directly and forwarding +decision is made based on information read from the input packet. + +Eventdev can optionally use S/W or H/W (if supported by platform) scheduler +implementation for packet I/O based on run time parameters. + +The lookup method is either hash-based or LPM-based and is selected at run time. When the selected lookup method is hash-based, +a hash object is used to emulate the flow classification stage. +The hash object is used in correlation with a flow table to map each input packet to its flow at runtime. + +The hash lookup key is represented by a DiffServ 5-tuple composed of the following fields read from the input packet: +Source IP Address, Destination IP Address, Protocol, Source Port and Destination Port. +The ID of the output interface for the input packet is read from the identified flow table entry. +The set of flows used by the application is statically configured and loaded into the hash at initialization time. +When the selected lookup method is LPM based, an LPM object is used to emulate the forwarding stage for IPv4 packets. +The LPM object is used as the routing table to identify the next hop for each input packet at runtime. + +The LPM lookup key is represented by the Destination IP Address field read from the input packet. +The ID of the output interface for the input packet is the next hop returned by the LPM lookup. +The set of LPM rules used by the application is statically configured and loaded into the LPM object at initialization time. + +In the sample application, hash-based forwarding supports IPv4 and IPv6. LPM-based forwarding supports IPv4 only. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l3fwd`` sub-directory. + +Running the Application +----------------------- + +The application has a number of command line options:: + + ./l3fwd [EAL options] -- -p PORTMASK + [-P] + [-E] + [-L] + --config(port,queue,lcore)[,(port,queue,lcore)] + [--eth-dest=X,MM:MM:MM:MM:MM:MM] + [--enable-jumbo [--max-pkt-len PKTLEN]] + [--no-numa] + [--hash-entry-num] + [--ipv6] + [--parse-ptype] + [--per-port-pool] + [--mode] + [--eventq-sched] + [--event-eth-rxqs] + +Where, + +* ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure + +* ``-P:`` Optional, sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address. + Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted. + +* ``-E:`` Optional, enable exact match. + +* ``-L:`` Optional, enable longest prefix match. + +* ``--config (port,queue,lcore)[,(port,queue,lcore)]:`` Determines which queues from which ports are mapped to which cores. + +* ``--eth-dest=X,MM:MM:MM:MM:MM:MM:`` Optional, ethernet destination for port X. + +* ``--enable-jumbo:`` Optional, enables jumbo frames. + +* ``--max-pkt-len:`` Optional, under the premise of enabling jumbo, maximum packet length in decimal (64-9600). + +* ``--no-numa:`` Optional, disables numa awareness. + +* ``--hash-entry-num:`` Optional, specifies the hash entry number in hexadecimal to be setup. + +* ``--ipv6:`` Optional, set if running ipv6 packets. + +* ``--parse-ptype:`` Optional, set to use software to analyze packet type. Without this option, hardware will check the packet type. + +* ``--per-port-pool:`` Optional, set to use independent buffer pools per port. Without this option, single buffer pool is used for all ports. + +* ``--mode:`` Optional, Packet transfer mode for I/O, poll or eventdev. + +* ``--eventq-sched:`` Optional, Event queue synchronization method, Ordered, Atomic or Parallel. Only valid if --mode=eventdev. + +* ``--event-eth-rxqs:`` Optional, Number of ethernet RX queues per device. Only valid if --mode=eventdev. + + +For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0, +while cores 8-15 and 24-31 appear on socket 1. + +To enable L3 forwarding between two ports, assuming that both ports are in the same socket, using two cores, cores 1 and 2, +(which are in the same socket too), use the following command: + +.. code-block:: console + + ./build/l3fwd -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)" + +In this command: + +* The -l option enables cores 1, 2 + +* The -p option enables ports 0 and 1 + +* The --config option enables one queue on each port and maps each (port,queue) pair to a specific core. + The following table shows the mapping in this example: + ++----------+-----------+-----------+-------------------------------------+ +| **Port** | **Queue** | **lcore** | **Description** | +| | | | | ++----------+-----------+-----------+-------------------------------------+ +| 0 | 0 | 1 | Map queue 0 from port 0 to lcore 1. | +| | | | | ++----------+-----------+-----------+-------------------------------------+ +| 1 | 0 | 2 | Map queue 0 from port 1 to lcore 2. | +| | | | | ++----------+-----------+-----------+-------------------------------------+ + +To use eventdev mode with sync method **ordered** on above mentioned environment, +Following is the sample command: + +.. code-block:: console + + ./build/l3fwd -l 0-3 -n 4 -w <event device> -- -p 0x3 --eventq-sched=ordered + +or + +.. code-block:: console + + ./build/l3fwd -l 0-3 -n 4 -w <event device> -- -p 0x03 --mode=eventdev --eventq-sched=ordered + +In this command: + +* -w option whitelist the event device supported by platform. Way to pass this device may vary based on platform. + +* The --mode option defines PMD to be used for packet I/O. + +* The --eventq-sched option enables synchronization menthod of event queue so that packets will be scheduled accordingly. + +If application uses S/W scheduler, it uses following DPDK services: + +* Software scheduler +* Rx adapter service function +* Tx adapter service function + +Application needs service cores to run above mentioned services. Service cores +must be provided as EAL parameters along with the --vdev=event_sw0 to enable S/W +scheduler. Following is the sample command: + +.. code-block:: console + + ./build/l3fwd -l 0-7 -s 0xf0000 -n 4 --vdev event_sw0 -- -p 0x3 --mode=eventdev --eventq-sched=ordered + +In case of eventdev mode, *--config* option is not used for ethernet port +configuration. Instead each ethernet port will be configured with mentioned +setup: + +* Single Rx/Tx queue + +* Each Rx queue will be connected to event queue via Rx adapter. + +* Each Tx queue will be connected via Tx adapter. + +Refer to the *DPDK Getting Started Guide* for general information on running applications and +the Environment Abstraction Layer (EAL) options. + +.. _l3_fwd_explanation: + +Explanation +----------- + +The following sections provide some explanation of the sample application code. As mentioned in the overview section, +the initialization and run-time paths are very similar to those of the :doc:`l2_forward_real_virtual` and :doc:`l2_forward_event`. +The following sections describe aspects that are specific to the L3 Forwarding sample application. + +Hash Initialization +~~~~~~~~~~~~~~~~~~~ + +The hash object is created and loaded with the pre-configured entries read from a global array, +and then generate the expected 5-tuple as key to keep consistence with those of real flow +for the convenience to execute hash performance test on 4M/8M/16M flows. + +.. note:: + + The Hash initialization will setup both ipv4 and ipv6 hash table, + and populate the either table depending on the value of variable ipv6. + To support the hash performance test with up to 8M single direction flows/16M bi-direction flows, + populate_ipv4_many_flow_into_table() function will populate the hash table with specified hash table entry number(default 4M). + +.. note:: + + Value of global variable ipv6 can be specified with --ipv6 in the command line. + Value of global variable hash_entry_number, + which is used to specify the total hash entry number for all used ports in hash performance test, + can be specified with --hash-entry-num VALUE in command line, being its default value 4. + +.. code-block:: c + + #if (APP_LOOKUP_METHOD == APP_LOOKUP_EXACT_MATCH) + + static void + setup_hash(int socketid) + { + // ... + + if (hash_entry_number != HASH_ENTRY_NUMBER_DEFAULT) { + if (ipv6 == 0) { + /* populate the ipv4 hash */ + populate_ipv4_many_flow_into_table(ipv4_l3fwd_lookup_struct[socketid], hash_entry_number); + } else { + /* populate the ipv6 hash */ + populate_ipv6_many_flow_into_table( ipv6_l3fwd_lookup_struct[socketid], hash_entry_number); + } + } else + if (ipv6 == 0) { + /* populate the ipv4 hash */ + populate_ipv4_few_flow_into_table(ipv4_l3fwd_lookup_struct[socketid]); + } else { + /* populate the ipv6 hash */ + populate_ipv6_few_flow_into_table(ipv6_l3fwd_lookup_struct[socketid]); + } + } + } + #endif + +LPM Initialization +~~~~~~~~~~~~~~~~~~ + +The LPM object is created and loaded with the pre-configured entries read from a global array. + +.. code-block:: c + + #if (APP_LOOKUP_METHOD == APP_LOOKUP_LPM) + + static void + setup_lpm(int socketid) + { + unsigned i; + int ret; + char s[64]; + + /* create the LPM table */ + + snprintf(s, sizeof(s), "IPV4_L3FWD_LPM_%d", socketid); + + ipv4_l3fwd_lookup_struct[socketid] = rte_lpm_create(s, socketid, IPV4_L3FWD_LPM_MAX_RULES, 0); + + if (ipv4_l3fwd_lookup_struct[socketid] == NULL) + rte_exit(EXIT_FAILURE, "Unable to create the l3fwd LPM table" + " on socket %d\n", socketid); + + /* populate the LPM table */ + + for (i = 0; i < IPV4_L3FWD_NUM_ROUTES; i++) { + /* skip unused ports */ + + if ((1 << ipv4_l3fwd_route_array[i].if_out & enabled_port_mask) == 0) + continue; + + ret = rte_lpm_add(ipv4_l3fwd_lookup_struct[socketid], ipv4_l3fwd_route_array[i].ip, + ipv4_l3fwd_route_array[i].depth, ipv4_l3fwd_route_array[i].if_out); + + if (ret < 0) { + rte_exit(EXIT_FAILURE, "Unable to add entry %u to the " + "l3fwd LPM table on socket %d\n", i, socketid); + } + + printf("LPM: Adding route 0x%08x / %d (%d)\n", + (unsigned)ipv4_l3fwd_route_array[i].ip, ipv4_l3fwd_route_array[i].depth, ipv4_l3fwd_route_array[i].if_out); + } + } + #endif + +Packet Forwarding for Hash-based Lookups +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For each input packet, the packet forwarding operation is done by the l3fwd_simple_forward() +or simple_ipv4_fwd_4pkts() function for IPv4 packets or the simple_ipv6_fwd_4pkts() function for IPv6 packets. +The l3fwd_simple_forward() function provides the basic functionality for both IPv4 and IPv6 packet forwarding +for any number of burst packets received, +and the packet forwarding decision (that is, the identification of the output interface for the packet) +for hash-based lookups is done by the get_ipv4_dst_port() or get_ipv6_dst_port() function. +The get_ipv4_dst_port() function is shown below: + +.. code-block:: c + + static inline uint8_t + get_ipv4_dst_port(void *ipv4_hdr, uint16_t portid, lookup_struct_t *ipv4_l3fwd_lookup_struct) + { + int ret = 0; + union ipv4_5tuple_host key; + + ipv4_hdr = (uint8_t *)ipv4_hdr + offsetof(struct rte_ipv4_hdr, time_to_live); + + m128i data = _mm_loadu_si128(( m128i*)(ipv4_hdr)); + + /* Get 5 tuple: dst port, src port, dst IP address, src IP address and protocol */ + + key.xmm = _mm_and_si128(data, mask0); + + /* Find destination port */ + + ret = rte_hash_lookup(ipv4_l3fwd_lookup_struct, (const void *)&key); + + return (uint8_t)((ret < 0)? portid : ipv4_l3fwd_out_if[ret]); + } + +The get_ipv6_dst_port() function is similar to the get_ipv4_dst_port() function. + +The simple_ipv4_fwd_4pkts() and simple_ipv6_fwd_4pkts() function are optimized for continuous 4 valid ipv4 and ipv6 packets, +they leverage the multiple buffer optimization to boost the performance of forwarding packets with the exact match on hash table. +The key code snippet of simple_ipv4_fwd_4pkts() is shown below: + +.. code-block:: c + + static inline void + simple_ipv4_fwd_4pkts(struct rte_mbuf* m[4], uint16_t portid, struct lcore_conf *qconf) + { + // ... + + data[0] = _mm_loadu_si128(( m128i*)(rte_pktmbuf_mtod(m[0], unsigned char *) + sizeof(struct rte_ether_hdr) + offsetof(struct rte_ipv4_hdr, time_to_live))); + data[1] = _mm_loadu_si128(( m128i*)(rte_pktmbuf_mtod(m[1], unsigned char *) + sizeof(struct rte_ether_hdr) + offsetof(struct rte_ipv4_hdr, time_to_live))); + data[2] = _mm_loadu_si128(( m128i*)(rte_pktmbuf_mtod(m[2], unsigned char *) + sizeof(struct rte_ether_hdr) + offsetof(struct rte_ipv4_hdr, time_to_live))); + data[3] = _mm_loadu_si128(( m128i*)(rte_pktmbuf_mtod(m[3], unsigned char *) + sizeof(struct rte_ether_hdr) + offsetof(struct rte_ipv4_hdr, time_to_live))); + + key[0].xmm = _mm_and_si128(data[0], mask0); + key[1].xmm = _mm_and_si128(data[1], mask0); + key[2].xmm = _mm_and_si128(data[2], mask0); + key[3].xmm = _mm_and_si128(data[3], mask0); + + const void *key_array[4] = {&key[0], &key[1], &key[2],&key[3]}; + + rte_hash_lookup_bulk(qconf->ipv4_lookup_struct, &key_array[0], 4, ret); + + dst_port[0] = (ret[0] < 0)? portid:ipv4_l3fwd_out_if[ret[0]]; + dst_port[1] = (ret[1] < 0)? portid:ipv4_l3fwd_out_if[ret[1]]; + dst_port[2] = (ret[2] < 0)? portid:ipv4_l3fwd_out_if[ret[2]]; + dst_port[3] = (ret[3] < 0)? portid:ipv4_l3fwd_out_if[ret[3]]; + + // ... + } + +The simple_ipv6_fwd_4pkts() function is similar to the simple_ipv4_fwd_4pkts() function. + +Known issue: IP packets with extensions or IP packets which are not TCP/UDP cannot work well at this mode. + +Packet Forwarding for LPM-based Lookups +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For each input packet, the packet forwarding operation is done by the l3fwd_simple_forward() function, +but the packet forwarding decision (that is, the identification of the output interface for the packet) +for LPM-based lookups is done by the get_ipv4_dst_port() function below: + +.. code-block:: c + + static inline uint16_t + get_ipv4_dst_port(struct rte_ipv4_hdr *ipv4_hdr, uint16_t portid, lookup_struct_t *ipv4_l3fwd_lookup_struct) + { + uint8_t next_hop; + + return ((rte_lpm_lookup(ipv4_l3fwd_lookup_struct, rte_be_to_cpu_32(ipv4_hdr->dst_addr), &next_hop) == 0)? next_hop : portid); + } + +Eventdev Driver Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Eventdev driver initialization is same as L2 forwarding eventdev application. +Refer :doc:`l2_forward_event` for more details. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_access_ctrl.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_access_ctrl.rst new file mode 100644 index 000000000..a44fbcd52 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_access_ctrl.rst @@ -0,0 +1,339 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +L3 Forwarding with Access Control Sample Application +==================================================== + +The L3 Forwarding with Access Control application is a simple example of packet processing using the DPDK. +The application performs a security check on received packets. +Packets that are in the Access Control List (ACL), which is loaded during initialization, are dropped. +Others are forwarded to the correct port. + +Overview +-------- + +The application demonstrates the use of the ACL library in the DPDK to implement access control +and packet L3 forwarding. +The application loads two types of rules at initialization: + +* Route information rules, which are used for L3 forwarding + +* Access Control List (ACL) rules that blacklist (or block) packets with a specific characteristic + +When packets are received from a port, +the application extracts the necessary information from the TCP/IP header of the received packet and +performs a lookup in the rule database to figure out whether the packets should be dropped (in the ACL range) +or forwarded to desired ports. +The initialization and run-time paths are similar to those of the :doc:`l3_forward`. +However, there are significant differences in the two applications. +For example, the original L3 forwarding application uses either LPM or +an exact match algorithm to perform forwarding port lookup, +while this application uses the ACL library to perform both ACL and route entry lookup. +The following sections provide more detail. + +Classification for both IPv4 and IPv6 packets is supported in this application. +The application also assumes that all the packets it processes are TCP/UDP packets and +always extracts source/destination port information from the packets. + +Tuple Packet Syntax +~~~~~~~~~~~~~~~~~~~ + +The application implements packet classification for the IPv4/IPv6 5-tuple syntax specifically. +The 5-tuple syntax consist of a source IP address, a destination IP address, +a source port, a destination port and a protocol identifier. +The fields in the 5-tuple syntax have the following formats: + +* **Source IP address and destination IP address** + : Each is either a 32-bit field (for IPv4), or a set of 4 32-bit fields (for IPv6) represented by a value and a mask length. + For example, an IPv4 range of 192.168.1.0 to 192.168.1.255 could be represented by a value = [192, 168, 1, 0] and a mask length = 24. + +* **Source port and destination port** + : Each is a 16-bit field, represented by a lower start and a higher end. + For example, a range of ports 0 to 8192 could be represented by lower = 0 and higher = 8192. + +* **Protocol identifier** + : An 8-bit field, represented by a value and a mask, that covers a range of values. + To verify that a value is in the range, use the following expression: "(VAL & mask) == value" + +The trick in how to represent a range with a mask and value is as follows. +A range can be enumerated in binary numbers with some bits that are never changed and some bits that are dynamically changed. +Set those bits that dynamically changed in mask and value with 0. +Set those bits that never changed in the mask with 1, in value with number expected. +For example, a range of 6 to 7 is enumerated as 0b110 and 0b111. +Bit 1-7 are bits never changed and bit 0 is the bit dynamically changed. +Therefore, set bit 0 in mask and value with 0, set bits 1-7 in mask with 1, and bits 1-7 in value with number 0b11. +So, mask is 0xfe, value is 0x6. + +.. note:: + + The library assumes that each field in the rule is in LSB or Little Endian order when creating the database. + It internally converts them to MSB or Big Endian order. + When performing a lookup, the library assumes the input is in MSB or Big Endian order. + +Access Rule Syntax +~~~~~~~~~~~~~~~~~~ + +In this sample application, each rule is a combination of the following: + +* 5-tuple field: This field has a format described in Section. + +* priority field: A weight to measure the priority of the rules. + The rule with the higher priority will ALWAYS be returned if the specific input has multiple matches in the rule database. + Rules with lower priority will NEVER be returned in any cases. + +* userdata field: A user-defined field that could be any value. + It can be the forwarding port number if the rule is a route table entry or it can be a pointer to a mapping address + if the rule is used for address mapping in the NAT application. + The key point is that it is a useful reserved field for user convenience. + +ACL and Route Rules +~~~~~~~~~~~~~~~~~~~ + +The application needs to acquire ACL and route rules before it runs. +Route rules are mandatory, while ACL rules are optional. +To simplify the complexity of the priority field for each rule, all ACL and route entries are assumed to be in the same file. +To read data from the specified file successfully, the application assumes the following: + +* Each rule occupies a single line. + +* Only the following four rule line types are valid in this application: + +* ACL rule line, which starts with a leading character '@' + +* Route rule line, which starts with a leading character 'R' + +* Comment line, which starts with a leading character '#' + +* Empty line, which consists of a space, form-feed ('\f'), newline ('\n'), + carriage return ('\r'), horizontal tab ('\t'), or vertical tab ('\v'). + +Other lines types are considered invalid. + +* Rules are organized in descending order of priority, + which means rules at the head of the file always have a higher priority than those further down in the file. + +* A typical IPv4 ACL rule line should have a format as shown below: + + +.. _figure_ipv4_acl_rule: + +.. figure:: img/ipv4_acl_rule.* + + A typical IPv4 ACL rule + + +IPv4 addresses are specified in CIDR format as specified in RFC 4632. +They consist of the dot notation for the address and a prefix length separated by '/'. +For example, 192.168.0.34/32, where the address is 192.168.0.34 and the prefix length is 32. + +Ports are specified as a range of 16-bit numbers in the format MIN:MAX, +where MIN and MAX are the inclusive minimum and maximum values of the range. +The range 0:65535 represents all possible ports in a range. +When MIN and MAX are the same value, a single port is represented, for example, 20:20. + +The protocol identifier is an 8-bit value and a mask separated by '/'. +For example: 6/0xfe matches protocol values 6 and 7. + +* Route rules start with a leading character 'R' and have the same format as ACL rules except an extra field at the tail + that indicates the forwarding port number. + +Rules File Example +~~~~~~~~~~~~~~~~~~ + +.. _figure_example_rules: + +.. figure:: img/example_rules.* + + Rules example + + +Each rule is explained as follows: + +* Rule 1 (the first line) tells the application to drop those packets with source IP address = [1.2.3.*], + destination IP address = [192.168.0.36], protocol = [6]/[7] + +* Rule 2 (the second line) is similar to Rule 1, except the source IP address is ignored. + It tells the application to forward packets with destination IP address = [192.168.0.36], + protocol = [6]/[7], destined to port 1. + +* Rule 3 (the third line) tells the application to forward all packets to port 0. + This is something like a default route entry. + +As described earlier, the application assume rules are listed in descending order of priority, +therefore Rule 1 has the highest priority, then Rule 2, and finally, +Rule 3 has the lowest priority. + +Consider the arrival of the following three packets: + +* Packet 1 has source IP address = [1.2.3.4], destination IP address = [192.168.0.36], and protocol = [6] + +* Packet 2 has source IP address = [1.2.4.4], destination IP address = [192.168.0.36], and protocol = [6] + +* Packet 3 has source IP address = [1.2.3.4], destination IP address = [192.168.0.36], and protocol = [8] + +Observe that: + +* Packet 1 matches all of the rules + +* Packet 2 matches Rule 2 and Rule 3 + +* Packet 3 only matches Rule 3 + +For priority reasons, Packet 1 matches Rule 1 and is dropped. +Packet 2 matches Rule 2 and is forwarded to port 1. +Packet 3 matches Rule 3 and is forwarded to port 0. + +For more details on the rule file format, +please refer to rule_ipv4.db and rule_ipv6.db files (inside <RTE_SDK>/examples/l3fwd-acl/). + +Application Phases +~~~~~~~~~~~~~~~~~~ + +Once the application starts, it transitions through three phases: + +* **Initialization Phase** + - Perform the following tasks: + +* Parse command parameters. Check the validity of rule file(s) name(s), number of logical cores, receive and transmit queues. + Bind ports, queues and logical cores. Check ACL search options, and so on. + +* Call Environmental Abstraction Layer (EAL) and Poll Mode Driver (PMD) functions to initialize the environment and detect possible NICs. + The EAL creates several threads and sets affinity to a specific hardware thread CPU based on the configuration specified + by the command line arguments. + +* Read the rule files and format the rules into the representation that the ACL library can recognize. + Call the ACL library function to add the rules into the database and compile them as a trie of pattern sets. + Note that application maintains a separate AC contexts for IPv4 and IPv6 rules. + +* **Runtime Phase** + - Process the incoming packets from a port. Packets are processed in three steps: + + * Retrieval: Gets a packet from the receive queue. Each logical core may process several queues for different ports. + This depends on the configuration specified by command line arguments. + + * Lookup: Checks that the packet type is supported (IPv4/IPv6) and performs a 5-tuple lookup over corresponding AC context. + If an ACL rule is matched, the packets will be dropped and return back to step 1. + If a route rule is matched, it indicates the packet is not in the ACL list and should be forwarded. + If there is no matches for the packet, then the packet is dropped. + + * Forwarding: Forwards the packet to the corresponding port. + +* **Final Phase** - Perform the following tasks: + + Calls the EAL, PMD driver and ACL library to free resource, then quits. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l3fwd-acl`` sub-directory. + +Running the Application +----------------------- + +The application has a number of command line options: + +.. code-block:: console + + ./build/l3fwd-acl [EAL options] -- -p PORTMASK [-P] --config(port,queue,lcore)[,(port,queue,lcore)] --rule_ipv4 FILENAME rule_ipv6 FILENAME [--scalar] [--enable-jumbo [--max-pkt-len PKTLEN]] [--no-numa] + + +where, + +* -p PORTMASK: Hexadecimal bitmask of ports to configure + +* -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address. + Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted. + +* --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores + +* --rule_ipv4 FILENAME: Specifies the IPv4 ACL and route rules file + +* --rule_ipv6 FILENAME: Specifies the IPv6 ACL and route rules file + +* --scalar: Use a scalar function to perform rule lookup + +* --enable-jumbo: optional, enables jumbo frames + +* --max-pkt-len: optional, maximum packet length in decimal (64-9600) + +* --no-numa: optional, disables numa awareness + +For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0, +while cores 8-15 and 24-31 appear on socket 1. + +To enable L3 forwarding between two ports, assuming that both ports are in the same socket, using two cores, cores 1 and 2, +(which are in the same socket too), use the following command: + +.. code-block:: console + + ./build/l3fwd-acl -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)" --rule_ipv4="./rule_ipv4.db" -- rule_ipv6="./rule_ipv6.db" --scalar + +In this command: + +* The -l option enables cores 1, 2 + +* The -p option enables ports 0 and 1 + +* The --config option enables one queue on each port and maps each (port,queue) pair to a specific core. + The following table shows the mapping in this example: + + +----------+------------+-----------+-------------------------------------+ + | **Port** | **Queue** | **lcore** | **Description** | + | | | | | + +==========+============+===========+=====================================+ + | 0 | 0 | 1 | Map queue 0 from port 0 to lcore 1. | + | | | | | + +----------+------------+-----------+-------------------------------------+ + | 1 | 0 | 2 | Map queue 0 from port 1 to lcore 2. | + | | | | | + +----------+------------+-----------+-------------------------------------+ + +* The --rule_ipv4 option specifies the reading of IPv4 rules sets from the ./ rule_ipv4.db file. + +* The --rule_ipv6 option specifies the reading of IPv6 rules sets from the ./ rule_ipv6.db file. + +* The --scalar option specifies the performing of rule lookup with a scalar function. + +Explanation +----------- + +The following sections provide some explanation of the sample application code. +The aspects of port, device and CPU configuration are similar to those of the :doc:`l3_forward`. +The following sections describe aspects that are specific to L3 forwarding with access control. + +Parse Rules from File +~~~~~~~~~~~~~~~~~~~~~ + +As described earlier, both ACL and route rules are assumed to be saved in the same file. +The application parses the rules from the file and adds them to the database by calling the ACL library function. +It ignores empty and comment lines, and parses and validates the rules it reads. +If errors are detected, the application exits with messages to identify the errors encountered. + +The application needs to consider the userdata and priority fields. +The ACL rules save the index to the specific rules in the userdata field, +while route rules save the forwarding port number. +In order to differentiate the two types of rules, ACL rules add a signature in the userdata field. +As for the priority field, the application assumes rules are organized in descending order of priority. +Therefore, the code only decreases the priority number with each rule it parses. + +Setting Up the ACL Context +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For each supported AC rule format (IPv4 5-tuple, IPv6 6-tuple) application creates a separate context handler +from the ACL library for each CPU socket on the board and adds parsed rules into that context. + +Note, that for each supported rule type, +application needs to calculate the expected offset of the fields from the start of the packet. +That's why only packets with fixed IPv4/ IPv6 header are supported. +That allows to perform ACL classify straight over incoming packet buffer - +no extra protocol field retrieval need to be performed. + +Subsequently, the application checks whether NUMA is enabled. +If it is, the application records the socket IDs of the CPU cores involved in the task. + +Finally, the application creates contexts handler from the ACL library, +adds rules parsed from the file into the database and build an ACL trie. +It is important to note that the application creates an independent copy of each database for each socket CPU +involved in the task to reduce the time for remote memory access. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_graph.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_graph.rst new file mode 100644 index 000000000..df50827ba --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_graph.rst @@ -0,0 +1,334 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2020 Marvell International Ltd. + +L3 Forwarding Graph Sample Application +====================================== + +The L3 Forwarding Graph application is a simple example of packet processing +using the DPDK Graph framework. The application performs L3 forwarding using +Graph framework and nodes written for graph framework. + +Overview +-------- + +The application demonstrates the use of the graph framework and graph nodes +``ethdev_rx``, ``ip4_lookup``, ``ip4_rewrite``, ``ethdev_tx`` and ``pkt_drop`` in DPDK to +implement packet forwarding. + +The initialization is very similar to those of the :doc:`l3_forward`. +There is also additional initialization of graph for graph object creation +and configuration per lcore. +Run-time path is main thing that differs from L3 forwarding sample application. +Difference is that forwarding logic starting from Rx, followed by LPM lookup, +TTL update and finally Tx is implemented inside graph nodes. These nodes are +interconnected in graph framework. Application main loop needs to walk over +graph using ``rte_graph_walk()`` with graph objects created one per slave lcore. + +The lookup method is as per implementation of ``ip4_lookup`` graph node. +The ID of the output interface for the input packet is the next hop returned by +the LPM lookup. The set of LPM rules used by the application is statically +configured and provided to ``ip4_lookup`` graph node and ``ip4_rewrite`` graph node +using node control API ``rte_node_ip4_route_add()`` and ``rte_node_ip4_rewrite_add()``. + +In the sample application, only IPv4 forwarding is supported as of now. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l3fwd-graph`` sub-directory. + +Running the Application +----------------------- + +The application has a number of command line options similar to l3fwd:: + + ./l3fwd-graph [EAL options] -- -p PORTMASK + [-P] + --config(port,queue,lcore)[,(port,queue,lcore)] + [--eth-dest=X,MM:MM:MM:MM:MM:MM] + [--enable-jumbo [--max-pkt-len PKTLEN]] + [--no-numa] + [--per-port-pool] + +Where, + +* ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure + +* ``-P:`` Optional, sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address. + Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted. + +* ``--config (port,queue,lcore)[,(port,queue,lcore)]:`` Determines which queues from which ports are mapped to which cores. + +* ``--eth-dest=X,MM:MM:MM:MM:MM:MM:`` Optional, ethernet destination for port X. + +* ``--enable-jumbo:`` Optional, enables jumbo frames. + +* ``--max-pkt-len:`` Optional, under the premise of enabling jumbo, maximum packet length in decimal (64-9600). + +* ``--no-numa:`` Optional, disables numa awareness. + +* ``--per-port-pool:`` Optional, set to use independent buffer pools per port. Without this option, single buffer pool is used for all ports. + +For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0, +while cores 8-15 and 24-31 appear on socket 1. + +To enable L3 forwarding between two ports, assuming that both ports are in the same socket, using two cores, cores 1 and 2, +(which are in the same socket too), use the following command: + +.. code-block:: console + + ./build/l3fwd-graph -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)" + +In this command: + +* The -l option enables cores 1, 2 + +* The -p option enables ports 0 and 1 + +* The --config option enables one queue on each port and maps each (port,queue) pair to a specific core. + The following table shows the mapping in this example: + ++----------+-----------+-----------+-------------------------------------+ +| **Port** | **Queue** | **lcore** | **Description** | +| | | | | ++----------+-----------+-----------+-------------------------------------+ +| 0 | 0 | 1 | Map queue 0 from port 0 to lcore 1. | +| | | | | ++----------+-----------+-----------+-------------------------------------+ +| 1 | 0 | 2 | Map queue 0 from port 1 to lcore 2. | +| | | | | ++----------+-----------+-----------+-------------------------------------+ + +Refer to the *DPDK Getting Started Guide* for general information on running applications and +the Environment Abstraction Layer (EAL) options. + +.. _l3_fwd_graph_explanation: + +Explanation +----------- + +The following sections provide some explanation of the sample application code. +As mentioned in the overview section, the initialization is similar to that of +the :doc:`l3_forward`. Run-time path though similar in functionality to that of +:doc:`l3_forward`, major part of the implementation is in graph nodes via used +via ``librte_node`` library. +The following sections describe aspects that are specific to the L3 Forwarding +Graph sample application. + +Graph Node Pre-Init Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After device configuration and device Rx, Tx queue setup is complete, +a minimal config of port id, num_rx_queues, num_tx_queues, mempools etc will +be passed to *ethdev_** node ctrl API ``rte_node_eth_config()``. This will be +lead to the clone of ``ethdev_rx`` and ``ethdev_tx`` nodes as ``ethdev_rx-X-Y`` and +``ethdev_tx-X`` where X, Y represent port id and queue id associated with them. +In case of ``ethdev_tx-X`` nodes, tx queue id assigned per instance of the node +is same as graph id. + +These cloned nodes along with existing static nodes such as ``ip4_lookup`` and +``ip4_rewrite`` will be used in graph creation to associate node's to lcore +specific graph object. + +.. code-block:: c + + RTE_ETH_FOREACH_DEV(portid) + { + + /* ... */ + ret = rte_eth_dev_configure(portid, nb_rx_queue, + n_tx_queue, &local_port_conf); + /* ... */ + + /* Init one TX queue per couple (lcore,port) */ + queueid = 0; + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + /* ... */ + ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd, + socketid, txconf); + /* ... */ + queueid++; + } + + /* Setup ethdev node config */ + ethdev_conf[nb_conf].port_id = portid; + ethdev_conf[nb_conf].num_rx_queues = nb_rx_queue; + ethdev_conf[nb_conf].num_tx_queues = n_tx_queue; + if (!per_port_pool) + ethdev_conf[nb_conf].mp = pktmbuf_pool[0]; + else + ethdev_conf[nb_conf].mp = pktmbuf_pool[portid]; + ethdev_conf[nb_conf].mp_count = NB_SOCKETS; + + nb_conf++; + printf("\n"); + } + + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + /* Init RX queues */ + for (queue = 0; queue < qconf->n_rx_queue; ++queue) { + /* ... */ + if (!per_port_pool) + ret = rte_eth_rx_queue_setup(portid, queueid, nb_rxd, socketid, + &rxq_conf, pktmbuf_pool[0][socketid]); + else + ret = rte_eth_rx_queue_setup(portid, queueid, nb_rxd, socketid, + &rxq_conf, pktmbuf_pool[portid][socketid]); + /* ... */ + } + } + + /* Ethdev node config, skip rx queue mapping */ + ret = rte_node_eth_config(ethdev_conf, nb_conf, nb_graphs); + +Graph Initialization +~~~~~~~~~~~~~~~~~~~~ + +Now a graph needs to be created with a specific set of nodes for every lcore. +A graph object returned after graph creation is a per lcore object and +cannot be shared between lcores. Since ``ethdev_tx-X`` node is per port node, +it can be associated with all the graphs created as all the lcores should have +Tx capability for every port. But ``ethdev_rx-X-Y`` node is created per +(port, rx_queue_id), so they should be associated with a graph based on +the application argument ``--config`` specifying rx queue mapping to lcore. + +.. note:: + + The Graph creation will fail if the passed set of shell node pattern's + are not sufficient to meet their inter-dependency or even one node is not + found with a given regex node pattern. + +.. code-block:: c + + static const char *const default_patterns[] = { + "ip4*", + "ethdev_tx-*", + "pkt_drop", + }; + const char **node_patterns; + uint16_t nb_pattern; + + /* ... */ + + /* Create a graph object per lcore with common nodes and + * lcore specific nodes based on application arguments + */ + nb_patterns = RTE_DIM(default_patterns); + node_patterns = malloc((MAX_RX_QUEUE_PER_LCORE + nb_patterns) * + sizeof(*node_patterns)); + memcpy(node_patterns, default_patterns, + nb_patterns * sizeof(*node_patterns)); + + memset(&graph_conf, 0, sizeof(graph_conf)); + + /* Common set of nodes in every lcore's graph object */ + graph_conf.node_patterns = node_patterns; + + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + /* ... */ + + /* Skip graph creation if no source exists */ + if (!qconf->n_rx_queue) + continue; + + /* Add rx node patterns of this lcore based on --config */ + for (i = 0; i < qconf->n_rx_queue; i++) { + graph_conf.node_patterns[nb_patterns + i] = + qconf->rx_queue_list[i].node_name; + } + + graph_conf.nb_node_patterns = nb_patterns + i; + graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id); + + snprintf(qconf->name, sizeof(qconf->name), "worker_%u", lcore_id); + + graph_id = rte_graph_create(qconf->name, &graph_conf); + + /* ... */ + + qconf->graph = rte_graph_lookup(qconf->name); + + /* ... */ + } + +Forwarding data(Route, Next-Hop) addition +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once graph objects are created, node specific info like routes and rewrite +headers will be provided run-time using ``rte_node_ip4_route_add()`` and +``rte_node_ip4_rewrite_add()`` API. + +.. note:: + + Since currently ``ip4_lookup`` and ``ip4_rewrite`` nodes don't support + lock-less mechanisms(RCU, etc) to add run-time forwarding data like route and + rewrite data, forwarding data is added before packet processing loop is + launched on slave lcore. + +.. code-block:: c + + /* Add route to ip4 graph infra */ + for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) { + /* ... */ + + dst_port = ipv4_l3fwd_lpm_route_array[i].if_out; + next_hop = i; + + /* ... */ + ret = rte_node_ip4_route_add(ipv4_l3fwd_lpm_route_array[i].ip, + ipv4_l3fwd_lpm_route_array[i].depth, next_hop, + RTE_NODE_IP4_LOOKUP_NEXT_REWRITE); + + /* ... */ + + memcpy(rewrite_data, val_eth + dst_port, rewrite_len); + + /* Add next hop for a given destination */ + ret = rte_node_ip4_rewrite_add(next_hop, rewrite_data, + rewrite_len, dst_port); + + RTE_LOG(INFO, L3FWD_GRAPH, "Added route %s, next_hop %u\n", + route_str, next_hop); + } + +Packet Forwarding using Graph Walk +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now that all the device configurations are done, graph creations are done and +forwarding data is updated with nodes, slave lcores will be launched with graph +main loop. Graph main loop is very simple in the sense that it needs to +continuously call a non-blocking API ``rte_graph_walk()`` with it's lcore +specific graph object that was already created. + +.. note:: + + rte_graph_walk() will walk over all the sources nodes i.e ``ethdev_rx-X-Y`` + associated with a given graph and Rx the available packets and enqueue them + to the following node ``ip4_lookup`` which then will enqueue them to ``ip4_rewrite`` + node if LPM lookup succeeds. ``ip4_rewrite`` node then will update Ethernet header + as per next-hop data and transmit the packet via port 'Z' by enqueuing + to ``ethdev_tx-Z`` node instance in its graph object. + +.. code-block:: c + + /* Main processing loop */ + static int + graph_main_loop(void *conf) + { + // ... + + lcore_id = rte_lcore_id(); + qconf = &lcore_conf[lcore_id]; + graph = qconf->graph; + + RTE_LOG(INFO, L3FWD_GRAPH, + "Entering main loop on lcore %u, graph %s(%p)\n", lcore_id, + qconf->name, graph); + + /* Walk over graph until signal to quit */ + while (likely(!force_quit)) + rte_graph_walk(graph); + return 0; + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_power_man.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_power_man.rst new file mode 100644 index 000000000..0cc6f2e62 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_power_man.rst @@ -0,0 +1,461 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +L3 Forwarding with Power Management Sample Application +====================================================== + +Introduction +------------ + +The L3 Forwarding with Power Management application is an example of power-aware packet processing using the DPDK. +The application is based on existing L3 Forwarding sample application, +with the power management algorithms to control the P-states and +C-states of the Intel processor via a power management library. + +Overview +-------- + +The application demonstrates the use of the Power libraries in the DPDK to implement packet forwarding. +The initialization and run-time paths are very similar to those of the :doc:`l3_forward`. +The main difference from the L3 Forwarding sample application is that this application introduces power-aware optimization algorithms +by leveraging the Power library to control P-state and C-state of processor based on packet load. + +The DPDK includes poll-mode drivers to configure Intel NIC devices and their receive (Rx) and transmit (Tx) queues. +The design principle of this PMD is to access the Rx and Tx descriptors directly without any interrupts to quickly receive, +process and deliver packets in the user space. + +In general, the DPDK executes an endless packet processing loop on dedicated IA cores that include the following steps: + +* Retrieve input packets through the PMD to poll Rx queue + +* Process each received packet or provide received packets to other processing cores through software queues + +* Send pending output packets to Tx queue through the PMD + +In this way, the PMD achieves better performance than a traditional interrupt-mode driver, +at the cost of keeping cores active and running at the highest frequency, +hence consuming the maximum power all the time. +However, during the period of processing light network traffic, +which happens regularly in communication infrastructure systems due to well-known "tidal effect", +the PMD is still busy waiting for network packets, which wastes a lot of power. + +Processor performance states (P-states) are the capability of an Intel processor +to switch between different supported operating frequencies and voltages. +If configured correctly, according to system workload, this feature provides power savings. +CPUFreq is the infrastructure provided by the Linux* kernel to control the processor performance state capability. +CPUFreq supports a user space governor that enables setting frequency via manipulating the virtual file device from a user space application. +The Power library in the DPDK provides a set of APIs for manipulating a virtual file device to allow user space application +to set the CPUFreq governor and set the frequency of specific cores. + +This application includes a P-state power management algorithm to generate a frequency hint to be sent to CPUFreq. +The algorithm uses the number of received and available Rx packets on recent polls to make a heuristic decision to scale frequency up/down. +Specifically, some thresholds are checked to see whether a specific core running an DPDK polling thread needs to increase frequency +a step up based on the near to full trend of polled Rx queues. +Also, it decreases frequency a step if packet processed per loop is far less than the expected threshold +or the thread's sleeping time exceeds a threshold. + +C-States are also known as sleep states. +They allow software to put an Intel core into a low power idle state from which it is possible to exit via an event, such as an interrupt. +However, there is a tradeoff between the power consumed in the idle state and the time required to wake up from the idle state (exit latency). +Therefore, as you go into deeper C-states, the power consumed is lower but the exit latency is increased. Each C-state has a target residency. +It is essential that when entering into a C-state, the core remains in this C-state for at least as long as the target residency in order +to fully realize the benefits of entering the C-state. +CPUIdle is the infrastructure provide by the Linux kernel to control the processor C-state capability. +Unlike CPUFreq, CPUIdle does not provide a mechanism that allows the application to change C-state. +It actually has its own heuristic algorithms in kernel space to select target C-state to enter by executing privileged instructions like HLT and MWAIT, +based on the speculative sleep duration of the core. +In this application, we introduce a heuristic algorithm that allows packet processing cores to sleep for a short period +if there is no Rx packet received on recent polls. +In this way, CPUIdle automatically forces the corresponding cores to enter deeper C-states +instead of always running to the C0 state waiting for packets. + +.. note:: + + To fully demonstrate the power saving capability of using C-states, + it is recommended to enable deeper C3 and C6 states in the BIOS during system boot up. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``l3fwd-power`` sub-directory. + +Running the Application +----------------------- + +The application has a number of command line options: + +.. code-block:: console + + ./build/l3fwd_power [EAL options] -- -p PORTMASK [-P] --config(port,queue,lcore)[,(port,queue,lcore)] [--enable-jumbo [--max-pkt-len PKTLEN]] [--no-numa] + +where, + +* -p PORTMASK: Hexadecimal bitmask of ports to configure + +* -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address. + Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted. + +* --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores. + +* --enable-jumbo: optional, enables jumbo frames + +* --max-pkt-len: optional, maximum packet length in decimal (64-9600) + +* --no-numa: optional, disables numa awareness + +* --empty-poll: Traffic Aware power management. See below for details + +* --telemetry: Telemetry mode. + +See :doc:`l3_forward` for details. +The L3fwd-power example reuses the L3fwd command line options. + +Explanation +----------- + +The following sections provide some explanation of the sample application code. +As mentioned in the overview section, +the initialization and run-time paths are identical to those of the L3 forwarding application. +The following sections describe aspects that are specific to the L3 Forwarding with Power Management sample application. + +Power Library Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Power library is initialized in the main routine. +It changes the P-state governor to userspace for specific cores that are under control. +The Timer library is also initialized and several timers are created later on, +responsible for checking if it needs to scale down frequency at run time by checking CPU utilization statistics. + +.. note:: + + Only the power management related initialization is shown. + +.. code-block:: c + + int main(int argc, char **argv) + { + struct lcore_conf *qconf; + int ret; + unsigned nb_ports; + uint16_t queueid, portid; + unsigned lcore_id; + uint64_t hz; + uint32_t n_tx_queue, nb_lcores; + uint8_t nb_rx_queue, queue, socketid; + + // ... + + /* init RTE timer library to be used to initialize per-core timers */ + + rte_timer_subsystem_init(); + + // ... + + + /* per-core initialization */ + + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + if (rte_lcore_is_enabled(lcore_id) == 0) + continue; + + /* init power management library for a specified core */ + + ret = rte_power_init(lcore_id); + if (ret) + rte_exit(EXIT_FAILURE, "Power management library " + "initialization failed on core%d\n", lcore_id); + + /* init timer structures for each enabled lcore */ + + rte_timer_init(&power_timers[lcore_id]); + + hz = rte_get_hpet_hz(); + + rte_timer_reset(&power_timers[lcore_id], hz/TIMER_NUMBER_PER_SECOND, SINGLE, lcore_id, power_timer_cb, NULL); + + // ... + } + + // ... + } + +Monitoring Loads of Rx Queues +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In general, the polling nature of the DPDK prevents the OS power management subsystem from knowing +if the network load is actually heavy or light. +In this sample, sampling network load work is done by monitoring received and +available descriptors on NIC Rx queues in recent polls. +Based on the number of returned and available Rx descriptors, +this example implements algorithms to generate frequency scaling hints and speculative sleep duration, +and use them to control P-state and C-state of processors via the power management library. +Frequency (P-state) control and sleep state (C-state) control work individually for each logical core, +and the combination of them contributes to a power efficient packet processing solution when serving light network loads. + +The rte_eth_rx_burst() function and the newly-added rte_eth_rx_queue_count() function are used in the endless packet processing loop +to return the number of received and available Rx descriptors. +And those numbers of specific queue are passed to P-state and C-state heuristic algorithms +to generate hints based on recent network load trends. + +.. note:: + + Only power control related code is shown. + +.. code-block:: c + + static + __rte_noreturn int main_loop(__rte_unused void *dummy) + { + // ... + + while (1) { + // ... + + /** + * Read packet from RX queues + */ + + lcore_scaleup_hint = FREQ_CURRENT; + lcore_rx_idle_count = 0; + + for (i = 0; i < qconf->n_rx_queue; ++i) + { + rx_queue = &(qconf->rx_queue_list[i]); + rx_queue->idle_hint = 0; + portid = rx_queue->port_id; + queueid = rx_queue->queue_id; + + nb_rx = rte_eth_rx_burst(portid, queueid, pkts_burst, MAX_PKT_BURST); + stats[lcore_id].nb_rx_processed += nb_rx; + + if (unlikely(nb_rx == 0)) { + /** + * no packet received from rx queue, try to + * sleep for a while forcing CPU enter deeper + * C states. + */ + + rx_queue->zero_rx_packet_count++; + + if (rx_queue->zero_rx_packet_count <= MIN_ZERO_POLL_COUNT) + continue; + + rx_queue->idle_hint = power_idle_heuristic(rx_queue->zero_rx_packet_count); + lcore_rx_idle_count++; + } else { + rx_ring_length = rte_eth_rx_queue_count(portid, queueid); + + rx_queue->zero_rx_packet_count = 0; + + /** + * do not scale up frequency immediately as + * user to kernel space communication is costly + * which might impact packet I/O for received + * packets. + */ + + rx_queue->freq_up_hint = power_freq_scaleup_heuristic(lcore_id, rx_ring_length); + } + + /* Prefetch and forward packets */ + + // ... + } + + if (likely(lcore_rx_idle_count != qconf->n_rx_queue)) { + for (i = 1, lcore_scaleup_hint = qconf->rx_queue_list[0].freq_up_hint; i < qconf->n_rx_queue; ++i) { + x_queue = &(qconf->rx_queue_list[i]); + + if (rx_queue->freq_up_hint > lcore_scaleup_hint) + + lcore_scaleup_hint = rx_queue->freq_up_hint; + } + + if (lcore_scaleup_hint == FREQ_HIGHEST) + + rte_power_freq_max(lcore_id); + + else if (lcore_scaleup_hint == FREQ_HIGHER) + rte_power_freq_up(lcore_id); + } else { + /** + * All Rx queues empty in recent consecutive polls, + * sleep in a conservative manner, meaning sleep as + * less as possible. + */ + + for (i = 1, lcore_idle_hint = qconf->rx_queue_list[0].idle_hint; i < qconf->n_rx_queue; ++i) { + rx_queue = &(qconf->rx_queue_list[i]); + if (rx_queue->idle_hint < lcore_idle_hint) + lcore_idle_hint = rx_queue->idle_hint; + } + + if ( lcore_idle_hint < SLEEP_GEAR1_THRESHOLD) + /** + * execute "pause" instruction to avoid context + * switch for short sleep. + */ + rte_delay_us(lcore_idle_hint); + else + /* long sleep force ruining thread to suspend */ + usleep(lcore_idle_hint); + + stats[lcore_id].sleep_time += lcore_idle_hint; + } + } + } + +P-State Heuristic Algorithm +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The power_freq_scaleup_heuristic() function is responsible for generating a frequency hint for the specified logical core +according to available descriptor number returned from rte_eth_rx_queue_count(). +On every poll for new packets, the length of available descriptor on an Rx queue is evaluated, +and the algorithm used for frequency hinting is as follows: + +* If the size of available descriptors exceeds 96, the maximum frequency is hinted. + +* If the size of available descriptors exceeds 64, a trend counter is incremented by 100. + +* If the length of the ring exceeds 32, the trend counter is incremented by 1. + +* When the trend counter reached 10000 the frequency hint is changed to the next higher frequency. + +.. note:: + + The assumption is that the Rx queue size is 128 and the thresholds specified above + must be adjusted accordingly based on actual hardware Rx queue size, + which are configured via the rte_eth_rx_queue_setup() function. + +In general, a thread needs to poll packets from multiple Rx queues. +Most likely, different queue have different load, so they would return different frequency hints. +The algorithm evaluates all the hints and then scales up frequency in an aggressive manner +by scaling up to highest frequency as long as one Rx queue requires. +In this way, we can minimize any negative performance impact. + +On the other hand, frequency scaling down is controlled in the timer callback function. +Specifically, if the sleep times of a logical core indicate that it is sleeping more than 25% of the sampling period, +or if the average packet per iteration is less than expectation, the frequency is decreased by one step. + +C-State Heuristic Algorithm +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Whenever recent rte_eth_rx_burst() polls return 5 consecutive zero packets, +an idle counter begins incrementing for each successive zero poll. +At the same time, the function power_idle_heuristic() is called to generate speculative sleep duration +in order to force logical to enter deeper sleeping C-state. +There is no way to control C- state directly, and the CPUIdle subsystem in OS is intelligent enough +to select C-state to enter based on actual sleep period time of giving logical core. +The algorithm has the following sleeping behavior depending on the idle counter: + +* If idle count less than 100, the counter value is used as a microsecond sleep value through rte_delay_us() + which execute pause instructions to avoid costly context switch but saving power at the same time. + +* If idle count is between 100 and 999, a fixed sleep interval of 100 μs is used. + A 100 μs sleep interval allows the core to enter the C1 state while keeping a fast response time in case new traffic arrives. + +* If idle count is greater than 1000, a fixed sleep value of 1 ms is used until the next timer expiration is used. + This allows the core to enter the C3/C6 states. + +.. note:: + + The thresholds specified above need to be adjusted for different Intel processors and traffic profiles. + +If a thread polls multiple Rx queues and different queue returns different sleep duration values, +the algorithm controls the sleep time in a conservative manner by sleeping for the least possible time +in order to avoid a potential performance impact. + +Empty Poll Mode +------------------------- +Additionally, there is a traffic aware mode of operation called "Empty +Poll" where the number of empty polls can be monitored to keep track +of how busy the application is. Empty poll mode can be enabled by the +command line option --empty-poll. + +See :doc:`Power Management<../prog_guide/power_man>` chapter in the DPDK Programmer's Guide for empty poll mode details. + +.. code-block:: console + + ./l3fwd-power -l xxx -n 4 -w 0000:xx:00.0 -w 0000:xx:00.1 -- -p 0x3 -P --config="(0,0,xx),(1,0,xx)" --empty-poll="0,0,0" -l 14 -m 9 -h 1 + +Where, + +--empty-poll: Enable the empty poll mode instead of original algorithm + +--empty-poll="training_flag, med_threshold, high_threshold" + +* ``training_flag`` : optional, enable/disable training mode. Default value is 0. If the training_flag is set as 1(true), then the application will start in training mode and print out the trained threshold values. If the training_flag is set as 0(false), the application will start in normal mode, and will use either the default thresholds or those supplied on the command line. The trained threshold values are specific to the user’s system, may give a better power profile when compared to the default threshold values. + +* ``med_threshold`` : optional, sets the empty poll threshold of a modestly busy system state. If this is not supplied, the application will apply the default value of 350000. + +* ``high_threshold`` : optional, sets the empty poll threshold of a busy system state. If this is not supplied, the application will apply the default value of 580000. + +* -l : optional, set up the LOW power state frequency index + +* -m : optional, set up the MED power state frequency index + +* -h : optional, set up the HIGH power state frequency index + +Empty Poll Mode Example Usage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To initially obtain the ideal thresholds for the system, the training +mode should be run first. This is achieved by running the l3fwd-power +app with the training flag set to “1”, and the other parameters set to +0. + +.. code-block:: console + + ./examples/l3fwd-power/build/l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --empty-poll "1,0,0" –P + +This will run the training algorithm for x seconds on each core (cores 2 +and 3), and then print out the recommended threshold values for those +cores. The thresholds should be very similar for each core. + +.. code-block:: console + + POWER: Bring up the Timer + POWER: set the power freq to MED + POWER: Low threshold is 230277 + POWER: MED threshold is 335071 + POWER: HIGH threshold is 523769 + POWER: Training is Complete for 2 + POWER: set the power freq to MED + POWER: Low threshold is 236814 + POWER: MED threshold is 344567 + POWER: HIGH threshold is 538580 + POWER: Training is Complete for 3 + +Once the values have been measured for a particular system, the app can +then be started without the training mode so traffic can start immediately. + +.. code-block:: console + + ./examples/l3fwd-power/build/l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --empty-poll "0,340000,540000" –P + +Telemetry Mode +-------------- + +The telemetry mode support for ``l3fwd-power`` is a standalone mode, in this mode +``l3fwd-power`` does simple l3fwding along with calculating empty polls, full polls, +and busy percentage for each forwarding core. The aggregation of these +values of all cores is reported as application level telemetry to metric +library for every 500ms from the master core. + +The busy percentage is calculated by recording the poll_count +and when the count reaches a defined value the total +cycles it took is measured and compared with minimum and maximum +reference cycles and accordingly busy rate is set to either 0% or +50% or 100%. + + .. Note:: + + * The CONFIG_RTE_LIBRTE_TELEMETRY should be set in order to get the stats in DPDK telemetry. + +.. code-block:: console + + ./examples/l3fwd-power/build/l3fwd-power --telemetry -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --telemetry + +The new stats ``empty_poll`` , ``full_poll`` and ``busy_percent`` can be viewed by running the script +``/usertools/dpdk-telemetry-client.py`` and selecting the menu option ``Send for global Metrics``. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/link_status_intr.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/link_status_intr.rst new file mode 100644 index 000000000..04c40f285 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/link_status_intr.rst @@ -0,0 +1,415 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Link Status Interrupt Sample Application +======================================== + +The Link Status Interrupt sample application is a simple example of packet processing using +the Data Plane Development Kit (DPDK) that +demonstrates how network link status changes for a network port can be captured and +used by a DPDK application. + +Overview +-------- + +The Link Status Interrupt sample application registers a user space callback for the link status interrupt of each port +and performs L2 forwarding for each packet that is received on an RX_PORT. +The following operations are performed: + +* RX_PORT and TX_PORT are paired with available ports one-by-one according to the core mask + +* The source MAC address is replaced by the TX_PORT MAC address + +* The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID + +This application can be used to demonstrate the usage of link status interrupt and its user space callbacks +and the behavior of L2 forwarding each time the link status changes. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``link_status_interrupt`` sub-directory. + +Running the Application +----------------------- + +The application requires a number of command line options: + +.. code-block:: console + + ./build/link_status_interrupt [EAL options] -- -p PORTMASK [-q NQ][-T PERIOD] + +where, + +* -p PORTMASK: A hexadecimal bitmask of the ports to configure + +* -q NQ: A number of queues (=ports) per lcore (default is 1) + +* -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default) + +To run the application in a linux environment with 4 lcores, 4 memory channels, 16 ports and 8 RX queues per lcore, +issue the command: + +.. code-block:: console + + $ ./build/link_status_interrupt -l 0-3 -n 4-- -q 8 -p ffff + +Refer to the *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. + +Command Line Arguments +~~~~~~~~~~~~~~~~~~~~~~ + +The Link Status Interrupt sample application takes specific parameters, +in addition to Environment Abstraction Layer (EAL) arguments (see Section `Running the Application`_). + +Command line parsing is done in the same way as it is done in the L2 Forwarding Sample Application. +See :ref:`l2_fwd_app_cmd_arguments` for more information. + +Mbuf Pool Initialization +~~~~~~~~~~~~~~~~~~~~~~~~ + +Mbuf pool initialization is done in the same way as it is done in the L2 Forwarding Sample Application. +See :ref:`l2_fwd_app_mbuf_init` for more information. + +Driver Initialization +~~~~~~~~~~~~~~~~~~~~~ + +The main part of the code in the main() function relates to the initialization of the driver. +To fully understand this code, it is recommended to study the chapters that related to the Poll Mode Driver in the +*DPDK Programmer's Guide and the DPDK API Reference*. + +.. code-block:: c + + /* + * Each logical core is assigned a dedicated TX queue on each port. + */ + + RTE_ETH_FOREACH_DEV(portid) { + /* skip ports that are not enabled */ + + if ((lsi_enabled_port_mask & (1 << portid)) == 0) + continue; + + /* save the destination port id */ + + if (nb_ports_in_mask % 2) { + lsi_dst_ports[portid] = portid_last; + lsi_dst_ports[portid_last] = portid; + } + else + portid_last = portid; + + nb_ports_in_mask++; + + rte_eth_dev_info_get((uint8_t) portid, &dev_info); + } + +The next step is to configure the RX and TX queues. +For each port, there is only one RX queue (only one lcore is able to poll a given port). +The number of TX queues depends on the number of available lcores. +The rte_eth_dev_configure() function is used to configure the number of queues for a port: + +.. code-block:: c + + ret = rte_eth_dev_configure((uint8_t) portid, 1, 1, &port_conf); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Cannot configure device: err=%d, port=%u\n", ret, portid); + +The global configuration is stored in a static structure: + +.. code-block:: c + + static const struct rte_eth_conf port_conf = { + .rxmode = { + .split_hdr_size = 0, + }, + .txmode = {}, + .intr_conf = { + .lsc = 1, /**< link status interrupt feature enabled */ + }, + }; + +Configuring lsc to 0 (the default) disables the generation of any link status change interrupts in kernel space +and no user space interrupt event is received. +The public interface rte_eth_link_get() accesses the NIC registers directly to update the link status. +Configuring lsc to non-zero enables the generation of link status change interrupts in kernel space +when a link status change is present and calls the user space callbacks registered by the application. +The public interface rte_eth_link_get() just reads the link status in a global structure +that would be updated in the interrupt host thread only. + +Interrupt Callback Registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The application can register one or more callbacks to a specific port and interrupt event. +An example callback function that has been written as indicated below. + +.. code-block:: c + + static void + lsi_event_callback(uint16_t port_id, enum rte_eth_event_type type, void *param) + { + struct rte_eth_link link; + int ret; + + RTE_SET_USED(param); + + printf("\n\nIn registered callback...\n"); + + printf("Event type: %s\n", type == RTE_ETH_EVENT_INTR_LSC ? "LSC interrupt" : "unknown event"); + + ret = rte_eth_link_get_nowait(port_id, &link); + if (ret < 0) { + printf("Failed to get port %d link status: %s\n\n", + port_id, rte_strerror(-ret)); + } else if (link.link_status) { + printf("Port %d Link Up - speed %u Mbps - %s\n\n", port_id, (unsigned)link.link_speed, + (link.link_duplex == ETH_LINK_FULL_DUPLEX) ? ("full-duplex") : ("half-duplex")); + } else + printf("Port %d Link Down\n\n", port_id); + } + +This function is called when a link status interrupt is present for the right port. +The port_id indicates which port the interrupt applies to. +The type parameter identifies the interrupt event type, +which currently can be RTE_ETH_EVENT_INTR_LSC only, but other types can be added in the future. +The param parameter is the address of the parameter for the callback. +This function should be implemented with care since it will be called in the interrupt host thread, +which is different from the main thread of its caller. + +The application registers the lsi_event_callback and a NULL parameter to the link status interrupt event on each port: + +.. code-block:: c + + rte_eth_dev_callback_register((uint8_t)portid, RTE_ETH_EVENT_INTR_LSC, lsi_event_callback, NULL); + +This registration can be done only after calling the rte_eth_dev_configure() function and before calling any other function. +If lsc is initialized with 0, the callback is never called since no interrupt event would ever be present. + +RX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +The application uses one lcore to poll one or several ports, depending on the -q option, +which specifies the number of queues per lcore. + +For example, if the user specifies -q 4, the application is able to poll four ports with one lcore. +If there are 16 ports on the target (and if the portmask argument is -p ffff), +the application will need four lcores to poll all the ports. + +.. code-block:: c + + ret = rte_eth_rx_queue_setup((uint8_t) portid, 0, nb_rxd, SOCKET0, &rx_conf, lsi_pktmbuf_pool); + if (ret < 0) + rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup: err=%d, port=%u\n", ret, portid); + +The list of queues that must be polled for a given lcore is stored in a private structure called struct lcore_queue_conf. + +.. code-block:: c + + struct lcore_queue_conf { + unsigned n_rx_port; + unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE]; unsigned tx_queue_id; + struct mbuf_table tx_mbufs[LSI_MAX_PORTS]; + } rte_cache_aligned; + + struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE]; + +The n_rx_port and rx_port_list[] fields are used in the main packet processing loop +(see `Receive, Process and Transmit Packets`_). + +The global configuration for the RX queues is stored in a static structure: + +.. code-block:: c + + static const struct rte_eth_rxconf rx_conf = { + .rx_thresh = { + .pthresh = RX_PTHRESH, + .hthresh = RX_HTHRESH, + .wthresh = RX_WTHRESH, + }, + }; + +TX Queue Initialization +~~~~~~~~~~~~~~~~~~~~~~~ + +Each lcore should be able to transmit on any port. +For every port, a single TX queue is initialized. + +.. code-block:: c + + /* init one TX queue logical core on each port */ + + fflush(stdout); + + ret = rte_eth_tx_queue_setup(portid, 0, nb_txd, rte_eth_dev_socket_id(portid), &tx_conf); + if (ret < 0) + rte_exit(EXIT_FAILURE, "rte_eth_tx_queue_setup: err=%d,port=%u\n", ret, (unsigned) portid); + +The global configuration for TX queues is stored in a static structure: + +.. code-block:: c + + static const struct rte_eth_txconf tx_conf = { + .tx_thresh = { + .pthresh = TX_PTHRESH, + .hthresh = TX_HTHRESH, + .wthresh = TX_WTHRESH, + }, + .tx_free_thresh = RTE_TEST_TX_DESC_DEFAULT + 1, /* disable feature */ + }; + +Receive, Process and Transmit Packets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the lsi_main_loop() function, the main task is to read ingress packets from the RX queues. +This is done using the following code: + +.. code-block:: c + + /* + * Read packet from RX queues + */ + + for (i = 0; i < qconf->n_rx_port; i++) { + portid = qconf->rx_port_list[i]; + nb_rx = rte_eth_rx_burst((uint8_t) portid, 0, pkts_burst, MAX_PKT_BURST); + port_statistics[portid].rx += nb_rx; + + for (j = 0; j < nb_rx; j++) { + m = pkts_burst[j]; + rte_prefetch0(rte_pktmbuf_mtod(m, void *)); + lsi_simple_forward(m, portid); + } + } + +Packets are read in a burst of size MAX_PKT_BURST. +The rte_eth_rx_burst() function writes the mbuf pointers in a local table and returns the number of available mbufs in the table. + +Then, each mbuf in the table is processed by the lsi_simple_forward() function. +The processing is very simple: processes the TX port from the RX port and then replaces the source and destination MAC addresses. + +.. note:: + + In the following code, the two lines for calculating the output port require some explanation. + If portId is even, the first line does nothing (as portid & 1 will be 0), and the second line adds 1. + If portId is odd, the first line subtracts one and the second line does nothing. + Therefore, 0 goes to 1, and 1 to 0, 2 goes to 3 and 3 to 2, and so on. + +.. code-block:: c + + static void + lsi_simple_forward(struct rte_mbuf *m, unsigned portid) + { + struct rte_ether_hdr *eth; + void *tmp; + unsigned dst_port = lsi_dst_ports[portid]; + + eth = rte_pktmbuf_mtod(m, struct rte_ether_hdr *); + + /* 02:00:00:00:00:xx */ + + tmp = ð->d_addr.addr_bytes[0]; + + *((uint64_t *)tmp) = 0x000000000002 + (dst_port << 40); + + /* src addr */ + rte_ether_addr_copy(&lsi_ports_eth_addr[dst_port], ð->s_addr); + + lsi_send_packet(m, dst_port); + } + +Then, the packet is sent using the lsi_send_packet(m, dst_port) function. +For this test application, the processing is exactly the same for all packets arriving on the same RX port. +Therefore, it would have been possible to call the lsi_send_burst() function directly from the main loop +to send all the received packets on the same TX port using +the burst-oriented send function, which is more efficient. + +However, in real-life applications (such as, L3 routing), +packet N is not necessarily forwarded on the same port as packet N-1. +The application is implemented to illustrate that so the same approach can be reused in a more complex application. + +The lsi_send_packet() function stores the packet in a per-lcore and per-txport table. +If the table is full, the whole packets table is transmitted using the lsi_send_burst() function: + +.. code-block:: c + + /* Send the packet on an output interface */ + + static int + lsi_send_packet(struct rte_mbuf *m, uint16_t port) + { + unsigned lcore_id, len; + struct lcore_queue_conf *qconf; + + lcore_id = rte_lcore_id(); + qconf = &lcore_queue_conf[lcore_id]; + len = qconf->tx_mbufs[port].len; + qconf->tx_mbufs[port].m_table[len] = m; + len++; + + /* enough pkts to be sent */ + + if (unlikely(len == MAX_PKT_BURST)) { + lsi_send_burst(qconf, MAX_PKT_BURST, port); + len = 0; + } + qconf->tx_mbufs[port].len = len; + + return 0; + } + +To ensure that no packets remain in the tables, each lcore does a draining of the TX queue in its main loop. +This technique introduces some latency when there are not many packets to send. +However, it improves performance: + +.. code-block:: c + + cur_tsc = rte_rdtsc(); + + /* + * TX burst queue drain + */ + + diff_tsc = cur_tsc - prev_tsc; + + if (unlikely(diff_tsc > drain_tsc)) { + /* this could be optimized (use queueid instead of * portid), but it is not called so often */ + + for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) { + if (qconf->tx_mbufs[portid].len == 0) + continue; + + lsi_send_burst(&lcore_queue_conf[lcore_id], + qconf->tx_mbufs[portid].len, (uint8_t) portid); + qconf->tx_mbufs[portid].len = 0; + } + + /* if timer is enabled */ + + if (timer_period > 0) { + /* advance the timer */ + + timer_tsc += diff_tsc; + + /* if timer has reached its timeout */ + + if (unlikely(timer_tsc >= (uint64_t) timer_period)) { + /* do this only on master core */ + + if (lcore_id == rte_get_master_lcore()) { + print_stats(); + + /* reset the timer */ + timer_tsc = 0; + } + } + } + prev_tsc = cur_tsc; + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/multi_process.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/multi_process.rst new file mode 100644 index 000000000..f2a79a639 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/multi_process.rst @@ -0,0 +1,323 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +.. _multi_process_app: + +Multi-process Sample Application +================================ + +This chapter describes the example applications for multi-processing that are included in the DPDK. + +Example Applications +-------------------- + +Building the Sample Applications +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The multi-process example applications are built in the same way as other sample applications, +and as documented in the *DPDK Getting Started Guide*. + + +To compile the sample application see :doc:`compiling`. + +The applications are located in the ``multi_process`` sub-directory. + +.. note:: + + If just a specific multi-process application needs to be built, + the final make command can be run just in that application's directory, + rather than at the top-level multi-process directory. + +Basic Multi-process Example +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The examples/simple_mp folder in the DPDK release contains a basic example application to demonstrate how +two DPDK processes can work together using queues and memory pools to share information. + +Running the Application +^^^^^^^^^^^^^^^^^^^^^^^ + +To run the application, start one copy of the simple_mp binary in one terminal, +passing at least two cores in the coremask/corelist, as follows: + +.. code-block:: console + + ./build/simple_mp -l 0-1 -n 4 --proc-type=primary + +For the first DPDK process run, the proc-type flag can be omitted or set to auto, +since all DPDK processes will default to being a primary instance, +meaning they have control over the hugepage shared memory regions. +The process should start successfully and display a command prompt as follows: + +.. code-block:: console + + $ ./build/simple_mp -l 0-1 -n 4 --proc-type=primary + EAL: coremask set to 3 + EAL: Detected lcore 0 on socket 0 + EAL: Detected lcore 1 on socket 0 + EAL: Detected lcore 2 on socket 0 + EAL: Detected lcore 3 on socket 0 + ... + + EAL: Requesting 2 pages of size 1073741824 + EAL: Requesting 768 pages of size 2097152 + EAL: Ask a virtual area of 0x40000000 bytes + EAL: Virtual area found at 0x7ff200000000 (size = 0x40000000) + ... + + EAL: check igb_uio module + EAL: check module finished + EAL: Master core 0 is ready (tid=54e41820) + EAL: Core 1 is ready (tid=53b32700) + + Starting core 1 + + simple_mp > + +To run the secondary process to communicate with the primary process, +again run the same binary setting at least two cores in the coremask/corelist: + +.. code-block:: console + + ./build/simple_mp -l 2-3 -n 4 --proc-type=secondary + +When running a secondary process such as that shown above, the proc-type parameter can again be specified as auto. +However, omitting the parameter altogether will cause the process to try and start as a primary rather than secondary process. + +Once the process type is specified correctly, +the process starts up, displaying largely similar status messages to the primary instance as it initializes. +Once again, you will be presented with a command prompt. + +Once both processes are running, messages can be sent between them using the send command. +At any stage, either process can be terminated using the quit command. + +.. code-block:: console + + EAL: Master core 10 is ready (tid=b5f89820) EAL: Master core 8 is ready (tid=864a3820) + EAL: Core 11 is ready (tid=84ffe700) EAL: Core 9 is ready (tid=85995700) + Starting core 11 Starting core 9 + simple_mp > send hello_secondary simple_mp > core 9: Received 'hello_secondary' + simple_mp > core 11: Received 'hello_primary' simple_mp > send hello_primary + simple_mp > quit simple_mp > quit + +.. note:: + + If the primary instance is terminated, the secondary instance must also be shut-down and restarted after the primary. + This is necessary because the primary instance will clear and reset the shared memory regions on startup, + invalidating the secondary process's pointers. + The secondary process can be stopped and restarted without affecting the primary process. + +How the Application Works +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The core of this example application is based on using two queues and a single memory pool in shared memory. +These three objects are created at startup by the primary process, +since the secondary process cannot create objects in memory as it cannot reserve memory zones, +and the secondary process then uses lookup functions to attach to these objects as it starts up. + +.. code-block:: c + + if (rte_eal_process_type() == RTE_PROC_PRIMARY){ + send_ring = rte_ring_create(_PRI_2_SEC, ring_size, SOCKET0, flags); + recv_ring = rte_ring_create(_SEC_2_PRI, ring_size, SOCKET0, flags); + message_pool = rte_mempool_create(_MSG_POOL, pool_size, string_size, pool_cache, priv_data_sz, NULL, NULL, NULL, NULL, SOCKET0, flags); + } else { + recv_ring = rte_ring_lookup(_PRI_2_SEC); + send_ring = rte_ring_lookup(_SEC_2_PRI); + message_pool = rte_mempool_lookup(_MSG_POOL); + } + +Note, however, that the named ring structure used as send_ring in the primary process is the recv_ring in the secondary process. + +Once the rings and memory pools are all available in both the primary and secondary processes, +the application simply dedicates two threads to sending and receiving messages respectively. +The receive thread simply dequeues any messages on the receive ring, prints them, +and frees the buffer space used by the messages back to the memory pool. +The send thread makes use of the command-prompt library to interactively request user input for messages to send. +Once a send command is issued by the user, a buffer is allocated from the memory pool, filled in with the message contents, +then enqueued on the appropriate rte_ring. + +Symmetric Multi-process Example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The second example of DPDK multi-process support demonstrates how a set of processes can run in parallel, +with each process performing the same set of packet- processing operations. +(Since each process is identical in functionality to the others, +we refer to this as symmetric multi-processing, to differentiate it from asymmetric multi- processing - +such as a client-server mode of operation seen in the next example, +where different processes perform different tasks, yet co-operate to form a packet-processing system.) +The following diagram shows the data-flow through the application, using two processes. + +.. _figure_sym_multi_proc_app: + +.. figure:: img/sym_multi_proc_app.* + + Example Data Flow in a Symmetric Multi-process Application + + +As the diagram shows, each process reads packets from each of the network ports in use. +RSS is used to distribute incoming packets on each port to different hardware RX queues. +Each process reads a different RX queue on each port and so does not contend with any other process for that queue access. +Similarly, each process writes outgoing packets to a different TX queue on each port. + +Running the Application +^^^^^^^^^^^^^^^^^^^^^^^ + +As with the simple_mp example, the first instance of the symmetric_mp process must be run as the primary instance, +though with a number of other application- specific parameters also provided after the EAL arguments. +These additional parameters are: + +* -p <portmask>, where portmask is a hexadecimal bitmask of what ports on the system are to be used. + For example: -p 3 to use ports 0 and 1 only. + +* --num-procs <N>, where N is the total number of symmetric_mp instances that will be run side-by-side to perform packet processing. + This parameter is used to configure the appropriate number of receive queues on each network port. + +* --proc-id <n>, where n is a numeric value in the range 0 <= n < N (number of processes, specified above). + This identifies which symmetric_mp instance is being run, so that each process can read a unique receive queue on each network port. + +The secondary symmetric_mp instances must also have these parameters specified, +and the first two must be the same as those passed to the primary instance, or errors result. + +For example, to run a set of four symmetric_mp instances, running on lcores 1-4, +all performing level-2 forwarding of packets between ports 0 and 1, +the following commands can be used (assuming run as root): + +.. code-block:: console + + # ./build/symmetric_mp -l 1 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=0 + # ./build/symmetric_mp -l 2 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=1 + # ./build/symmetric_mp -l 3 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=2 + # ./build/symmetric_mp -l 4 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=3 + +.. note:: + + In the above example, the process type can be explicitly specified as primary or secondary, rather than auto. + When using auto, the first process run creates all the memory structures needed for all processes - + irrespective of whether it has a proc-id of 0, 1, 2 or 3. + +.. note:: + + For the symmetric multi-process example, since all processes work in the same manner, + once the hugepage shared memory and the network ports are initialized, + it is not necessary to restart all processes if the primary instance dies. + Instead, that process can be restarted as a secondary, + by explicitly setting the proc-type to secondary on the command line. + (All subsequent instances launched will also need this explicitly specified, + as auto-detection will detect no primary processes running and therefore attempt to re-initialize shared memory.) + +How the Application Works +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The initialization calls in both the primary and secondary instances are the same for the most part, +calling the rte_eal_init(), 1 G and 10 G driver initialization and then probing devices. +Thereafter, the initialization done depends on whether the process is configured as a primary or secondary instance. + +In the primary instance, a memory pool is created for the packet mbufs and the network ports to be used are initialized - +the number of RX and TX queues per port being determined by the num-procs parameter passed on the command-line. +The structures for the initialized network ports are stored in shared memory and +therefore will be accessible by the secondary process as it initializes. + +.. code-block:: c + + if (num_ports & 1) + rte_exit(EXIT_FAILURE, "Application must use an even number of ports\n"); + + for(i = 0; i < num_ports; i++){ + if(proc_type == RTE_PROC_PRIMARY) + if (smp_port_init(ports[i], mp, (uint16_t)num_procs) < 0) + rte_exit(EXIT_FAILURE, "Error initializing ports\n"); + } + +In the secondary instance, rather than initializing the network ports, the port information exported by the primary process is used, +giving the secondary process access to the hardware and software rings for each network port. +Similarly, the memory pool of mbufs is accessed by doing a lookup for it by name: + +.. code-block:: c + + mp = (proc_type == RTE_PROC_SECONDARY) ? rte_mempool_lookup(_SMP_MBUF_POOL) : rte_mempool_create(_SMP_MBUF_POOL, NB_MBUFS, MBUF_SIZE, ... ) + +Once this initialization is complete, the main loop of each process, both primary and secondary, +is exactly the same - each process reads from each port using the queue corresponding to its proc-id parameter, +and writes to the corresponding transmit queue on the output port. + +Client-Server Multi-process Example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The third example multi-process application included with the DPDK shows how one can +use a client-server type multi-process design to do packet processing. +In this example, a single server process performs the packet reception from the ports being used and +distributes these packets using round-robin ordering among a set of client processes, +which perform the actual packet processing. +In this case, the client applications just perform level-2 forwarding of packets by sending each packet out on a different network port. + +The following diagram shows the data-flow through the application, using two client processes. + +.. _figure_client_svr_sym_multi_proc_app: + +.. figure:: img/client_svr_sym_multi_proc_app.* + + Example Data Flow in a Client-Server Symmetric Multi-process Application + + +Running the Application +^^^^^^^^^^^^^^^^^^^^^^^ + +The server process must be run initially as the primary process to set up all memory structures for use by the clients. +In addition to the EAL parameters, the application- specific parameters are: + +* -p <portmask >, where portmask is a hexadecimal bitmask of what ports on the system are to be used. + For example: -p 3 to use ports 0 and 1 only. + +* -n <num-clients>, where the num-clients parameter is the number of client processes that will process the packets received + by the server application. + +.. note:: + + In the server process, a single thread, the master thread, that is, the lowest numbered lcore in the coremask/corelist, performs all packet I/O. + If a coremask/corelist is specified with more than a single lcore bit set in it, + an additional lcore will be used for a thread to periodically print packet count statistics. + +Since the server application stores configuration data in shared memory, including the network ports to be used, +the only application parameter needed by a client process is its client instance ID. +Therefore, to run a server application on lcore 1 (with lcore 2 printing statistics) along with two client processes running on lcores 3 and 4, +the following commands could be used: + +.. code-block:: console + + # ./mp_server/build/mp_server -l 1-2 -n 4 -- -p 3 -n 2 + # ./mp_client/build/mp_client -l 3 -n 4 --proc-type=auto -- -n 0 + # ./mp_client/build/mp_client -l 4 -n 4 --proc-type=auto -- -n 1 + +.. note:: + + If the server application dies and needs to be restarted, all client applications also need to be restarted, + as there is no support in the server application for it to run as a secondary process. + Any client processes that need restarting can be restarted without affecting the server process. + +How the Application Works +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The server process performs the network port and data structure initialization much as the symmetric multi-process application does when run as primary. +One additional enhancement in this sample application is that the server process stores its port configuration data in a memory zone in hugepage shared memory. +This eliminates the need for the client processes to have the portmask parameter passed into them on the command line, +as is done for the symmetric multi-process application, and therefore eliminates mismatched parameters as a potential source of errors. + +In the same way that the server process is designed to be run as a primary process instance only, +the client processes are designed to be run as secondary instances only. +They have no code to attempt to create shared memory objects. +Instead, handles to all needed rings and memory pools are obtained via calls to rte_ring_lookup() and rte_mempool_lookup(). +The network ports for use by the processes are obtained by loading the network port drivers and probing the PCI bus, +which will, as in the symmetric multi-process example, +automatically get access to the network ports using the settings already configured by the primary/server process. + +Once all applications are initialized, the server operates by reading packets from each network port in turn and +distributing those packets to the client queues (software rings, one for each client process) in round-robin order. +On the client side, the packets are read from the rings in as big of bursts as possible, then routed out to a different network port. +The routing used is very simple. All packets received on the first NIC port are transmitted back out on the second port and vice versa. +Similarly, packets are routed between the 3rd and 4th network ports and so on. +The sending of packets is done by writing the packets directly to the network ports; they are not transferred back via the server process. + +In both the server and the client processes, outgoing packets are buffered before being sent, +so as to allow the sending of multiple packets in a single burst to improve efficiency. +For example, the client process will buffer packets to send, +until either the buffer is full or until we receive no further packets from the server. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ntb.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ntb.rst new file mode 100644 index 000000000..93fb752f2 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ntb.rst @@ -0,0 +1,94 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +NTB Sample Application +====================== + +The ntb sample application shows how to use ntb rawdev driver. +This sample provides interactive mode to do packet based processing +between two systems. + +This sample supports 4 types of packet forwarding mode. + +* ``file-trans``: transmit files between two systems. The sample will + be polling to receive files from the peer and save the file as + ``ntb_recv_file[N]``, [N] represents the number of received file. +* ``rxonly``: NTB receives packets but doesn't transmit them. +* ``txonly``: NTB generates and transmits packets without receiving any. +* ``iofwd``: iofwd between NTB device and ethdev. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ntb`` sub-directory. + +Running the Application +----------------------- + +The application requires an available core for each port, plus one. +The only available options are the standard ones for the EAL: + +.. code-block:: console + + ./build/ntb_fwd -c 0xf -n 6 -- -i + +Refer to the *DPDK Getting Started Guide* for general information on +running applications and the Environment Abstraction Layer (EAL) +options. + +Command-line Options +-------------------- + +The application supports the following command-line options. + +* ``--buf-size=N`` + + Set the data size of the mbufs used to N bytes, where N < 65536. + The default value is 2048. + +* ``--fwd-mode=mode`` + + Set the packet forwarding mode as ``file-trans``, ``txonly``, + ``rxonly`` or ``iofwd``. + +* ``--nb-desc=N`` + + Set number of descriptors of queue as N, namely queue size, + where 64 <= N <= 1024. The default value is 1024. + +* ``--txfreet=N`` + + Set the transmit free threshold of TX rings to N, where 0 <= N <= + the value of ``--nb-desc``. The default value is 256. + +* ``--burst=N`` + + Set the number of packets per burst to N, where 1 <= N <= 32. + The default value is 32. + +* ``--qp=N`` + + Set the number of queues as N, where qp > 0. The default value is 1. + +Using the application +--------------------- + +The application is console-driven using the cmdline DPDK interface: + +.. code-block:: console + + ntb> + +From this interface the available commands and descriptions of what +they do as follows: + +* ``send [filepath]``: Send file to the peer host. Need to be in + file-trans forwarding mode first. +* ``start``: Start transmission. +* ``stop``: Stop transmission. +* ``show/clear port stats``: Show/Clear port stats and throughput. +* ``set fwd file-trans/rxonly/txonly/iofwd``: Set packet forwarding + mode. +* ``quit``: Exit program. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/packet_ordering.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/packet_ordering.rst new file mode 100644 index 000000000..1c8ee5d04 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/packet_ordering.rst @@ -0,0 +1,60 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015 Intel Corporation. + +Packet Ordering Application +============================ + +The Packet Ordering sample app simply shows the impact of reordering a stream. +It's meant to stress the library with different configurations for performance. + +Overview +-------- + +The application uses at least three CPU cores: + +* RX core (maser core) receives traffic from the NIC ports and feeds Worker + cores with traffic through SW queues. + +* Worker core (slave core) basically do some light work on the packet. + Currently it modifies the output port of the packet for configurations with + more than one port enabled. + +* TX Core (slave core) receives traffic from Worker cores through software queues, + inserts out-of-order packets into reorder buffer, extracts ordered packets + from the reorder buffer and sends them to the NIC ports for transmission. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``packet_ordering`` sub-directory. + +Running the Application +----------------------- + +Refer to *DPDK Getting Started Guide* for general information on running applications +and the Environment Abstraction Layer (EAL) options. + +Application Command Line +~~~~~~~~~~~~~~~~~~~~~~~~ + +The application execution command line is: + +.. code-block:: console + + ./packet_ordering [EAL options] -- -p PORTMASK [--disable-reorder] [--insight-worker] + +The -c EAL CPU_COREMASK option has to contain at least 3 CPU cores. +The first CPU core in the core mask is the master core and would be assigned to +RX core, the last to TX core and the rest to Worker cores. + +The PORTMASK parameter must contain either 1 or even enabled port numbers. +When setting more than 1 port, traffic would be forwarded in pairs. +For example, if we enable 4 ports, traffic from port 0 to 1 and from 1 to 0, +then the other pair from 2 to 3 and from 3 to 2, having [0,1] and [2,3] pairs. + +The disable-reorder long option does, as its name implies, disable the reordering +of traffic, which should help evaluate reordering performance impact. + +The insight-worker long option enables output the packet statistics of each worker thread. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/performance_thread.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/performance_thread.rst new file mode 100644 index 000000000..b04d0ba44 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/performance_thread.rst @@ -0,0 +1,1221 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015 Intel Corporation. + +Performance Thread Sample Application +===================================== + +The performance thread sample application is a derivative of the standard L3 +forwarding application that demonstrates different threading models. + +Overview +-------- +For a general description of the L3 forwarding applications capabilities +please refer to the documentation of the standard application in +:doc:`l3_forward`. + +The performance thread sample application differs from the standard L3 +forwarding example in that it divides the TX and RX processing between +different threads, and makes it possible to assign individual threads to +different cores. + +Three threading models are considered: + +#. When there is one EAL thread per physical core. +#. When there are multiple EAL threads per physical core. +#. When there are multiple lightweight threads per EAL thread. + +Since DPDK release 2.0 it is possible to launch applications using the +``--lcores`` EAL parameter, specifying cpu-sets for a physical core. With the +performance thread sample application its is now also possible to assign +individual RX and TX functions to different cores. + +As an alternative to dividing the L3 forwarding work between different EAL +threads the performance thread sample introduces the possibility to run the +application threads as lightweight threads (L-threads) within one or +more EAL threads. + +In order to facilitate this threading model the example includes a primitive +cooperative scheduler (L-thread) subsystem. More details of the L-thread +subsystem can be found in :ref:`lthread_subsystem`. + +**Note:** Whilst theoretically possible it is not anticipated that multiple +L-thread schedulers would be run on the same physical core, this mode of +operation should not be expected to yield useful performance and is considered +invalid. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the `performance-thread/l3fwd-thread` sub-directory. + +Running the Application +----------------------- + +The application has a number of command line options:: + + ./build/l3fwd-thread [EAL options] -- + -p PORTMASK [-P] + --rx(port,queue,lcore,thread)[,(port,queue,lcore,thread)] + --tx(lcore,thread)[,(lcore,thread)] + [--enable-jumbo] [--max-pkt-len PKTLEN]] [--no-numa] + [--hash-entry-num] [--ipv6] [--no-lthreads] [--stat-lcore lcore] + [--parse-ptype] + +Where: + +* ``-p PORTMASK``: Hexadecimal bitmask of ports to configure. + +* ``-P``: optional, sets all ports to promiscuous mode so that packets are + accepted regardless of the packet's Ethernet MAC destination address. + Without this option, only packets with the Ethernet MAC destination address + set to the Ethernet address of the port are accepted. + +* ``--rx (port,queue,lcore,thread)[,(port,queue,lcore,thread)]``: the list of + NIC RX ports and queues handled by the RX lcores and threads. The parameters + are explained below. + +* ``--tx (lcore,thread)[,(lcore,thread)]``: the list of TX threads identifying + the lcore the thread runs on, and the id of RX thread with which it is + associated. The parameters are explained below. + +* ``--enable-jumbo``: optional, enables jumbo frames. + +* ``--max-pkt-len``: optional, maximum packet length in decimal (64-9600). + +* ``--no-numa``: optional, disables numa awareness. + +* ``--hash-entry-num``: optional, specifies the hash entry number in hex to be + setup. + +* ``--ipv6``: optional, set it if running ipv6 packets. + +* ``--no-lthreads``: optional, disables l-thread model and uses EAL threading + model. See below. + +* ``--stat-lcore``: optional, run CPU load stats collector on the specified + lcore. + +* ``--parse-ptype:`` optional, set to use software to analyze packet type. + Without this option, hardware will check the packet type. + +The parameters of the ``--rx`` and ``--tx`` options are: + +* ``--rx`` parameters + + .. _table_l3fwd_rx_parameters: + + +--------+------------------------------------------------------+ + | port | RX port | + +--------+------------------------------------------------------+ + | queue | RX queue that will be read on the specified RX port | + +--------+------------------------------------------------------+ + | lcore | Core to use for the thread | + +--------+------------------------------------------------------+ + | thread | Thread id (continuously from 0 to N) | + +--------+------------------------------------------------------+ + + +* ``--tx`` parameters + + .. _table_l3fwd_tx_parameters: + + +--------+------------------------------------------------------+ + | lcore | Core to use for L3 route match and transmit | + +--------+------------------------------------------------------+ + | thread | Id of RX thread to be associated with this TX thread | + +--------+------------------------------------------------------+ + +The ``l3fwd-thread`` application allows you to start packet processing in two +threading models: L-Threads (default) and EAL Threads (when the +``--no-lthreads`` parameter is used). For consistency all parameters are used +in the same way for both models. + + +Running with L-threads +~~~~~~~~~~~~~~~~~~~~~~ + +When the L-thread model is used (default option), lcore and thread parameters +in ``--rx/--tx`` are used to affinitize threads to the selected scheduler. + +For example, the following places every l-thread on different lcores:: + + l3fwd-thread -l 0-7 -n 2 -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,1,1)" \ + --tx="(2,0)(3,1)" + +The following places RX l-threads on lcore 0 and TX l-threads on lcore 1 and 2 +and so on:: + + l3fwd-thread -l 0-7 -n 2 -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,0,1)" \ + --tx="(1,0)(2,1)" + + +Running with EAL threads +~~~~~~~~~~~~~~~~~~~~~~~~ + +When the ``--no-lthreads`` parameter is used, the L-threading model is turned +off and EAL threads are used for all processing. EAL threads are enumerated in +the same way as L-threads, but the ``--lcores`` EAL parameter is used to +affinitize threads to the selected cpu-set (scheduler). Thus it is possible to +place every RX and TX thread on different lcores. + +For example, the following places every EAL thread on different lcores:: + + l3fwd-thread -l 0-7 -n 2 -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,1,1)" \ + --tx="(2,0)(3,1)" \ + --no-lthreads + + +To affinitize two or more EAL threads to one cpu-set, the EAL ``--lcores`` +parameter is used. + +The following places RX EAL threads on lcore 0 and TX EAL threads on lcore 1 +and 2 and so on:: + + l3fwd-thread -l 0-7 -n 2 --lcores="(0,1)@0,(2,3)@1" -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,1,1)" \ + --tx="(2,0)(3,1)" \ + --no-lthreads + + +Examples +~~~~~~~~ + +For selected scenarios the command line configuration of the application for L-threads +and its corresponding EAL threads command line can be realized as follows: + +a) Start every thread on different scheduler (1:1):: + + l3fwd-thread -l 0-7 -n 2 -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,1,1)" \ + --tx="(2,0)(3,1)" + + EAL thread equivalent:: + + l3fwd-thread -l 0-7 -n 2 -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,1,1)" \ + --tx="(2,0)(3,1)" \ + --no-lthreads + +b) Start all threads on one core (N:1). + + Start 4 L-threads on lcore 0:: + + l3fwd-thread -l 0-7 -n 2 -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,0,1)" \ + --tx="(0,0)(0,1)" + + Start 4 EAL threads on cpu-set 0:: + + l3fwd-thread -l 0-7 -n 2 --lcores="(0-3)@0" -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,0,1)" \ + --tx="(2,0)(3,1)" \ + --no-lthreads + +c) Start threads on different cores (N:M). + + Start 2 L-threads for RX on lcore 0, and 2 L-threads for TX on lcore 1:: + + l3fwd-thread -l 0-7 -n 2 -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,0,1)" \ + --tx="(1,0)(1,1)" + + Start 2 EAL threads for RX on cpu-set 0, and 2 EAL threads for TX on + cpu-set 1:: + + l3fwd-thread -l 0-7 -n 2 --lcores="(0-1)@0,(2-3)@1" -- -P -p 3 \ + --rx="(0,0,0,0)(1,0,1,1)" \ + --tx="(2,0)(3,1)" \ + --no-lthreads + +Explanation +----------- + +To a great extent the sample application differs little from the standard L3 +forwarding application, and readers are advised to familiarize themselves with +the material covered in the :doc:`l3_forward` documentation before proceeding. + +The following explanation is focused on the way threading is handled in the +performance thread example. + + +Mode of operation with EAL threads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The performance thread sample application has split the RX and TX functionality +into two different threads, and the RX and TX threads are +interconnected via software rings. With respect to these rings the RX threads +are producers and the TX threads are consumers. + +On initialization the TX and RX threads are started according to the command +line parameters. + +The RX threads poll the network interface queues and post received packets to a +TX thread via a corresponding software ring. + +The TX threads poll software rings, perform the L3 forwarding hash/LPM match, +and assemble packet bursts before performing burst transmit on the network +interface. + +As with the standard L3 forward application, burst draining of residual packets +is performed periodically with the period calculated from elapsed time using +the timestamps counter. + +The diagram below illustrates a case with two RX threads and three TX threads. + +.. _figure_performance_thread_1: + +.. figure:: img/performance_thread_1.* + + +Mode of operation with L-threads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Like the EAL thread configuration the application has split the RX and TX +functionality into different threads, and the pairs of RX and TX threads are +interconnected via software rings. + +On initialization an L-thread scheduler is started on every EAL thread. On all +but the master EAL thread only a dummy L-thread is initially started. +The L-thread started on the master EAL thread then spawns other L-threads on +different L-thread schedulers according the command line parameters. + +The RX threads poll the network interface queues and post received packets +to a TX thread via the corresponding software ring. + +The ring interface is augmented by means of an L-thread condition variable that +enables the TX thread to be suspended when the TX ring is empty. The RX thread +signals the condition whenever it posts to the TX ring, causing the TX thread +to be resumed. + +Additionally the TX L-thread spawns a worker L-thread to take care of +polling the software rings, whilst it handles burst draining of the transmit +buffer. + +The worker threads poll the software rings, perform L3 route lookup and +assemble packet bursts. If the TX ring is empty the worker thread suspends +itself by waiting on the condition variable associated with the ring. + +Burst draining of residual packets, less than the burst size, is performed by +the TX thread which sleeps (using an L-thread sleep function) and resumes +periodically to flush the TX buffer. + +This design means that L-threads that have no work, can yield the CPU to other +L-threads and avoid having to constantly poll the software rings. + +The diagram below illustrates a case with two RX threads and three TX functions +(each comprising a thread that processes forwarding and a thread that +periodically drains the output buffer of residual packets). + +.. _figure_performance_thread_2: + +.. figure:: img/performance_thread_2.* + + +CPU load statistics +~~~~~~~~~~~~~~~~~~~ + +It is possible to display statistics showing estimated CPU load on each core. +The statistics indicate the percentage of CPU time spent: processing +received packets (forwarding), polling queues/rings (waiting for work), +and doing any other processing (context switch and other overhead). + +When enabled statistics are gathered by having the application threads set and +clear flags when they enter and exit pertinent code sections. The flags are +then sampled in real time by a statistics collector thread running on another +core. This thread displays the data in real time on the console. + +This feature is enabled by designating a statistics collector core, using the +``--stat-lcore`` parameter. + + +.. _lthread_subsystem: + +The L-thread subsystem +---------------------- + +The L-thread subsystem resides in the examples/performance-thread/common +directory and is built and linked automatically when building the +``l3fwd-thread`` example. + +The subsystem provides a simple cooperative scheduler to enable arbitrary +functions to run as cooperative threads within a single EAL thread. +The subsystem provides a pthread like API that is intended to assist in +reuse of legacy code written for POSIX pthreads. + +The following sections provide some detail on the features, constraints, +performance and porting considerations when using L-threads. + + +.. _comparison_between_lthreads_and_pthreads: + +Comparison between L-threads and POSIX pthreads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The fundamental difference between the L-thread and pthread models is the +way in which threads are scheduled. The simplest way to think about this is to +consider the case of a processor with a single CPU. To run multiple threads +on a single CPU, the scheduler must frequently switch between the threads, +in order that each thread is able to make timely progress. +This is the basis of any multitasking operating system. + +This section explores the differences between the pthread model and the +L-thread model as implemented in the provided L-thread subsystem. If needed a +theoretical discussion of preemptive vs cooperative multi-threading can be +found in any good text on operating system design. + + +Scheduling and context switching +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The POSIX pthread library provides an application programming interface to +create and synchronize threads. Scheduling policy is determined by the host OS, +and may be configurable. The OS may use sophisticated rules to determine which +thread should be run next, threads may suspend themselves or make other threads +ready, and the scheduler may employ a time slice giving each thread a maximum +time quantum after which it will be preempted in favor of another thread that +is ready to run. To complicate matters further threads may be assigned +different scheduling priorities. + +By contrast the L-thread subsystem is considerably simpler. Logically the +L-thread scheduler performs the same multiplexing function for L-threads +within a single pthread as the OS scheduler does for pthreads within an +application process. The L-thread scheduler is simply the main loop of a +pthread, and in so far as the host OS is concerned it is a regular pthread +just like any other. The host OS is oblivious about the existence of and +not at all involved in the scheduling of L-threads. + +The other and most significant difference between the two models is that +L-threads are scheduled cooperatively. L-threads cannot not preempt each +other, nor can the L-thread scheduler preempt a running L-thread (i.e. +there is no time slicing). The consequence is that programs implemented with +L-threads must possess frequent rescheduling points, meaning that they must +explicitly and of their own volition return to the scheduler at frequent +intervals, in order to allow other L-threads an opportunity to proceed. + +In both models switching between threads requires that the current CPU +context is saved and a new context (belonging to the next thread ready to run) +is restored. With pthreads this context switching is handled transparently +and the set of CPU registers that must be preserved between context switches +is as per an interrupt handler. + +An L-thread context switch is achieved by the thread itself making a function +call to the L-thread scheduler. Thus it is only necessary to preserve the +callee registers. The caller is responsible to save and restore any other +registers it is using before a function call, and restore them on return, +and this is handled by the compiler. For ``X86_64`` on both Linux and BSD the +System V calling convention is used, this defines registers RSP, RBP, and +R12-R15 as callee-save registers (for more detailed discussion a good reference +is `X86 Calling Conventions <https://en.wikipedia.org/wiki/X86_calling_conventions>`_). + +Taking advantage of this, and due to the absence of preemption, an L-thread +context switch is achieved with less than 20 load/store instructions. + +The scheduling policy for L-threads is fixed, there is no prioritization of +L-threads, all L-threads are equal and scheduling is based on a FIFO +ready queue. + +An L-thread is a struct containing the CPU context of the thread +(saved on context switch) and other useful items. The ready queue contains +pointers to threads that are ready to run. The L-thread scheduler is a simple +loop that polls the ready queue, reads from it the next thread ready to run, +which it resumes by saving the current context (the current position in the +scheduler loop) and restoring the context of the next thread from its thread +struct. Thus an L-thread is always resumed at the last place it yielded. + +A well behaved L-thread will call the context switch regularly (at least once +in its main loop) thus returning to the scheduler's own main loop. Yielding +inserts the current thread at the back of the ready queue, and the process of +servicing the ready queue is repeated, thus the system runs by flipping back +and forth the between L-threads and scheduler loop. + +In the case of pthreads, the preemptive scheduling, time slicing, and support +for thread prioritization means that progress is normally possible for any +thread that is ready to run. This comes at the price of a relatively heavier +context switch and scheduling overhead. + +With L-threads the progress of any particular thread is determined by the +frequency of rescheduling opportunities in the other L-threads. This means that +an errant L-thread monopolizing the CPU might cause scheduling of other threads +to be stalled. Due to the lower cost of context switching, however, voluntary +rescheduling to ensure progress of other threads, if managed sensibly, is not +a prohibitive overhead, and overall performance can exceed that of an +application using pthreads. + + +Mutual exclusion +^^^^^^^^^^^^^^^^ + +With pthreads preemption means that threads that share data must observe +some form of mutual exclusion protocol. + +The fact that L-threads cannot preempt each other means that in many cases +mutual exclusion devices can be completely avoided. + +Locking to protect shared data can be a significant bottleneck in +multi-threaded applications so a carefully designed cooperatively scheduled +program can enjoy significant performance advantages. + +So far we have considered only the simplistic case of a single core CPU, +when multiple CPUs are considered things are somewhat more complex. + +First of all it is inevitable that there must be multiple L-thread schedulers, +one running on each EAL thread. So long as these schedulers remain isolated +from each other the above assertions about the potential advantages of +cooperative scheduling hold true. + +A configuration with isolated cooperative schedulers is less flexible than the +pthread model where threads can be affinitized to run on any CPU. With isolated +schedulers scaling of applications to utilize fewer or more CPUs according to +system demand is very difficult to achieve. + +The L-thread subsystem makes it possible for L-threads to migrate between +schedulers running on different CPUs. Needless to say if the migration means +that threads that share data end up running on different CPUs then this will +introduce the need for some kind of mutual exclusion system. + +Of course ``rte_ring`` software rings can always be used to interconnect +threads running on different cores, however to protect other kinds of shared +data structures, lock free constructs or else explicit locking will be +required. This is a consideration for the application design. + +In support of this extended functionality, the L-thread subsystem implements +thread safe mutexes and condition variables. + +The cost of affinitizing and of condition variable signaling is significantly +lower than the equivalent pthread operations, and so applications using these +features will see a performance benefit. + + +Thread local storage +^^^^^^^^^^^^^^^^^^^^ + +As with applications written for pthreads an application written for L-threads +can take advantage of thread local storage, in this case local to an L-thread. +An application may save and retrieve a single pointer to application data in +the L-thread struct. + +For legacy and backward compatibility reasons two alternative methods are also +offered, the first is modeled directly on the pthread get/set specific APIs, +the second approach is modeled on the ``RTE_PER_LCORE`` macros, whereby +``PER_LTHREAD`` macros are introduced, in both cases the storage is local to +the L-thread. + + +.. _constraints_and_performance_implications: + +Constraints and performance implications when using L-threads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +.. _API_compatibility: + +API compatibility +^^^^^^^^^^^^^^^^^ + +The L-thread subsystem provides a set of functions that are logically equivalent +to the corresponding functions offered by the POSIX pthread library, however not +all pthread functions have a corresponding L-thread equivalent, and not all +features available to pthreads are implemented for L-threads. + +The pthread library offers considerable flexibility via programmable attributes +that can be associated with threads, mutexes, and condition variables. + +By contrast the L-thread subsystem has fixed functionality, the scheduler policy +cannot be varied, and L-threads cannot be prioritized. There are no variable +attributes associated with any L-thread objects. L-threads, mutexes and +conditional variables, all have fixed functionality. (Note: reserved parameters +are included in the APIs to facilitate possible future support for attributes). + +The table below lists the pthread and equivalent L-thread APIs with notes on +differences and/or constraints. Where there is no L-thread entry in the table, +then the L-thread subsystem provides no equivalent function. + +.. _table_lthread_pthread: + +.. table:: Pthread and equivalent L-thread APIs. + + +----------------------------+------------------------+-------------------+ + | **Pthread function** | **L-thread function** | **Notes** | + +============================+========================+===================+ + | pthread_barrier_destroy | | | + +----------------------------+------------------------+-------------------+ + | pthread_barrier_init | | | + +----------------------------+------------------------+-------------------+ + | pthread_barrier_wait | | | + +----------------------------+------------------------+-------------------+ + | pthread_cond_broadcast | lthread_cond_broadcast | See note 1 | + +----------------------------+------------------------+-------------------+ + | pthread_cond_destroy | lthread_cond_destroy | | + +----------------------------+------------------------+-------------------+ + | pthread_cond_init | lthread_cond_init | | + +----------------------------+------------------------+-------------------+ + | pthread_cond_signal | lthread_cond_signal | See note 1 | + +----------------------------+------------------------+-------------------+ + | pthread_cond_timedwait | | | + +----------------------------+------------------------+-------------------+ + | pthread_cond_wait | lthread_cond_wait | See note 5 | + +----------------------------+------------------------+-------------------+ + | pthread_create | lthread_create | See notes 2, 3 | + +----------------------------+------------------------+-------------------+ + | pthread_detach | lthread_detach | See note 4 | + +----------------------------+------------------------+-------------------+ + | pthread_equal | | | + +----------------------------+------------------------+-------------------+ + | pthread_exit | lthread_exit | | + +----------------------------+------------------------+-------------------+ + | pthread_getspecific | lthread_getspecific | | + +----------------------------+------------------------+-------------------+ + | pthread_getcpuclockid | | | + +----------------------------+------------------------+-------------------+ + | pthread_join | lthread_join | | + +----------------------------+------------------------+-------------------+ + | pthread_key_create | lthread_key_create | | + +----------------------------+------------------------+-------------------+ + | pthread_key_delete | lthread_key_delete | | + +----------------------------+------------------------+-------------------+ + | pthread_mutex_destroy | lthread_mutex_destroy | | + +----------------------------+------------------------+-------------------+ + | pthread_mutex_init | lthread_mutex_init | | + +----------------------------+------------------------+-------------------+ + | pthread_mutex_lock | lthread_mutex_lock | See note 6 | + +----------------------------+------------------------+-------------------+ + | pthread_mutex_trylock | lthread_mutex_trylock | See note 6 | + +----------------------------+------------------------+-------------------+ + | pthread_mutex_timedlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_mutex_unlock | lthread_mutex_unlock | | + +----------------------------+------------------------+-------------------+ + | pthread_once | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_destroy | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_init | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_rdlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_timedrdlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_timedwrlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_tryrdlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_trywrlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_unlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_rwlock_wrlock | | | + +----------------------------+------------------------+-------------------+ + | pthread_self | lthread_current | | + +----------------------------+------------------------+-------------------+ + | pthread_setspecific | lthread_setspecific | | + +----------------------------+------------------------+-------------------+ + | pthread_spin_init | | See note 10 | + +----------------------------+------------------------+-------------------+ + | pthread_spin_destroy | | See note 10 | + +----------------------------+------------------------+-------------------+ + | pthread_spin_lock | | See note 10 | + +----------------------------+------------------------+-------------------+ + | pthread_spin_trylock | | See note 10 | + +----------------------------+------------------------+-------------------+ + | pthread_spin_unlock | | See note 10 | + +----------------------------+------------------------+-------------------+ + | pthread_cancel | lthread_cancel | | + +----------------------------+------------------------+-------------------+ + | pthread_setcancelstate | | | + +----------------------------+------------------------+-------------------+ + | pthread_setcanceltype | | | + +----------------------------+------------------------+-------------------+ + | pthread_testcancel | | | + +----------------------------+------------------------+-------------------+ + | pthread_getschedparam | | | + +----------------------------+------------------------+-------------------+ + | pthread_setschedparam | | | + +----------------------------+------------------------+-------------------+ + | pthread_yield | lthread_yield | See note 7 | + +----------------------------+------------------------+-------------------+ + | pthread_setaffinity_np | lthread_set_affinity | See notes 2, 3, 8 | + +----------------------------+------------------------+-------------------+ + | | lthread_sleep | See note 9 | + +----------------------------+------------------------+-------------------+ + | | lthread_sleep_clks | See note 9 | + +----------------------------+------------------------+-------------------+ + + +**Note 1**: + +Neither lthread signal nor broadcast may be called concurrently by L-threads +running on different schedulers, although multiple L-threads running in the +same scheduler may freely perform signal or broadcast operations. L-threads +running on the same or different schedulers may always safely wait on a +condition variable. + + +**Note 2**: + +Pthread attributes may be used to affinitize a pthread with a cpu-set. The +L-thread subsystem does not support a cpu-set. An L-thread may be affinitized +only with a single CPU at any time. + + +**Note 3**: + +If an L-thread is intended to run on a different NUMA node than the node that +creates the thread then, when calling ``lthread_create()`` it is advantageous +to specify the destination core as a parameter of ``lthread_create()``. See +:ref:`memory_allocation_and_NUMA_awareness` for details. + + +**Note 4**: + +An L-thread can only detach itself, and cannot detach other L-threads. + + +**Note 5**: + +A wait operation on a pthread condition variable is always associated with and +protected by a mutex which must be owned by the thread at the time it invokes +``pthread_wait()``. By contrast L-thread condition variables are thread safe +(for waiters) and do not use an associated mutex. Multiple L-threads (including +L-threads running on other schedulers) can safely wait on a L-thread condition +variable. As a consequence the performance of an L-thread condition variables +is typically an order of magnitude faster than its pthread counterpart. + + +**Note 6**: + +Recursive locking is not supported with L-threads, attempts to take a lock +recursively will be detected and rejected. + + +**Note 7**: + +``lthread_yield()`` will save the current context, insert the current thread +to the back of the ready queue, and resume the next ready thread. Yielding +increases ready queue backlog, see :ref:`ready_queue_backlog` for more details +about the implications of this. + + +N.B. The context switch time as measured from immediately before the call to +``lthread_yield()`` to the point at which the next ready thread is resumed, +can be an order of magnitude faster that the same measurement for +pthread_yield. + + +**Note 8**: + +``lthread_set_affinity()`` is similar to a yield apart from the fact that the +yielding thread is inserted into a peer ready queue of another scheduler. +The peer ready queue is actually a separate thread safe queue, which means that +threads appearing in the peer ready queue can jump any backlog in the local +ready queue on the destination scheduler. + +The context switch time as measured from the time just before the call to +``lthread_set_affinity()`` to just after the same thread is resumed on the new +scheduler can be orders of magnitude faster than the same measurement for +``pthread_setaffinity_np()``. + + +**Note 9**: + +Although there is no ``pthread_sleep()`` function, ``lthread_sleep()`` and +``lthread_sleep_clks()`` can be used wherever ``sleep()``, ``usleep()`` or +``nanosleep()`` might ordinarily be used. The L-thread sleep functions suspend +the current thread, start an ``rte_timer`` and resume the thread when the +timer matures. The ``rte_timer_manage()`` entry point is called on every pass +of the scheduler loop. This means that the worst case jitter on timer expiry +is determined by the longest period between context switches of any running +L-threads. + +In a synthetic test with many threads sleeping and resuming then the measured +jitter is typically orders of magnitude lower than the same measurement made +for ``nanosleep()``. + + +**Note 10**: + +Spin locks are not provided because they are problematical in a cooperative +environment, see :ref:`porting_locks_and_spinlocks` for a more detailed +discussion on how to avoid spin locks. + + +.. _Thread_local_storage_performance: + +Thread local storage +^^^^^^^^^^^^^^^^^^^^ + +Of the three L-thread local storage options the simplest and most efficient is +storing a single application data pointer in the L-thread struct. + +The ``PER_LTHREAD`` macros involve a run time computation to obtain the address +of the variable being saved/retrieved and also require that the accesses are +de-referenced via a pointer. This means that code that has used +``RTE_PER_LCORE`` macros being ported to L-threads might need some slight +adjustment (see :ref:`porting_thread_local_storage` for hints about porting +code that makes use of thread local storage). + +The get/set specific APIs are consistent with their pthread counterparts both +in use and in performance. + + +.. _memory_allocation_and_NUMA_awareness: + +Memory allocation and NUMA awareness +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All memory allocation is from DPDK huge pages, and is NUMA aware. Each +scheduler maintains its own caches of objects: lthreads, their stacks, TLS, +mutexes and condition variables. These caches are implemented as unbounded lock +free MPSC queues. When objects are created they are always allocated from the +caches on the local core (current EAL thread). + +If an L-thread has been affinitized to a different scheduler, then it can +always safely free resources to the caches from which they originated (because +the caches are MPSC queues). + +If the L-thread has been affinitized to a different NUMA node then the memory +resources associated with it may incur longer access latency. + +The commonly used pattern of setting affinity on entry to a thread after it has +started, means that memory allocation for both the stack and TLS will have been +made from caches on the NUMA node on which the threads creator is running. +This has the side effect that access latency will be sub-optimal after +affinitizing. + +This side effect can be mitigated to some extent (although not completely) by +specifying the destination CPU as a parameter of ``lthread_create()`` this +causes the L-thread's stack and TLS to be allocated when it is first scheduled +on the destination scheduler, if the destination is a on another NUMA node it +results in a more optimal memory allocation. + +Note that the lthread struct itself remains allocated from memory on the +creating node, this is unavoidable because an L-thread is known everywhere by +the address of this struct. + + +.. _object_cache_sizing: + +Object cache sizing +^^^^^^^^^^^^^^^^^^^ + +The per lcore object caches pre-allocate objects in bulk whenever a request to +allocate an object finds a cache empty. By default 100 objects are +pre-allocated, this is defined by ``LTHREAD_PREALLOC`` in the public API +header file lthread_api.h. This means that the caches constantly grow to meet +system demand. + +In the present implementation there is no mechanism to reduce the cache sizes +if system demand reduces. Thus the caches will remain at their maximum extent +indefinitely. + +A consequence of the bulk pre-allocation of objects is that every 100 (default +value) additional new object create operations results in a call to +``rte_malloc()``. For creation of objects such as L-threads, which trigger the +allocation of even more objects (i.e. their stacks and TLS) then this can +cause outliers in scheduling performance. + +If this is a problem the simplest mitigation strategy is to dimension the +system, by setting the bulk object pre-allocation size to some large number +that you do not expect to be exceeded. This means the caches will be populated +once only, the very first time a thread is created. + + +.. _Ready_queue_backlog: + +Ready queue backlog +^^^^^^^^^^^^^^^^^^^ + +One of the more subtle performance considerations is managing the ready queue +backlog. The fewer threads that are waiting in the ready queue then the faster +any particular thread will get serviced. + +In a naive L-thread application with N L-threads simply looping and yielding, +this backlog will always be equal to the number of L-threads, thus the cost of +a yield to a particular L-thread will be N times the context switch time. + +This side effect can be mitigated by arranging for threads to be suspended and +wait to be resumed, rather than polling for work by constantly yielding. +Blocking on a mutex or condition variable or even more obviously having a +thread sleep if it has a low frequency workload are all mechanisms by which a +thread can be excluded from the ready queue until it really does need to be +run. This can have a significant positive impact on performance. + + +.. _Initialization_and_shutdown_dependencies: + +Initialization, shutdown and dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The L-thread subsystem depends on DPDK for huge page allocation and depends on +the ``rte_timer subsystem``. The DPDK EAL initialization and +``rte_timer_subsystem_init()`` **MUST** be completed before the L-thread sub +system can be used. + +Thereafter initialization of the L-thread subsystem is largely transparent to +the application. Constructor functions ensure that global variables are properly +initialized. Other than global variables each scheduler is initialized +independently the first time that an L-thread is created by a particular EAL +thread. + +If the schedulers are to be run as isolated and independent schedulers, with +no intention that L-threads running on different schedulers will migrate between +schedulers or synchronize with L-threads running on other schedulers, then +initialization consists simply of creating an L-thread, and then running the +L-thread scheduler. + +If there will be interaction between L-threads running on different schedulers, +then it is important that the starting of schedulers on different EAL threads +is synchronized. + +To achieve this an additional initialization step is necessary, this is simply +to set the number of schedulers by calling the API function +``lthread_num_schedulers_set(n)``, where ``n`` is the number of EAL threads +that will run L-thread schedulers. Setting the number of schedulers to a +number greater than 0 will cause all schedulers to wait until the others have +started before beginning to schedule L-threads. + +The L-thread scheduler is started by calling the function ``lthread_run()`` +and should be called from the EAL thread and thus become the main loop of the +EAL thread. + +The function ``lthread_run()``, will not return until all threads running on +the scheduler have exited, and the scheduler has been explicitly stopped by +calling ``lthread_scheduler_shutdown(lcore)`` or +``lthread_scheduler_shutdown_all()``. + +All these function do is tell the scheduler that it can exit when there are no +longer any running L-threads, neither function forces any running L-thread to +terminate. Any desired application shutdown behavior must be designed and +built into the application to ensure that L-threads complete in a timely +manner. + +**Important Note:** It is assumed when the scheduler exits that the application +is terminating for good, the scheduler does not free resources before exiting +and running the scheduler a subsequent time will result in undefined behavior. + + +.. _porting_legacy_code_to_run_on_lthreads: + +Porting legacy code to run on L-threads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Legacy code originally written for a pthread environment may be ported to +L-threads if the considerations about differences in scheduling policy, and +constraints discussed in the previous sections can be accommodated. + +This section looks in more detail at some of the issues that may have to be +resolved when porting code. + + +.. _pthread_API_compatibility: + +pthread API compatibility +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The first step is to establish exactly which pthread APIs the legacy +application uses, and to understand the requirements of those APIs. If there +are corresponding L-lthread APIs, and where the default pthread functionality +is used by the application then, notwithstanding the other issues discussed +here, it should be feasible to run the application with L-threads. If the +legacy code modifies the default behavior using attributes then if may be +necessary to make some adjustments to eliminate those requirements. + + +.. _blocking_system_calls: + +Blocking system API calls +^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is important to understand what other system services the application may be +using, bearing in mind that in a cooperatively scheduled environment a thread +cannot block without stalling the scheduler and with it all other cooperative +threads. Any kind of blocking system call, for example file or socket IO, is a +potential problem, a good tool to analyze the application for this purpose is +the ``strace`` utility. + +There are many strategies to resolve these kind of issues, each with it +merits. Possible solutions include: + +* Adopting a polled mode of the system API concerned (if available). + +* Arranging for another core to perform the function and synchronizing with + that core via constructs that will not block the L-thread. + +* Affinitizing the thread to another scheduler devoted (as a matter of policy) + to handling threads wishing to make blocking calls, and then back again when + finished. + + +.. _porting_locks_and_spinlocks: + +Locks and spinlocks +^^^^^^^^^^^^^^^^^^^ + +Locks and spinlocks are another source of blocking behavior that for the same +reasons as system calls will need to be addressed. + +If the application design ensures that the contending L-threads will always +run on the same scheduler then it its probably safe to remove locks and spin +locks completely. + +The only exception to the above rule is if for some reason the +code performs any kind of context switch whilst holding the lock +(e.g. yield, sleep, or block on a different lock, or on a condition variable). +This will need to determined before deciding to eliminate a lock. + +If a lock cannot be eliminated then an L-thread mutex can be substituted for +either kind of lock. + +An L-thread blocking on an L-thread mutex will be suspended and will cause +another ready L-thread to be resumed, thus not blocking the scheduler. When +default behavior is required, it can be used as a direct replacement for a +pthread mutex lock. + +Spin locks are typically used when lock contention is likely to be rare and +where the period during which the lock may be held is relatively short. +When the contending L-threads are running on the same scheduler then an +L-thread blocking on a spin lock will enter an infinite loop stopping the +scheduler completely (see :ref:`porting_infinite_loops` below). + +If the application design ensures that contending L-threads will always run +on different schedulers then it might be reasonable to leave a short spin lock +that rarely experiences contention in place. + +If after all considerations it appears that a spin lock can neither be +eliminated completely, replaced with an L-thread mutex, or left in place as +is, then an alternative is to loop on a flag, with a call to +``lthread_yield()`` inside the loop (n.b. if the contending L-threads might +ever run on different schedulers the flag will need to be manipulated +atomically). + +Spinning and yielding is the least preferred solution since it introduces +ready queue backlog (see also :ref:`ready_queue_backlog`). + + +.. _porting_sleeps_and_delays: + +Sleeps and delays +^^^^^^^^^^^^^^^^^ + +Yet another kind of blocking behavior (albeit momentary) are delay functions +like ``sleep()``, ``usleep()``, ``nanosleep()`` etc. All will have the +consequence of stalling the L-thread scheduler and unless the delay is very +short (e.g. a very short nanosleep) calls to these functions will need to be +eliminated. + +The simplest mitigation strategy is to use the L-thread sleep API functions, +of which two variants exist, ``lthread_sleep()`` and ``lthread_sleep_clks()``. +These functions start an rte_timer against the L-thread, suspend the L-thread +and cause another ready L-thread to be resumed. The suspended L-thread is +resumed when the rte_timer matures. + + +.. _porting_infinite_loops: + +Infinite loops +^^^^^^^^^^^^^^ + +Some applications have threads with loops that contain no inherent +rescheduling opportunity, and rely solely on the OS time slicing to share +the CPU. In a cooperative environment this will stop everything dead. These +kind of loops are not hard to identify, in a debug session you will find the +debugger is always stopping in the same loop. + +The simplest solution to this kind of problem is to insert an explicit +``lthread_yield()`` or ``lthread_sleep()`` into the loop. Another solution +might be to include the function performed by the loop into the execution path +of some other loop that does in fact yield, if this is possible. + + +.. _porting_thread_local_storage: + +Thread local storage +^^^^^^^^^^^^^^^^^^^^ + +If the application uses thread local storage, the use case should be +studied carefully. + +In a legacy pthread application either or both the ``__thread`` prefix, or the +pthread set/get specific APIs may have been used to define storage local to a +pthread. + +In some applications it may be a reasonable assumption that the data could +or in fact most likely should be placed in L-thread local storage. + +If the application (like many DPDK applications) has assumed a certain +relationship between a pthread and the CPU to which it is affinitized, there +is a risk that thread local storage may have been used to save some data items +that are correctly logically associated with the CPU, and others items which +relate to application context for the thread. Only a good understanding of the +application will reveal such cases. + +If the application requires an that an L-thread is to be able to move between +schedulers then care should be taken to separate these kinds of data, into per +lcore, and per L-thread storage. In this way a migrating thread will bring with +it the local data it needs, and pick up the new logical core specific values +from pthread local storage at its new home. + + +.. _pthread_shim: + +Pthread shim +~~~~~~~~~~~~ + +A convenient way to get something working with legacy code can be to use a +shim that adapts pthread API calls to the corresponding L-thread ones. +This approach will not mitigate any of the porting considerations mentioned +in the previous sections, but it will reduce the amount of code churn that +would otherwise been involved. It is a reasonable approach to evaluate +L-threads, before investing effort in porting to the native L-thread APIs. + + +Overview +^^^^^^^^ +The L-thread subsystem includes an example pthread shim. This is a partial +implementation but does contain the API stubs needed to get basic applications +running. There is a simple "hello world" application that demonstrates the +use of the pthread shim. + +A subtlety of working with a shim is that the application will still need +to make use of the genuine pthread library functions, at the very least in +order to create the EAL threads in which the L-thread schedulers will run. +This is the case with DPDK initialization, and exit. + +To deal with the initialization and shutdown scenarios, the shim is capable of +switching on or off its adaptor functionality, an application can control this +behavior by the calling the function ``pt_override_set()``. The default state +is disabled. + +The pthread shim uses the dynamic linker loader and saves the loaded addresses +of the genuine pthread API functions in an internal table, when the shim +functionality is enabled it performs the adaptor function, when disabled it +invokes the genuine pthread function. + +The function ``pthread_exit()`` has additional special handling. The standard +system header file pthread.h declares ``pthread_exit()`` with +``__rte_noreturn`` this is an optimization that is possible because +the pthread is terminating and this enables the compiler to omit the normal +handling of stack and protection of registers since the function is not +expected to return, and in fact the thread is being destroyed. These +optimizations are applied in both the callee and the caller of the +``pthread_exit()`` function. + +In our cooperative scheduling environment this behavior is inadmissible. The +pthread is the L-thread scheduler thread, and, although an L-thread is +terminating, there must be a return to the scheduler in order that the system +can continue to run. Further, returning from a function with attribute +``noreturn`` is invalid and may result in undefined behavior. + +The solution is to redefine the ``pthread_exit`` function with a macro, +causing it to be mapped to a stub function in the shim that does not have the +``noreturn`` attribute. This macro is defined in the file +``pthread_shim.h``. The stub function is otherwise no different than any of +the other stub functions in the shim, and will switch between the real +``pthread_exit()`` function or the ``lthread_exit()`` function as +required. The only difference is that the mapping to the stub by macro +substitution. + +A consequence of this is that the file ``pthread_shim.h`` must be included in +legacy code wishing to make use of the shim. It also means that dynamic +linkage of a pre-compiled binary that did not include pthread_shim.h is not be +supported. + +Given the requirements for porting legacy code outlined in +:ref:`porting_legacy_code_to_run_on_lthreads` most applications will require at +least some minimal adjustment and recompilation to run on L-threads so +pre-compiled binaries are unlikely to be met in practice. + +In summary the shim approach adds some overhead but can be a useful tool to help +establish the feasibility of a code reuse project. It is also a fairly +straightforward task to extend the shim if necessary. + +**Note:** Bearing in mind the preceding discussions about the impact of making +blocking calls then switching the shim in and out on the fly to invoke any +pthread API this might block is something that should typically be avoided. + + +Building and running the pthread shim +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The shim example application is located in the sample application +in the performance-thread folder + +To build and run the pthread shim example + +#. Go to the example applications folder + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK}/examples/performance-thread/pthread_shim + + +#. Set the target (a default target is used if not specified). For example: + + .. code-block:: console + + export RTE_TARGET=x86_64-native-linux-gcc + + See the DPDK Getting Started Guide for possible RTE_TARGET values. + +#. Build the application: + + .. code-block:: console + + make + +#. To run the pthread_shim example + + .. code-block:: console + + lthread-pthread-shim -c core_mask -n number_of_channels + +.. _lthread_diagnostics: + +L-thread Diagnostics +~~~~~~~~~~~~~~~~~~~~ + +When debugging you must take account of the fact that the L-threads are run in +a single pthread. The current scheduler is defined by +``RTE_PER_LCORE(this_sched)``, and the current lthread is stored at +``RTE_PER_LCORE(this_sched)->current_lthread``. Thus on a breakpoint in a GDB +session the current lthread can be obtained by displaying the pthread local +variable ``per_lcore_this_sched->current_lthread``. + +Another useful diagnostic feature is the possibility to trace significant +events in the life of an L-thread, this feature is enabled by changing the +value of LTHREAD_DIAG from 0 to 1 in the file ``lthread_diag_api.h``. + +Tracing of events can be individually masked, and the mask may be programmed +at run time. An unmasked event results in a callback that provides information +about the event. The default callback simply prints trace information. The +default mask is 0 (all events off) the mask can be modified by calling the +function ``lthread_diagniostic_set_mask()``. + +It is possible register a user callback function to implement more +sophisticated diagnostic functions. +Object creation events (lthread, mutex, and condition variable) accept, and +store in the created object, a user supplied reference value returned by the +callback function. + +The lthread reference value is passed back in all subsequent event callbacks, +the mutex and APIs are provided to retrieve the reference value from +mutexes and condition variables. This enables a user to monitor, count, or +filter for specific events, on specific objects, for example to monitor for a +specific thread signaling a specific condition variable, or to monitor +on all timer events, the possibilities and combinations are endless. + +The callback function can be set by calling the function +``lthread_diagnostic_enable()`` supplying a callback function pointer and an +event mask. + +Setting ``LTHREAD_DIAG`` also enables counting of statistics about cache and +queue usage, and these statistics can be displayed by calling the function +``lthread_diag_stats_display()``. This function also performs a consistency +check on the caches and queues. The function should only be called from the +master EAL thread after all slave threads have stopped and returned to the C +main program, otherwise the consistency check will fail. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/ptpclient.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/ptpclient.rst new file mode 100644 index 000000000..12b4f13d5 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/ptpclient.rst @@ -0,0 +1,252 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015 Intel Corporation. + +PTP Client Sample Application +============================= + +The PTP (Precision Time Protocol) client sample application is a simple +example of using the DPDK IEEE1588 API to communicate with a PTP master clock +to synchronize the time on the NIC and, optionally, on the Linux system. + +Note, PTP is a time syncing protocol and cannot be used within DPDK as a +time-stamping mechanism. See the following for an explanation of the protocol: +`Precision Time Protocol +<https://en.wikipedia.org/wiki/Precision_Time_Protocol>`_. + + +Limitations +----------- + +The PTP sample application is intended as a simple reference implementation of +a PTP client using the DPDK IEEE1588 API. +In order to keep the application simple the following assumptions are made: + +* The first discovered master is the master for the session. +* Only L2 PTP packets are supported. +* Only the PTP v2 protocol is supported. +* Only the slave clock is implemented. + + +How the Application Works +------------------------- + +.. _figure_ptpclient_highlevel: + +.. figure:: img/ptpclient.* + + PTP Synchronization Protocol + +The PTP synchronization in the sample application works as follows: + +* Master sends *Sync* message - the slave saves it as T2. +* Master sends *Follow Up* message and sends time of T1. +* Slave sends *Delay Request* frame to PTP Master and stores T3. +* Master sends *Delay Response* T4 time which is time of received T3. + +The adjustment for slave can be represented as: + + adj = -[(T2-T1)-(T4 - T3)]/2 + +If the command line parameter ``-T 1`` is used the application also +synchronizes the PTP PHC clock with the Linux kernel clock. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ptpclient`` sub-directory. + +.. note:: + To compile the application edit the ``config/common_linux`` configuration file to enable IEEE1588 + and then recompile DPDK: + + .. code-block:: console + + CONFIG_RTE_LIBRTE_IEEE1588=y + +Running the Application +----------------------- + +To run the example in a ``linux`` environment: + +.. code-block:: console + + ./build/ptpclient -l 1 -n 4 -- -p 0x1 -T 0 + +Refer to *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + +* ``-p portmask``: Hexadecimal portmask. +* ``-T 0``: Update only the PTP slave clock. +* ``-T 1``: Update the PTP slave clock and synchronize the Linux Kernel to the PTP clock. + + +Code Explanation +---------------- + +The following sections provide an explanation of the main components of the +code. + +All DPDK library functions used in the sample code are prefixed with ``rte_`` +and are explained in detail in the *DPDK API Documentation*. + + +The Main Function +~~~~~~~~~~~~~~~~~ + +The ``main()`` function performs the initialization and calls the execution +threads for each lcore. + +The first task is to initialize the Environment Abstraction Layer (EAL). The +``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` +function. The value returned is the number of parsed arguments: + +.. code-block:: c + + int ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); + +And than we parse application specific arguments + +.. code-block:: c + + argc -= ret; + argv += ret; + + ret = ptp_parse_args(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with PTP initialization\n"); + +The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers) +used by the application: + +.. code-block:: c + + mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports, + MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id()); + +Mbufs are the packet buffer structure used by DPDK. They are explained in +detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*. + +The ``main()`` function also initializes all the ports using the user defined +``port_init()`` function with portmask provided by user: + +.. code-block:: c + + for (portid = 0; portid < nb_ports; portid++) + if ((ptp_enabled_port_mask & (1 << portid)) != 0) { + + if (port_init(portid, mbuf_pool) == 0) { + ptp_enabled_ports[ptp_enabled_port_nb] = portid; + ptp_enabled_port_nb++; + } else { + rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n", + portid); + } + } + + +Once the initialization is complete, the application is ready to launch a +function on an lcore. In this example ``lcore_main()`` is called on a single +lcore. + +.. code-block:: c + + lcore_main(); + +The ``lcore_main()`` function is explained below. + + +The Lcores Main +~~~~~~~~~~~~~~~ + +As we saw above the ``main()`` function calls an application function on the +available lcores. + +The main work of the application is done within the loop: + +.. code-block:: c + + for (portid = 0; portid < ptp_enabled_port_nb; portid++) { + + portid = ptp_enabled_ports[portid]; + nb_rx = rte_eth_rx_burst(portid, 0, &m, 1); + + if (likely(nb_rx == 0)) + continue; + + if (m->ol_flags & PKT_RX_IEEE1588_PTP) + parse_ptp_frames(portid, m); + + rte_pktmbuf_free(m); + } + +Packets are received one by one on the RX ports and, if required, PTP response +packets are transmitted on the TX ports. + +If the offload flags in the mbuf indicate that the packet is a PTP packet then +the packet is parsed to determine which type: + +.. code-block:: c + + if (m->ol_flags & PKT_RX_IEEE1588_PTP) + parse_ptp_frames(portid, m); + + +All packets are freed explicitly using ``rte_pktmbuf_free()``. + +The forwarding loop can be interrupted and the application closed using +``Ctrl-C``. + + +PTP parsing +~~~~~~~~~~~ + +The ``parse_ptp_frames()`` function processes PTP packets, implementing slave +PTP IEEE1588 L2 functionality. + +.. code-block:: c + + void + parse_ptp_frames(uint16_t portid, struct rte_mbuf *m) { + struct ptp_header *ptp_hdr; + struct rte_ether_hdr *eth_hdr; + uint16_t eth_type; + + eth_hdr = rte_pktmbuf_mtod(m, struct rte_ether_hdr *); + eth_type = rte_be_to_cpu_16(eth_hdr->ether_type); + + if (eth_type == PTP_PROTOCOL) { + ptp_data.m = m; + ptp_data.portid = portid; + ptp_hdr = (struct ptp_header *)(rte_pktmbuf_mtod(m, char *) + + sizeof(struct rte_ether_hdr)); + + switch (ptp_hdr->msgtype) { + case SYNC: + parse_sync(&ptp_data); + break; + case FOLLOW_UP: + parse_fup(&ptp_data); + break; + case DELAY_RESP: + parse_drsp(&ptp_data); + print_clock_info(&ptp_data); + break; + default: + break; + } + } + } + +There are 3 types of packets on the RX path which we must parse to create a minimal +implementation of the PTP slave client: + +* SYNC packet. +* FOLLOW UP packet +* DELAY RESPONSE packet. + +When we parse the *FOLLOW UP* packet we also create and send a *DELAY_REQUEST* packet. +Also when we parse the *DELAY RESPONSE* packet, and all conditions are met we adjust the PTP slave clock. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/qos_metering.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/qos_metering.rst new file mode 100644 index 000000000..d75f7da52 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/qos_metering.rst @@ -0,0 +1,155 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +QoS Metering Sample Application +=============================== + +The QoS meter sample application is an example that demonstrates the use of DPDK to provide QoS marking and metering, +as defined by RFC2697 for Single Rate Three Color Marker (srTCM) and RFC 2698 for Two Rate Three Color Marker (trTCM) algorithm. + +Overview +-------- + +The application uses a single thread for reading the packets from the RX port, +metering, marking them with the appropriate color (green, yellow or red) and writing them to the TX port. + +A policing scheme can be applied before writing the packets to the TX port by dropping or +changing the color of the packet in a static manner depending on both the input and output colors of the packets that are processed by the meter. + +The operation mode can be selected as compile time out of the following options: + +* Simple forwarding + +* srTCM color blind + +* srTCM color aware + +* srTCM color blind + +* srTCM color aware + +Please refer to RFC2697 and RFC2698 for details about the srTCM and trTCM configurable parameters +(CIR, CBS and EBS for srTCM; CIR, PIR, CBS and PBS for trTCM). + +The color blind modes are functionally equivalent with the color-aware modes when +all the incoming packets are colored as green. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``qos_meter`` sub-directory. + +Running the Application +----------------------- + +The application execution command line is as below: + +.. code-block:: console + + ./qos_meter [EAL options] -- -p PORTMASK + +The application is constrained to use a single core in the EAL core mask and 2 ports only in the application port mask +(first port from the port mask is used for RX and the other port in the core mask is used for TX). + +Refer to *DPDK Getting Started Guide* for general information on running applications and +the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +Selecting one of the metering modes is done with these defines: + +.. code-block:: c + + #define APP_MODE_FWD 0 + #define APP_MODE_SRTCM_COLOR_BLIND 1 + #define APP_MODE_SRTCM_COLOR_AWARE 2 + #define APP_MODE_TRTCM_COLOR_BLIND 3 + #define APP_MODE_TRTCM_COLOR_AWARE 4 + + #define APP_MODE APP_MODE_SRTCM_COLOR_BLIND + +To simplify debugging (for example, by using the traffic generator RX side MAC address based packet filtering feature), +the color is defined as the LSB byte of the destination MAC address. + +The traffic meter parameters are configured in the application source code with following default values: + +.. code-block:: c + + struct rte_meter_srtcm_params app_srtcm_params[] = { + + {.cir = 1000000 * 46, .cbs = 2048, .ebs = 2048}, + + }; + + struct rte_meter_trtcm_params app_trtcm_params[] = { + + {.cir = 1000000 * 46, .pir = 1500000 * 46, .cbs = 2048, .pbs = 2048}, + + }; + +Assuming the input traffic is generated at line rate and all packets are 64 bytes Ethernet frames (IPv4 packet size of 46 bytes) +and green, the expected output traffic should be marked as shown in the following table: + +.. _table_qos_metering_1: + +.. table:: Output Traffic Marking + + +-------------+------------------+-------------------+----------------+ + | **Mode** | **Green (Mpps)** | **Yellow (Mpps)** | **Red (Mpps)** | + | | | | | + +=============+==================+===================+================+ + | srTCM blind | 1 | 1 | 12.88 | + | | | | | + +-------------+------------------+-------------------+----------------+ + | srTCM color | 1 | 1 | 12.88 | + | | | | | + +-------------+------------------+-------------------+----------------+ + | trTCM blind | 1 | 0.5 | 13.38 | + | | | | | + +-------------+------------------+-------------------+----------------+ + | trTCM color | 1 | 0.5 | 13.38 | + | | | | | + +-------------+------------------+-------------------+----------------+ + | FWD | 14.88 | 0 | 0 | + | | | | | + +-------------+------------------+-------------------+----------------+ + +To set up the policing scheme as desired, it is necessary to modify the main.h source file, +where this policy is implemented as a static structure, as follows: + +.. code-block:: c + + int policer_table[e_RTE_METER_COLORS][e_RTE_METER_COLORS] = + { + { GREEN, RED, RED}, + { DROP, YELLOW, RED}, + { DROP, DROP, RED} + }; + +Where rows indicate the input color, columns indicate the output color, +and the value that is stored in the table indicates the action to be taken for that particular case. + +There are four different actions: + +* GREEN: The packet's color is changed to green. + +* YELLOW: The packet's color is changed to yellow. + +* RED: The packet's color is changed to red. + +* DROP: The packet is dropped. + +In this particular case: + +* Every packet which input and output color are the same, keeps the same color. + +* Every packet which color has improved is dropped (this particular case can't happen, so these values will not be used). + +* For the rest of the cases, the color is changed to red. + +.. note:: + * In color blind mode, first row GREEN color is only valid. + * To drop the packet, policer_table action has to be set to DROP. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/qos_scheduler.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/qos_scheduler.rst new file mode 100644 index 000000000..b5010657a --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/qos_scheduler.rst @@ -0,0 +1,375 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +QoS Scheduler Sample Application +================================ + +The QoS sample application demonstrates the use of the DPDK to provide QoS scheduling. + +Overview +-------- + +The architecture of the QoS scheduler application is shown in the following figure. + +.. _figure_qos_sched_app_arch: + +.. figure:: img/qos_sched_app_arch.* + + QoS Scheduler Application Architecture + + +There are two flavors of the runtime execution for this application, +with two or three threads per each packet flow configuration being used. +The RX thread reads packets from the RX port, +classifies the packets based on the double VLAN (outer and inner) and +the lower byte of the IP destination address and puts them into the ring queue. +The worker thread dequeues the packets from the ring and calls the QoS scheduler enqueue/dequeue functions. +If a separate TX core is used, these are sent to the TX ring. +Otherwise, they are sent directly to the TX port. +The TX thread, if present, reads from the TX ring and write the packets to the TX port. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``qos_sched`` sub-directory. + + .. note:: + + This application is intended as a linux only. + +.. note:: + + To get statistics on the sample app using the command line interface as described in the next section, + DPDK must be compiled defining *CONFIG_RTE_SCHED_COLLECT_STATS*, + which can be done by changing the configuration file for the specific target to be compiled. + +Running the Application +----------------------- + +.. note:: + + In order to run the application, a total of at least 4 + G of huge pages must be set up for each of the used sockets (depending on the cores in use). + +The application has a number of command line options: + +.. code-block:: console + + ./qos_sched [EAL options] -- <APP PARAMS> + +Mandatory application parameters include: + +* --pfc "RX PORT, TX PORT, RX LCORE, WT LCORE, TX CORE": Packet flow configuration. + Multiple pfc entities can be configured in the command line, + having 4 or 5 items (if TX core defined or not). + +Optional application parameters include: + +* -i: It makes the application to start in the interactive mode. + In this mode, the application shows a command line that can be used for obtaining statistics while + scheduling is taking place (see interactive mode below for more information). + +* --mst n: Master core index (the default value is 1). + +* --rsz "A, B, C": Ring sizes: + +* A = Size (in number of buffer descriptors) of each of the NIC RX rings read + by the I/O RX lcores (the default value is 128). + +* B = Size (in number of elements) of each of the software rings used + by the I/O RX lcores to send packets to worker lcores (the default value is 8192). + +* C = Size (in number of buffer descriptors) of each of the NIC TX rings written + by worker lcores (the default value is 256) + +* --bsz "A, B, C, D": Burst sizes + +* A = I/O RX lcore read burst size from the NIC RX (the default value is 64) + +* B = I/O RX lcore write burst size to the output software rings, + worker lcore read burst size from input software rings,QoS enqueue size (the default value is 64) + +* C = QoS dequeue size (the default value is 32) + +* D = Worker lcore write burst size to the NIC TX (the default value is 64) + +* --msz M: Mempool size (in number of mbufs) for each pfc (default 2097152) + +* --rth "A, B, C": The RX queue threshold parameters + +* A = RX prefetch threshold (the default value is 8) + +* B = RX host threshold (the default value is 8) + +* C = RX write-back threshold (the default value is 4) + +* --tth "A, B, C": TX queue threshold parameters + +* A = TX prefetch threshold (the default value is 36) + +* B = TX host threshold (the default value is 0) + +* C = TX write-back threshold (the default value is 0) + +* --cfg FILE: Profile configuration to load + +Refer to *DPDK Getting Started Guide* for general information on running applications and +the Environment Abstraction Layer (EAL) options. + +The profile configuration file defines all the port/subport/pipe/traffic class/queue parameters +needed for the QoS scheduler configuration. + +The profile file has the following format: + +:: + + ; port configuration [port] + + frame overhead = 24 + number of subports per port = 1 + + ; Subport configuration + + [subport 0] + number of pipes per subport = 4096 + queue sizes = 64 64 64 64 64 64 64 64 64 64 64 64 64 + tb rate = 1250000000; Bytes per second + tb size = 1000000; Bytes + tc 0 rate = 1250000000; Bytes per second + tc 1 rate = 1250000000; Bytes per second + tc 2 rate = 1250000000; Bytes per second + tc 3 rate = 1250000000; Bytes per second + tc 4 rate = 1250000000; Bytes per second + tc 5 rate = 1250000000; Bytes per second + tc 6 rate = 1250000000; Bytes per second + tc 7 rate = 1250000000; Bytes per second + tc 8 rate = 1250000000; Bytes per second + tc 9 rate = 1250000000; Bytes per second + tc 10 rate = 1250000000; Bytes per second + tc 11 rate = 1250000000; Bytes per second + tc 12 rate = 1250000000; Bytes per second + + tc period = 10; Milliseconds + tc oversubscription period = 10; Milliseconds + + pipe 0-4095 = 0; These pipes are configured with pipe profile 0 + + ; Pipe configuration + + [pipe profile 0] + tb rate = 305175; Bytes per second + tb size = 1000000; Bytes + + tc 0 rate = 305175; Bytes per second + tc 1 rate = 305175; Bytes per second + tc 2 rate = 305175; Bytes per second + tc 3 rate = 305175; Bytes per second + tc 4 rate = 305175; Bytes per second + tc 5 rate = 305175; Bytes per second + tc 6 rate = 305175; Bytes per second + tc 7 rate = 305175; Bytes per second + tc 8 rate = 305175; Bytes per second + tc 9 rate = 305175; Bytes per second + tc 10 rate = 305175; Bytes per second + tc 11 rate = 305175; Bytes per second + tc 12 rate = 305175; Bytes per second + tc period = 40; Milliseconds + + tc 0 oversubscription weight = 1 + tc 1 oversubscription weight = 1 + tc 2 oversubscription weight = 1 + tc 3 oversubscription weight = 1 + tc 4 oversubscription weight = 1 + tc 5 oversubscription weight = 1 + tc 6 oversubscription weight = 1 + tc 7 oversubscription weight = 1 + tc 8 oversubscription weight = 1 + tc 9 oversubscription weight = 1 + tc 10 oversubscription weight = 1 + tc 11 oversubscription weight = 1 + tc 12 oversubscription weight = 1 + + tc 12 wrr weights = 1 1 1 1 + + ; RED params per traffic class and color (Green / Yellow / Red) + + [red] + tc 0 wred min = 48 40 32 + tc 0 wred max = 64 64 64 + tc 0 wred inv prob = 10 10 10 + tc 0 wred weight = 9 9 9 + + tc 1 wred min = 48 40 32 + tc 1 wred max = 64 64 64 + tc 1 wred inv prob = 10 10 10 + tc 1 wred weight = 9 9 9 + + tc 2 wred min = 48 40 32 + tc 2 wred max = 64 64 64 + tc 2 wred inv prob = 10 10 10 + tc 2 wred weight = 9 9 9 + + tc 3 wred min = 48 40 32 + tc 3 wred max = 64 64 64 + tc 3 wred inv prob = 10 10 10 + tc 3 wred weight = 9 9 9 + + tc 4 wred min = 48 40 32 + tc 4 wred max = 64 64 64 + tc 4 wred inv prob = 10 10 10 + tc 4 wred weight = 9 9 9 + + tc 5 wred min = 48 40 32 + tc 5 wred max = 64 64 64 + tc 5 wred inv prob = 10 10 10 + tc 5 wred weight = 9 9 9 + + tc 6 wred min = 48 40 32 + tc 6 wred max = 64 64 64 + tc 6 wred inv prob = 10 10 10 + tc 6 wred weight = 9 9 9 + + tc 7 wred min = 48 40 32 + tc 7 wred max = 64 64 64 + tc 7 wred inv prob = 10 10 10 + tc 7 wred weight = 9 9 9 + + tc 8 wred min = 48 40 32 + tc 8 wred max = 64 64 64 + tc 8 wred inv prob = 10 10 10 + tc 8 wred weight = 9 9 9 + + tc 9 wred min = 48 40 32 + tc 9 wred max = 64 64 64 + tc 9 wred inv prob = 10 10 10 + tc 9 wred weight = 9 9 9 + + tc 10 wred min = 48 40 32 + tc 10 wred max = 64 64 64 + tc 10 wred inv prob = 10 10 10 + tc 10 wred weight = 9 9 9 + + tc 11 wred min = 48 40 32 + tc 11 wred max = 64 64 64 + tc 11 wred inv prob = 10 10 10 + tc 11 wred weight = 9 9 9 + + tc 12 wred min = 48 40 32 + tc 12 wred max = 64 64 64 + tc 12 wred inv prob = 10 10 10 + tc 12 wred weight = 9 9 9 + +Interactive mode +~~~~~~~~~~~~~~~~ + +These are the commands that are currently working under the command line interface: + +* Control Commands + +* --quit: Quits the application. + +* General Statistics + + * stats app: Shows a table with in-app calculated statistics. + + * stats port X subport Y: For a specific subport, it shows the number of packets that + went through the scheduler properly and the number of packets that were dropped. + The same information is shown in bytes. + The information is displayed in a table separating it in different traffic classes. + + * stats port X subport Y pipe Z: For a specific pipe, it shows the number of packets that + went through the scheduler properly and the number of packets that were dropped. + The same information is shown in bytes. + This information is displayed in a table separating it in individual queues. + +* Average queue size + +All of these commands work the same way, averaging the number of packets throughout a specific subset of queues. + +Two parameters can be configured for this prior to calling any of these commands: + + * qavg n X: n is the number of times that the calculation will take place. + Bigger numbers provide higher accuracy. The default value is 10. + + * qavg period X: period is the number of microseconds that will be allowed between each calculation. + The default value is 100. + +The commands that can be used for measuring average queue size are: + +* qavg port X subport Y: Show average queue size per subport. + +* qavg port X subport Y tc Z: Show average queue size per subport for a specific traffic class. + +* qavg port X subport Y pipe Z: Show average queue size per pipe. + +* qavg port X subport Y pipe Z tc A: Show average queue size per pipe for a specific traffic class. + +* qavg port X subport Y pipe Z tc A q B: Show average queue size of a specific queue. + +Example +~~~~~~~ + +The following is an example command with a single packet flow configuration: + +.. code-block:: console + + ./qos_sched -l 1,5,7 -n 4 -- --pfc "3,2,5,7" --cfg ./profile.cfg + +This example uses a single packet flow configuration which creates one RX thread on lcore 5 reading +from port 3 and a worker thread on lcore 7 writing to port 2. + +Another example with 2 packet flow configurations using different ports but sharing the same core for QoS scheduler is given below: + +.. code-block:: console + + ./qos_sched -l 1,2,6,7 -n 4 -- --pfc "3,2,2,6,7" --pfc "1,0,2,6,7" --cfg ./profile.cfg + +Note that independent cores for the packet flow configurations for each of the RX, WT and TX thread are also supported, +providing flexibility to balance the work. + +The EAL coremask/corelist is constrained to contain the default mastercore 1 and the RX, WT and TX cores only. + +Explanation +----------- + +The Port/Subport/Pipe/Traffic Class/Queue are the hierarchical entities in a typical QoS application: + +* A subport represents a predefined group of users. + +* A pipe represents an individual user/subscriber. + +* A traffic class is the representation of a different traffic type with a specific loss rate, + delay and jitter requirements; such as data voice, video or data transfers. + +* A queue hosts packets from one or multiple connections of the same type belonging to the same user. + +The traffic flows that need to be configured are application dependent. +This application classifies based on the QinQ double VLAN tags and the IP destination address as indicated in the following table. + +.. _table_qos_scheduler_1: + +.. table:: Entity Types + + +----------------+-------------------------+--------------------------------------------------+----------------------------------+ + | **Level Name** | **Siblings per Parent** | **QoS Functional Description** | **Selected By** | + | | | | | + +================+=========================+==================================================+==================================+ + | Port | - | Ethernet port | Physical port | + | | | | | + +----------------+-------------------------+--------------------------------------------------+----------------------------------+ + | Subport | Config (8) | Traffic shaped (token bucket) | Outer VLAN tag | + | | | | | + +----------------+-------------------------+--------------------------------------------------+----------------------------------+ + | Pipe | Config (4k) | Traffic shaped (token bucket) | Inner VLAN tag | + | | | | | + +----------------+-------------------------+--------------------------------------------------+----------------------------------+ + | Traffic Class | 13 | TCs of the same pipe services in strict priority | Destination IP address (0.0.0.X) | + | | | | | + +----------------+-------------------------+--------------------------------------------------+----------------------------------+ + | Queue | High Priority TC: 1, | Queue of lowest priority traffic | Destination IP address (0.0.0.X) | + | | Lowest Priority TC: 4 | class (Best effort) serviced in WRR | | + +----------------+-------------------------+--------------------------------------------------+----------------------------------+ + +Please refer to the "QoS Scheduler" chapter in the *DPDK Programmer's Guide* for more information about these parameters. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/rxtx_callbacks.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/rxtx_callbacks.rst new file mode 100644 index 000000000..0a69ec71a --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/rxtx_callbacks.rst @@ -0,0 +1,207 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015 Intel Corporation. + +RX/TX Callbacks Sample Application +================================== + +The RX/TX Callbacks sample application is a packet forwarding application that +demonstrates the use of user defined callbacks on received and transmitted +packets. The application performs a simple latency check, using callbacks, to +determine the time packets spend within the application. + +In the sample application a user defined callback is applied to all received +packets to add a timestamp. A separate callback is applied to all packets +prior to transmission to calculate the elapsed time, in CPU cycles. + +If hardware timestamping is supported by the NIC, the sample application will +also display the average latency since the packet was timestamped in hardware, +on top of the latency since the packet was received and processed by the RX +callback. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``rxtx_callbacks`` sub-directory. + +The callbacks feature requires that the ``CONFIG_RTE_ETHDEV_RXTX_CALLBACKS`` +setting is on in the ``config/common_`` config file that applies to the +target. This is generally on by default: + +.. code-block:: console + + CONFIG_RTE_ETHDEV_RXTX_CALLBACKS=y + +Running the Application +----------------------- + +To run the example in a ``linux`` environment: + +.. code-block:: console + + ./build/rxtx_callbacks -l 1 -n 4 -- [-t] + +Use -t to enable hardware timestamping. If not supported by the NIC, an error +will be displayed. + +Refer to *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + + + +Explanation +----------- + +The ``rxtx_callbacks`` application is mainly a simple forwarding application +based on the :doc:`skeleton`. See that section of the documentation for more +details of the forwarding part of the application. + +The sections below explain the additional RX/TX callback code. + + +The Main Function +~~~~~~~~~~~~~~~~~ + +The ``main()`` function performs the application initialization and calls the +execution threads for each lcore. This function is effectively identical to +the ``main()`` function explained in :doc:`skeleton`. + +The ``lcore_main()`` function is also identical. + +The main difference is in the user defined ``port_init()`` function where the +callbacks are added. This is explained in the next section: + + +The Port Initialization Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The main functional part of the port initialization is shown below with +comments: + +.. code-block:: c + + static inline int + port_init(uint16_t port, struct rte_mempool *mbuf_pool) + { + struct rte_eth_conf port_conf = port_conf_default; + const uint16_t rx_rings = 1, tx_rings = 1; + struct rte_ether_addr addr; + int retval; + uint16_t q; + + /* Configure the Ethernet device. */ + retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf); + if (retval != 0) + return retval; + + /* Allocate and set up 1 RX queue per Ethernet port. */ + for (q = 0; q < rx_rings; q++) { + retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE, + rte_eth_dev_socket_id(port), NULL, mbuf_pool); + if (retval < 0) + return retval; + } + + /* Allocate and set up 1 TX queue per Ethernet port. */ + for (q = 0; q < tx_rings; q++) { + retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE, + rte_eth_dev_socket_id(port), NULL); + if (retval < 0) + return retval; + } + + /* Start the Ethernet port. */ + retval = rte_eth_dev_start(port); + if (retval < 0) + return retval; + + /* Enable RX in promiscuous mode for the Ethernet device. */ + retval = rte_eth_promiscuous_enable(port); + if (retval != 0) + return retval; + + /* Add the callbacks for RX and TX.*/ + rte_eth_add_rx_callback(port, 0, add_timestamps, NULL); + rte_eth_add_tx_callback(port, 0, calc_latency, NULL); + + return 0; + } + + +The RX and TX callbacks are added to the ports/queues as function pointers: + +.. code-block:: c + + rte_eth_add_rx_callback(port, 0, add_timestamps, NULL); + rte_eth_add_tx_callback(port, 0, calc_latency, NULL); + +More than one callback can be added and additional information can be passed +to callback function pointers as a ``void*``. In the examples above ``NULL`` +is used. + +The ``add_timestamps()`` and ``calc_latency()`` functions are explained below. + + +The add_timestamps() Callback +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``add_timestamps()`` callback is added to the RX port and is applied to +all packets received: + +.. code-block:: c + + static uint16_t + add_timestamps(uint16_t port __rte_unused, uint16_t qidx __rte_unused, + struct rte_mbuf **pkts, uint16_t nb_pkts, void *_ __rte_unused) + { + unsigned i; + uint64_t now = rte_rdtsc(); + + for (i = 0; i < nb_pkts; i++) + pkts[i]->udata64 = now; + + return nb_pkts; + } + +The DPDK function ``rte_rdtsc()`` is used to add a cycle count timestamp to +each packet (see the *cycles* section of the *DPDK API Documentation* for +details). + + +The calc_latency() Callback +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``calc_latency()`` callback is added to the TX port and is applied to all +packets prior to transmission: + +.. code-block:: c + + static uint16_t + calc_latency(uint16_t port __rte_unused, uint16_t qidx __rte_unused, + struct rte_mbuf **pkts, uint16_t nb_pkts, void *_ __rte_unused) + { + uint64_t cycles = 0; + uint64_t now = rte_rdtsc(); + unsigned i; + + for (i = 0; i < nb_pkts; i++) + cycles += now - pkts[i]->udata64; + + latency_numbers.total_cycles += cycles; + latency_numbers.total_pkts += nb_pkts; + + if (latency_numbers.total_pkts > (100 * 1000 * 1000ULL)) { + printf("Latency = %"PRIu64" cycles\n", + latency_numbers.total_cycles / latency_numbers.total_pkts); + + latency_numbers.total_cycles = latency_numbers.total_pkts = 0; + } + + return nb_pkts; + } + +The ``calc_latency()`` function accumulates the total number of packets and +the total number of cycles used. Once more than 100 million packets have been +transmitted the average cycle count per packet is printed out and the counters +are reset. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/server_node_efd.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/server_node_efd.rst new file mode 100644 index 000000000..12208f425 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/server_node_efd.rst @@ -0,0 +1,450 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2016-2017 Intel Corporation. + +Server-Node EFD Sample Application +================================== + +This sample application demonstrates the use of EFD library as a flow-level +load balancer, for more information about the EFD Library please refer to the +DPDK programmer's guide. + +This sample application is a variant of the +:ref:`client-server sample application <multi_process_app>` +where a specific target node is specified for every and each flow +(not in a round-robin fashion as the original load balancing sample application). + +Overview +-------- + +The architecture of the EFD flow-based load balancer sample application is +presented in the following figure. + +.. _figure_efd_sample_app_overview: + +.. figure:: img/server_node_efd.* + + Using EFD as a Flow-Level Load Balancer + +As shown in :numref:`figure_efd_sample_app_overview`, +the sample application consists of a front-end node (server) +using the EFD library to create a load-balancing table for flows, +for each flow a target backend worker node is specified. The EFD table does not +store the flow key (unlike a regular hash table), and hence, it can +individually load-balance millions of flows (number of targets * maximum number +of flows fit in a flow table per target) while still fitting in CPU cache. + +It should be noted that although they are referred to as nodes, the frontend +server and worker nodes are processes running on the same platform. + +Front-end Server +~~~~~~~~~~~~~~~~ + +Upon initializing, the frontend server node (process) creates a flow +distributor table (based on the EFD library) which is populated with flow +information and its intended target node. + +The sample application assigns a specific target node_id (process) for each of +the IP destination addresses as follows: + +.. code-block:: c + + node_id = i % num_nodes; /* Target node id is generated */ + ip_dst = rte_cpu_to_be_32(i); /* Specific ip destination address is + assigned to this target node */ + +then the pair of <key,target> is inserted into the flow distribution table. + +The main loop of the server process receives a burst of packets, then for +each packet, a flow key (IP destination address) is extracted. The flow +distributor table is looked up and the target node id is returned. Packets are +then enqueued to the specified target node id. + +It should be noted that flow distributor table is not a membership test table. +I.e. if the key has already been inserted the target node id will be correct, +but for new keys the flow distributor table will return a value (which can be +valid). + +Backend Worker Nodes +~~~~~~~~~~~~~~~~~~~~ + +Upon initializing, the worker node (process) creates a flow table (a regular +hash table that stores the key default size 1M flows) which is populated with +only the flow information that is serviced at this node. This flow key is +essential to point out new keys that have not been inserted before. + +The worker node's main loop is simply receiving packets then doing a hash table +lookup. If a match occurs then statistics are updated for flows serviced by +this node. If no match is found in the local hash table then this indicates +that this is a new flow, which is dropped. + + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``server_node_efd`` sub-directory. + +Running the Application +----------------------- + +The application has two binaries to be run: the front-end server +and the back-end node. + +The frontend server (server) has the following command line options:: + + ./server [EAL options] -- -p PORTMASK -n NUM_NODES -f NUM_FLOWS + +Where, + +* ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure +* ``-n NUM_NODES:`` Number of back-end nodes that will be used +* ``-f NUM_FLOWS:`` Number of flows to be added in the EFD table (1 million, by default) + +The back-end node (node) has the following command line options:: + + ./node [EAL options] -- -n NODE_ID + +Where, + +* ``-n NODE_ID:`` Node ID, which cannot be equal or higher than NUM_MODES + + +First, the server app must be launched, with the number of nodes that will be run. +Once it has been started, the node instances can be run, with different NODE_ID. +These instances have to be run as secondary processes, with ``--proc-type=secondary`` +in the EAL options, which will attach to the primary process memory, and therefore, +they can access the queues created by the primary process to distribute packets. + +To successfully run the application, the command line used to start the +application has to be in sync with the traffic flows configured on the traffic +generator side. + +For examples of application command lines and traffic generator flows, please +refer to the DPDK Test Report. For more details on how to set up and run the +sample applications provided with DPDK package, please refer to the +:ref:`DPDK Getting Started Guide for Linux <linux_gsg>` and +:ref:`DPDK Getting Started Guide for FreeBSD <freebsd_gsg>`. + + +Explanation +----------- + +As described in previous sections, there are two processes in this example. + +The first process, the front-end server, creates and populates the EFD table, +which is used to distribute packets to nodes, which the number of flows +specified in the command line (1 million, by default). + + +.. code-block:: c + + static void + create_efd_table(void) + { + uint8_t socket_id = rte_socket_id(); + + /* create table */ + efd_table = rte_efd_create("flow table", num_flows * 2, sizeof(uint32_t), + 1 << socket_id, socket_id); + + if (efd_table == NULL) + rte_exit(EXIT_FAILURE, "Problem creating the flow table\n"); + } + + static void + populate_efd_table(void) + { + unsigned int i; + int32_t ret; + uint32_t ip_dst; + uint8_t socket_id = rte_socket_id(); + uint64_t node_id; + + /* Add flows in table */ + for (i = 0; i < num_flows; i++) { + node_id = i % num_nodes; + + ip_dst = rte_cpu_to_be_32(i); + ret = rte_efd_update(efd_table, socket_id, + (void *)&ip_dst, (efd_value_t)node_id); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Unable to add entry %u in " + "EFD table\n", i); + } + + printf("EFD table: Adding 0x%x keys\n", num_flows); + } + +After initialization, packets are received from the enabled ports, and the IPv4 +address from the packets is used as a key to look up in the EFD table, +which tells the node where the packet has to be distributed. + +.. code-block:: c + + static void + process_packets(uint32_t port_num __rte_unused, struct rte_mbuf *pkts[], + uint16_t rx_count, unsigned int socket_id) + { + uint16_t i; + uint8_t node; + efd_value_t data[EFD_BURST_MAX]; + const void *key_ptrs[EFD_BURST_MAX]; + + struct rte_ipv4_hdr *ipv4_hdr; + uint32_t ipv4_dst_ip[EFD_BURST_MAX]; + + for (i = 0; i < rx_count; i++) { + /* Handle IPv4 header.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(pkts[i], struct rte_ipv4_hdr *, + sizeof(struct rte_ether_hdr)); + ipv4_dst_ip[i] = ipv4_hdr->dst_addr; + key_ptrs[i] = (void *)&ipv4_dst_ip[i]; + } + + rte_efd_lookup_bulk(efd_table, socket_id, rx_count, + (const void **) key_ptrs, data); + for (i = 0; i < rx_count; i++) { + node = (uint8_t) ((uintptr_t)data[i]); + + if (node >= num_nodes) { + /* + * Node is out of range, which means that + * flow has not been inserted + */ + flow_dist_stats.drop++; + rte_pktmbuf_free(pkts[i]); + } else { + flow_dist_stats.distributed++; + enqueue_rx_packet(node, pkts[i]); + } + } + + for (i = 0; i < num_nodes; i++) + flush_rx_queue(i); + } + +The burst of packets received is enqueued in temporary buffers (per node), +and enqueued in the shared ring between the server and the node. +After this, a new burst of packets is received and this process is +repeated infinitely. + +.. code-block:: c + + static void + flush_rx_queue(uint16_t node) + { + uint16_t j; + struct node *cl; + + if (cl_rx_buf[node].count == 0) + return; + + cl = &nodes[node]; + if (rte_ring_enqueue_bulk(cl->rx_q, (void **)cl_rx_buf[node].buffer, + cl_rx_buf[node].count, NULL) != cl_rx_buf[node].count){ + for (j = 0; j < cl_rx_buf[node].count; j++) + rte_pktmbuf_free(cl_rx_buf[node].buffer[j]); + cl->stats.rx_drop += cl_rx_buf[node].count; + } else + cl->stats.rx += cl_rx_buf[node].count; + + cl_rx_buf[node].count = 0; + } + +The second process, the back-end node, receives the packets from the shared +ring with the server and send them out, if they belong to the node. + +At initialization, it attaches to the server process memory, to have +access to the shared ring, parameters and statistics. + +.. code-block:: c + + rx_ring = rte_ring_lookup(get_rx_queue_name(node_id)); + if (rx_ring == NULL) + rte_exit(EXIT_FAILURE, "Cannot get RX ring - " + "is server process running?\n"); + + mp = rte_mempool_lookup(PKTMBUF_POOL_NAME); + if (mp == NULL) + rte_exit(EXIT_FAILURE, "Cannot get mempool for mbufs\n"); + + mz = rte_memzone_lookup(MZ_SHARED_INFO); + if (mz == NULL) + rte_exit(EXIT_FAILURE, "Cannot get port info structure\n"); + info = mz->addr; + tx_stats = &(info->tx_stats[node_id]); + filter_stats = &(info->filter_stats[node_id]); + +Then, the hash table that contains the flows that will be handled +by the node is created and populated. + +.. code-block:: c + + static struct rte_hash * + create_hash_table(const struct shared_info *info) + { + uint32_t num_flows_node = info->num_flows / info->num_nodes; + char name[RTE_HASH_NAMESIZE]; + struct rte_hash *h; + + /* create table */ + struct rte_hash_parameters hash_params = { + .entries = num_flows_node * 2, /* table load = 50% */ + .key_len = sizeof(uint32_t), /* Store IPv4 dest IP address */ + .socket_id = rte_socket_id(), + .hash_func_init_val = 0, + }; + + snprintf(name, sizeof(name), "hash_table_%d", node_id); + hash_params.name = name; + h = rte_hash_create(&hash_params); + + if (h == NULL) + rte_exit(EXIT_FAILURE, + "Problem creating the hash table for node %d\n", + node_id); + return h; + } + + static void + populate_hash_table(const struct rte_hash *h, const struct shared_info *info) + { + unsigned int i; + int32_t ret; + uint32_t ip_dst; + uint32_t num_flows_node = 0; + uint64_t target_node; + + /* Add flows in table */ + for (i = 0; i < info->num_flows; i++) { + target_node = i % info->num_nodes; + if (target_node != node_id) + continue; + + ip_dst = rte_cpu_to_be_32(i); + + ret = rte_hash_add_key(h, (void *) &ip_dst); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Unable to add entry %u " + "in hash table\n", i); + else + num_flows_node++; + + } + + printf("Hash table: Adding 0x%x keys\n", num_flows_node); + } + +After initialization, packets are dequeued from the shared ring +(from the server) and, like in the server process, +the IPv4 address from the packets is used as a key to look up in the hash table. +If there is a hit, packet is stored in a buffer, to be eventually transmitted +in one of the enabled ports. If key is not there, packet is dropped, since the +flow is not handled by the node. + +.. code-block:: c + + static inline void + handle_packets(struct rte_hash *h, struct rte_mbuf **bufs, uint16_t num_packets) + { + struct rte_ipv4_hdr *ipv4_hdr; + uint32_t ipv4_dst_ip[PKT_READ_SIZE]; + const void *key_ptrs[PKT_READ_SIZE]; + unsigned int i; + int32_t positions[PKT_READ_SIZE] = {0}; + + for (i = 0; i < num_packets; i++) { + /* Handle IPv4 header.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(bufs[i], struct rte_ipv4_hdr *, + sizeof(struct rte_ether_hdr)); + ipv4_dst_ip[i] = ipv4_hdr->dst_addr; + key_ptrs[i] = &ipv4_dst_ip[i]; + } + /* Check if packets belongs to any flows handled by this node */ + rte_hash_lookup_bulk(h, key_ptrs, num_packets, positions); + + for (i = 0; i < num_packets; i++) { + if (likely(positions[i] >= 0)) { + filter_stats->passed++; + transmit_packet(bufs[i]); + } else { + filter_stats->drop++; + /* Drop packet, as flow is not handled by this node */ + rte_pktmbuf_free(bufs[i]); + } + } + } + +Finally, note that both processes updates statistics, such as transmitted, received +and dropped packets, which are shown and refreshed by the server app. + +.. code-block:: c + + static void + do_stats_display(void) + { + unsigned int i, j; + const char clr[] = {27, '[', '2', 'J', '\0'}; + const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'}; + uint64_t port_tx[RTE_MAX_ETHPORTS], port_tx_drop[RTE_MAX_ETHPORTS]; + uint64_t node_tx[MAX_NODES], node_tx_drop[MAX_NODES]; + + /* to get TX stats, we need to do some summing calculations */ + memset(port_tx, 0, sizeof(port_tx)); + memset(port_tx_drop, 0, sizeof(port_tx_drop)); + memset(node_tx, 0, sizeof(node_tx)); + memset(node_tx_drop, 0, sizeof(node_tx_drop)); + + for (i = 0; i < num_nodes; i++) { + const struct tx_stats *tx = &info->tx_stats[i]; + + for (j = 0; j < info->num_ports; j++) { + const uint64_t tx_val = tx->tx[info->id[j]]; + const uint64_t drop_val = tx->tx_drop[info->id[j]]; + + port_tx[j] += tx_val; + port_tx_drop[j] += drop_val; + node_tx[i] += tx_val; + node_tx_drop[i] += drop_val; + } + } + + /* Clear screen and move to top left */ + printf("%s%s", clr, topLeft); + + printf("PORTS\n"); + printf("-----\n"); + for (i = 0; i < info->num_ports; i++) + printf("Port %u: '%s'\t", (unsigned int)info->id[i], + get_printable_mac_addr(info->id[i])); + printf("\n\n"); + for (i = 0; i < info->num_ports; i++) { + printf("Port %u - rx: %9"PRIu64"\t" + "tx: %9"PRIu64"\n", + (unsigned int)info->id[i], info->rx_stats.rx[i], + port_tx[i]); + } + + printf("\nSERVER\n"); + printf("-----\n"); + printf("distributed: %9"PRIu64", drop: %9"PRIu64"\n", + flow_dist_stats.distributed, flow_dist_stats.drop); + + printf("\nNODES\n"); + printf("-------\n"); + for (i = 0; i < num_nodes; i++) { + const unsigned long long rx = nodes[i].stats.rx; + const unsigned long long rx_drop = nodes[i].stats.rx_drop; + const struct filter_stats *filter = &info->filter_stats[i]; + + printf("Node %2u - rx: %9llu, rx_drop: %9llu\n" + " tx: %9"PRIu64", tx_drop: %9"PRIu64"\n" + " filter_passed: %9"PRIu64", " + "filter_drop: %9"PRIu64"\n", + i, rx, rx_drop, node_tx[i], node_tx_drop[i], + filter->passed, filter->drop); + } + + printf("\n"); + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/service_cores.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/service_cores.rst new file mode 100644 index 000000000..cd0f4717a --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/service_cores.rst @@ -0,0 +1,145 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017 Intel Corporation. + +Service Cores Sample Application +================================ + +The service cores sample application demonstrates the service cores capabilities +of DPDK. The service cores infrastructure is part of the DPDK EAL, and allows +any DPDK component to register a service. A service is a work item or task, that +requires CPU time to perform its duty. + +This sample application registers 5 dummy services. These 5 services are used +to show how the service_cores API can be used to orchestrate these services to +run on different service lcores. This orchestration is done by calling the +service cores APIs, however the sample application introduces a "profile" +concept to contain the service mapping details. Note that the profile concept +is application specific, and not a part of the service cores API. + + +Compiling the Application +------------------------- + +#. Go to the example directory: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK}/examples/service_cores + +#. Set the target (a default target is used if not specified). For example: + + .. code-block:: console + + export RTE_TARGET=x86_64-native-linux-gcc + + See the *DPDK Getting Started* Guide for possible RTE_TARGET values. + +#. Build the application: + + .. code-block:: console + + make + +Running the Application +----------------------- + +To run the example, just execute the binary. Since the application dynamically +adds service cores in the application code itself, there is no requirement to +pass a service core-mask as an EAL argument at startup time. + +.. code-block:: console + + $ ./build/service_cores + + +Explanation +----------- + +The following sections provide some explanation of code focusing on +registering applications from an applications point of view, and modifying the +service core counts and mappings at runtime. + + +Registering a Service +~~~~~~~~~~~~~~~~~~~~~ + +The following code section shows how to register a service as an application. +Note that the service component header must be included by the application in +order to register services: ``rte_service_component.h``, in addition +to the ordinary service cores header ``rte_service.h`` which provides +the runtime functions to add, remove and remap service cores. + +.. code-block:: c + + struct rte_service_spec service = { + .name = "service_name", + }; + int ret = rte_service_component_register(services, &id); + if (ret) + return -1; + + /* set the service itself to be ready to run. In the case of + * ethdev, eventdev etc PMDs, this will be set when the + * appropriate configure or setup function is called. + */ + rte_service_component_runstate_set(id, 1); + + /* Collect statistics for the service */ + rte_service_set_stats_enable(id, 1); + + /* The application sets the service to running state. Note that this + * function enables the service to run - while the 'component' version + * of this function (as above) marks the service itself as ready */ + ret = rte_service_runstate_set(id, 1); + + +Controlling A Service Core +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section demonstrates how to add a service core. The ``rte_service.h`` +header file provides the functions for dynamically adding and removing cores. +The APIs to add and remove cores use lcore IDs similar to existing DPDK +functions. + +These are the functions to start a service core, and have it run a service: + +.. code-block:: c + + /* the lcore ID to use as a service core */ + uint32_t service_core_id = 7; + ret = rte_service_lcore_add(service_core_id); + if(ret) + return -1; + + /* service cores are in "stopped" state when added, so start it */ + ret = rte_service_lcore_start(service_core_id); + if(ret) + return -1; + + /* map a service to the service core, causing it to run the service */ + uint32_t service_id; /* ID of a registered service */ + uint32_t enable = 1; /* 1 maps the service, 0 unmaps */ + ret = rte_service_map_lcore_set(service_id, service_core_id, enable); + if(ret) + return -1; + + +Removing A Service Core +~~~~~~~~~~~~~~~~~~~~~~~ + +To remove a service core, the steps are similar to adding but in reverse order. +Note that it is not allowed to remove a service core if the service is running, +and the service-core is the only core running that service (see documentation +for ``rte_service_lcore_stop`` function for details). + + +Conclusion +~~~~~~~~~~ + +The service cores infrastructure provides DPDK with two main features. The first +is to abstract away hardware differences: the service core can CPU cycles to +a software fallback implementation, allowing the application to be abstracted +from the difference in HW / SW availability. The second feature is a flexible +method of registering functions to be run, allowing the running of the +functions to be scaled across multiple CPUs. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/skeleton.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/skeleton.rst new file mode 100644 index 000000000..34f6f9aea --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/skeleton.rst @@ -0,0 +1,290 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2015 Intel Corporation. + +Basic Forwarding Sample Application +=================================== + +The Basic Forwarding sample application is a simple *skeleton* example of a +forwarding application. + +It is intended as a demonstration of the basic components of a DPDK forwarding +application. For more detailed implementations see the L2 and L3 forwarding +sample applications. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``skeleton`` sub-directory. + +Running the Application +----------------------- + +To run the example in a ``linux`` environment: + +.. code-block:: console + + ./build/basicfwd -l 1 -n 4 + +Refer to *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + + +Explanation +----------- + +The following sections provide an explanation of the main components of the +code. + +All DPDK library functions used in the sample code are prefixed with ``rte_`` +and are explained in detail in the *DPDK API Documentation*. + + +The Main Function +~~~~~~~~~~~~~~~~~ + +The ``main()`` function performs the initialization and calls the execution +threads for each lcore. + +The first task is to initialize the Environment Abstraction Layer (EAL). The +``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` +function. The value returned is the number of parsed arguments: + +.. code-block:: c + + int ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); + + +The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers) +used by the application: + +.. code-block:: c + + mbuf_pool = rte_mempool_create("MBUF_POOL", + NUM_MBUFS * nb_ports, + MBUF_SIZE, + MBUF_CACHE_SIZE, + sizeof(struct rte_pktmbuf_pool_private), + rte_pktmbuf_pool_init, NULL, + rte_pktmbuf_init, NULL, + rte_socket_id(), + 0); + +Mbufs are the packet buffer structure used by DPDK. They are explained in +detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*. + +The ``main()`` function also initializes all the ports using the user defined +``port_init()`` function which is explained in the next section: + +.. code-block:: c + + RTE_ETH_FOREACH_DEV(portid) { + if (port_init(portid, mbuf_pool) != 0) { + rte_exit(EXIT_FAILURE, + "Cannot init port %" PRIu8 "\n", portid); + } + } + + +Once the initialization is complete, the application is ready to launch a +function on an lcore. In this example ``lcore_main()`` is called on a single +lcore. + + +.. code-block:: c + + lcore_main(); + +The ``lcore_main()`` function is explained below. + + + +The Port Initialization Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The main functional part of the port initialization used in the Basic +Forwarding application is shown below: + +.. code-block:: c + + static inline int + port_init(uint16_t port, struct rte_mempool *mbuf_pool) + { + struct rte_eth_conf port_conf = port_conf_default; + const uint16_t rx_rings = 1, tx_rings = 1; + struct rte_ether_addr addr; + int retval; + uint16_t q; + + if (!rte_eth_dev_is_valid_port(port)) + return -1; + + /* Configure the Ethernet device. */ + retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf); + if (retval != 0) + return retval; + + /* Allocate and set up 1 RX queue per Ethernet port. */ + for (q = 0; q < rx_rings; q++) { + retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE, + rte_eth_dev_socket_id(port), NULL, mbuf_pool); + if (retval < 0) + return retval; + } + + /* Allocate and set up 1 TX queue per Ethernet port. */ + for (q = 0; q < tx_rings; q++) { + retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE, + rte_eth_dev_socket_id(port), NULL); + if (retval < 0) + return retval; + } + + /* Start the Ethernet port. */ + retval = rte_eth_dev_start(port); + if (retval < 0) + return retval; + + /* Enable RX in promiscuous mode for the Ethernet device. */ + retval = rte_eth_promiscuous_enable(port); + if (retval != 0) + return retval; + + return 0; + } + +The Ethernet ports are configured with default settings using the +``rte_eth_dev_configure()`` function and the ``port_conf_default`` struct: + +.. code-block:: c + + static const struct rte_eth_conf port_conf_default = { + .rxmode = { .max_rx_pkt_len = RTE_ETHER_MAX_LEN } + }; + +For this example the ports are set up with 1 RX and 1 TX queue using the +``rte_eth_rx_queue_setup()`` and ``rte_eth_tx_queue_setup()`` functions. + +The Ethernet port is then started: + +.. code-block:: c + + retval = rte_eth_dev_start(port); + + +Finally the RX port is set in promiscuous mode: + +.. code-block:: c + + retval = rte_eth_promiscuous_enable(port); + + +The Lcores Main +~~~~~~~~~~~~~~~ + +As we saw above the ``main()`` function calls an application function on the +available lcores. For the Basic Forwarding application the lcore function +looks like the following: + +.. code-block:: c + + static __rte_noreturn void + lcore_main(void) + { + uint16_t port; + + /* + * Check that the port is on the same NUMA node as the polling thread + * for best performance. + */ + RTE_ETH_FOREACH_DEV(port) + if (rte_eth_dev_socket_id(port) > 0 && + rte_eth_dev_socket_id(port) != + (int)rte_socket_id()) + printf("WARNING, port %u is on remote NUMA node to " + "polling thread.\n\tPerformance will " + "not be optimal.\n", port); + + printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n", + rte_lcore_id()); + + /* Run until the application is quit or killed. */ + for (;;) { + /* + * Receive packets on a port and forward them on the paired + * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc. + */ + RTE_ETH_FOREACH_DEV(port) { + + /* Get burst of RX packets, from first port of pair. */ + struct rte_mbuf *bufs[BURST_SIZE]; + const uint16_t nb_rx = rte_eth_rx_burst(port, 0, + bufs, BURST_SIZE); + + if (unlikely(nb_rx == 0)) + continue; + + /* Send burst of TX packets, to second port of pair. */ + const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0, + bufs, nb_rx); + + /* Free any unsent packets. */ + if (unlikely(nb_tx < nb_rx)) { + uint16_t buf; + for (buf = nb_tx; buf < nb_rx; buf++) + rte_pktmbuf_free(bufs[buf]); + } + } + } + } + + +The main work of the application is done within the loop: + +.. code-block:: c + + for (;;) { + RTE_ETH_FOREACH_DEV(port) { + + /* Get burst of RX packets, from first port of pair. */ + struct rte_mbuf *bufs[BURST_SIZE]; + const uint16_t nb_rx = rte_eth_rx_burst(port, 0, + bufs, BURST_SIZE); + + if (unlikely(nb_rx == 0)) + continue; + + /* Send burst of TX packets, to second port of pair. */ + const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0, + bufs, nb_rx); + + /* Free any unsent packets. */ + if (unlikely(nb_tx < nb_rx)) { + uint16_t buf; + for (buf = nb_tx; buf < nb_rx; buf++) + rte_pktmbuf_free(bufs[buf]); + } + } + } + +Packets are received in bursts on the RX ports and transmitted in bursts on +the TX ports. The ports are grouped in pairs with a simple mapping scheme +using the an XOR on the port number:: + + 0 -> 1 + 1 -> 0 + + 2 -> 3 + 3 -> 2 + + etc. + +The ``rte_eth_tx_burst()`` function frees the memory buffers of packets that +are transmitted. If packets fail to transmit, ``(nb_tx < nb_rx)``, then they +must be freed explicitly using ``rte_pktmbuf_free()``. + +The forwarding loop can be interrupted and the application closed using +``Ctrl-C``. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/tep_termination.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/tep_termination.rst new file mode 100644 index 000000000..df159355d --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/tep_termination.rst @@ -0,0 +1,233 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2015 Intel Corporation. + +TEP termination Sample Application +================================== + +The TEP (Tunnel End point) termination sample application simulates a VXLAN +Tunnel Endpoint (VTEP) termination in DPDK, which is used to demonstrate +the offload and filtering capabilities of Intel® XL710 10/40 Gigabit Ethernet +Controller for VXLAN packet. +This sample uses the basic virtio devices management mechanism from vhost example, +and also uses the us-vHost interface and tunnel filtering mechanism to direct +a specified traffic to a specific VM. +In addition, this sample is also designed to show how tunneling protocols can be handled. + +Background +---------- + +With virtualization, overlay networks allow a network structure to be built +or imposed across physical nodes which is abstracted away from the actual +underlining physical network connections. +This allows network isolation, QOS, etc to be provided on a per client basis. + +.. _figure_overlay_networking: + +.. figure:: img/overlay_networking.* + + Overlay Networking. + +In a typical setup, the network overlay tunnel is terminated at the Virtual/Tunnel End Point (VEP/TEP). +The TEP is normally located at the physical host level ideally in the software switch. +Due to processing constraints and the inevitable bottleneck that the switch +becomes, the ability to offload overlay support features becomes an important requirement. +Intel® XL710 10/40 Gigabit Ethernet network card provides hardware filtering +and offload capabilities to support overlay networks implementations such as MAC in UDP and MAC in GRE. + +Sample Code Overview +-------------------- + +The DPDK TEP termination sample code demonstrates the offload and filtering +capabilities of Intel® XL710 10/40 Gigabit Ethernet Controller for VXLAN packet. + +The sample code is based on vhost library. +The vhost library is developed for user space Ethernet switch to easily integrate with vhost functionality. + +The sample will support the followings: + +* Tunneling packet recognition. + +* The port of UDP tunneling is configurable + +* Directing incoming traffic to the correct queue based on the tunnel filter type. + The supported filter type are listed below. + + * Inner MAC and VLAN and tenant ID + + * Inner MAC and tenant ID, and Outer MAC + + * Inner MAC and tenant ID + + The tenant ID will be assigned from a static internal table based on the us-vhost device ID. + Each device will receive a unique device ID. + The inner MAC will be learned by the first packet transmitted from a device. + +* Decapsulation of RX VXLAN traffic. This is a software only operation. + +* Encapsulation of TX VXLAN traffic. This is a software only operation. + +* Inner IP and inner L4 checksum offload. + +* TSO offload support for tunneling packet. + +The following figure shows the framework of the TEP termination sample +application based on DPDK vhost lib. + +.. _figure_tep_termination_arch: + +.. figure:: img/tep_termination_arch.* + + TEP termination Framework Overview + +Supported Distributions +----------------------- + +The example in this section have been validated with the following distributions: + +* Fedora* 18 + +* Fedora* 19 + +* Fedora* 20 + +Compiling the Sample Code +------------------------- + +To enable vhost, turn on vhost library in the configure file +``config/common_linux``. + + .. code-block:: console + + CONFIG_RTE_LIBRTE_VHOST=y + +Then following the to compile the sample application shown in +:doc:`compiling`. + +Running the Sample Code +----------------------- + +#. Go to the examples directory: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK}/examples/tep_termination + +#. Run the tep_termination sample code: + + .. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + -p 0x1 --dev-basename tep-termination --nb-devices 4 + --udp-port 4789 --filter-type 1 + +.. note:: + + Please note the huge-dir parameter instructs the DPDK to allocate its memory from the 2 MB page hugetlbfs. + +Parameters +~~~~~~~~~~ + +**The same parameters with the vhost sample.** + +Refer to :ref:`vhost_app_parameters` for detailed explanation. + +**Number of Devices.** + +The nb-devices option specifies the number of virtIO device. +The default value is 2. + +.. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + --nb-devices 2 + +**Tunneling UDP port.** + +The udp-port option is used to specify the destination UDP number for UDP tunneling packet. +The default value is 4789. + +.. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + --nb-devices 2 --udp-port 4789 + +**Filter Type.** + +The filter-type option is used to specify which filter type is used to +filter UDP tunneling packet to a specified queue. +The default value is 1, which means the filter type of inner MAC and tenant ID is used. + +.. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + --nb-devices 2 --udp-port 4789 --filter-type 1 + +**TX Checksum.** + +The tx-checksum option is used to enable or disable the inner header checksum offload. +The default value is 0, which means the checksum offload is disabled. + +.. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + --nb-devices 2 --tx-checksum + +**TCP segment size.** + +The tso-segsz option specifies the TCP segment size for TSO offload for tunneling packet. +The default value is 0, which means TSO offload is disabled. + +.. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + --tx-checksum --tso-segsz 800 + +**Decapsulation option.** + +The decap option is used to enable or disable decapsulation operation for received VXLAN packet. +The default value is 1. + +.. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + --nb-devices 4 --udp-port 4789 --decap 1 + +**Encapsulation option.** + +The encap option is used to enable or disable encapsulation operation for transmitted packet. +The default value is 1. + +.. code-block:: console + + user@target:~$ ./build/app/tep_termination -l 0-3 -n 4 --huge-dir /mnt/huge -- + --nb-devices 4 --udp-port 4789 --encap 1 + + +Running the Virtual Machine (QEMU) +---------------------------------- + +Refer to :ref:`vhost_app_run_vm`. + +Running DPDK in the Virtual Machine +----------------------------------- + +Refer to :ref:`vhost_app_run_dpdk_inside_guest`. + +Passing Traffic to the Virtual Machine Device +--------------------------------------------- + +For a virtio-net device to receive traffic, the traffic's Layer 2 header must include +both the virtio-net device's MAC address. +The DPDK sample code behaves in a similar manner to a learning switch in that +it learns the MAC address of the virtio-net devices from the first transmitted packet. +On learning the MAC address, +the DPDK vhost sample code prints a message with the MAC address and tenant ID virtio-net device. +For example: + +.. code-block:: console + + DATA: (0) MAC_ADDRESS cc:bb:bb:bb:bb:bb and VNI 1000 registered + +The above message indicates that device 0 has been registered with MAC address cc:bb:bb:bb:bb:bb and VNI 1000. +Any packets received on the NIC with these values are placed on the devices receive queue. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/test_pipeline.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/test_pipeline.rst new file mode 100644 index 000000000..5aefd8d89 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/test_pipeline.rst @@ -0,0 +1,239 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Test Pipeline Application +========================= + +The Test Pipeline application illustrates the use of the DPDK Packet Framework tool suite. +Its purpose is to demonstrate the performance of single-table DPDK pipelines. + +Overview +-------- + +The application uses three CPU cores: + +* Core A ("RX core") receives traffic from the NIC ports and feeds core B with traffic through SW queues. + +* Core B ("Pipeline core") implements a single-table DPDK pipeline + whose type is selectable through specific command line parameter. + Core B receives traffic from core A through software queues, + processes it according to the actions configured in the table entries that + are hit by the input packets and feeds it to core C through another set of software queues. + +* Core C ("TX core") receives traffic from core B through software queues and sends it to the NIC ports for transmission. + +.. _figure_test_pipeline_app: + +.. figure:: img/test_pipeline_app.* + + Test Pipeline Application + +Compiling the Application +------------------------- +To compile the sample application see :doc:`compiling` + +The application is located in the ``$RTE_SDK/app/test-pipeline`` directory. + + +Running the Application +----------------------- + +Application Command Line +~~~~~~~~~~~~~~~~~~~~~~~~ + +The application execution command line is: + +.. code-block:: console + + ./test-pipeline [EAL options] -- -p PORTMASK --TABLE_TYPE + +The -c or -l EAL CPU coremask/corelist option has to contain exactly 3 CPU cores. +The first CPU core in the core mask is assigned for core A, the second for core B and the third for core C. + +The PORTMASK parameter must contain 2 or 4 ports. + +Table Types and Behavior +~~~~~~~~~~~~~~~~~~~~~~~~ + +:numref:`table_test_pipeline_1` describes the table types used and how they are populated. + +The hash tables are pre-populated with 16 million keys. +For hash tables, the following parameters can be selected: + +* **Configurable key size implementation or fixed (specialized) key size implementation (e.g. hash-8-ext or hash-spec-8-ext).** + The key size specialized implementations are expected to provide better performance for 8-byte and 16-byte key sizes, + while the key-size-non-specialized implementation is expected to provide better performance for larger key sizes; + +* **Key size (e.g. hash-spec-8-ext or hash-spec-16-ext).** + The available options are 8, 16 and 32 bytes; + +* **Table type (e.g. hash-spec-16-ext or hash-spec-16-lru).** + The available options are ext (extendable bucket) or lru (least recently used). + +.. _table_test_pipeline_1: + +.. table:: Table Types + + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | **#** | **TABLE_TYPE** | **Description of Core B Table** | **Pre-added Table Entries** | + | | | | | + +=======+========================+==========================================================+=======================================================+ + | 1 | none | Core B is not implementing a DPDK pipeline. | N/A | + | | | Core B is implementing a pass-through from its input set | | + | | | of software queues to its output set of software queues. | | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 2 | stub | Stub table. Core B is implementing the same pass-through | N/A | + | | | functionality as described for the "none" option by | | + | | | using the DPDK Packet Framework by using one | | + | | | stub table for each input NIC port. | | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 3 | hash-[spec]-8-lru | LRU hash table with 8-byte key size and 16 million | 16 million entries are successfully added to the | + | | | entries. | hash table with the following key format: | + | | | | | + | | | | [4-byte index, 4 bytes of 0] | + | | | | | + | | | | The action configured for all table entries is | + | | | | "Sendto output port", with the output port index | + | | | | uniformly distributed for the range of output ports. | + | | | | | + | | | | The default table rule (used in the case of a lookup | + | | | | miss) is to drop the packet. | + | | | | | + | | | | At run time, core A is creating the following lookup | + | | | | key and storing it into the packet meta data for | + | | | | core B to use for table lookup: | + | | | | | + | | | | [destination IPv4 address, 4 bytes of 0] | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 4 | hash-[spec]-8-ext | Extendable bucket hash table with 8-byte key size | Same as hash-[spec]-8-lru table entries, above. | + | | | and 16 million entries. | | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 5 | hash-[spec]-16-lru | LRU hash table with 16-byte key size and 16 million | 16 million entries are successfully added to the hash | + | | | entries. | table with the following key format: | + | | | | | + | | | | [4-byte index, 12 bytes of 0] | + | | | | | + | | | | The action configured for all table entries is | + | | | | "Send to output port", with the output port index | + | | | | uniformly distributed for the range of output ports. | + | | | | | + | | | | The default table rule (used in the case of a lookup | + | | | | miss) is to drop the packet. | + | | | | | + | | | | At run time, core A is creating the following lookup | + | | | | key and storing it into the packet meta data for core | + | | | | B to use for table lookup: | + | | | | | + | | | | [destination IPv4 address, 12 bytes of 0] | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 6 | hash-[spec]-16-ext | Extendable bucket hash table with 16-byte key size | Same as hash-[spec]-16-lru table entries, above. | + | | | and 16 million entries. | | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 7 | hash-[spec]-32-lru | LRU hash table with 32-byte key size and 16 million | 16 million entries are successfully added to the hash | + | | | entries. | table with the following key format: | + | | | | | + | | | | [4-byte index, 28 bytes of 0]. | + | | | | | + | | | | The action configured for all table entries is | + | | | | "Send to output port", with the output port index | + | | | | uniformly distributed for the range of output ports. | + | | | | | + | | | | The default table rule (used in the case of a lookup | + | | | | miss) is to drop the packet. | + | | | | | + | | | | At run time, core A is creating the following lookup | + | | | | key and storing it into the packet meta data for | + | | | | Lpmcore B to use for table lookup: | + | | | | | + | | | | [destination IPv4 address, 28 bytes of 0] | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 8 | hash-[spec]-32-ext | Extendable bucket hash table with 32-byte key size | Same as hash-[spec]-32-lru table entries, above. | + | | | and 16 million entries. | | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 9 | lpm | Longest Prefix Match (LPM) IPv4 table. | In the case of two ports, two routes | + | | | | are added to the table: | + | | | | | + | | | | [0.0.0.0/9 => send to output port 0] | + | | | | | + | | | | [0.128.0.0/9 => send to output port 1] | + | | | | | + | | | | In case of four ports, four entries are added to the | + | | | | table: | + | | | | | + | | | | [0.0.0.0/10 => send to output port 0] | + | | | | | + | | | | [0.64.0.0/10 => send to output port 1] | + | | | | | + | | | | [0.128.0.0/10 => send to output port 2] | + | | | | | + | | | | [0.192.0.0/10 => send to output port 3] | + | | | | | + | | | | The default table rule (used in the case of a lookup | + | | | | miss) is to drop the packet. | + | | | | | + | | | | At run time, core A is storing the IPv4 destination | + | | | | within the packet meta data to be later used by core | + | | | | B as the lookup key. | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + | 10 | acl | Access Control List (ACL) table | In the case of two ports, two ACL rules are added to | + | | | | the table: | + | | | | | + | | | | [priority = 0 (highest), | + | | | | | + | | | | IPv4 source = ANY, | + | | | | | + | | | | IPv4 destination = 0.0.0.0/9, | + | | | | | + | | | | L4 protocol = ANY, | + | | | | | + | | | | TCP source port = ANY, | + | | | | | + | | | | TCP destination port = ANY | + | | | | | + | | | | => send to output port 0] | + | | | | | + | | | | | + | | | | [priority = 0 (highest), | + | | | | | + | | | | IPv4 source = ANY, | + | | | | | + | | | | IPv4 destination = 0.128.0.0/9, | + | | | | | + | | | | L4 protocol = ANY, | + | | | | | + | | | | TCP source port = ANY, | + | | | | | + | | | | TCP destination port = ANY | + | | | | | + | | | | => send to output port 0]. | + | | | | | + | | | | | + | | | | The default table rule (used in the case of a lookup | + | | | | miss) is to drop the packet. | + | | | | | + +-------+------------------------+----------------------------------------------------------+-------------------------------------------------------+ + +Input Traffic +~~~~~~~~~~~~~ + +Regardless of the table type used for the core B pipeline, +the same input traffic can be used to hit all table entries with uniform distribution, +which results in uniform distribution of packets sent out on the set of output NIC ports. +The profile for input traffic is TCP/IPv4 packets with: + +* destination IP address as A.B.C.D with A fixed to 0 and B, C,D random + +* source IP address fixed to 0.0.0.0 + +* destination TCP port fixed to 0 + +* source TCP port fixed to 0 diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/timer.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/timer.rst new file mode 100644 index 000000000..98d762d23 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/timer.rst @@ -0,0 +1,176 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Timer Sample Application +======================== + +The Timer sample application is a simple application that demonstrates the use of a timer in a DPDK application. +This application prints some messages from different lcores regularly, demonstrating the use of timers. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``timer`` sub-directory. + +Running the Application +----------------------- + +To run the example in linux environment: + +.. code-block:: console + + $ ./build/timer -l 0-3 -n 4 + +Refer to the *DPDK Getting Started Guide* for general information on running applications and +the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. + +Initialization and Main Loop +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In addition to EAL initialization, the timer subsystem must be initialized, by calling the rte_timer_subsystem_init() function. + +.. code-block:: c + + /* init EAL */ + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_panic("Cannot init EAL\n"); + + /* init RTE timer library */ + + rte_timer_subsystem_init(); + +After timer creation (see the next paragraph), +the main loop is executed on each slave lcore using the well-known rte_eal_remote_launch() and also on the master. + +.. code-block:: c + + /* call lcore_mainloop() on every slave lcore */ + + RTE_LCORE_FOREACH_SLAVE(lcore_id) { + rte_eal_remote_launch(lcore_mainloop, NULL, lcore_id); + } + + /* call it on master lcore too */ + + (void) lcore_mainloop(NULL); + +The main loop is very simple in this example: + +.. code-block:: c + + while (1) { + /* + * Call the timer handler on each core: as we don't + * need a very precise timer, so only call + * rte_timer_manage() every ~10ms (at 2 GHz). In a real + * application, this will enhance performances as + * reading the HPET timer is not efficient. + */ + + cur_tsc = rte_rdtsc(); + + diff_tsc = cur_tsc - prev_tsc; + + if (diff_tsc > TIMER_RESOLUTION_CYCLES) { + rte_timer_manage(); + prev_tsc = cur_tsc; + } + } + +As explained in the comment, it is better to use the TSC register (as it is a per-lcore register) to check if the +rte_timer_manage() function must be called or not. +In this example, the resolution of the timer is 10 milliseconds. + +Managing Timers +~~~~~~~~~~~~~~~ + +In the main() function, the two timers are initialized. +This call to rte_timer_init() is necessary before doing any other operation on the timer structure. + +.. code-block:: c + + /* init timer structures */ + + rte_timer_init(&timer0); + rte_timer_init(&timer1); + +Then, the two timers are configured: + +* The first timer (timer0) is loaded on the master lcore and expires every second. + Since the PERIODICAL flag is provided, the timer is reloaded automatically by the timer subsystem. + The callback function is timer0_cb(). + +* The second timer (timer1) is loaded on the next available lcore every 333 ms. + The SINGLE flag means that the timer expires only once and must be reloaded manually if required. + The callback function is timer1_cb(). + +.. code-block:: c + + /* load timer0, every second, on master lcore, reloaded automatically */ + + hz = rte_get_hpet_hz(); + + lcore_id = rte_lcore_id(); + + rte_timer_reset(&timer0, hz, PERIODICAL, lcore_id, timer0_cb, NULL); + + /* load timer1, every second/3, on next lcore, reloaded manually */ + + lcore_id = rte_get_next_lcore(lcore_id, 0, 1); + + rte_timer_reset(&timer1, hz/3, SINGLE, lcore_id, timer1_cb, NULL); + +The callback for the first timer (timer0) only displays a message until a global counter reaches 20 (after 20 seconds). +In this case, the timer is stopped using the rte_timer_stop() function. + +.. code-block:: c + + /* timer0 callback */ + + static void + timer0_cb(__rte_unused struct rte_timer *tim, __rte_unused void *arg) + { + static unsigned counter = 0; + + unsigned lcore_id = rte_lcore_id(); + + printf("%s() on lcore %u\n", FUNCTION , lcore_id); + + /* this timer is automatically reloaded until we decide to stop it, when counter reaches 20. */ + + if ((counter ++) == 20) + rte_timer_stop(tim); + } + +The callback for the second timer (timer1) displays a message and reloads the timer on the next lcore, using the +rte_timer_reset() function: + +.. code-block:: c + + /* timer1 callback */ + + static void + timer1_cb(__rte_unused struct rte_timer *tim, __rte_unused void *arg) + { + unsigned lcore_id = rte_lcore_id(); + uint64_t hz; + + printf("%s() on lcore %u\\n", FUNCTION , lcore_id); + + /* reload it on another lcore */ + + hz = rte_get_hpet_hz(); + + lcore_id = rte_get_next_lcore(lcore_id, 0, 1); + + rte_timer_reset(&timer1, hz/3, SINGLE, lcore_id, timer1_cb, NULL); + } diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/vdpa.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/vdpa.rst new file mode 100644 index 000000000..745f196ca --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/vdpa.rst @@ -0,0 +1,120 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +Vdpa Sample Application +======================= + +The vdpa sample application creates vhost-user sockets by using the +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes +virtio ring compatible devices to serve virtio driver directly to enable +datapath acceleration. As vDPA driver can help to set up vhost datapath, +this application doesn't need to launch dedicated worker threads for vhost +enqueue/dequeue operations. + +Testing steps +------------- + +This section shows the steps of how to start VMs with vDPA vhost-user +backend and verify network connection & live migration. + +Build +~~~~~ + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``vdpa`` sub-directory. + +Start the vdpa example +~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ./vdpa [EAL options] -- [--client] [--interactive|-i] or [--iface SOCKET_PATH] + +where + +* --client means running vdpa app in client mode, in the client mode, QEMU needs + to run as the server mode and take charge of socket file creation. +* --iface specifies the path prefix of the UNIX domain socket file, e.g. + /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n> + (n starts from 0). +* --interactive means run the vdpa sample in interactive mode, currently 4 + internal cmds are supported: + + 1. help: show help message + 2. list: list all available vdpa devices + 3. create: create a new vdpa port with socket file and vdpa device address + 4. quit: unregister vhost driver and exit the application + +Take IFCVF driver for example: + +.. code-block:: console + + ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \ + -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \ + -- --interactive + +.. note:: + Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices, + and we need to bind vfio-pci to them before running vdpa sample. + + * modprobe vfio-pci + * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4 + +Then we can create 2 vdpa ports in interactive cmdline. + +.. code-block:: console + + vdpa> list + device id device address queue num supported features + 0 0000:06:00.3 1 0x14c238020 + 1 0000:06:00.4 1 0x14c238020 + 2 0000:06:00.5 1 0x14c238020 + + vdpa> create /tmp/vdpa-socket0 0000:06:00.3 + vdpa> create /tmp/vdpa-socket1 0000:06:00.4 + +.. _vdpa_app_run_vm: + +Start the VMs +~~~~~~~~~~~~~ + +.. code-block:: console + + qemu-system-x86_64 -cpu host -enable-kvm \ + <snip> + -mem-prealloc \ + -chardev socket,id=char0,path=<socket_file created in above steps> \ + -netdev type=vhost-user,id=vdpa,chardev=char0 \ + -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \ + +After the VMs launches, we can login the VMs and configure the ip, verify the +network connection via ping or netperf. + +.. note:: + Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA. + +Live Migration +~~~~~~~~~~~~~~ +vDPA supports cross-backend live migration, user can migrate SW vhost backend +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is +the source host with SW vhost VM and B is the destination host with vDPA. + +1. Start vdpa sample and launch a VM with exact same parameters as the VM on A, + in migration-listen mode: + +.. code-block:: console + + B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT)) + +2. Start the migration (on source host): + +.. code-block:: console + + A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT) + +3. Check the status (on source host): + +.. code-block:: console + + A: (qemu) info migrate diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/vhost.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/vhost.rst new file mode 100644 index 000000000..b7ed4f8bd --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/vhost.rst @@ -0,0 +1,210 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2016 Intel Corporation. + +Vhost Sample Application +======================== + +The vhost sample application demonstrates integration of the Data Plane +Development Kit (DPDK) with the Linux* KVM hypervisor by implementing the +vhost-net offload API. The sample application performs simple packet +switching between virtual machines based on Media Access Control (MAC) +address or Virtual Local Area Network (VLAN) tag. The splitting of Ethernet +traffic from an external switch is performed in hardware by the Virtual +Machine Device Queues (VMDQ) and Data Center Bridging (DCB) features of +the Intel® 82599 10 Gigabit Ethernet Controller. + +Testing steps +------------- + +This section shows the steps how to test a typical PVP case with this +vhost-switch sample, whereas packets are received from the physical NIC +port first and enqueued to the VM's Rx queue. Through the guest testpmd's +default forwarding mode (io forward), those packets will be put into +the Tx queue. The vhost-switch example, in turn, gets the packets and +puts back to the same physical NIC port. + +Build +~~~~~ + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``vhost`` sub-directory. + +.. note:: + In this example, you need build DPDK both on the host and inside guest. + +Start the vswitch example +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ./vhost-switch -l 0-3 -n 4 --socket-mem 1024 \ + -- --socket-file /tmp/sock0 --client \ + ... + +Check the `Parameters`_ section for the explanations on what do those +parameters mean. + +.. _vhost_app_run_vm: + +Start the VM +~~~~~~~~~~~~ + +.. code-block:: console + + qemu-system-x86_64 -machine accel=kvm -cpu host \ + -m $mem -object memory-backend-file,id=mem,size=$mem,mem-path=/dev/hugepages,share=on \ + -mem-prealloc -numa node,memdev=mem \ + \ + -chardev socket,id=char1,path=/tmp/sock0,server \ + -netdev type=vhost-user,id=hostnet1,chardev=char1 \ + -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:00:00:14 \ + ... + +.. note:: + For basic vhost-user support, QEMU 2.2 (or above) is required. For + some specific features, a higher version might be need. Such as + QEMU 2.7 (or above) for the reconnect feature. + +.. _vhost_app_run_dpdk_inside_guest: + +Run testpmd inside guest +~~~~~~~~~~~~~~~~~~~~~~~~ + +Make sure you have DPDK built inside the guest. Also make sure the +corresponding virtio-net PCI device is bond to a uio driver, which +could be done by: + +.. code-block:: console + + modprobe uio_pci_generic + $RTE_SDK/usertools/dpdk-devbind.py -b uio_pci_generic 0000:00:04.0 + +Then start testpmd for packet forwarding testing. + +.. code-block:: console + + ./x86_64-native-gcc/app/testpmd -l 0-1 -- -i + > start tx_first + +Inject packets +-------------- + +While a virtio-net is connected to vhost-switch, a VLAN tag starts with +1000 is assigned to it. So make sure configure your packet generator +with the right MAC and VLAN tag, you should be able to see following +log from the vhost-switch console. It means you get it work:: + + VHOST_DATA: (0) mac 52:54:00:00:00:14 and vlan 1000 registered + + +.. _vhost_app_parameters: + +Parameters +---------- + +**--socket-file path** +Specifies the vhost-user socket file path. + +**--client** +DPDK vhost-user will act as the client mode when such option is given. +In the client mode, QEMU will create the socket file. Otherwise, DPDK +will create it. Put simply, it's the server to create the socket file. + + +**--vm2vm mode** +The vm2vm parameter sets the mode of packet switching between guests in +the host. + +- 0 disables vm2vm, implying that VM's packets will always go to the NIC port. +- 1 means a normal mac lookup packet routing. +- 2 means hardware mode packet forwarding between guests, it allows packets + go to the NIC port, hardware L2 switch will determine which guest the + packet should forward to or need send to external, which bases on the + packet destination MAC address and VLAN tag. + +**--mergeable 0|1** +Set 0/1 to disable/enable the mergeable Rx feature. It's disabled by default. + +**--stats interval** +The stats parameter controls the printing of virtio-net device statistics. +The parameter specifies an interval (in unit of seconds) to print statistics, +with an interval of 0 seconds disabling statistics. + +**--rx-retry 0|1** +The rx-retry option enables/disables enqueue retries when the guests Rx queue +is full. This feature resolves a packet loss that is observed at high data +rates, by allowing it to delay and retry in the receive path. This option is +enabled by default. + +**--rx-retry-num num** +The rx-retry-num option specifies the number of retries on an Rx burst, it +takes effect only when rx retry is enabled. The default value is 4. + +**--rx-retry-delay msec** +The rx-retry-delay option specifies the timeout (in micro seconds) between +retries on an RX burst, it takes effect only when rx retry is enabled. The +default value is 15. + +**--dequeue-zero-copy** +Dequeue zero copy will be enabled when this option is given. it is worth to +note that if NIC is bound to driver with iommu enabled, dequeue zero copy +cannot work at VM2NIC mode (vm2vm=0) due to currently we don't setup iommu +dma mapping for guest memory. + +**--vlan-strip 0|1** +VLAN strip option is removed, because different NICs have different behaviors +when disabling VLAN strip. Such feature, which heavily depends on hardware, +should be removed from this example to reduce confusion. Now, VLAN strip is +enabled and cannot be disabled. + +**--builtin-net-driver** +A very simple vhost-user net driver which demonstrates how to use the generic +vhost APIs will be used when this option is given. It is disabled by default. + +Common Issues +------------- + +* QEMU fails to allocate memory on hugetlbfs, with an error like the + following:: + + file_ram_alloc: can't mmap RAM pages: Cannot allocate memory + + When running QEMU the above error indicates that it has failed to allocate + memory for the Virtual Machine on the hugetlbfs. This is typically due to + insufficient hugepages being free to support the allocation request. The + number of free hugepages can be checked as follows: + + .. code-block:: console + + cat /sys/kernel/mm/hugepages/hugepages-<pagesize>/nr_hugepages + + The command above indicates how many hugepages are free to support QEMU's + allocation request. + +* Failed to build DPDK in VM + + Make sure "-cpu host" QEMU option is given. + +* Device start fails if NIC's max queues > the default number of 128 + + mbuf pool size is dependent on the MAX_QUEUES configuration, if NIC's + max queue number is larger than 128, device start will fail due to + insufficient mbuf. + + Change the default number to make it work as below, just set the number + according to the NIC's property. :: + + make EXTRA_CFLAGS="-DMAX_QUEUES=320" + +* Option "builtin-net-driver" is incompatible with QEMU + + QEMU vhost net device start will fail if protocol feature is not negotiated. + DPDK virtio-user pmd can be the replacement of QEMU. + +* Device start fails when enabling "builtin-net-driver" without memory + pre-allocation + + The builtin example doesn't support dynamic memory allocation. When vhost + backend enables "builtin-net-driver", "--socket-mem" option should be + added at virtio-user pmd side as a startup item. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/vhost_blk.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/vhost_blk.rst new file mode 100644 index 000000000..681de6f3e --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/vhost_blk.rst @@ -0,0 +1,65 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2017 Intel Corporation. + +Vhost_blk Sample Application +============================= + +The vhost_blk sample application implemented a simple block device, +which used as the backend of Qemu vhost-user-blk device. Users can extend +the exist example to use other type of block device(e.g. AIO) besides +memory based block device. Similar with vhost-user-net device, the sample +application used domain socket to communicate with Qemu, and the virtio +ring (split or packed format) was processed by vhost_blk sample application. + +The sample application reuse lots codes from SPDK(Storage Performance +Development Kit, https://github.com/spdk/spdk) vhost-user-blk target, +for DPDK vhost library used in storage area, user can take SPDK as +reference as well. + +Testing steps +------------- + +This section shows the steps how to start a VM with the block device as +fast data path for critical application. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``examples`` sub-directory. + +You will also need to build DPDK both on the host and inside the guest + +Start the vhost_blk example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ./vhost_blk -m 1024 + +.. _vhost_blk_app_run_vm: + +Start the VM +~~~~~~~~~~~~ + +.. code-block:: console + + qemu-system-x86_64 -machine accel=kvm \ + -m $mem -object memory-backend-file,id=mem,size=$mem,\ + mem-path=/dev/hugepages,share=on -numa node,memdev=mem \ + -drive file=os.img,if=none,id=disk \ + -device ide-hd,drive=disk,bootindex=0 \ + -chardev socket,id=char0,reconnect=1,path=/tmp/vhost.socket \ + -device vhost-user-blk-pci,packed=on,chardev=char0,num-queues=1 \ + ... + +.. note:: + You must check whether your Qemu can support "vhost-user-blk" or not, + Qemu v4.0 or newer version is required. + reconnect=1 means live recovery support that qemu can reconnect vhost_blk + after we restart vhost_blk example. + packed=on means the device support packed ring but need the guest kernel + version >= 5.0. + Now Qemu commit 9bb73502321d46f4d320fa17aa38201445783fc4 both support the + vhost-blk reconnect and packed ring. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/vhost_crypto.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/vhost_crypto.rst new file mode 100644 index 000000000..bbc25bde1 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/vhost_crypto.rst @@ -0,0 +1,80 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017-2018 Intel Corporation. + +Vhost_Crypto Sample Application +=============================== + +The vhost_crypto sample application implemented a simple Crypto device, +which used as the backend of Qemu vhost-user-crypto device. Similar with +vhost-user-net and vhost-user-scsi device, the sample application used +domain socket to communicate with Qemu, and the virtio ring was processed +by vhost_crypto sample application. + +Testing steps +------------- + +This section shows the steps how to start a VM with the crypto device as +fast data path for critical application. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``examples`` sub-directory. + +Start the vhost_crypto example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ./vhost_crypto [EAL options] -- + --config (lcore,cdev-id,queue-id)[,(lcore,cdev-id,queue-id)] + --socket-file lcore,PATH + [--zero-copy] + [--guest-polling] + +where, + +* config (lcore,cdev-id,queue-id): build the lcore-cryptodev id-queue id + connection. Once specified, the specified lcore will only work with + specified cryptodev's queue. + +* socket-file lcore,PATH: the path of UNIX socket file to be created and + the lcore id that will deal with the all workloads of the socket. Multiple + instances of this config item is supported and one lcore supports processing + multiple sockets. + +* zero-copy: the presence of this item means the ZERO-COPY feature will be + enabled. Otherwise it is disabled. PLEASE NOTE the ZERO-COPY feature is still + in experimental stage and may cause the problem like segmentation fault. If + the user wants to use LKCF in the guest, this feature shall be turned off. + +* guest-polling: the presence of this item means the application assumes the + guest works in polling mode, thus will NOT notify the guest completion of + processing. + +The application requires that crypto devices capable of performing +the specified crypto operation are available on application initialization. +This means that HW crypto device/s must be bound to a DPDK driver or +a SW crypto device/s (virtual crypto PMD) must be created (using --vdev). + +.. _vhost_crypto_app_run_vm: + +Start the VM +~~~~~~~~~~~~ + +.. code-block:: console + + qemu-system-x86_64 -machine accel=kvm \ + -m $mem -object memory-backend-file,id=mem,size=$mem,\ + mem-path=/dev/hugepages,share=on -numa node,memdev=mem \ + -drive file=os.img,if=none,id=disk \ + -device ide-hd,drive=disk,bootindex=0 \ + -chardev socket,id={chardev_id},path={PATH} \ + -object cryptodev-vhost-user,id={obj_id},chardev={chardev_id} \ + -device virtio-crypto-pci,id={dev_id},cryptodev={obj_id} \ + ... + +.. note:: + You must check whether your Qemu can support "vhost-user-crypto" or not. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/vm_power_management.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/vm_power_management.rst new file mode 100644 index 000000000..e98277ccb --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/vm_power_management.rst @@ -0,0 +1,942 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +Virtual Machine Power Management Application +============================================ + +Applications running in virtual environments have an abstract view of +the underlying hardware on the host. Specifically, applications cannot +see the binding of virtual components to physical hardware. When looking +at CPU resourcing, the pinning of Virtual CPUs (vCPUs) to Physical CPUs +(pCPUs) on the host is not apparent to an application and this pinning +may change over time. In addition, operating systems on Virtual Machines +(VMs) do not have the ability to govern their own power policy. The +Machine Specific Registers (MSRs) for enabling P-state transitions are +not exposed to the operating systems running on the VMs. + +The solution demonstrated in this sample application shows an example of +how a DPDK application can indicate its processing requirements using +VM-local only information (vCPU/lcore, and so on) to a host resident VM +Power Manager. The VM Power Manager is responsible for: + +- **Accepting requests for frequency changes for a vCPU** +- **Translating the vCPU to a pCPU using libvirt** +- **Performing the change in frequency** + +This application demonstrates the following features: + +- **The handling of VM application requests to change frequency.** + VM applications can request frequency changes for a vCPU. The VM + Power Management Application uses libvirt to translate that + virtual CPU (vCPU) request to a physical CPU (pCPU) request and + performs the frequency change. + +- **The acceptance of power management policies from VM applications.** + A VM application can send a policy to the host application. The + policy contains rules that define the power management behaviour + of the VM. The host application then applies the rules of the + policy independent of the VM application. For example, the + policy can contain time-of-day information for busy/quiet + periods, and the host application can scale up/down the relevant + cores when required. See :ref:`sending_policy` for information on + setting policy values. + +- **Out-of-band monitoring of workloads using core hardware event counters.** + The host application can manage power for an application by looking + at the event counters of the cores and taking action based on the + branch miss/hit ratio. See :ref:`enabling_out_of_band`. + + **Note**: This functionality also applies in non-virtualised environments. + +In addition to the ``librte_power`` library used on the host, the +application uses a special version of ``librte_power`` on each VM, which +directs frequency changes and policies to the host monitor rather than +the APCI ``cpufreq`` ``sysfs`` interface used on the host in non-virtualised +environments. + +.. _figure_vm_power_mgr_highlevel: + +.. figure:: img/vm_power_mgr_highlevel.* + + Highlevel Solution + +In the above diagram, the DPDK Applications are shown running in +virtual machines, and the VM Power Monitor application is shown running +in the host. + +**DPDK VM Application** + +- Reuse ``librte_power`` interface, but uses an implementation that + forwards frequency requests to the host using a ``virtio-serial`` channel +- Each lcore has exclusive access to a single channel +- Sample application reuses ``l3fwd_power`` +- A CLI for changing frequency from within a VM is also included + +**VM Power Monitor** + +- Accepts VM commands over ``virtio-serial`` endpoints, monitored + using ``epoll`` +- Commands include the virtual core to be modified, using ``libvirt`` to get + the physical core mapping +- Uses ``librte_power`` to affect frequency changes using Linux userspace + power governor (``acpi_cpufreq`` OR ``intel_pstate`` driver) +- CLI: For adding VM channels to monitor, inspecting and changing channel + state, manually altering CPU frequency. Also allows for the changings + of vCPU to pCPU pinning + +Sample Application Architecture Overview +---------------------------------------- + +The VM power management solution employs ``qemu-kvm`` to provide +communications channels between the host and VMs in the form of a +``virtio-serial`` connection that appears as a para-virtualised serial +device on a VM and can be configured to use various backends on the +host. For this example, the configuration of each ``virtio-serial`` endpoint +on the host as an ``AF_UNIX`` file socket, supporting poll/select and +``epoll`` for event notification. In this example, each channel endpoint on +the host is monitored for ``EPOLLIN`` events using ``epoll``. Each channel +is specified as ``qemu-kvm`` arguments or as ``libvirt`` XML for each VM, +where each VM can have several channels up to a maximum of 64 per VM. In this +example, each DPDK lcore on a VM has exclusive access to a channel. + +To enable frequency changes from within a VM, the VM forwards a +``librte_power`` request over the ``virtio-serial`` channel to the host. Each +request contains the vCPU and power command (scale up/down/min/max). The +API for the host ``librte_power`` and guest ``librte_power`` is consistent +across environments, with the selection of VM or host implementation +determined automatically at runtime based on the environment. On +receiving a request, the host translates the vCPU to a pCPU using the +libvirt API before forwarding it to the host ``librte_power``. + + +.. _figure_vm_power_mgr_vm_request_seq: + +.. figure:: img/vm_power_mgr_vm_request_seq.* + +In addition to the ability to send power management requests to the +host, a VM can send a power management policy to the host. In some +cases, using a power management policy is a preferred option because it +can eliminate possible latency issues that can occur when sending power +management requests. Once the VM sends the policy to the host, the VM no +longer needs to worry about power management, because the host now +manages the power for the VM based on the policy. The policy can specify +power behavior that is based on incoming traffic rates or time-of-day +power adjustment (busy/quiet hour power adjustment for example). See +:ref:`sending_policy` for more information. + +One method of power management is to sense how busy a core is when +processing packets and adjusting power accordingly. One technique for +doing this is to monitor the ratio of the branch miss to branch hits +counters and scale the core power accordingly. This technique is based +on the premise that when a core is not processing packets, the ratio of +branch misses to branch hits is very low, but when the core is +processing packets, it is measurably higher. The implementation of this +capability is as a policy of type ``BRANCH_RATIO``. +See :ref:`sending_policy` for more information on using the +BRANCH_RATIO policy option. + +A JSON interface enables the specification of power management requests +and policies in JSON format. The JSON interfaces provide a more +convenient and more easily interpreted interface for the specification +of requests and policies. See :ref:`power_man_requests` for more information. + +Performance Considerations +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +While the Haswell microarchitecture allows for independent power control +for each core, earlier microarchitectures do not offer such fine-grained +control. When deploying on pre-Haswell platforms, greater care must be +taken when selecting which cores are assigned to a VM, for example, a +core does not scale down in frequency until all of its siblings are +similarly scaled down. + +Configuration +------------- + +BIOS +~~~~ + +To use the power management features of the DPDK, you must enable +Enhanced Intel SpeedStep® Technology in the platform BIOS. Otherwise, +the ``sys`` file folder ``/sys/devices/system/cpu/cpu0/cpufreq`` does not +exist, and you cannot use CPU frequency-based power management. Refer to the +relevant BIOS documentation to determine how to access these settings. + +Host Operating System +~~~~~~~~~~~~~~~~~~~~~ + +The DPDK Power Management library can use either the ``acpi_cpufreq`` or +the ``intel_pstate`` kernel driver for the management of core frequencies. In +many cases, the ``intel_pstate`` driver is the default power management +environment. + +Should the ``acpi-cpufreq driver`` be required, the ``intel_pstate`` +module must be disabled, and the ``acpi-cpufreq`` module loaded in its place. + +To disable the ``intel_pstate`` driver, add the following to the ``grub`` +Linux command line: + + ``intel_pstate=disable`` + +On reboot, load the ``acpi_cpufreq`` module: + + ``modprobe acpi_cpufreq`` + +Hypervisor Channel Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configure ``virtio-serial`` channels using ``libvirt`` XML. +The XML structure is as follows: + +.. code-block:: XML + + <name>{vm_name}</name> + <controller type='virtio-serial' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> + </controller> + <channel type='unix'> + <source mode='bind' path='/tmp/powermonitor/{vm_name}.{channel_num}'/> + <target type='virtio' name='virtio.serial.port.poweragent.{vm_channel_num}'/> + <address type='virtio-serial' controller='0' bus='0' port='{N}'/> + </channel> + +Where a single controller of type ``virtio-serial`` is created, up to 32 +channels can be associated with a single controller, and multiple +controllers can be specified. The convention is to use the name of the +VM in the host path ``{vm_name}`` and to increment ``{channel_num}`` for each +channel. Likewise, the port value ``{N}`` must be incremented for each +channel. + +On the host, for each channel to appear in the path, ensure the creation +of the ``/tmp/powermonitor/`` directory and the assignment of ``qemu`` +permissions: + +.. code-block:: console + + mkdir /tmp/powermonitor/ + chown qemu:qemu /tmp/powermonitor + +Note that files and directories in ``/tmp`` are generally removed when +rebooting the host and you may need to perform the previous steps after +each reboot. + +The serial device as it appears on a VM is configured with the target +element attribute name and must be in the form: +``virtio.serial.port.poweragent.{vm_channel_num}``, where +``vm_channel_num`` is typically the lcore channel to be used in +DPDK VM applications. + +Each channel on a VM is present at: + +``/dev/virtio-ports/virtio.serial.port.poweragent.{vm_channel_num}`` + +Compiling and Running the Host Application +------------------------------------------ + +Compiling the Host Application +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For information on compiling the DPDK and sample applications, see +see :doc:`compiling`. + +The application is located in the ``vm_power_manager`` subdirectory. + +To build just the ``vm_power_manager`` application using ``make``: + +.. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + export RTE_TARGET=build + cd ${RTE_SDK}/examples/vm_power_manager/ + make + +The resulting binary is ``${RTE_SDK}/build/examples/vm_power_manager``. + +To build just the ``vm_power_manager`` application using ``meson``/``ninja``: + +.. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK} + meson build + cd build + ninja + meson configure -Dexamples=vm_power_manager + ninja + +The resulting binary is ``${RTE_SDK}/build/examples/dpdk-vm_power_manager``. + +Running the Host Application +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The application does not have any specific command line options other +than the EAL options: + +.. code-block:: console + + ./build/vm_power_mgr [EAL options] + +The application requires exactly two cores to run. One core for the CLI +and the other for the channel endpoint monitor. For example, to run on +cores 0 and 1 on a system with four memory channels, issue the command: + +.. code-block:: console + + ./build/vm_power_mgr -l 0-1 -n 4 + +After successful initialization, the VM Power Manager CLI prompt appears: + +.. code-block:: console + + vm_power> + +Now, it is possible to add virtual machines to the VM Power Manager: + +.. code-block:: console + + vm_power> add_vm {vm_name} + +When a ``{vm_name}`` is specified with the ``add_vm`` command, a lookup is +performed with ``libvirt`` to ensure that the VM exists. ``{vm_name}`` is a +unique identifier to associate channels with a particular VM and for +executing operations on a VM within the CLI. VMs do not have to be +running to add them. + +It is possible to issue several commands from the CLI to manage VMs. + +Remove the virtual machine identified by ``{vm_name}`` from the VM Power +Manager using the command: + +.. code-block:: console + + rm_vm {vm_name} + +Add communication channels for the specified VM using the following +command. The ``virtio`` channels must be enabled in the VM configuration +(``qemu/libvirt``) and the associated VM must be active. ``{list}`` is a +comma-separated list of channel numbers to add. Specifying the keyword +``all`` attempts to add all channels for the VM: + +.. code-block:: console + + set_pcpu {vm_name} {vcpu} {pcpu} + + Enable query of physical core information from a VM: + +.. code-block:: console + + set_query {vm_name} enable|disable + +Manual control and inspection can also be carried in relation CPU frequency scaling: + + Get the current frequency for each core specified in the mask: + +.. code-block:: console + + show_cpu_freq_mask {mask} + + Set the current frequency for the cores specified in {core_mask} by scaling each up/down/min/max: + +.. code-block:: console + + add_channels {vm_name} {list}|all + +Enable or disable the communication channels in ``{list}`` (comma-separated) +for the specified VM. Alternatively, replace ``list`` with the keyword +``all``. Disabled channels receive packets on the host. However, the commands +they specify are ignored. Set the status to enabled to begin processing +requests again: + +.. code-block:: console + + set_channel_status {vm_name} {list}|all enabled|disabled + +Print to the CLI information on the specified VM. The information lists +the number of vCPUs, the pinning to pCPU(s) as a bit mask, along with +any communication channels associated with each VM, and the status of +each channel: + +.. code-block:: console + + show_vm {vm_name} + +Set the binding of a virtual CPU on a VM with name ``{vm_name}`` to the +physical CPU mask: + +.. code-block:: console + + set_pcpu_mask {vm_name} {vcpu} {pcpu} + +Set the binding of the virtual CPU on the VM to the physical CPU: + + .. code-block:: console + + set_pcpu {vm_name} {vcpu} {pcpu} + +It is also possible to perform manual control and inspection in relation +to CPU frequency scaling. + +Get the current frequency for each core specified in the mask: + +.. code-block:: console + + show_cpu_freq_mask {mask} + +Set the current frequency for the cores specified in ``{core_mask}`` by +scaling each up/down/min/max: + +.. code-block:: console + + set_cpu_freq {core_mask} up|down|min|max + +Get the current frequency for the specified core: + +.. code-block:: console + + show_cpu_freq {core_num} + +Set the current frequency for the specified core by scaling up/down/min/max: + +.. code-block:: console + + set_cpu_freq {core_num} up|down|min|max + +.. _enabling_out_of_band: + +Command Line Options for Enabling Out-of-band Branch Ratio Monitoring +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are a couple of command line parameters for enabling the out-of-band +monitoring of branch ratios on cores doing busy polling using PMDs as +described below: + +``--core-list {list of cores}`` + Specify the list of cores to monitor the ratio of branch misses + to branch hits. A tightly-polling PMD thread has a very low + branch ratio, therefore the core frequency scales down to the + minimum allowed value. On receiving packets, the code path changes, + causing the branch ratio to increase. When the ratio goes above + the ratio threshold, the core frequency scales up to the maximum + allowed value. + +``--branch-ratio {ratio}`` + Specify a floating-point number that identifies the threshold at which + to scale up or down for the given workload. The default branch ratio + is 0.01 and needs adjustment for different workloads. + + +Compiling and Running the Guest Applications +-------------------------------------------- + +It is possible to use the ``l3fwd-power`` application (for example) with the +``vm_power_manager``. + +The distribution also provides a guest CLI for validating the setup. + +For both ``l3fwd-power`` and the guest CLI, the host application must use +the ``add_channels`` command to monitor the channels for the VM. To do this, +issue the following commands in the host application: + +.. code-block:: console + + vm_power> add_vm vmname + vm_power> add_channels vmname all + vm_power> set_channel_status vmname all enabled + vm_power> show_vm vmname + +Compiling the Guest Application +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For information on compiling DPDK and the sample applications in general, +see :doc:`compiling`. + +For compiling and running the ``l3fwd-power`` sample application, see +:doc:`l3_forward_power_man`. + +The application is in the ``guest_cli`` subdirectory under ``vm_power_manager``. + +To build just the ``guest_vm_power_manager`` application using ``make``, issue +the following commands: + +.. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + export RTE_TARGET=build + cd ${RTE_SDK}/examples/vm_power_manager/guest_cli/ + make + +The resulting binary is ``${RTE_SDK}/build/examples/guest_cli``. + +**Note**: This sample application conditionally links in the Jansson JSON +library. Consequently, if you are using a multilib or cross-compile +environment, you may need to set the ``PKG_CONFIG_LIBDIR`` environmental +variable to point to the relevant ``pkgconfig`` folder so that the correct +library is linked in. + +For example, if you are building for a 32-bit target, you could find the +correct directory using the following find command: + +.. code-block:: console + + # find /usr -type d -name pkgconfig + /usr/lib/i386-linux-gnu/pkgconfig + /usr/lib/x86_64-linux-gnu/pkgconfig + +Then use: + +.. code-block:: console + + export PKG_CONFIG_LIBDIR=/usr/lib/i386-linux-gnu/pkgconfig + +You then use the ``make`` command as normal, which should find the 32-bit +version of the library, if it installed. If not, the application builds +without the JSON interface functionality. + +To build just the ``vm_power_manager`` application using ``meson``/``ninja``: + +.. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK} + meson build + cd build + ninja + meson configure -Dexamples=vm_power_manager/guest_cli + ninja + +The resulting binary is ``${RTE_SDK}/build/examples/guest_cli``. + +Running the Guest Application +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The standard EAL command line parameters are necessary: + +.. code-block:: console + + ./build/vm_power_mgr [EAL options] -- [guest options] + +The guest example uses a channel for each lcore enabled. For example, to +run on cores 0, 1, 2 and 3: + +.. code-block:: console + + ./build/guest_vm_power_mgr -l 0-3 + +.. _sending_policy: + +Command Line Options Available When Sending a Policy to the Host +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Optionally, there are several command line options for a user who needs +to send a power policy to the host application: + +``--vm-name {name of guest vm}`` + Allows the user to change the virtual machine name + passed down to the host application using the power policy. + The default is ubuntu2. + +``--vcpu-list {list vm cores}`` + A comma-separated list of cores in the VM that the user + wants the host application to monitor. + The list of cores in any VM starts at zero, + and the host application maps these to the physical cores + once the policy passes down to the host. + Valid syntax includes individual cores 2,3,4, + a range of cores 2-4, or a combination of both 1,3,5-7. + +``--busy-hours {list of busy hours}`` + A comma-separated list of hours in which to set the core + frequency to the maximum. + Valid syntax includes individual hours 2,3,4, + a range of hours 2-4, or a combination of both 1,3,5-7. + Valid hour values are 0 to 23. + +``--quiet-hours {list of quiet hours}`` + A comma-separated list of hours in which to set the core frequency + to minimum. Valid syntax includes individual hours 2,3,4, + a range of hours 2-4, or a combination of both 1,3,5-7. + Valid hour values are 0 to 23. + +``--policy {policy type}`` + The type of policy. This can be one of the following values: + + - TRAFFIC - Based on incoming traffic rates on the NIC. + - TIME - Uses a busy/quiet hours policy. + - BRANCH_RATIO - Uses branch ratio counters to determine core busyness. + - WORKLOAD - Sets the frequency to low, medium or high + based on the received policy setting. + + **Note**: Not all policy types need all parameters. + For example, BRANCH_RATIO only needs the vcpu-list parameter. + +After successful initialization, the VM Power Manager Guest CLI prompt +appears: + +.. code-block:: console + + vm_power(guest)> + +To change the frequency of an lcore, use a ``set_cpu_freq`` command similar +to the following: + +.. code-block:: console + + set_cpu_freq {core_num} up|down|min|max + +where, ``{core_num}`` is the lcore and channel to change frequency by +scaling up/down/min/max. + +To start an application, configure the power policy, and send it to the +host, use a command like the following: + +.. code-block:: console + + ./build/guest_vm_power_mgr -l 0-3 -n 4 -- --vm-name=ubuntu --policy=BRANCH_RATIO --vcpu-list=2-4 + +Once the VM Power Manager Guest CLI appears, issuing the 'send_policy now' command +will send the policy to the host: + +.. code-block:: console + + send_policy now + +Once the policy is sent to the host, the host application takes over the power monitoring +of the specified cores in the policy. + +.. _power_man_requests: + +JSON Interface for Power Management Requests and Policies +--------------------------------------------------------- + +In addition to the command line interface for the host command, and a +``virtio-serial`` interface for VM power policies, there is also a JSON +interface through which power commands and policies can be sent. + +**Note**: This functionality adds a dependency on the Jansson library. +Install the Jansson development package on the system to avail of the +JSON parsing functionality in the app. Issue the ``apt-get install +libjansson-dev`` command to install the development package. The command +and package name may be different depending on your operating system. It +is worth noting that the app builds successfully if this package is not +present, but a warning displays during compilation, and the JSON parsing +functionality is not present in the app. + +Send a request or policy to the VM Power Manager by simply opening a +fifo file at ``/tmp/powermonitor/fifo``, writing a JSON string to that file, +and closing the file. + +The JSON string can be a power management request or a policy, and takes +the following format: + +.. code-block:: javascript + + {"packet_type": { + "pair_1": value, + "pair_2": value + }} + +The ``packet_type`` header can contain one of two values, depending on +whether a power management request or policy is being sent. The two +possible values are ``instruction`` and ``policy`` and the expected name-value +pairs are different depending on which type is sent. + +The pairs are in the format of standard JSON name-value pairs. The value +type varies between the different name-value pairs, and may be integers, +strings, arrays, and so on. See :ref:`json_interface_ex` +for examples of policies and instructions and +:ref:`json_name_value_pair` for the supported names and value types. + +.. _json_interface_ex: + +JSON Interface Examples +~~~~~~~~~~~~~~~~~~~~~~~ + +The following is an example JSON string that creates a time-profile +policy. + +.. code-block:: JSON + + {"policy": { + "name": "ubuntu", + "command": "create", + "policy_type": "TIME", + "busy_hours":[ 17, 18, 19, 20, 21, 22, 23 ], + "quiet_hours":[ 2, 3, 4, 5, 6 ], + "core_list":[ 11 ] + }} + +The following is an example JSON string that removes the named policy. + +.. code-block:: JSON + + {"policy": { + "name": "ubuntu", + "command": "destroy", + }} + +The following is an example JSON string for a power management request. + +.. code-block:: JSON + + {"instruction": { + "name": "ubuntu", + "command": "power", + "unit": "SCALE_MAX", + "resource_id": 10 + }} + +To query the available frequences of an lcore, use the query_cpu_freq command. +Where {core_num} is the lcore to query. +Before using this command, please enable responses via the set_query command on the host. + +.. code-block:: console + + query_cpu_freq {core_num}|all + +To query the capabilities of an lcore, use the query_cpu_caps command. +Where {core_num} is the lcore to query. +Before using this command, please enable responses via the set_query command on the host. + +.. code-block:: console + + query_cpu_caps {core_num}|all + +To start the application and configure the power policy, and send it to the host: + +.. code-block:: console + + ./build/guest_vm_power_mgr -l 0-3 -n 4 -- --vm-name=ubuntu --policy=BRANCH_RATIO --vcpu-list=2-4 + +Once the VM Power Manager Guest CLI appears, issuing the 'send_policy now' command +will send the policy to the host: + +.. code-block:: console + + send_policy now + +Once the policy is sent to the host, the host application takes over the power monitoring +of the specified cores in the policy. + +.. _json_name_value_pair: + +JSON Name-value Pairs +~~~~~~~~~~~~~~~~~~~~~ + +The following are the name-value pairs supported by the JSON interface: + +- `avg_packet_thresh`_ +- `busy_hours`_ +- `command`_ +- `core_list`_ +- `mac_list`_ +- `max_packet_thresh`_ +- `name`_ +- `policy_type`_ +- `quiet_hours`_ +- `resource_id`_ +- `unit`_ +- `workload`_ + +avg_packet_thresh +^^^^^^^^^^^^^^^^^ + +Description + The threshold below which the frequency is set to the minimum value + for the TRAFFIC policy. + If the traffic rate is above this value and below the maximum value, + the frequency is set to medium. +Type + integer +Values + The number of packets below which the TRAFFIC policy applies + the minimum frequency, or the medium frequency + if between the average and maximum thresholds. +Required + Yes +Example + ``"avg_packet_thresh": 100000`` + +busy_hours +^^^^^^^^^^ + +Description + The hours of the day in which we scale up the cores for busy times. +Type + array of integers +Values + An array with a list of hour values (0-23). +Required + For the TIME policy only. +Example + ``"busy_hours":[ 17, 18, 19, 20, 21, 22, 23 ]`` + +command +^^^^^^^ + +Description + The type of packet to send to the VM Power Manager. + It is possible to create or destroy a policy or send a direct command + to adjust the frequency of a core, + as is possible on the command line interface. +Type + string +Values + Possible values are: + - CREATE: Create a new policy. + - DESTROY: Remove an existing policy. + - POWER: Send an immediate command, max, min, and so on. +Required + Yes +Example + ``"command": "CREATE"`` + +core_list +^^^^^^^^^ + +Description + The cores to which to apply a policy. +Type + array of integers +Values + An array with a list of virtual CPUs. +Required + For CREATE/DESTROY policy requests only. +Example + ``"core_list":[ 10, 11 ]`` + +mac_list +^^^^^^^^ + +Description + When the policy is of type TRAFFIC, + it is necessary to specify the MAC addresses that the host must monitor. +Type + array of strings +Values + An array with a list of MAC address strings. +Required + For TRAFFIC policy types only. +Example + ``"mac_list":[ "de:ad:be:ef:01:01","de:ad:be:ef:01:02" ]`` + +max_packet_thresh +^^^^^^^^^^^^^^^^^ + +Description + In a policy of type TRAFFIC, + the threshold value above which the frequency is set to a maximum. +Type + integer +Values + The number of packets per interval above which + the TRAFFIC policy applies the maximum frequency. +Required + For the TRAFFIC policy only. +Example + ``"max_packet_thresh": 500000`` + +name +^^^^ + +Description + The name of the VM or host. + Allows the parser to associate the policy with the relevant VM or host OS. +Type + string +Values + Any valid string. +Required + Yes +Example + ``"name": "ubuntu2"`` + +policy_type +^^^^^^^^^^^ + +Description + The type of policy to apply. + See the ``--policy`` option description for more information. +Type + string +Values + Possible values are: + + - TIME: Time-of-day policy. + Scale the frequencies of the relevant cores up/down + depending on busy and quiet hours. + - TRAFFIC: Use statistics from the NIC and scale up and down accordingly. + - WORKLOAD: Determine how heavily loaded the cores are + and scale up and down accordingly. + - BRANCH_RATIO: An out-of-band policy that looks at the ratio + between branch hits and misses on a core + and uses that information to determine how much packet processing + a core is doing. + +Required + For ``CREATE`` and ``DESTROY`` policy requests only. +Example + ``"policy_type": "TIME"`` + +quiet_hours +^^^^^^^^^^^ + +Description + The hours of the day to scale down the cores for quiet times. +Type + array of integers +Values + An array with a list of hour numbers with values in the range 0 to 23. +Required + For the TIME policy only. +Example + ``"quiet_hours":[ 2, 3, 4, 5, 6 ]`` + +resource_id +^^^^^^^^^^^ + +Description + The core to which to apply a power command. +Type + integer +Values + A valid core ID for the VM or host OS. +Required + For the ``POWER`` instruction only. +Example + ``"resource_id": 10`` + +unit +^^^^ + +Description + The type of power operation to apply in the command. +Type + string +Values + - SCALE_MAX: Scale the frequency of this core to the maximum. + - SCALE_MIN: Scale the frequency of this core to the minimum. + - SCALE_UP: Scale up the frequency of this core. + - SCALE_DOWN: Scale down the frequency of this core. + - ENABLE_TURBO: Enable Intel® Turbo Boost Technology for this core. + - DISABLE_TURBO: Disable Intel® Turbo Boost Technology for this core. +Required + For the ``POWER`` instruction only. +Example + ``"unit": "SCALE_MAX"`` + +workload +^^^^^^^^ + +Description + In a policy of type WORKLOAD, + it is necessary to specify how heavy the workload is. +Type + string +Values + - HIGH: Scale the frequency of this core to maximum. + - MEDIUM: Scale the frequency of this core to minimum. + - LOW: Scale up the frequency of this core. +Required + For the ``WORKLOAD`` policy only. +Example + ``"workload": "MEDIUM"`` diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst new file mode 100644 index 000000000..8e1774d9e --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst @@ -0,0 +1,266 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2014 Intel Corporation. + +VMDQ and DCB Forwarding Sample Application +========================================== + +The VMDQ and DCB Forwarding sample application is a simple example of packet processing using the DPDK. +The application performs L2 forwarding using VMDQ and DCB to divide the incoming traffic into queues. +The traffic splitting is performed in hardware by the VMDQ and DCB features of the Intel® 82599 and X710/XL710 Ethernet Controllers. + +Overview +-------- + +This sample application can be used as a starting point for developing a new application that is based on the DPDK and +uses VMDQ and DCB for traffic partitioning. + +The VMDQ and DCB filters work on MAC and VLAN traffic to divide the traffic into input queues on the basis of the Destination MAC +address, VLAN ID and VLAN user priority fields. +VMDQ filters split the traffic into 16 or 32 groups based on the Destination MAC and VLAN ID. +Then, DCB places each packet into one of queues within that group, based upon the VLAN user priority field. + +All traffic is read from a single incoming port (port 0) and output on port 1, without any processing being performed. +With Intel® 82599 NIC, for example, the traffic is split into 128 queues on input, where each thread of the application reads from +multiple queues. When run with 8 threads, that is, with the -c FF option, each thread receives and forwards packets from 16 queues. + +As supplied, the sample application configures the VMDQ feature to have 32 pools with 4 queues each as indicated in :numref:`figure_vmdq_dcb_example`. +The Intel® 82599 10 Gigabit Ethernet Controller NIC also supports the splitting of traffic into 16 pools of 8 queues. While the +Intel® X710 or XL710 Ethernet Controller NICs support many configurations of VMDQ pools of 4 or 8 queues each. For simplicity, only 16 +or 32 pools is supported in this sample. And queues numbers for each VMDQ pool can be changed by setting CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM +in config/common_* file. +The nb-pools, nb-tcs and enable-rss parameters can be passed on the command line, after the EAL parameters: + +.. code-block:: console + + ./build/vmdq_dcb [EAL options] -- -p PORTMASK --nb-pools NP --nb-tcs TC --enable-rss + +where, NP can be 16 or 32, TC can be 4 or 8, rss is disabled by default. + +.. _figure_vmdq_dcb_example: + +.. figure:: img/vmdq_dcb_example.* + + Packet Flow Through the VMDQ and DCB Sample Application + + +In Linux* user space, the application can display statistics with the number of packets received on each queue. +To have the application display the statistics, send a SIGHUP signal to the running application process. + +The VMDQ and DCB Forwarding sample application is in many ways simpler than the L2 Forwarding application +(see :doc:`l2_forward_real_virtual`) +as it performs unidirectional L2 forwarding of packets from one port to a second port. +No command-line options are taken by this application apart from the standard EAL command-line options. + +.. note:: + + Since VMD queues are being used for VMM, this application works correctly + when VTd is disabled in the BIOS or Linux* kernel (intel_iommu=off). + +Compiling the Application +------------------------- + + + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``vmdq_dcb`` sub-directory. + +Running the Application +----------------------- + +To run the example in a linux environment: + +.. code-block:: console + + user@target:~$ ./build/vmdq_dcb -l 0-3 -n 4 -- -p 0x3 --nb-pools 32 --nb-tcs 4 + +Refer to the *DPDK Getting Started Guide* for general information on running applications and +the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. + +Initialization +~~~~~~~~~~~~~~ + +The EAL, driver and PCI configuration is performed largely as in the L2 Forwarding sample application, +as is the creation of the mbuf pool. +See :doc:`l2_forward_real_virtual`. +Where this example application differs is in the configuration of the NIC port for RX. + +The VMDQ and DCB hardware feature is configured at port initialization time by setting the appropriate values in the +rte_eth_conf structure passed to the rte_eth_dev_configure() API. +Initially in the application, +a default structure is provided for VMDQ and DCB configuration to be filled in later by the application. + +.. code-block:: c + + /* empty vmdq+dcb configuration structure. Filled in programmatically */ + static const struct rte_eth_conf vmdq_dcb_conf_default = { + .rxmode = { + .mq_mode = ETH_MQ_RX_VMDQ_DCB, + .split_hdr_size = 0, + }, + .txmode = { + .mq_mode = ETH_MQ_TX_VMDQ_DCB, + }, + /* + * should be overridden separately in code with + * appropriate values + */ + .rx_adv_conf = { + .vmdq_dcb_conf = { + .nb_queue_pools = ETH_32_POOLS, + .enable_default_pool = 0, + .default_pool = 0, + .nb_pool_maps = 0, + .pool_map = {{0, 0},}, + .dcb_tc = {0}, + }, + .dcb_rx_conf = { + .nb_tcs = ETH_4_TCS, + /** Traffic class each UP mapped to. */ + .dcb_tc = {0}, + }, + .vmdq_rx_conf = { + .nb_queue_pools = ETH_32_POOLS, + .enable_default_pool = 0, + .default_pool = 0, + .nb_pool_maps = 0, + .pool_map = {{0, 0},}, + }, + }, + .tx_adv_conf = { + .vmdq_dcb_tx_conf = { + .nb_queue_pools = ETH_32_POOLS, + .dcb_tc = {0}, + }, + }, + }; + +The get_eth_conf() function fills in an rte_eth_conf structure with the appropriate values, +based on the global vlan_tags array, +and dividing up the possible user priority values equally among the individual queues +(also referred to as traffic classes) within each pool. With Intel® 82599 NIC, +if the number of pools is 32, then the user priority fields are allocated 2 to a queue. +If 16 pools are used, then each of the 8 user priority fields is allocated to its own queue within the pool. +With Intel® X710/XL710 NICs, if number of tcs is 4, and number of queues in pool is 8, +then the user priority fields are allocated 2 to one tc, and a tc has 2 queues mapping to it, then +RSS will determine the destination queue in 2. +For the VLAN IDs, each one can be allocated to possibly multiple pools of queues, +so the pools parameter in the rte_eth_vmdq_dcb_conf structure is specified as a bitmask value. +For destination MAC, each VMDQ pool will be assigned with a MAC address. In this sample, each VMDQ pool +is assigned to the MAC like 52:54:00:12:<port_id>:<pool_id>, that is, +the MAC of VMDQ pool 2 on port 1 is 52:54:00:12:01:02. + +.. code-block:: c + + const uint16_t vlan_tags[] = { + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15, + 16, 17, 18, 19, 20, 21, 22, 23, + 24, 25, 26, 27, 28, 29, 30, 31 + }; + + /* pool mac addr template, pool mac addr is like: 52 54 00 12 port# pool# */ + static struct rte_ether_addr pool_addr_template = { + .addr_bytes = {0x52, 0x54, 0x00, 0x12, 0x00, 0x00} + }; + + /* Builds up the correct configuration for vmdq+dcb based on the vlan tags array + * given above, and the number of traffic classes available for use. */ + static inline int + get_eth_conf(struct rte_eth_conf *eth_conf) + { + struct rte_eth_vmdq_dcb_conf conf; + struct rte_eth_vmdq_rx_conf vmdq_conf; + struct rte_eth_dcb_rx_conf dcb_conf; + struct rte_eth_vmdq_dcb_tx_conf tx_conf; + uint8_t i; + + conf.nb_queue_pools = (enum rte_eth_nb_pools)num_pools; + vmdq_conf.nb_queue_pools = (enum rte_eth_nb_pools)num_pools; + tx_conf.nb_queue_pools = (enum rte_eth_nb_pools)num_pools; + conf.nb_pool_maps = num_pools; + vmdq_conf.nb_pool_maps = num_pools; + conf.enable_default_pool = 0; + vmdq_conf.enable_default_pool = 0; + conf.default_pool = 0; /* set explicit value, even if not used */ + vmdq_conf.default_pool = 0; + + for (i = 0; i < conf.nb_pool_maps; i++) { + conf.pool_map[i].vlan_id = vlan_tags[i]; + vmdq_conf.pool_map[i].vlan_id = vlan_tags[i]; + conf.pool_map[i].pools = 1UL << i ; + vmdq_conf.pool_map[i].pools = 1UL << i; + } + for (i = 0; i < ETH_DCB_NUM_USER_PRIORITIES; i++){ + conf.dcb_tc[i] = i % num_tcs; + dcb_conf.dcb_tc[i] = i % num_tcs; + tx_conf.dcb_tc[i] = i % num_tcs; + } + dcb_conf.nb_tcs = (enum rte_eth_nb_tcs)num_tcs; + (void)(rte_memcpy(eth_conf, &vmdq_dcb_conf_default, sizeof(*eth_conf))); + (void)(rte_memcpy(ð_conf->rx_adv_conf.vmdq_dcb_conf, &conf, + sizeof(conf))); + (void)(rte_memcpy(ð_conf->rx_adv_conf.dcb_rx_conf, &dcb_conf, + sizeof(dcb_conf))); + (void)(rte_memcpy(ð_conf->rx_adv_conf.vmdq_rx_conf, &vmdq_conf, + sizeof(vmdq_conf))); + (void)(rte_memcpy(ð_conf->tx_adv_conf.vmdq_dcb_tx_conf, &tx_conf, + sizeof(tx_conf))); + if (rss_enable) { + eth_conf->rxmode.mq_mode= ETH_MQ_RX_VMDQ_DCB_RSS; + eth_conf->rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | + ETH_RSS_UDP | + ETH_RSS_TCP | + ETH_RSS_SCTP; + } + return 0; + } + + ...... + + /* Set mac for each pool.*/ + for (q = 0; q < num_pools; q++) { + struct rte_ether_addr mac; + mac = pool_addr_template; + mac.addr_bytes[4] = port; + mac.addr_bytes[5] = q; + printf("Port %u vmdq pool %u set mac %02x:%02x:%02x:%02x:%02x:%02x\n", + port, q, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); + retval = rte_eth_dev_mac_addr_add(port, &mac, + q + vmdq_pool_base); + if (retval) { + printf("mac addr add failed at pool %d\n", q); + return retval; + } + } + +Once the network port has been initialized using the correct VMDQ and DCB values, +the initialization of the port's RX and TX hardware rings is performed similarly to that +in the L2 Forwarding sample application. +See :doc:`l2_forward_real_virtual` for more information. + +Statistics Display +~~~~~~~~~~~~~~~~~~ + +When run in a linux environment, +the VMDQ and DCB Forwarding sample application can display statistics showing the number of packets read from each RX queue. +This is provided by way of a signal handler for the SIGHUP signal, +which simply prints to standard output the packet counts in grid form. +Each row of the output is a single pool with the columns being the queue number within that pool. + +To generate the statistics output, use the following command: + +.. code-block:: console + + user@host$ sudo killall -HUP vmdq_dcb_app + +Please note that the statistics output will appear on the terminal where the vmdq_dcb_app is running, +rather than the terminal from which the HUP signal was sent. diff --git a/src/spdk/dpdk/doc/guides/sample_app_ug/vmdq_forwarding.rst b/src/spdk/dpdk/doc/guides/sample_app_ug/vmdq_forwarding.rst new file mode 100644 index 000000000..c3a3439e5 --- /dev/null +++ b/src/spdk/dpdk/doc/guides/sample_app_ug/vmdq_forwarding.rst @@ -0,0 +1,208 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2020 Intel Corporation. + +VMDq Forwarding Sample Application +========================================== + +The VMDq Forwarding sample application is a simple example of packet processing using the DPDK. +The application performs L2 forwarding using VMDq to divide the incoming traffic into queues. +The traffic splitting is performed in hardware by the VMDq feature of the Intel® 82599 and X710/XL710 Ethernet Controllers. + +Overview +-------- + +This sample application can be used as a starting point for developing a new application that is based on the DPDK and +uses VMDq for traffic partitioning. + +VMDq filters split the incoming packets up into different "pools" - each with its own set of RX queues - based upon +the MAC address and VLAN ID within the VLAN tag of the packet. + +All traffic is read from a single incoming port and output on another port, without any processing being performed. +With Intel® 82599 NIC, for example, the traffic is split into 128 queues on input, where each thread of the application reads from +multiple queues. When run with 8 threads, that is, with the -c FF option, each thread receives and forwards packets from 16 queues. + +As supplied, the sample application configures the VMDq feature to have 32 pools with 4 queues each. +The Intel® 82599 10 Gigabit Ethernet Controller NIC also supports the splitting of traffic into 16 pools of 2 queues. +While the Intel® X710 or XL710 Ethernet Controller NICs support many configurations of VMDq pools of 4 or 8 queues each. +And queues numbers for each VMDq pool can be changed by setting CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM +in config/common_* file. +The nb-pools and enable-rss parameters can be passed on the command line, after the EAL parameters: + +.. code-block:: console + + ./build/vmdq_app [EAL options] -- -p PORTMASK --nb-pools NP --enable-rss + +where, NP can be 8, 16 or 32, rss is disabled by default. + +In Linux* user space, the application can display statistics with the number of packets received on each queue. +To have the application display the statistics, send a SIGHUP signal to the running application process. + +The VMDq Forwarding sample application is in many ways simpler than the L2 Forwarding application +(see :doc:`l2_forward_real_virtual`) +as it performs unidirectional L2 forwarding of packets from one port to a second port. +No command-line options are taken by this application apart from the standard EAL command-line options. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``vmdq`` sub-directory. + +Running the Application +----------------------- + +To run the example in a Linux environment: + +.. code-block:: console + + user@target:~$ ./build/vmdq_app -l 0-3 -n 4 -- -p 0x3 --nb-pools 16 + +Refer to the *DPDK Getting Started Guide* for general information on running applications and +the Environment Abstraction Layer (EAL) options. + +Explanation +----------- + +The following sections provide some explanation of the code. + +Initialization +~~~~~~~~~~~~~~ + +The EAL, driver and PCI configuration is performed largely as in the L2 Forwarding sample application, +as is the creation of the mbuf pool. +See :doc:`l2_forward_real_virtual`. +Where this example application differs is in the configuration of the NIC port for RX. + +The VMDq hardware feature is configured at port initialization time by setting the appropriate values in the +rte_eth_conf structure passed to the rte_eth_dev_configure() API. +Initially in the application, +a default structure is provided for VMDq configuration to be filled in later by the application. + +.. code-block:: c + + /* empty vmdq configuration structure. Filled in programmatically */ + static const struct rte_eth_conf vmdq_conf_default = { + .rxmode = { + .mq_mode = ETH_MQ_RX_VMDQ_ONLY, + .split_hdr_size = 0, + }, + + .txmode = { + .mq_mode = ETH_MQ_TX_NONE, + }, + .rx_adv_conf = { + /* + * should be overridden separately in code with + * appropriate values + */ + .vmdq_rx_conf = { + .nb_queue_pools = ETH_8_POOLS, + .enable_default_pool = 0, + .default_pool = 0, + .nb_pool_maps = 0, + .pool_map = {{0, 0},}, + }, + }, + }; + +The get_eth_conf() function fills in an rte_eth_conf structure with the appropriate values, +based on the global vlan_tags array. +For the VLAN IDs, each one can be allocated to possibly multiple pools of queues. +For destination MAC, each VMDq pool will be assigned with a MAC address. In this sample, each VMDq pool +is assigned to the MAC like 52:54:00:12:<port_id>:<pool_id>, that is, +the MAC of VMDq pool 2 on port 1 is 52:54:00:12:01:02. + +.. code-block:: c + + const uint16_t vlan_tags[] = { + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15, + 16, 17, 18, 19, 20, 21, 22, 23, + 24, 25, 26, 27, 28, 29, 30, 31, + 32, 33, 34, 35, 36, 37, 38, 39, + 40, 41, 42, 43, 44, 45, 46, 47, + 48, 49, 50, 51, 52, 53, 54, 55, + 56, 57, 58, 59, 60, 61, 62, 63, + }; + + /* pool mac addr template, pool mac addr is like: 52 54 00 12 port# pool# */ + static struct rte_ether_addr pool_addr_template = { + .addr_bytes = {0x52, 0x54, 0x00, 0x12, 0x00, 0x00} + }; + + /* + * Builds up the correct configuration for vmdq based on the vlan tags array + * given above, and determine the queue number and pool map number according to + * valid pool number + */ + static inline int + get_eth_conf(struct rte_eth_conf *eth_conf, uint32_t num_pools) + { + struct rte_eth_vmdq_rx_conf conf; + unsigned i; + + conf.nb_queue_pools = (enum rte_eth_nb_pools)num_pools; + conf.nb_pool_maps = num_pools; + conf.enable_default_pool = 0; + conf.default_pool = 0; /* set explicit value, even if not used */ + + for (i = 0; i < conf.nb_pool_maps; i++) { + conf.pool_map[i].vlan_id = vlan_tags[i]; + conf.pool_map[i].pools = (1UL << (i % num_pools)); + } + + (void)(rte_memcpy(eth_conf, &vmdq_conf_default, sizeof(*eth_conf))); + (void)(rte_memcpy(ð_conf->rx_adv_conf.vmdq_rx_conf, &conf, + sizeof(eth_conf->rx_adv_conf.vmdq_rx_conf))); + return 0; + } + + ...... + + /* + * Set mac for each pool. + * There is no default mac for the pools in i40. + * Removes this after i40e fixes this issue. + */ + for (q = 0; q < num_pools; q++) { + struct rte_ether_addr mac; + mac = pool_addr_template; + mac.addr_bytes[4] = port; + mac.addr_bytes[5] = q; + printf("Port %u vmdq pool %u set mac %02x:%02x:%02x:%02x:%02x:%02x\n", + port, q, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); + retval = rte_eth_dev_mac_addr_add(port, &mac, + q + vmdq_pool_base); + if (retval) { + printf("mac addr add failed at pool %d\n", q); + return retval; + } + } + +Once the network port has been initialized using the correct VMDq values, +the initialization of the port's RX and TX hardware rings is performed similarly to that +in the L2 Forwarding sample application. +See :doc:`l2_forward_real_virtual` for more information. + +Statistics Display +~~~~~~~~~~~~~~~~~~ + +When run in a Linux environment, +the VMDq Forwarding sample application can display statistics showing the number of packets read from each RX queue. +This is provided by way of a signal handler for the SIGHUP signal, +which simply prints to standard output the packet counts in grid form. +Each row of the output is a single pool with the columns being the queue number within that pool. + +To generate the statistics output, use the following command: + +.. code-block:: console + + user@host$ sudo killall -HUP vmdq_app + +Please note that the statistics output will appear on the terminal where the vmdq_app is running, +rather than the terminal from which the HUP signal was sent. + |
