Adding upstream version 18.2.2.upstream/18.2.2

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 271 insertions, 0 deletions

diff --git a/src/spdk/doc/nvmf.md b/src/spdk/doc/nvmf.md
new file mode 100644
index 000000000..34a69b66f
--- /dev/null
+++ b/src/spdk/doc/nvmf.md
@@ -0,0 +1,271 @@
+# NVMe over Fabrics Target {#nvmf}
+
+@sa @ref nvme_fabrics_host
+@sa @ref nvmf_tgt_tracepoints
+
+# NVMe-oF Target Getting Started Guide {#nvmf_getting_started}
+
+The SPDK NVMe over Fabrics target is a user space application that presents block devices over a fabrics
+such as Ethernet, Infiniband or Fibre Channel. SPDK currently supports RDMA and TCP transports.
+
+The NVMe over Fabrics specification defines subsystems that can be exported over different transports.
+SPDK has chosen to call the software that exports these subsystems a "target", which is the term used
+for iSCSI. The specification refers to the "client" that connects to the target as a "host". Many
+people will also refer to the host as an "initiator", which is the equivalent thing in iSCSI
+parlance. SPDK will try to stick to the terms "target" and "host" to match the specification.
+
+The Linux kernel also implements an NVMe-oF target and host, and SPDK is tested for
+interoperability with the Linux kernel implementations.
+
+If you want to kill the application using signal, make sure use the SIGTERM, then the application
+will release all the share memory resource before exit, the SIGKILL will make the share memory
+resource have no chance to be released by application, you may need to release the resource manually.
+
+## RDMA transport support {#nvmf_rdma_transport}
+
+It requires an RDMA-capable NIC with its corresponding OFED (OpenFabrics Enterprise Distribution)
+software package installed to run. Maybe OS distributions provide packages, but OFED is also
+available [here](https://downloads.openfabrics.org/OFED/).
+
+### Prerequisites {#nvmf_prereqs}
+
+To build nvmf_tgt with the RDMA transport, there are some additional dependencies,
+which can be install using pkgdep.sh script.
+
+~~~{.sh}
+sudo scripts/pkgdep.sh --rdma
+~~~
+
+Then build SPDK with RDMA enabled:
+
+~~~{.sh}
+./configure --with-rdma <other config parameters>
+make
+~~~
+
+Once built, the binary will be in `build/bin`.
+
+### Prerequisites for InfiniBand/RDMA Verbs {#nvmf_prereqs_verbs}
+
+Before starting our NVMe-oF target with the RDMA transport we must load the InfiniBand and RDMA modules
+that allow userspace processes to use InfiniBand/RDMA verbs directly.
+
+~~~{.sh}
+modprobe ib_cm
+modprobe ib_core
+# Please note that ib_ucm does not exist in newer versions of the kernel and is not required.
+modprobe ib_ucm || true
+modprobe ib_umad
+modprobe ib_uverbs
+modprobe iw_cm
+modprobe rdma_cm
+modprobe rdma_ucm
+~~~
+
+### Prerequisites for RDMA NICs {#nvmf_prereqs_rdma_nics}
+
+Before starting our NVMe-oF target we must detect RDMA NICs and assign them IP addresses.
+
+### Finding RDMA NICs and associated network interfaces
+
+~~~{.sh}
+ls /sys/class/infiniband/*/device/net
+~~~
+
+#### Mellanox ConnectX-3 RDMA NICs
+
+~~~{.sh}
+modprobe mlx4_core
+modprobe mlx4_ib
+modprobe mlx4_en
+~~~
+
+#### Mellanox ConnectX-4 RDMA NICs
+
+~~~{.sh}
+modprobe mlx5_core
+modprobe mlx5_ib
+~~~
+
+#### Assigning IP addresses to RDMA NICs
+
+~~~{.sh}
+ifconfig eth1 192.168.100.8 netmask 255.255.255.0 up
+ifconfig eth2 192.168.100.9 netmask 255.255.255.0 up
+~~~
+
+### RDMA Limitations {#nvmf_rdma_limitations}
+
+As RDMA NICs put a limitation on the number of memory regions registered, the SPDK NVMe-oF
+target application may eventually start failing to allocate more DMA-able memory. This is
+an imperfection of the DPDK dynamic memory management and is most likely to occur with too
+many 2MB hugepages reserved at runtime. One type of memory bottleneck is the number of NIC memory
+regions, e.g., some NICs report as many as 2048 for the maximum number of memory regions. This
+gives us a 4GB memory limit with 2MB hugepages for the total memory regions. It can be overcome by
+using 1GB hugepages or by pre-reserving memory at application startup with `--mem-size` or `-s`
+option. All pre-reserved memory will be registered as a single region, but won't be returned to the
+system until the SPDK application is terminated.
+
+## TCP transport support {#nvmf_tcp_transport}
+
+The transport is built into the nvmf_tgt by default, and it does not need any special libraries.
+
+## Configuring the SPDK NVMe over Fabrics Target {#nvmf_config}
+
+An NVMe over Fabrics target can be configured using JSON RPCs.
+The basic RPCs needed to configure the NVMe-oF subsystem are detailed below. More information about
+working with NVMe over Fabrics specific RPCs can be found on the @ref jsonrpc_components_nvmf_tgt RPC page.
+
+Using .ini style configuration files for configuration of the NVMe-oF target is deprecated and should
+be replaced with JSON based RPCs. .ini style configuration files can be converted to json format by way
+of the new script `scripts/config_converter.py`.
+
+## FC transport support {#nvmf_fc_transport}
+
+To build nvmf_tgt with the FC transport, there is an additional FC LLD (Low Level Driver) code dependency.
+Please contact your FC vendor for instructions to obtain FC driver module.
+
+### Broadcom FC LLD code
+
+FC LLD driver for Broadcom FC NVMe capable adapters can be obtained from,
+https://github.com/ecdufcdrvr/bcmufctdrvr.
+
+### Fetch FC LLD module and then build SPDK with FC enabled
+
+After cloning SPDK repo and initialize submodules, FC LLD library is built which then can be linked with
+the fc transport.
+
+~~~{.sh}
+git clone https://github.com/spdk/spdk spdk
+git clone https://github.com/ecdufcdrvr/bcmufctdrvr fc
+cd spdk
+git submodule update --init
+cd ../fc
+make DPDK_DIR=../spdk/dpdk/build SPDK_DIR=../spdk
+cd ../spdk
+./configure --with-fc=../fc/build
+make
+~~~
+
+### Using RPCs {#nvmf_config_rpc}
+
+Start the nvmf_tgt application with elevated privileges. Once the target is started,
+the nvmf_create_transport rpc can be used to initialize a given transport. Below is an
+example where the target is started and configured with two different transports.
+The RDMA transport is configured with an I/O unit size of 8192 bytes, 4 max qpairs per controller,
+and an in capsule data size of 0 bytes. The TCP transport is configured with an I/O unit size of
+16384 bytes, 8 max qpairs per controller, and an in capsule data size of 8192 bytes.
+
+~~~{.sh}
+build/bin/nvmf_tgt
+scripts/rpc.py nvmf_create_transport -t RDMA -u 8192 -p 4 -c 0
+scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
+~~~
+
+Below is an example of creating a malloc bdev and assigning it to a subsystem. Adjust the bdevs,
+NQN, serial number, and IP address with RDMA transport to your own circumstances. If you replace
+"rdma" with "TCP", then the subsystem will add a listener with TCP transport.
+
+~~~{.sh}
+scripts/rpc.py bdev_malloc_create -b Malloc0 512 512
+scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1
+scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0
+scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t rdma -a 192.168.100.8 -s 4420
+~~~
+
+### NQN Formal Definition
+
+NVMe qualified names or NQNs are defined in section 7.9 of the
+[NVMe specification](http://nvmexpress.org/wp-content/uploads/NVM_Express_Revision_1.3.pdf). SPDK has attempted to
+formalize that definition using [Extended Backus-Naur form](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form).
+SPDK modules use this formal definition (provided below) when validating NQNs.
+
+~~~{.sh}
+
+Basic Types
+year = 4 * digit ;
+month = '01' | '02' | '03' | '04' | '05' | '06' | '07' | '08' | '09' | '10' | '11' | '12' ;
+digit = '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' ;
+hex digit = 'A' | 'B' | 'C' | 'D' | 'E' | 'F' | 'a' | 'b' | 'c' | 'd' | 'e' | 'f' | '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' ;
+
+NQN Definition
+NVMe Qualified Name = ( NVMe-oF Discovery NQN | NVMe UUID NQN | NVMe Domain NQN ), '\0' ;
+NVMe-oF Discovery NQN = "nqn.2014-08.org.nvmexpress.discovery" ;
+NVMe UUID NQN = "nqn.2014-08.org.nvmexpress:uuid:", string UUID ;
+string UUID = 8 * hex digit, '-', 3 * (4 * hex digit, '-'), 12 * hex digit ;
+NVMe Domain NQN = "nqn.", year, '-', month, '.', reverse domain, ':', utf-8 string ;
+
+~~~
+
+Please note that the following types from the definition above are defined elsewhere:
+
+1. utf-8 string: Defined in [rfc 3629](https://tools.ietf.org/html/rfc3629).
+2. reverse domain: Equivalent to domain name as defined in [rfc 1034](https://tools.ietf.org/html/rfc1034).
+
+While not stated in the formal definition, SPDK enforces the requirement from the spec that the
+"maximum name is 223 bytes in length". SPDK does not include the null terminating character when
+defining the length of an nqn, and will accept an nqn containing up to 223 valid bytes with an
+additional null terminator. To be precise, SPDK follows the same conventions as the c standard
+library function [strlen()](http://man7.org/linux/man-pages/man3/strlen.3.html).
+
+#### NQN Comparisons
+
+SPDK compares NQNs byte for byte without case matching or unicode normalization. This has specific implications for
+uuid based NQNs. The following pair of NQNs, for example, would not match when compared in the SPDK NVMe-oF Target:
+
+nqn.2014-08.org.nvmexpress:uuid:11111111-aaaa-bbdd-ffee-123456789abc
+nqn.2014-08.org.nvmexpress:uuid:11111111-AAAA-BBDD-FFEE-123456789ABC
+
+In order to ensure the consistency of uuid based NQNs while using SPDK, users should use lowercase when representing
+alphabetic hex digits in their NQNs.
+
+### Assigning CPU Cores to the NVMe over Fabrics Target {#nvmf_config_lcore}
+
+SPDK uses the [DPDK Environment Abstraction Layer](http://dpdk.org/doc/guides/prog_guide/env_abstraction_layer.html)
+to gain access to hardware resources such as huge memory pages and CPU core(s). DPDK EAL provides
+functions to assign threads to specific cores.
+To ensure the SPDK NVMe-oF target has the best performance, configure the NICs and NVMe devices to
+be located on the same NUMA node.
+
+The `-m` core mask option specifies a bit mask of the CPU cores that
+SPDK is allowed to execute work items on.
+For example, to allow SPDK to use cores 24, 25, 26 and 27:
+~~~{.sh}
+build/bin/nvmf_tgt -m 0xF000000
+~~~
+
+## Configuring the Linux NVMe over Fabrics Host {#nvmf_host}
+
+Both the Linux kernel and SPDK implement an NVMe over Fabrics host.
+The Linux kernel NVMe-oF RDMA host support is provided by the `nvme-rdma` driver
+(to support RDMA transport) and `nvme-tcp` (to support TCP transport). And the
+following shows two different commands for loading the driver.
+
+~~~{.sh}
+modprobe nvme-rdma
+modprobe nvme-tcp
+~~~
+
+The nvme-cli tool may be used to interface with the Linux kernel NVMe over Fabrics host.
+See below for examples of the discover, connect and disconnect commands. In all three instances, the
+transport can be changed to TCP by interchanging 'rdma' for 'tcp'.
+
+Discovery:
+~~~{.sh}
+nvme discover -t rdma -a 192.168.100.8 -s 4420
+~~~
+
+Connect:
+~~~{.sh}
+nvme connect -t rdma -n "nqn.2016-06.io.spdk:cnode1" -a 192.168.100.8 -s 4420
+~~~
+
+Disconnect:
+~~~{.sh}
+nvme disconnect -n "nqn.2016-06.io.spdk:cnode1"
+~~~
+
+## Enabling NVMe-oF target tracepoints for offline analysis and debug {#nvmf_trace}
+
+SPDK has a tracing framework for capturing low-level event information at runtime.
+@ref nvmf_tgt_tracepoints enable analysis of both performance and application crashes.