1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
|
nvmetcli(8)
===========
NAME
----
nvmetcli - Configure NVMe-over-Fabrics Target.
USAGE
------
[verse]
nvmetcli
nvmetcli clear
nvmetcli restore [filename.json]
DESCRIPTION
-----------
*nvmetcli* is a program used for viewing, editing, saving,
and starting a Linux kernel NVMe Target, used for an NVMe-over-Fabrics
network configuration. It allows an administrator to export
a storage resource (such as NVMe devices, files, and volumes)
to a local block device and expose them to remote systems
based on the NVMe-over-Fabrics specification from http://www.nvmexpress.org.
*nvmetcli* is run as root and has two modes:
1. An interactive configuration shell
2. Command-line mode which uses an argument
BACKGROUND
----------
The term *NQN* used throughout this man page is the *NVMe Qualified
Name* format which an NVMe endpoint (device, subsystem, etc) must
follow to guarantee a unique name under the NVMe standard. Any
name in a network system setup can be used, but if it does not
follow the NQN format, it may not be unique on an NVMe-over-Fabrics network.
Note that some of the fields set for an NVMe Target port under
interactive mode are defined in the "Discovery Log Page" section of
NVMe-over-Fabrics specification. Each NVMe Target has a
discovery controller mechanism that an NVMe Host can use to determine
the NVM subsystems it can access. *nvmetcli* can be used to add
a new record to the discovery controller upon each new subsystem
entry and port entry that the newly created subsystem entry binds
to (see *OPTIONS* and *EXAMPLES* sections). Each NVMe
Host only gets to see the discovery entries defined in
*/subsystems/[NQN NAME]/allowed_hosts* and the IP port it is connected
to the NVMe Target. An NVMe Host can retrieve these discovery logs via
the nvme-cli tool (https://github.com/linux-nvme/nvme-cli).
OPTIONS
-------
*Interactive Configuration Shell*
To start the interactive configuration shell, type *nvmetcli* on
the command-line. nvmetcli interacts with the Linux kernel
NVMe Target configfs subsystem starting at base
nvmetcli directories **/port**, **/subsystem**, and **/host**.
Configuration changes entered by the administrator are made
immediately to the kernel target configuration. The
following commands can be used while in the interactive configuration
shell mode:
[]
|==================
| cd | Allows to move around the tree.
| ls | Lists contents of current tree node.
| create [NQN name]/[#] | Create a new object using the specified name
or number. If a [NQN name]/[#] is not specified,
a random entry will be used.
| delete [NQN name]/[#] | Delete an object with the specified name or number.
| set attr allow_any_host=[0/1] | Used under */subsystems/[NQN name]* to
specify if any NVMe Host can connect to
the subsystem.
| set device path=[device path] | Used under
*/subsystems/[NQN name]/namespaces*
to set the (storage) device to be used.
| set device nguid=[string] | Used under
*/subsystems/[NQN name]/namespaces*
to set the unique id of the device to
the defined namespace.
| enable/disable | Used under
*/subsystems/[NQN name]/namespaces*
to enable and disable the namespace.
| set addr [discovery log page field]=[string] | Used under */ports/[#]*
to create a port which
access is allowed. See
*EXAMPLES* for more
information.
| saveconfig [filename.json] | Save the NVMe Target configuration in .json
format. Without specifying the
filename this will save as
*/etc/nvmet/config.json*. This file
is in JSON format and can be edited directly
using a preferred file editor.
| exit | Quits interactive configuration shell mode.
|==================
*Command Line Mode*
Typing *nvmetcli [cmd]* on the command-line will execute a command
and not enter the interactive configuration shell.
[]
|==================
| restore [filename.json] | Loads a saved NVMe Target configuration.
Without specifying the filename this will use
*/etc/nvmet/config.json*.
| clear | Clears a current NVMe Target configuration.
| ls | Dumps the current NVMe Target configuration.
|==================
EXAMPLES
--------
Make sure to run nvmetcli as root, the nvmet module is loaded,
your devices and all dependent modules are loaded,
and configfs is mounted on /sys/kernel/config
using:
mount -t configfs none /sys/kernel/config
The following section walks through a configuration example.
* To get started with the interactive mode and the nvmetcli command prompt,
type (in root):
--------------
# ./nvmetcli
...>
--------------
* Create a subsystem. If you do not specify a name a NQN will be generated,
which is probably the best choice. We don't do it here as the name
would be random:
--------------
> cd /subsystems
...> create testnqn
--------------
* Add access for a specific NVMe Host by it's NQN:
--------------
...> cd /hosts
...> create hostnqn
...> cd /subsystems/testnqn
...> set attr allow_any_host=0
...> cd /subsystems/testnqn/allowed_hosts/
...> create hostnqn
--------------
* Remove access of a subsystem by deleting the Host NQN:
--------------
...> cd /subsystems/testnqn/allowed_hosts/
...> delete hostnqn
--------------
* Alternatively this allows any Host to connect to the subsystsem. Only
use this in tightly controlled environments:
--------------
...> cd /subsystems/testnqn/
...> set attr allow_any_host=1
--------------
* Create a new namespace. If you do not specify a namespace ID the fist
unused one will be used:
--------------
...> cd /subsystems/testnqn/namespaces
...> create 1
...> cd 1
...> set device path=/dev/nvme0n1
...> enable
--------------
Note that in the above setup the 'device_nguid' attribute
does not have to be set for correct NVMe Target functionality (but
to correctly match a namespace to the exact device upon
clear and restore operations, it is advised to set the
'device_nguid' parameter).
* Create a loopback port that can be used with nvme-loop module
on the same physical machine...
--------------
...> cd /ports/
...> create 1
...> cd 1/
...> set addr trtype=loop
...> cd subsystems/
...> create testnqn
--------------
* or create an RDMA (IB, RoCE, iWarp) port using IPv4 addressing. 4420 is the
IANA assigned default port for NVMe over Fabrics using RDMA:
--------------
...> cd /ports/
...> create 2
...> cd 2/
...> set addr trtype=rdma
...> set addr adrfam=ipv4
...> set addr traddr=192.168.6.68
...> set addr trsvcid=4420
...> cd subsystems/
...> create testnqn
--------------
* or create an FC port. traddr is the WWNN/WWPN of the FC port.
--------------
...> cd /ports/
...> create 3
...> cd 3/
...> set addr trtype=fc
...> set addr adrfam=fc
...> set addr traddr=nn-0x1000000044001123:pn-0x2000000055001123
...> set addr trsvcid=none
...> cd subsystems/
...> create testnqn
--------------
* Saving the NVMe Target configuration:
--------------
./nvmetcli
...> saveconfig test.json
--------------
* Loading an NVMe Target configuration:
--------------
./nvmetcli restore test.json
--------------
* Clearing a current NVMe Target configuration:
--------------
./nvmetcli clear
--------------
ADDITIONAL INFORMATION
----------------------
nvmetcli has the ability to start and stop the NVMe Target configuration
on boot and shutdown through the *systemctl* Linux utility via a .service file.
nvmetcli package comes with *nvmet.service* which when installed, it can
automatically restore the default, saved NVMe Target configuration from
*/etc/nvmet/config.json*. *nvmet.service* can be installed in directories
such as */lib/systemd/system*.
To explicitly enable the service, type:
--------------
systemctl enable nvmet
--------------
To explicitly disable the service, type:
--------------
systemctl disable nvmet
--------------
See also systemctl(1).
AUTHORS
-------
This man page was written by
mailto:james.p.freyensee@intel.com[Jay Freyensee]. nvmetcli was
originally written by mailto:hch@infradead.org[Christoph Hellwig].
REPORTING BUGS & DEVELOPMENT
-----------------------------
Please send patches and bug reports to linux-nvme@lists.infradead.org
for review and acceptance.
LICENSE
-------
nvmetcli is licensed under the *Apache License, Version 2.0*. Software
distributed under this license is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied.
|
