1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
|
.. _orchestrator-cli-module:
================
Orchestrator CLI
================
This module provides a command line interface (CLI) to orchestrator
modules (ceph-mgr modules which interface with external orchestation services)
As the orchestrator CLI unifies different external orchestrators, a common nomenclature
for the orchestrator module is needed.
+--------------------------------------+---------------------------------------+
| host | hostname (not DNS name) of the |
| | physical host. Not the podname, |
| | container name, or hostname inside |
| | the container. |
+--------------------------------------+---------------------------------------+
| service type | The type of the service. e.g., nfs, |
| | mds, osd, mon, rgw, mgr, iscsi |
+--------------------------------------+---------------------------------------+
| service | A logical service, Typically |
| | comprised of multiple service |
| | instances on multiple hosts for HA |
| | |
| | * ``fs_name`` for mds type |
| | * ``rgw_zone`` for rgw type |
| | * ``ganesha_cluster_id`` for nfs type |
+--------------------------------------+---------------------------------------+
| service instance | A single instance of a service. |
| | Usually a daemon, but maybe not |
| | (e.g., might be a kernel service |
| | like LIO or knfsd or whatever) |
| | |
| | This identifier should |
| | uniquely identify the instance |
+--------------------------------------+---------------------------------------+
| daemon | A running process on a host; use |
| | “service instance” instead |
+--------------------------------------+---------------------------------------+
The relation between the names is the following:
* a service belongs to a service type
* a service instance belongs to a service type
* a service instance belongs to a single service group
Configuration
=============
To enable the orchestrator, please select the orchestrator module to use
with the ``set backend`` command::
ceph orchestrator set backend <module>
For example, to enable the Rook orchestrator module and use it with the CLI::
ceph mgr module enable rook
ceph orchestrator set backend rook
You can then check backend is properly configured::
ceph orchestrator status
Disable the Orchestrator
~~~~~~~~~~~~~~~~~~~~~~~~
To disable the orchestrator again, use the empty string ``""``::
ceph orchestrator set backend ""``
ceph mgr module disable rook
Usage
=====
.. warning::
The orchestrator CLI is unfinished and work in progress. Some commands will not
exist, or return a different result.
.. note::
Orchestrator modules may only implement a subset of the commands listed below.
Also, the implementation of the commands are orchestrator module dependent and will
differ between implementations.
Status
~~~~~~
::
ceph orchestrator status
Show current orchestrator mode and high-level status (whether the module able
to talk to it)
Also show any in-progress actions.
Host Management
~~~~~~~~~~~~~~~
List hosts associated with the cluster::
ceph orchestrator host ls
Add and remove hosts::
ceph orchestrator host add <host>
ceph orchestrator host rm <host>
OSD Management
~~~~~~~~~~~~~~
List Devices
^^^^^^^^^^^^
Print a list of discovered devices, grouped by node and optionally
filtered to a particular node:
::
ceph orchestrator device ls [--host=...] [--refresh]
Create OSDs
^^^^^^^^^^^
Create OSDs on a group of devices on a single host::
ceph orchestrator osd create <host>:<drive>
ceph orchestrator osd create -i <path-to-drive-group.json>
The output of ``osd create`` is not specified and may vary between orchestrator backends.
Where ``drive.group.json`` is a JSON file containing the fields defined in :class:`orchestrator.DriveGroupSpec`
Decommission an OSD
^^^^^^^^^^^^^^^^^^^
::
ceph orchestrator osd rm <osd-id> [osd-id...]
Removes one or more OSDs from the cluster and the host, if the OSDs are marked as
``destroyed``.
..
Blink Device Lights
^^^^^^^^^^^^^^^^^^^
::
ceph orchestrator device ident-on <host> <devname>
ceph orchestrator device ident-off <host> <devname>
ceph orchestrator device fault-on <host> <devname>
ceph orchestrator device fault-off <host> <devname>
ceph orchestrator osd ident-on {primary,journal,db,wal,all} <osd-id>
ceph orchestrator osd ident-off {primary,journal,db,wal,all} <osd-id>
ceph orchestrator osd fault-on {primary,journal,db,wal,all} <osd-id>
ceph orchestrator osd fault-off {primary,journal,db,wal,all} <osd-id>
Where ``journal`` is the filestore journal, ``wal`` is the write ahead log of
bluestore and ``all`` stands for all devices associated with the osd
Monitor and manager management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Creates or removes MONs or MGRs from the cluster. Orchestrator may return an
error if it doesn't know how to do this transition.
Update the number of monitor nodes::
ceph orchestrator mon update <num> [host, host:network...]
Each host can optionally specificy a network for the monitor to listen on.
Update the number of manager nodes::
ceph orchestrator mgr update <num> [host...]
..
.. note::
The host lists are the new full list of mon/mgr hosts
.. note::
specifying hosts is optional for some orchestrator modules
and mandatory for others (e.g. Ansible).
Service Status
~~~~~~~~~~~~~~
Print a list of services known to the orchestrator. The list can be limited to
services on a particular host with the optional --host parameter and/or
services of a particular type via optional --type parameter
(mon, osd, mgr, mds, rgw):
::
ceph orchestrator service ls [--host host] [--svc_type type] [--refresh]
Discover the status of a particular service::
ceph orchestrator service ls --svc_type type --svc_id <name> [--refresh]
Query the status of a particular service instance (mon, osd, mds, rgw). For OSDs
the id is the numeric OSD ID, for MDS services it is the filesystem name::
ceph orchestrator service-instance status <type> <instance-name> [--refresh]
Stateless services (MDS/RGW/NFS/rbd-mirror/iSCSI)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The orchestrator is not responsible for configuring the services. Please look into the corresponding
documentation for details.
The ``name`` parameter is an identifier of the group of instances:
* a CephFS filesystem for a group of MDS daemons,
* a zone name for a group of RGWs
Sizing: the ``size`` parameter gives the number of daemons in the cluster
(e.g. the number of MDS daemons for a particular CephFS filesystem).
Creating/growing/shrinking/removing services::
ceph orchestrator {mds,rgw} update <name> <size> [host…]
ceph orchestrator {mds,rgw} add <name>
ceph orchestrator nfs update <name> <size> [host…]
ceph orchestrator nfs add <name> <pool> [--namespace=<namespace>]
ceph orchestrator {mds,rgw,nfs} rm <name>
e.g., ``ceph orchestrator mds update myfs 3 host1 host2 host3``
Start/stop/reload::
ceph orchestrator service {stop,start,reload} <type> <name>
ceph orchestrator service-instance {start,stop,reload} <type> <instance-name>
Current Implementation Status
=============================
This is an overview of the current implementation status of the orchestrators.
=================================== ========= ====== ========= =====
Command Ansible Rook DeepSea SSH
=================================== ========= ====== ========= =====
host add ⚪ ⚪ ⚪ ✔️
host ls ⚪ ⚪ ⚪ ✔️
host rm ⚪ ⚪ ⚪ ✔️
mgr update ⚪ ⚪ ⚪ ✔️
mon update ⚪ ✔️ ⚪ ✔️
osd create ✔️ ✔️ ⚪ ✔️
osd device {ident,fault}-{on,off} ⚪ ⚪ ⚪ ⚪
osd rm ✔️ ⚪ ⚪ ⚪
device {ident,fault}-(on,off} ⚪ ⚪ ⚪ ⚪
device ls ✔️ ✔️ ✔️ ✔️
service ls ⚪ ✔️ ✔️ ⚪
service-instance status ⚪ ⚪ ⚪ ⚪
iscsi {stop,start,reload} ⚪ ⚪ ⚪ ⚪
iscsi add ⚪ ⚪ ⚪ ⚪
iscsi rm ⚪ ⚪ ⚪ ⚪
iscsi update ⚪ ⚪ ⚪ ⚪
mds {stop,start,reload} ⚪ ⚪ ⚪ ⚪
mds add ⚪ ✔️ ⚪ ⚪
mds rm ⚪ ✔️ ⚪ ⚪
mds update ⚪ ⚪ ⚪ ⚪
nfs {stop,start,reload} ⚪ ⚪ ⚪ ⚪
nfs add ⚪ ✔️ ⚪ ⚪
nfs rm ⚪ ✔️ ⚪ ⚪
nfs update ⚪ ⚪ ⚪ ⚪
rbd-mirror {stop,start,reload} ⚪ ⚪ ⚪ ⚪
rbd-mirror add ⚪ ⚪ ⚪ ⚪
rbd-mirror rm ⚪ ⚪ ⚪ ⚪
rbd-mirror update ⚪ ⚪ ⚪ ⚪
rgw {stop,start,reload} ⚪ ⚪ ⚪ ⚪
rgw add ⚪ ✔️ ⚪ ⚪
rgw rm ⚪ ✔️ ⚪ ⚪
rgw update ⚪ ⚪ ⚪ ⚪
=================================== ========= ====== ========= =====
where
* ⚪ = not yet implemented
* ❌ = not applicable
* ✔ = implemented
|
