1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
|
==============
Upgrading Ceph
==============
Cephadm can safely upgrade Ceph from one bugfix release to the next. For
example, you can upgrade from v15.2.0 (the first Octopus release) to the next
point release, v15.2.1.
The automated upgrade process follows Ceph best practices. For example:
* The upgrade order starts with managers, monitors, then other daemons.
* Each daemon is restarted only after Ceph indicates that the cluster
will remain available.
.. note::
The Ceph cluster health status is likely to switch to
``HEALTH_WARNING`` during the upgrade.
.. note::
In case a host of the cluster is offline, the upgrade is paused.
Starting the upgrade
====================
Before you use cephadm to upgrade Ceph, verify that all hosts are currently online and that your cluster is healthy by running the following command:
.. prompt:: bash #
ceph -s
To upgrade (or downgrade) to a specific release, run the following command:
.. prompt:: bash #
ceph orch upgrade start --ceph-version <version>
For example, to upgrade to v16.2.6, run the following command:
.. prompt:: bash #
ceph orch upgrade start --ceph-version 16.2.6
.. note::
From version v16.2.6 the Docker Hub registry is no longer used, so if you use Docker you have to point it to the image in the quay.io registry:
.. prompt:: bash #
ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.6
Monitoring the upgrade
======================
Determine (1) whether an upgrade is in progress and (2) which version the
cluster is upgrading to by running the following command:
.. prompt:: bash #
ceph orch upgrade status
Watching the progress bar during a Ceph upgrade
-----------------------------------------------
During the upgrade, a progress bar is visible in the ceph status output. It
looks like this:
.. code-block:: console
# ceph -s
[...]
progress:
Upgrade to docker.io/ceph/ceph:v15.2.1 (00h 20m 12s)
[=======.....................] (time remaining: 01h 43m 31s)
Watching the cephadm log during an upgrade
------------------------------------------
Watch the cephadm log by running the following command:
.. prompt:: bash #
ceph -W cephadm
Canceling an upgrade
====================
You can stop the upgrade process at any time by running the following command:
.. prompt:: bash #
ceph orch upgrade stop
Post upgrade actions
====================
In case the new version is based on ``cephadm``, once done with the upgrade the user
has to update the ``cephadm`` package (or ceph-common package in case the user
doesn't use ``cephadm shell``) to a version compatible with the new version.
Potential problems
==================
There are a few health alerts that can arise during the upgrade process.
UPGRADE_NO_STANDBY_MGR
----------------------
This alert (``UPGRADE_NO_STANDBY_MGR``) means that Ceph does not detect an
active standby manager daemon. In order to proceed with the upgrade, Ceph
requires an active standby manager daemon (which you can think of in this
context as "a second manager").
You can ensure that Cephadm is configured to run 2 (or more) managers by
running the following command:
.. prompt:: bash #
ceph orch apply mgr 2 # or more
You can check the status of existing mgr daemons by running the following
command:
.. prompt:: bash #
ceph orch ps --daemon-type mgr
If an existing mgr daemon has stopped, you can try to restart it by running the
following command:
.. prompt:: bash #
ceph orch daemon restart <name>
UPGRADE_FAILED_PULL
-------------------
This alert (``UPGRADE_FAILED_PULL``) means that Ceph was unable to pull the
container image for the target version. This can happen if you specify a
version or container image that does not exist (e.g. "1.2.3"), or if the
container registry can not be reached by one or more hosts in the cluster.
To cancel the existing upgrade and to specify a different target version, run
the following commands:
.. prompt:: bash #
ceph orch upgrade stop
ceph orch upgrade start --ceph-version <version>
Using customized container images
=================================
For most users, upgrading requires nothing more complicated than specifying the
Ceph version number to upgrade to. In such cases, cephadm locates the specific
Ceph container image to use by combining the ``container_image_base``
configuration option (default: ``docker.io/ceph/ceph``) with a tag of
``vX.Y.Z``.
But it is possible to upgrade to an arbitrary container image, if that's what
you need. For example, the following command upgrades to a development build:
.. prompt:: bash #
ceph orch upgrade start --image quay.io/ceph-ci/ceph:recent-git-branch-name
For more information about available container images, see :ref:`containers`.
Staggered Upgrade
=================
Some users may prefer to upgrade components in phases rather than all at once.
The upgrade command, starting in 16.2.11 and 17.2.1 allows parameters
to limit which daemons are upgraded by a single upgrade command. The options in
include ``daemon_types``, ``services``, ``hosts`` and ``limit``. ``daemon_types``
takes a comma-separated list of daemon types and will only upgrade daemons of those
types. ``services`` is mutually exclusive with ``daemon_types``, only takes services
of one type at a time (e.g. can't provide an OSD and RGW service at the same time), and
will only upgrade daemons belonging to those services. ``hosts`` can be combined
with ``daemon_types`` or ``services`` or provided on its own. The ``hosts`` parameter
follows the same format as the command line options for :ref:`orchestrator-cli-placement-spec`.
``limit`` takes an integer > 0 and provides a numerical limit on the number of
daemons cephadm will upgrade. ``limit`` can be combined with any of the other
parameters. For example, if you specify to upgrade daemons of type osd on host
Host1 with ``limit`` set to 3, cephadm will upgrade (up to) 3 osd daemons on
Host1.
Example: specifying daemon types and hosts:
.. prompt:: bash #
ceph orch upgrade start --image <image-name> --daemon-types mgr,mon --hosts host1,host2
Example: specifying services and using limit:
.. prompt:: bash #
ceph orch upgrade start --image <image-name> --services rgw.example1,rgw.example2 --limit 2
.. note::
Cephadm strictly enforces an order to the upgrade of daemons that is still present
in staggered upgrade scenarios. The current upgrade ordering is
``mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -> iscsi -> nfs``.
If you specify parameters that would upgrade daemons out of order, the upgrade
command will block and note which daemons will be missed if you proceed.
.. note::
Upgrade commands with limiting parameters will validate the options before beginning the
upgrade, which may require pulling the new container image. Do not be surprised
if the upgrade start command takes a while to return when limiting parameters are provided.
.. note::
In staggered upgrade scenarios (when a limiting parameter is provided) monitoring
stack daemons including Prometheus and node-exporter are refreshed after the Manager
daemons have been upgraded. Do not be surprised if Manager upgrades thus take longer
than expected. Note that the versions of monitoring stack daemons may not change between
Ceph releases, in which case they are only redeployed.
Upgrading to a version that supports staggered upgrade from one that doesn't
----------------------------------------------------------------------------
While upgrading from a version that already supports staggered upgrades the process
simply requires providing the necessary arguments. However, if you wish to upgrade
to a version that supports staggered upgrade from one that does not, there is a
workaround. It requires first manually upgrading the Manager daemons and then passing
the limiting parameters as usual.
.. warning::
Make sure you have multiple running mgr daemons before attempting this procedure.
To start with, determine which Manager is your active one and which are standby. This
can be done in a variety of ways such as looking at the ``ceph -s`` output. Then,
manually upgrade each standby mgr daemon with:
.. prompt:: bash #
ceph orch daemon redeploy mgr.example1.abcdef --image <new-image-name>
.. note::
If you are on a very early version of cephadm (early Octopus) the ``orch daemon redeploy``
command may not have the ``--image`` flag. In that case, you must manually set the
Manager container image ``ceph config set mgr container_image <new-image-name>`` and then
redeploy the Manager ``ceph orch daemon redeploy mgr.example1.abcdef``
At this point, a Manager fail over should allow us to have the active Manager be one
running the new version.
.. prompt:: bash #
ceph mgr fail
Verify the active Manager is now one running the new version. To complete the Manager
upgrading:
.. prompt:: bash #
ceph orch upgrade start --image <new-image-name> --daemon-types mgr
You should now have all your Manager daemons on the new version and be able to
specify the limiting parameters for the rest of the upgrade.
|
