summaryrefslogtreecommitdiffstats
path: root/doc/cephadm/upgrade.rst
diff options
context:
space:
mode:
Diffstat (limited to 'doc/cephadm/upgrade.rst')
-rw-r--r--doc/cephadm/upgrade.rst295
1 files changed, 295 insertions, 0 deletions
diff --git a/doc/cephadm/upgrade.rst b/doc/cephadm/upgrade.rst
new file mode 100644
index 000000000..e0a9f610a
--- /dev/null
+++ b/doc/cephadm/upgrade.rst
@@ -0,0 +1,295 @@
+==============
+Upgrading Ceph
+==============
+
+Cephadm can safely upgrade Ceph from one bugfix release to the next. For
+example, you can upgrade from v15.2.0 (the first Octopus release) to the next
+point release, v15.2.1.
+
+The automated upgrade process follows Ceph best practices. For example:
+
+* The upgrade order starts with managers, monitors, then other daemons.
+* Each daemon is restarted only after Ceph indicates that the cluster
+ will remain available.
+
+.. note::
+
+ The Ceph cluster health status is likely to switch to
+ ``HEALTH_WARNING`` during the upgrade.
+
+.. note::
+
+ In case a host of the cluster is offline, the upgrade is paused.
+
+
+Starting the upgrade
+====================
+
+.. note::
+ .. note::
+ `Staggered Upgrade`_ of the mons/mgrs may be necessary to have access
+ to this new feature.
+
+ Cephadm by default reduces `max_mds` to `1`. This can be disruptive for large
+ scale CephFS deployments because the cluster cannot quickly reduce active MDS(s)
+ to `1` and a single active MDS cannot easily handle the load of all clients
+ even for a short time. Therefore, to upgrade MDS(s) without reducing `max_mds`,
+ the `fail_fs` option can to be set to `true` (default value is `false`) prior
+ to initiating the upgrade:
+
+ .. prompt:: bash #
+
+ ceph config set mgr mgr/orchestrator/fail_fs true
+
+ This would:
+ #. Fail CephFS filesystems, bringing active MDS daemon(s) to
+ `up:standby` state.
+
+ #. Upgrade MDS daemons safely.
+
+ #. Bring CephFS filesystems back up, bringing the state of active
+ MDS daemon(s) from `up:standby` to `up:active`.
+
+Before you use cephadm to upgrade Ceph, verify that all hosts are currently online and that your cluster is healthy by running the following command:
+
+.. prompt:: bash #
+
+ ceph -s
+
+To upgrade (or downgrade) to a specific release, run the following command:
+
+.. prompt:: bash #
+
+ ceph orch upgrade start --ceph-version <version>
+
+For example, to upgrade to v16.2.6, run the following command:
+
+.. prompt:: bash #
+
+ ceph orch upgrade start --ceph-version 16.2.6
+
+.. note::
+
+ From version v16.2.6 the Docker Hub registry is no longer used, so if you use Docker you have to point it to the image in the quay.io registry:
+
+.. prompt:: bash #
+
+ ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.6
+
+
+Monitoring the upgrade
+======================
+
+Determine (1) whether an upgrade is in progress and (2) which version the
+cluster is upgrading to by running the following command:
+
+.. prompt:: bash #
+
+ ceph orch upgrade status
+
+Watching the progress bar during a Ceph upgrade
+-----------------------------------------------
+
+During the upgrade, a progress bar is visible in the ceph status output. It
+looks like this:
+
+.. code-block:: console
+
+ # ceph -s
+
+ [...]
+ progress:
+ Upgrade to docker.io/ceph/ceph:v15.2.1 (00h 20m 12s)
+ [=======.....................] (time remaining: 01h 43m 31s)
+
+Watching the cephadm log during an upgrade
+------------------------------------------
+
+Watch the cephadm log by running the following command:
+
+.. prompt:: bash #
+
+ ceph -W cephadm
+
+
+Canceling an upgrade
+====================
+
+You can stop the upgrade process at any time by running the following command:
+
+.. prompt:: bash #
+
+ ceph orch upgrade stop
+
+Post upgrade actions
+====================
+
+In case the new version is based on ``cephadm``, once done with the upgrade the user
+has to update the ``cephadm`` package (or ceph-common package in case the user
+doesn't use ``cephadm shell``) to a version compatible with the new version.
+
+Potential problems
+==================
+
+There are a few health alerts that can arise during the upgrade process.
+
+UPGRADE_NO_STANDBY_MGR
+----------------------
+
+This alert (``UPGRADE_NO_STANDBY_MGR``) means that Ceph does not detect an
+active standby manager daemon. In order to proceed with the upgrade, Ceph
+requires an active standby manager daemon (which you can think of in this
+context as "a second manager").
+
+You can ensure that Cephadm is configured to run 2 (or more) managers by
+running the following command:
+
+.. prompt:: bash #
+
+ ceph orch apply mgr 2 # or more
+
+You can check the status of existing mgr daemons by running the following
+command:
+
+.. prompt:: bash #
+
+ ceph orch ps --daemon-type mgr
+
+If an existing mgr daemon has stopped, you can try to restart it by running the
+following command:
+
+.. prompt:: bash #
+
+ ceph orch daemon restart <name>
+
+UPGRADE_FAILED_PULL
+-------------------
+
+This alert (``UPGRADE_FAILED_PULL``) means that Ceph was unable to pull the
+container image for the target version. This can happen if you specify a
+version or container image that does not exist (e.g. "1.2.3"), or if the
+container registry can not be reached by one or more hosts in the cluster.
+
+To cancel the existing upgrade and to specify a different target version, run
+the following commands:
+
+.. prompt:: bash #
+
+ ceph orch upgrade stop
+ ceph orch upgrade start --ceph-version <version>
+
+
+Using customized container images
+=================================
+
+For most users, upgrading requires nothing more complicated than specifying the
+Ceph version number to upgrade to. In such cases, cephadm locates the specific
+Ceph container image to use by combining the ``container_image_base``
+configuration option (default: ``docker.io/ceph/ceph``) with a tag of
+``vX.Y.Z``.
+
+But it is possible to upgrade to an arbitrary container image, if that's what
+you need. For example, the following command upgrades to a development build:
+
+.. prompt:: bash #
+
+ ceph orch upgrade start --image quay.io/ceph-ci/ceph:recent-git-branch-name
+
+For more information about available container images, see :ref:`containers`.
+
+Staggered Upgrade
+=================
+
+Some users may prefer to upgrade components in phases rather than all at once.
+The upgrade command, starting in 16.2.11 and 17.2.1 allows parameters
+to limit which daemons are upgraded by a single upgrade command. The options in
+include ``daemon_types``, ``services``, ``hosts`` and ``limit``. ``daemon_types``
+takes a comma-separated list of daemon types and will only upgrade daemons of those
+types. ``services`` is mutually exclusive with ``daemon_types``, only takes services
+of one type at a time (e.g. can't provide an OSD and RGW service at the same time), and
+will only upgrade daemons belonging to those services. ``hosts`` can be combined
+with ``daemon_types`` or ``services`` or provided on its own. The ``hosts`` parameter
+follows the same format as the command line options for :ref:`orchestrator-cli-placement-spec`.
+``limit`` takes an integer > 0 and provides a numerical limit on the number of
+daemons cephadm will upgrade. ``limit`` can be combined with any of the other
+parameters. For example, if you specify to upgrade daemons of type osd on host
+Host1 with ``limit`` set to 3, cephadm will upgrade (up to) 3 osd daemons on
+Host1.
+
+Example: specifying daemon types and hosts:
+
+.. prompt:: bash #
+
+ ceph orch upgrade start --image <image-name> --daemon-types mgr,mon --hosts host1,host2
+
+Example: specifying services and using limit:
+
+.. prompt:: bash #
+
+ ceph orch upgrade start --image <image-name> --services rgw.example1,rgw.example2 --limit 2
+
+.. note::
+
+ Cephadm strictly enforces an order to the upgrade of daemons that is still present
+ in staggered upgrade scenarios. The current upgrade ordering is
+ ``mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -> iscsi -> nfs``.
+ If you specify parameters that would upgrade daemons out of order, the upgrade
+ command will block and note which daemons will be missed if you proceed.
+
+.. note::
+
+ Upgrade commands with limiting parameters will validate the options before beginning the
+ upgrade, which may require pulling the new container image. Do not be surprised
+ if the upgrade start command takes a while to return when limiting parameters are provided.
+
+.. note::
+
+ In staggered upgrade scenarios (when a limiting parameter is provided) monitoring
+ stack daemons including Prometheus and node-exporter are refreshed after the Manager
+ daemons have been upgraded. Do not be surprised if Manager upgrades thus take longer
+ than expected. Note that the versions of monitoring stack daemons may not change between
+ Ceph releases, in which case they are only redeployed.
+
+Upgrading to a version that supports staggered upgrade from one that doesn't
+----------------------------------------------------------------------------
+
+While upgrading from a version that already supports staggered upgrades the process
+simply requires providing the necessary arguments. However, if you wish to upgrade
+to a version that supports staggered upgrade from one that does not, there is a
+workaround. It requires first manually upgrading the Manager daemons and then passing
+the limiting parameters as usual.
+
+.. warning::
+ Make sure you have multiple running mgr daemons before attempting this procedure.
+
+To start with, determine which Manager is your active one and which are standby. This
+can be done in a variety of ways such as looking at the ``ceph -s`` output. Then,
+manually upgrade each standby mgr daemon with:
+
+.. prompt:: bash #
+
+ ceph orch daemon redeploy mgr.example1.abcdef --image <new-image-name>
+
+.. note::
+
+ If you are on a very early version of cephadm (early Octopus) the ``orch daemon redeploy``
+ command may not have the ``--image`` flag. In that case, you must manually set the
+ Manager container image ``ceph config set mgr container_image <new-image-name>`` and then
+ redeploy the Manager ``ceph orch daemon redeploy mgr.example1.abcdef``
+
+At this point, a Manager fail over should allow us to have the active Manager be one
+running the new version.
+
+.. prompt:: bash #
+
+ ceph mgr fail
+
+Verify the active Manager is now one running the new version. To complete the Manager
+upgrading:
+
+.. prompt:: bash #
+
+ ceph orch upgrade start --image <new-image-name> --daemon-types mgr
+
+You should now have all your Manager daemons on the new version and be able to
+specify the limiting parameters for the rest of the upgrade.