summaryrefslogtreecommitdiffstats
path: root/doc/cephadm/troubleshooting.rst
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--doc/cephadm/troubleshooting.rst514
1 files changed, 514 insertions, 0 deletions
diff --git a/doc/cephadm/troubleshooting.rst b/doc/cephadm/troubleshooting.rst
new file mode 100644
index 000000000..d891ebaf2
--- /dev/null
+++ b/doc/cephadm/troubleshooting.rst
@@ -0,0 +1,514 @@
+Troubleshooting
+===============
+
+This section explains how to investigate why a cephadm command failed or why a
+certain service no longer runs properly.
+
+Cephadm deploys daemons within containers. Troubleshooting containerized
+daemons requires a different process than does troubleshooting traditional
+daemons that were installed by means of packages.
+
+Here are some tools and commands to help you troubleshoot your Ceph
+environment.
+
+.. _cephadm-pause:
+
+Pausing or Disabling cephadm
+----------------------------
+
+If something goes wrong and cephadm is behaving badly, pause most of the Ceph
+cluster's background activity by running the following command:
+
+.. prompt:: bash #
+
+ ceph orch pause
+
+This stops all changes in the Ceph cluster, but cephadm will still periodically
+check hosts to refresh its inventory of daemons and devices. Disable cephadm
+completely by running the following commands:
+
+.. prompt:: bash #
+
+ ceph orch set backend ''
+ ceph mgr module disable cephadm
+
+These commands disable all of the ``ceph orch ...`` CLI commands. All
+previously deployed daemon containers continue to run and will start just as
+they were before you ran these commands.
+
+See :ref:`cephadm-spec-unmanaged` for more on disabling individual services.
+
+
+Per-service and Per-daemon Events
+---------------------------------
+
+To make it easier to debug failed daemons, cephadm stores events per service
+and per daemon. These events often contain information relevant to
+the troubleshooting of your Ceph cluster.
+
+Listing Service Events
+~~~~~~~~~~~~~~~~~~~~~~
+
+To see the events associated with a certain service, run a command of the
+following form:
+
+.. prompt:: bash #
+
+ ceph orch ls --service_name=<service-name> --format yaml
+
+This will return something in the following form:
+
+.. code-block:: yaml
+
+ service_type: alertmanager
+ service_name: alertmanager
+ placement:
+ hosts:
+ - unknown_host
+ status:
+ ...
+ running: 1
+ size: 1
+ events:
+ - 2021-02-01T08:58:02.741162 service:alertmanager [INFO] "service was created"
+ - '2021-02-01T12:09:25.264584 service:alertmanager [ERROR] "Failed to apply: Cannot
+ place <AlertManagerSpec for service_name=alertmanager> on unknown_host: Unknown hosts"'
+
+Listing Daemon Events
+~~~~~~~~~~~~~~~~~~~~~
+
+To see the events associated with a certain daemon, run a command of the
+following form:
+
+.. prompt:: bash #
+
+ ceph orch ps --service-name <service-name> --daemon-id <daemon-id> --format yaml
+
+This will return something in the following form:
+
+.. code-block:: yaml
+
+ daemon_type: mds
+ daemon_id: cephfs.hostname.ppdhsz
+ hostname: hostname
+ status_desc: running
+ ...
+ events:
+ - 2021-02-01T08:59:43.845866 daemon:mds.cephfs.hostname.ppdhsz [INFO] "Reconfigured
+ mds.cephfs.hostname.ppdhsz on host 'hostname'"
+
+
+Checking Cephadm Logs
+---------------------
+
+To learn how to monitor cephadm logs as they are generated, read
+:ref:`watching_cephadm_logs`.
+
+If your Ceph cluster has been configured to log events to files, there will be
+a ``ceph.cephadm.log`` file on all monitor hosts. See :ref:`cephadm-logs` for a
+more complete explanation.
+
+Gathering Log Files
+-------------------
+
+Use ``journalctl`` to gather the log files of all daemons:
+
+.. note:: By default cephadm now stores logs in journald. This means
+ that you will no longer find daemon logs in ``/var/log/ceph/``.
+
+To read the log file of one specific daemon, run a command of the following
+form:
+
+.. prompt:: bash
+
+ cephadm logs --name <name-of-daemon>
+
+.. Note:: This works only when run on the same host that is running the daemon.
+ To get the logs of a daemon that is running on a different host, add the
+ ``--fsid`` option to the command, as in the following example:
+
+ .. prompt:: bash
+
+ cephadm logs --fsid <fsid> --name <name-of-daemon>
+
+ In this example, ``<fsid>`` corresponds to the cluster ID returned by the
+ ``ceph status`` command.
+
+To fetch all log files of all daemons on a given host, run the following
+for-loop::
+
+ for name in $(cephadm ls | jq -r '.[].name') ; do
+ cephadm logs --fsid <fsid> --name "$name" > $name;
+ done
+
+Collecting Systemd Status
+-------------------------
+
+To print the state of a systemd unit, run a command of the following form:
+
+.. prompt:: bash
+
+ systemctl status "ceph-$(cephadm shell ceph fsid)@<service name>.service";
+
+
+To fetch the state of all daemons of a given host, run the following shell
+script::
+
+ fsid="$(cephadm shell ceph fsid)"
+ for name in $(cephadm ls | jq -r '.[].name') ; do
+ systemctl status "ceph-$fsid@$name.service" > $name;
+ done
+
+
+List all Downloaded Container Images
+------------------------------------
+
+To list all container images that are downloaded on a host, run the following
+commands:
+
+.. prompt:: bash #
+
+ podman ps -a --format json | jq '.[].Image' "docker.io/library/centos:8" "registry.opensuse.org/opensuse/leap:15.2"
+
+.. note:: ``Image`` might also be called ``ImageID``.
+
+
+Manually Running Containers
+---------------------------
+
+Cephadm uses small wrappers when running containers. Refer to
+``/var/lib/ceph/<cluster-fsid>/<service-name>/unit.run`` for the container
+execution command.
+
+.. _cephadm-ssh-errors:
+
+SSH Errors
+----------
+
+Error message::
+
+ execnet.gateway_bootstrap.HostNotFound: -F /tmp/cephadm-conf-73z09u6g -i /tmp/cephadm-identity-ky7ahp_5 root@10.10.1.2
+ ...
+ raise OrchestratorError(msg) from e
+ orchestrator._interface.OrchestratorError: Failed to connect to 10.10.1.2 (10.10.1.2).
+ Please make sure that the host is reachable and accepts connections using the cephadm SSH key
+ ...
+
+If you receive the above error message, try the following things to
+troubleshoot the SSH connection between ``cephadm`` and the monitor:
+
+1. Ensure that ``cephadm`` has an SSH identity key::
+
+ [root@mon1~]# cephadm shell -- ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key
+ INFO:cephadm:Inferring fsid f8edc08a-7f17-11ea-8707-000c2915dd98
+ INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 obtained 'mgr/cephadm/ssh_identity_key'
+ [root@mon1 ~] # chmod 0600 ~/cephadm_private_key
+
+ If this fails, cephadm doesn't have a key. Fix this by running the following command::
+
+ [root@mon1 ~]# cephadm shell -- ceph cephadm generate-ssh-key
+
+ or::
+
+ [root@mon1 ~]# cat ~/cephadm_private_key | cephadm shell -- ceph cephadm set-ssh-key -i -
+
+2. Ensure that the SSH config is correct::
+
+ [root@mon1 ~]# cephadm shell -- ceph cephadm get-ssh-config > config
+
+3. Verify that it is possible to connect to the host::
+
+ [root@mon1 ~]# ssh -F config -i ~/cephadm_private_key root@mon1
+
+Verifying that the Public Key is Listed in the authorized_keys file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To verify that the public key is in the ``authorized_keys`` file, run the
+following commands::
+
+ [root@mon1 ~]# cephadm shell -- ceph cephadm get-pub-key > ~/ceph.pub
+ [root@mon1 ~]# grep "`cat ~/ceph.pub`" /root/.ssh/authorized_keys
+
+Failed to Infer CIDR network error
+----------------------------------
+
+If you see this error::
+
+ ERROR: Failed to infer CIDR network for mon ip ***; pass --skip-mon-network to configure it later
+
+Or this error::
+
+ Must set public_network config option or specify a CIDR network, ceph addrvec, or plain IP
+
+This means that you must run a command of this form:
+
+.. prompt:: bash
+
+ ceph config set mon public_network <mon_network>
+
+For more detail on operations of this kind, see
+:ref:`deploy_additional_monitors`.
+
+Accessing the Admin Socket
+--------------------------
+
+Each Ceph daemon provides an admin socket that bypasses the MONs (See
+:ref:`rados-monitoring-using-admin-socket`).
+
+#. To access the admin socket, enter the daemon container on the host::
+
+ [root@mon1 ~]# cephadm enter --name <daemon-name>
+
+#. Run a command of the following form to see the admin socket's configuration::
+
+ [ceph: root@mon1 /]# ceph --admin-daemon /var/run/ceph/ceph-<daemon-name>.asok config show
+
+Running Various Ceph Tools
+--------------------------------
+
+To run Ceph tools such as ``ceph-objectstore-tool`` or
+``ceph-monstore-tool``, invoke the cephadm CLI with
+``cephadm shell --name <daemon-name>``. For example::
+
+ root@myhostname # cephadm unit --name mon.myhostname stop
+ root@myhostname # cephadm shell --name mon.myhostname
+ [ceph: root@myhostname /]# ceph-monstore-tool /var/lib/ceph/mon/ceph-myhostname get monmap > monmap
+ [ceph: root@myhostname /]# monmaptool --print monmap
+ monmaptool: monmap file monmap
+ epoch 1
+ fsid 28596f44-3b56-11ec-9034-482ae35a5fbb
+ last_changed 2021-11-01T20:57:19.755111+0000
+ created 2021-11-01T20:57:19.755111+0000
+ min_mon_release 17 (quincy)
+ election_strategy: 1
+ 0: [v2:127.0.0.1:3300/0,v1:127.0.0.1:6789/0] mon.myhostname
+
+The cephadm shell sets up the environment in a way that is suitable for
+extended daemon maintenance and for the interactive running of daemons.
+
+.. _cephadm-restore-quorum:
+
+Restoring the Monitor Quorum
+----------------------------
+
+If the Ceph Monitor daemons (mons) cannot form a quorum, ``cephadm`` will not
+be able to manage the cluster until quorum is restored.
+
+In order to restore the quorum, remove unhealthy monitors
+form the monmap by following these steps:
+
+1. Stop all Monitors. Use ``ssh`` to connect to each Monitor's host, and then
+ while connected to the Monitor's host use ``cephadm`` to stop the Monitor
+ daemon:
+
+ .. prompt:: bash
+
+ ssh {mon-host}
+ cephadm unit --name {mon.hostname} stop
+
+
+2. Identify a surviving Monitor and log in to its host:
+
+ .. prompt:: bash
+
+ ssh {mon-host}
+ cephadm enter --name {mon.hostname}
+
+3. Follow the steps in :ref:`rados-mon-remove-from-unhealthy`.
+
+.. _cephadm-manually-deploy-mgr:
+
+Manually Deploying a Manager Daemon
+-----------------------------------
+At least one Manager (``mgr``) daemon is required by cephadm in order to manage
+the cluster. If the last remaining Manager has been removed from the Ceph
+cluster, follow these steps in order to deploy a fresh Manager on an arbitrary
+host in your cluster. In this example, the freshly-deployed Manager daemon is
+called ``mgr.hostname.smfvfd``.
+
+#. Disable the cephadm scheduler, in order to prevent ``cephadm`` from removing
+ the new Manager. See :ref:`cephadm-enable-cli`:
+
+ .. prompt:: bash #
+
+ ceph config-key set mgr/cephadm/pause true
+
+#. Retrieve or create the "auth entry" for the new Manager:
+
+ .. prompt:: bash #
+
+ ceph auth get-or-create mgr.hostname.smfvfd mon "profile mgr" osd "allow *" mds "allow *"
+
+#. Retrieve the Monitor's configuration:
+
+ .. prompt:: bash #
+
+ ceph config generate-minimal-conf
+
+#. Retrieve the container image:
+
+ .. prompt:: bash #
+
+ ceph config get "mgr.hostname.smfvfd" container_image
+
+#. Create a file called ``config-json.json``, which contains the information
+ necessary to deploy the daemon:
+
+ .. code-block:: json
+
+ {
+ "config": "# minimal ceph.conf for 8255263a-a97e-4934-822c-00bfe029b28f\n[global]\n\tfsid = 8255263a-a97e-4934-822c-00bfe029b28f\n\tmon_host = [v2:192.168.0.1:40483/0,v1:192.168.0.1:40484/0]\n",
+ "keyring": "[mgr.hostname.smfvfd]\n\tkey = V2VyIGRhcyBsaWVzdCBpc3QgZG9vZi4=\n"
+ }
+
+#. Deploy the Manager daemon:
+
+ .. prompt:: bash #
+
+ cephadm --image <container-image> deploy --fsid <fsid> --name mgr.hostname.smfvfd --config-json config-json.json
+
+Capturing Core Dumps
+---------------------
+
+A Ceph cluster that uses ``cephadm`` can be configured to capture core dumps.
+The initial capture and processing of the coredump is performed by
+`systemd-coredump
+<https://www.man7.org/linux/man-pages/man8/systemd-coredump.8.html>`_.
+
+
+To enable coredump handling, run the following command
+
+.. prompt:: bash #
+
+ ulimit -c unlimited
+
+
+.. note::
+
+ Core dumps are not namespaced by the kernel. This means that core dumps are
+ written to ``/var/lib/systemd/coredump`` on the container host. The ``ulimit
+ -c unlimited`` setting will persist only until the system is rebooted.
+
+Wait for the crash to happen again. To simulate the crash of a daemon, run for
+example ``killall -3 ceph-mon``.
+
+
+Running the Debugger with cephadm
+----------------------------------
+
+Running a single debugging session
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Initiate a debugging session by using the ``cephadm shell`` command.
+From within the shell container we need to install the debugger and debuginfo
+packages. To debug a core file captured by systemd, run the following:
+
+
+#. Start the shell session:
+
+ .. prompt:: bash #
+
+ cephadm shell --mount /var/lib/system/coredump
+
+#. From within the shell session, run the following commands:
+
+ .. prompt:: bash #
+
+ dnf install ceph-debuginfo gdb zstd
+
+ .. prompt:: bash #
+
+ unzstd /var/lib/systemd/coredump/core.ceph-*.zst
+
+ .. prompt:: bash #
+
+ gdb /usr/bin/ceph-mon /mnt/coredump/core.ceph-*.zst
+
+#. Run debugger commands at gdb's prompt:
+
+ .. prompt:: bash (gdb)
+
+ bt
+
+ ::
+
+ #0 0x00007fa9117383fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+ #1 0x00007fa910d7f8f0 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
+ #2 0x00007fa913d3f48f in AsyncMessenger::wait() () from /usr/lib64/ceph/libceph-common.so.2
+ #3 0x0000563085ca3d7e in main ()
+
+
+Running repeated debugging sessions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When using ``cephadm shell``, as in the example above, any changes made to the
+container that is spawned by the shell command are ephemeral. After the shell
+session exits, the files that were downloaded and installed cease to be
+available. You can simply re-run the same commands every time ``cephadm
+shell`` is invoked, but in order to save time and resources one can create a
+new container image and use it for repeated debugging sessions.
+
+In the following example, we create a simple file that will construct the
+container image. The command below uses podman but it is expected to work
+correctly even if ``podman`` is replaced with ``docker``::
+
+ cat >Containerfile <<EOF
+ ARG BASE_IMG=quay.io/ceph/ceph:v18
+ FROM \${BASE_IMG}
+ # install ceph debuginfo packages, gdb and other potentially useful packages
+ RUN dnf install --enablerepo='*debug*' -y ceph-debuginfo gdb zstd strace python3-debuginfo
+ EOF
+ podman build -t ceph:debugging -f Containerfile .
+ # pass --build-arg=BASE_IMG=<your image> to customize the base image
+
+The above file creates a new local image named ``ceph:debugging``. This image
+can be used on the same machine that built it. The image can also be pushed to
+a container repository or saved and copied to a node runing other Ceph
+containers. Consult the ``podman`` or ``docker`` documentation for more
+information about the container workflow.
+
+After the image has been built, it can be used to initiate repeat debugging
+sessions. By using an image in this way, you avoid the trouble of having to
+re-install the debug tools and debuginfo packages every time you need to run a
+debug session. To debug a core file using this image, in the same way as
+previously described, run:
+
+.. prompt:: bash #
+
+ cephadm --image ceph:debugging shell --mount /var/lib/system/coredump
+
+
+Debugging live processes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gdb debugger can attach to running processes to debug them. This can be
+achieved with a containerized process by using the debug image and attaching it
+to the same PID namespace in which the process to be debugged resides.
+
+This requires running a container command with some custom arguments. We can
+generate a script that can debug a process in a running container.
+
+.. prompt:: bash #
+
+ cephadm --image ceph:debugging shell --dry-run > /tmp/debug.sh
+
+This creates a script that includes the container command that ``cephadm``
+would use to create a shell. Modify the script by removing the ``--init``
+argument and replace it with the argument that joins to the namespace used for
+a running running container. For example, assume we want to debug the Manager
+and have determnined that the Manager is running in a container named
+``ceph-bc615290-685b-11ee-84a6-525400220000-mgr-ceph0-sluwsk``. In this case,
+the argument
+``--pid=container:ceph-bc615290-685b-11ee-84a6-525400220000-mgr-ceph0-sluwsk``
+should be used.
+
+We can run our debugging container with ``sh /tmp/debug.sh``. Within the shell,
+we can run commands such as ``ps`` to get the PID of the Manager process. In
+the following example this is ``2``. While running gdb, we can attach to the
+running process:
+
+.. prompt:: bash (gdb)
+
+ attach 2
+ info threads
+ bt