From 483eb2f56657e8e7f419ab1a4fab8dce9ade8609 Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sat, 27 Apr 2024 20:24:20 +0200 Subject: Adding upstream version 14.2.21. Signed-off-by: Daniel Baumann --- doc/rados/troubleshooting/troubleshooting-mon.rst | 582 ++++++++++++++++++++++ 1 file changed, 582 insertions(+) create mode 100644 doc/rados/troubleshooting/troubleshooting-mon.rst (limited to 'doc/rados/troubleshooting/troubleshooting-mon.rst') diff --git a/doc/rados/troubleshooting/troubleshooting-mon.rst b/doc/rados/troubleshooting/troubleshooting-mon.rst new file mode 100644 index 00000000..38cc4966 --- /dev/null +++ b/doc/rados/troubleshooting/troubleshooting-mon.rst @@ -0,0 +1,582 @@ +================================= + Troubleshooting Monitors +================================= + +.. index:: monitor, high availability + +When a cluster encounters monitor-related troubles there's a tendency to +panic, and some times with good reason. You should keep in mind that losing +a monitor, or a bunch of them, don't necessarily mean that your cluster is +down, as long as a majority is up, running and with a formed quorum. +Regardless of how bad the situation is, the first thing you should do is to +calm down, take a breath and try answering our initial troubleshooting script. + + +Initial Troubleshooting +======================== + + +**Are the monitors running?** + + First of all, we need to make sure the monitors are running. You would be + amazed by how often people forget to run the monitors, or restart them after + an upgrade. There's no shame in that, but let's try not losing a couple of + hours chasing an issue that is not there. + +**Are you able to connect to the monitor's servers?** + + Doesn't happen often, but sometimes people do have ``iptables`` rules that + block accesses to monitor servers or monitor ports. Usually leftovers from + monitor stress-testing that were forgotten at some point. Try ssh'ing into + the server and, if that succeeds, try connecting to the monitor's port + using you tool of choice (telnet, nc,...). + +**Does ceph -s run and obtain a reply from the cluster?** + + If the answer is yes then your cluster is up and running. One thing you + can take for granted is that the monitors will only answer to a ``status`` + request if there is a formed quorum. + + If ``ceph -s`` blocked however, without obtaining a reply from the cluster + or showing a lot of ``fault`` messages, then it is likely that your monitors + are either down completely or just a portion is up -- a portion that is not + enough to form a quorum (keep in mind that a quorum if formed by a majority + of monitors). + +**What if ceph -s doesn't finish?** + + If you haven't gone through all the steps so far, please go back and do. + + For those running on Emperor 0.72-rc1 and forward, you will be able to + contact each monitor individually asking them for their status, regardless + of a quorum being formed. This can be achieved using ``ceph ping mon.ID``, + ID being the monitor's identifier. You should perform this for each monitor + in the cluster. In section `Understanding mon_status`_ we will explain how + to interpret the output of this command. + + For the rest of you who don't tread on the bleeding edge, you will need to + ssh into the server and use the monitor's admin socket. Please jump to + `Using the monitor's admin socket`_. + +For other specific issues, keep on reading. + + +Using the monitor's admin socket +================================= + +The admin socket allows you to interact with a given daemon directly using a +Unix socket file. This file can be found in your monitor's ``run`` directory. +By default, the admin socket will be kept in ``/var/run/ceph/ceph-mon.ID.asok`` +but this can vary if you defined it otherwise. If you don't find it there, +please check your ``ceph.conf`` for an alternative path or run:: + + ceph-conf --name mon.ID --show-config-value admin_socket + +Please bear in mind that the admin socket will only be available while the +monitor is running. When the monitor is properly shutdown, the admin socket +will be removed. If however the monitor is not running and the admin socket +still persists, it is likely that the monitor was improperly shutdown. +Regardless, if the monitor is not running, you will not be able to use the +admin socket, with ``ceph`` likely returning ``Error 111: Connection Refused``. + +Accessing the admin socket is as simple as telling the ``ceph`` tool to use +the ``asok`` file. In pre-Dumpling Ceph, this can be achieved by:: + + ceph --admin-daemon /var/run/ceph/ceph-mon..asok + +while in Dumpling and beyond you can use the alternate (and recommended) +format:: + + ceph daemon mon. + +Using ``help`` as the command to the ``ceph`` tool will show you the +supported commands available through the admin socket. Please take a look +at ``config get``, ``config show``, ``mon_status`` and ``quorum_status``, +as those can be enlightening when troubleshooting a monitor. + + +Understanding mon_status +========================= + +``mon_status`` can be obtained through the ``ceph`` tool when you have +a formed quorum, or via the admin socket if you don't. This command will +output a multitude of information about the monitor, including the same +output you would get with ``quorum_status``. + +Take the following example of ``mon_status``:: + + + { "name": "c", + "rank": 2, + "state": "peon", + "election_epoch": 38, + "quorum": [ + 1, + 2], + "outside_quorum": [], + "extra_probe_peers": [], + "sync_provider": [], + "monmap": { "epoch": 3, + "fsid": "5c4e9d53-e2e1-478a-8061-f543f8be4cf8", + "modified": "2013-10-30 04:12:01.945629", + "created": "2013-10-29 14:14:41.914786", + "mons": [ + { "rank": 0, + "name": "a", + "addr": "127.0.0.1:6789\/0"}, + { "rank": 1, + "name": "b", + "addr": "127.0.0.1:6790\/0"}, + { "rank": 2, + "name": "c", + "addr": "127.0.0.1:6795\/0"}]}} + +A couple of things are obvious: we have three monitors in the monmap (*a*, *b* +and *c*), the quorum is formed by only two monitors, and *c* is in the quorum +as a *peon*. + +Which monitor is out of the quorum? + + The answer would be **a**. + +Why? + + Take a look at the ``quorum`` set. We have two monitors in this set: *1* + and *2*. These are not monitor names. These are monitor ranks, as established + in the current monmap. We are missing the monitor with rank 0, and according + to the monmap that would be ``mon.a``. + +By the way, how are ranks established? + + Ranks are (re)calculated whenever you add or remove monitors and follow a + simple rule: the **greater** the ``IP:PORT`` combination, the **lower** the + rank is. In this case, considering that ``127.0.0.1:6789`` is lower than all + the remaining ``IP:PORT`` combinations, ``mon.a`` has rank 0. + +Most Common Monitor Issues +=========================== + +Have Quorum but at least one Monitor is down +--------------------------------------------- + +When this happens, depending on the version of Ceph you are running, +you should be seeing something similar to:: + + $ ceph health detail + [snip] + mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum) + +How to troubleshoot this? + + First, make sure ``mon.a`` is running. + + Second, make sure you are able to connect to ``mon.a``'s server from the + other monitors' servers. Check the ports as well. Check ``iptables`` on + all your monitor nodes and make sure you are not dropping/rejecting + connections. + + If this initial troubleshooting doesn't solve your problems, then it's + time to go deeper. + + First, check the problematic monitor's ``mon_status`` via the admin + socket as explained in `Using the monitor's admin socket`_ and + `Understanding mon_status`_. + + Considering the monitor is out of the quorum, its state should be one of + ``probing``, ``electing`` or ``synchronizing``. If it happens to be either + ``leader`` or ``peon``, then the monitor believes to be in quorum, while + the remaining cluster is sure it is not; or maybe it got into the quorum + while we were troubleshooting the monitor, so check you ``ceph -s`` again + just to make sure. Proceed if the monitor is not yet in the quorum. + +What if the state is ``probing``? + + This means the monitor is still looking for the other monitors. Every time + you start a monitor, the monitor will stay in this state for some time + while trying to find the rest of the monitors specified in the ``monmap``. + The time a monitor will spend in this state can vary. For instance, when on + a single-monitor cluster, the monitor will pass through the probing state + almost instantaneously, since there are no other monitors around. On a + multi-monitor cluster, the monitors will stay in this state until they + find enough monitors to form a quorum -- this means that if you have 2 out + of 3 monitors down, the one remaining monitor will stay in this state + indefinitely until you bring one of the other monitors up. + + If you have a quorum, however, the monitor should be able to find the + remaining monitors pretty fast, as long as they can be reached. If your + monitor is stuck probing and you have gone through with all the communication + troubleshooting, then there is a fair chance that the monitor is trying + to reach the other monitors on a wrong address. ``mon_status`` outputs the + ``monmap`` known to the monitor: check if the other monitor's locations + match reality. If they don't, jump to + `Recovering a Monitor's Broken monmap`_; if they do, then it may be related + to severe clock skews amongst the monitor nodes and you should refer to + `Clock Skews`_ first, but if that doesn't solve your problem then it is + the time to prepare some logs and reach out to the community (please refer + to `Preparing your logs`_ on how to best prepare your logs). + + +What if state is ``electing``? + + This means the monitor is in the middle of an election. These should be + fast to complete, but at times the monitors can get stuck electing. This + is usually a sign of a clock skew among the monitor nodes; jump to + `Clock Skews`_ for more infos on that. If all your clocks are properly + synchronized, it is best if you prepare some logs and reach out to the + community. This is not a state that is likely to persist and aside from + (*really*) old bugs there is not an obvious reason besides clock skews on + why this would happen. + +What if state is ``synchronizing``? + + This means the monitor is synchronizing with the rest of the cluster in + order to join the quorum. The synchronization process is as faster as + smaller your monitor store is, so if you have a big store it may + take a while. Don't worry, it should be finished soon enough. + + However, if you notice that the monitor jumps from ``synchronizing`` to + ``electing`` and then back to ``synchronizing``, then you do have a + problem: the cluster state is advancing (i.e., generating new maps) way + too fast for the synchronization process to keep up. This used to be a + thing in early Cuttlefish, but since then the synchronization process was + quite refactored and enhanced to avoid just this sort of behavior. If this + happens in later versions let us know. And bring some logs + (see `Preparing your logs`_). + +What if state is ``leader`` or ``peon``? + + This should not happen. There is a chance this might happen however, and + it has a lot to do with clock skews -- see `Clock Skews`_. If you are not + suffering from clock skews, then please prepare your logs (see + `Preparing your logs`_) and reach out to us. + + +Recovering a Monitor's Broken monmap +------------------------------------- + +This is how a ``monmap`` usually looks like, depending on the number of +monitors:: + + + epoch 3 + fsid 5c4e9d53-e2e1-478a-8061-f543f8be4cf8 + last_changed 2013-10-30 04:12:01.945629 + created 2013-10-29 14:14:41.914786 + 0: 127.0.0.1:6789/0 mon.a + 1: 127.0.0.1:6790/0 mon.b + 2: 127.0.0.1:6795/0 mon.c + +This may not be what you have however. For instance, in some versions of +early Cuttlefish there was this one bug that could cause your ``monmap`` +to be nullified. Completely filled with zeros. This means that not even +``monmaptool`` would be able to read it because it would find it hard to +make sense of only-zeros. Some other times, you may end up with a monitor +with a severely outdated monmap, thus being unable to find the remaining +monitors (e.g., say ``mon.c`` is down; you add a new monitor ``mon.d``, +then remove ``mon.a``, then add a new monitor ``mon.e`` and remove +``mon.b``; you will end up with a totally different monmap from the one +``mon.c`` knows). + +In this sort of situations, you have two possible solutions: + +Scrap the monitor and create a new one + + You should only take this route if you are positive that you won't + lose the information kept by that monitor; that you have other monitors + and that they are running just fine so that your new monitor is able + to synchronize from the remaining monitors. Keep in mind that destroying + a monitor, if there are no other copies of its contents, may lead to + loss of data. + +Inject a monmap into the monitor + + Usually the safest path. You should grab the monmap from the remaining + monitors and inject it into the monitor with the corrupted/lost monmap. + + These are the basic steps: + + 1. Is there a formed quorum? If so, grab the monmap from the quorum:: + + $ ceph mon getmap -o /tmp/monmap + + 2. No quorum? Grab the monmap directly from another monitor (this + assumes the monitor you are grabbing the monmap from has id ID-FOO + and has been stopped):: + + $ ceph-mon -i ID-FOO --extract-monmap /tmp/monmap + + 3. Stop the monitor you are going to inject the monmap into. + + 4. Inject the monmap:: + + $ ceph-mon -i ID --inject-monmap /tmp/monmap + + 5. Start the monitor + + Please keep in mind that the ability to inject monmaps is a powerful + feature that can cause havoc with your monitors if misused as it will + overwrite the latest, existing monmap kept by the monitor. + + +Clock Skews +------------ + +Monitors can be severely affected by significant clock skews across the +monitor nodes. This usually translates into weird behavior with no obvious +cause. To avoid such issues, you should run a clock synchronization tool +on your monitor nodes. + + +What's the maximum tolerated clock skew? + + By default the monitors will allow clocks to drift up to ``0.05 seconds``. + + +Can I increase the maximum tolerated clock skew? + + This value is configurable via the ``mon-clock-drift-allowed`` option, and + although you *CAN* it doesn't mean you *SHOULD*. The clock skew mechanism + is in place because clock skewed monitor may not properly behave. We, as + developers and QA aficionados, are comfortable with the current default + value, as it will alert the user before the monitors get out hand. Changing + this value without testing it first may cause unforeseen effects on the + stability of the monitors and overall cluster healthiness, although there is + no risk of dataloss. + + +How do I know there's a clock skew? + + The monitors will warn you in the form of a ``HEALTH_WARN``. ``ceph health + detail`` should show something in the form of:: + + mon.c addr 10.10.0.1:6789/0 clock skew 0.08235s > max 0.05s (latency 0.0045s) + + That means that ``mon.c`` has been flagged as suffering from a clock skew. + + +What should I do if there's a clock skew? + + Synchronize your clocks. Running an NTP client may help. If you are already + using one and you hit this sort of issues, check if you are using some NTP + server remote to your network and consider hosting your own NTP server on + your network. This last option tends to reduce the amount of issues with + monitor clock skews. + + +Client Can't Connect or Mount +------------------------------ + +Check your IP tables. Some OS install utilities add a ``REJECT`` rule to +``iptables``. The rule rejects all clients trying to connect to the host except +for ``ssh``. If your monitor host's IP tables have such a ``REJECT`` rule in +place, clients connecting from a separate node will fail to mount with a timeout +error. You need to address ``iptables`` rules that reject clients trying to +connect to Ceph daemons. For example, you would need to address rules that look +like this appropriately:: + + REJECT all -- anywhere anywhere reject-with icmp-host-prohibited + +You may also need to add rules to IP tables on your Ceph hosts to ensure +that clients can access the ports associated with your Ceph monitors (i.e., port +6789 by default) and Ceph OSDs (i.e., 6800 through 7300 by default). For +example:: + + iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:7300 -j ACCEPT + +Monitor Store Failures +====================== + +Symptoms of store corruption +---------------------------- + +Ceph monitor stores the :term:`cluster map` in a key/value store such as LevelDB. If +a monitor fails due to the key/value store corruption, following error messages +might be found in the monitor log:: + + Corruption: error in middle of record + +or:: + + Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/mon.foo/store.db/1234567.ldb + +Recovery using healthy monitor(s) +--------------------------------- + +If there are any survivors, we can always :ref:`replace ` the corrupted one with a +new one. After booting up, the new joiner will sync up with a healthy +peer, and once it is fully sync'ed, it will be able to serve the clients. + +Recovery using OSDs +------------------- + +But what if all monitors fail at the same time? Since users are encouraged to +deploy at least three (and preferably five) monitors in a Ceph cluster, the chance of simultaneous +failure is rare. But unplanned power-downs in a data center with improperly +configured disk/fs settings could fail the underlying filesystem, and hence +kill all the monitors. In this case, we can recover the monitor store with the +information stored in OSDs.:: + + ms=/root/mon-store + mkdir $ms + + # collect the cluster map from OSDs + for host in $hosts; do + rsync -avz $ms/. user@host:$ms.remote + rm -rf $ms + ssh user@host <