1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
|
.. _mds-standby:
Terminology
-----------
A Ceph cluster may have zero or more CephFS *file systems*. CephFS
file systems have a human readable name (set in ``fs new``)
and an integer ID. The ID is called the file system cluster ID,
or *FSCID*.
Each CephFS file system has a number of *ranks*, one by default,
which start at zero. A rank may be thought of as a metadata shard.
Controlling the number of ranks in a file system is described
in :doc:`/cephfs/multimds`
Each CephFS ceph-mds process (a *daemon*) initially starts up
without a rank. It may be assigned one by the monitor cluster.
A daemon may only hold one rank at a time. Daemons only give up
a rank when the ceph-mds process stops.
If a rank is not associated with a daemon, the rank is
considered *failed*. Once a rank is assigned to a daemon,
the rank is considered *up*.
A daemon has a *name* that is set statically by the administrator
when the daemon is first configured. Typical configurations
use the hostname where the daemon runs as the daemon name.
A ceph-mds daemons can be assigned to a particular file system by
setting the `mds_join_fs` configuration option to the file system
name.
Each time a daemon starts up, it is also assigned a *GID*, which
is unique to this particular process lifetime of the daemon. The
GID is an integer.
Referring to MDS daemons
------------------------
Most of the administrative commands that refer to an MDS daemon
accept a flexible argument format that may contain a rank, a GID
or a name.
Where a rank is used, this may optionally be qualified with
a leading file system name or ID. If a daemon is a standby (i.e.
it is not currently assigned a rank), then it may only be
referred to by GID or name.
For example, if we had an MDS daemon which was called 'myhost',
had GID 5446, and was assigned rank 0 in the file system 'myfs'
which had FSCID 3, then any of the following would be suitable
forms of the 'fail' command:
::
ceph mds fail 5446 # GID
ceph mds fail myhost # Daemon name
ceph mds fail 0 # Unqualified rank
ceph mds fail 3:0 # FSCID and rank
ceph mds fail myfs:0 # File System name and rank
Managing failover
-----------------
If an MDS daemon stops communicating with the monitor, the monitor will wait
``mds_beacon_grace`` seconds (default 15 seconds) before marking the daemon as
*laggy*. If a standby is available, the monitor will immediately replace the
laggy daemon.
Each file system may specify a number of standby daemons to be considered
healthy. This number includes daemons in standby-replay waiting for a rank to
fail (remember that a standby-replay daemon will not be assigned to take over a
failure for another rank or a failure in a another CephFS file system). The
pool of standby daemons not in replay count towards any file system count.
Each file system may set the number of standby daemons wanted using:
::
ceph fs set <fs name> standby_count_wanted <count>
Setting ``count`` to 0 will disable the health check.
.. _mds-standby-replay:
Configuring standby-replay
--------------------------
Each CephFS file system may be configured to add standby-replay daemons. These
standby daemons follow the active MDS's metadata journal to reduce failover
time in the event the active MDS becomes unavailable. Each active MDS may have
only one standby-replay daemon following it.
Configuring standby-replay on a file system is done using:
::
ceph fs set <fs name> allow_standby_replay <bool>
Once set, the monitors will assign available standby daemons to follow the
active MDSs in that file system.
Once an MDS has entered the standby-replay state, it will only be used as a
standby for the rank that it is following. If another rank fails, this
standby-replay daemon will not be used as a replacement, even if no other
standbys are available. For this reason, it is advised that if standby-replay
is used then every active MDS should have a standby-replay daemon.
.. _mds-join-fs:
Configuring MDS file system affinity
------------------------------------
You may want to have an MDS used for a particular file system. Or, perhaps you
have larger MDSs on better hardware that should be preferred over a last-resort
standby on lesser or over-provisioned hardware. To express this preference,
CephFS provides a configuration option for MDS called ``mds_join_fs`` which
enforces this `affinity`.
As part of any failover, the Ceph monitors will prefer standby daemons with
``mds_join_fs`` equal to the file system name with the failed rank. If no
standby exists with ``mds_join_fs`` equal to the file system name, it will
choose a `vanilla` standby (no setting for ``mds_join_fs``) for the replacement
or any other available standby as a last resort. Note, this does not change the
behavior that ``standby-replay`` daemons are always selected before looking at
other standbys.
Even further, the monitors will regularly examine the CephFS file systems when
stable to check if a standby with stronger affinity is available to replace an
MDS with lower affinity. This process is also done for standby-replay daemons:
if a regular standby has stronger affinity than the standby-replay MDS, it will
replace the standby-replay MDS.
For example, given this stable and healthy file system:
::
$ ceph fs dump
dumped fsmap epoch 399
...
Filesystem 'cephfs' (27)
...
e399
max_mds 1
in 0
up {0=20384}
failed
damaged
stopped
...
[mds.a{0:20384} state up:active seq 239 addr [v2:127.0.0.1:6854/966242805,v1:127.0.0.1:6855/966242805]]
Standby daemons:
[mds.b{-1:10420} state up:standby seq 2 addr [v2:127.0.0.1:6856/2745199145,v1:127.0.0.1:6857/2745199145]]
You may set ``mds_join_fs`` on the standby to enforce your preference: ::
$ ceph config set mds.b mds_join_fs cephfs
after automatic failover: ::
$ ceph fs dump
dumped fsmap epoch 405
e405
...
Filesystem 'cephfs' (27)
...
max_mds 1
in 0
up {0=10420}
failed
damaged
stopped
...
[mds.b{0:10420} state up:active seq 274 join_fscid=27 addr [v2:127.0.0.1:6856/2745199145,v1:127.0.0.1:6857/2745199145]]
Standby daemons:
[mds.a{-1:10720} state up:standby seq 2 addr [v2:127.0.0.1:6854/1340357658,v1:127.0.0.1:6855/1340357658]]
Note in the above example that ``mds.b`` now has ``join_fscid=27``. In this
output, the file system name from ``mds_join_fs`` is changed to the file system
identifier (27). If the file system is recreated with the same name, the
standby will follow the new file system as expected.
Finally, if the file system is degraded or undersized, no failover will occur
to enforce ``mds_join_fs``.
|
