summaryrefslogtreecommitdiffstats
path: root/doc/rados/configuration/ceph-conf.rst
blob: ad93598de3bacef1f3dc0b9a93dc6c8e47f01cca (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
.. _configuring-ceph:

==================
 Configuring Ceph
==================

When Ceph services start, the initialization process activates a series
of daemons that run in the background. A :term:`Ceph Storage Cluster` runs 
at a minimum three types of daemons:

- :term:`Ceph Monitor` (``ceph-mon``)
- :term:`Ceph Manager` (``ceph-mgr``)
- :term:`Ceph OSD Daemon` (``ceph-osd``)

Ceph Storage Clusters that support the :term:`Ceph File System` also run at
least one :term:`Ceph Metadata Server` (``ceph-mds``). Clusters that
support :term:`Ceph Object Storage` run Ceph RADOS Gateway daemons
(``radosgw``) as well.

Each daemon has a number of configuration options, each of which has a
default value.  You may adjust the behavior of the system by changing these
configuration options.  Be careful to understand the consequences before
overriding default values, as it is possible to significantly degrade the
performance and stability of your cluster.  Also note that default values
sometimes change between releases, so it is best to review the version of
this documentation that aligns with your Ceph release.

Option names
============

All Ceph configuration options have a unique name consisting of words
formed with lower-case characters and connected with underscore
(``_``) characters.

When option names are specified on the command line, either underscore
(``_``) or dash (``-``) characters can be used interchangeable (e.g.,
``--mon-host`` is equivalent to ``--mon_host``).

When option names appear in configuration files, spaces can also be
used in place of underscore or dash.  We suggest, though, that for
clarity and convenience you consistently use underscores, as we do
throughout this documentation.

Config sources
==============

Each Ceph daemon, process, and library will pull its configuration
from several sources, listed below.  Sources later in the list will
override those earlier in the list when both are present.

- the compiled-in default value
- the monitor cluster's centralized configuration database
- a configuration file stored on the local host
- environment variables
- command line arguments
- runtime overrides set by an administrator

One of the first things a Ceph process does on startup is parse the
configuration options provided via the command line, environment, and
local configuration file.  The process will then contact the monitor
cluster to retrieve configuration stored centrally for the entire
cluster.  Once a complete view of the configuration is available, the
daemon or process startup will proceed.

.. _bootstrap-options:

Bootstrap options
-----------------

Because some configuration options affect the process's ability to
contact the monitors, authenticate, and retrieve the cluster-stored
configuration, they may need to be stored locally on the node and set
in a local configuration file.  These options include:

  - ``mon_host``, the list of monitors for the cluster
  - ``mon_host_override``, the list of monitors for the cluster to
    **initially** contact when beginning a new instance of communication with the
    Ceph cluster.  This overrides the known monitor list derived from MonMap
    updates sent to older Ceph instances (like librados cluster handles).  It is
    expected this option is primarily useful for debugging.
  - ``mon_dns_srv_name`` (default: `ceph-mon`), the name of the DNS
    SRV record to check to identify the cluster monitors via DNS
  - ``mon_data``, ``osd_data``, ``mds_data``, ``mgr_data``, and
    similar options that define which local directory the daemon
    stores its data in.
  - ``keyring``, ``keyfile``, and/or ``key``, which can be used to
    specify the authentication credential to use to authenticate with
    the monitor.  Note that in most cases the default keyring location
    is in the data directory specified above.

In the vast majority of cases the default values of these are
appropriate, with the exception of the ``mon_host`` option that
identifies the addresses of the cluster's monitors.  When DNS is used
to identify monitors a local ceph configuration file can be avoided
entirely.

Skipping monitor config
-----------------------

Any process may be passed the option ``--no-mon-config`` to skip the
step that retrieves configuration from the cluster monitors.  This is
useful in cases where configuration is managed entirely via
configuration files or where the monitor cluster is currently down but
some maintenance activity needs to be done.


.. _ceph-conf-file:


Configuration sections
======================

Any given process or daemon has a single value for each configuration
option.  However, values for an option may vary across different
daemon types even daemons of the same type.  Ceph options that are
stored in the monitor configuration database or in local configuration
files are grouped into sections to indicate which daemons or clients
they apply to.

These sections include:

``global``

:Description: Settings under ``global`` affect all daemons and clients
              in a Ceph Storage Cluster.

:Example: ``log_file = /var/log/ceph/$cluster-$type.$id.log``

``mon``

:Description: Settings under ``mon`` affect all ``ceph-mon`` daemons in
              the Ceph Storage Cluster, and override the same setting in 
              ``global``.

:Example: ``mon_cluster_log_to_syslog = true``


``mgr``

:Description: Settings in the ``mgr`` section affect all ``ceph-mgr`` daemons in
              the Ceph Storage Cluster, and override the same setting in 
              ``global``.

:Example: ``mgr_stats_period = 10``

``osd``

:Description: Settings under ``osd`` affect all ``ceph-osd`` daemons in
              the Ceph Storage Cluster, and override the same setting in
              ``global``.

:Example: ``osd_op_queue = wpq``

``mds``

:Description: Settings in the ``mds`` section affect all ``ceph-mds`` daemons in
              the Ceph Storage Cluster, and override the same setting in
              ``global``.

:Example: ``mds_cache_memory_limit = 10G``

``client``

:Description: Settings under ``client`` affect all Ceph Clients
              (e.g., mounted Ceph File Systems, mounted Ceph Block Devices,
              etc.) as well as Rados Gateway (RGW) daemons.

:Example: ``objecter_inflight_ops = 512``


Sections may also specify an individual daemon or client name.  For example,
``mon.foo``, ``osd.123``, and ``client.smith`` are all valid section names.


Any given daemon will draw its settings from the global section, the
daemon or client type section, and the section sharing its name.
Settings in the most-specific section take precedence, so for example
if the same option is specified in both ``global``, ``mon``, and
``mon.foo`` on the same source (i.e., in the same configurationfile),
the ``mon.foo`` value will be used.

If multiple values of the same configuration option are specified in the same
section, the last value wins.

Note that values from the local configuration file always take
precedence over values from the monitor configuration database,
regardless of which section they appear in.


.. _ceph-metavariables:

Metavariables
=============

Metavariables simplify Ceph Storage Cluster configuration
dramatically. When a metavariable is set in a configuration value,
Ceph expands the metavariable into a concrete value at the time the
configuration value is used. Ceph metavariables are similar to variable expansion in the Bash shell.

Ceph supports the following metavariables: 

``$cluster``

:Description: Expands to the Ceph Storage Cluster name. Useful when running 
              multiple Ceph Storage Clusters on the same hardware.

:Example: ``/etc/ceph/$cluster.keyring``
:Default: ``ceph``


``$type``

:Description: Expands to a daemon or process type (e.g., ``mds``, ``osd``, or ``mon``)

:Example: ``/var/lib/ceph/$type``


``$id``

:Description: Expands to the daemon or client identifier. For
              ``osd.0``, this would be ``0``; for ``mds.a``, it would
              be ``a``.

:Example: ``/var/lib/ceph/$type/$cluster-$id``


``$host``

:Description: Expands to the host name where the process is running.


``$name``

:Description: Expands to ``$type.$id``.
:Example: ``/var/run/ceph/$cluster-$name.asok``

``$pid``

:Description: Expands to daemon pid.
:Example: ``/var/run/ceph/$cluster-$name-$pid.asok``



The Configuration File
======================

On startup, Ceph processes search for a configuration file in the
following locations:

#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF``
   environment variable)
#. ``-c path/path``  (*i.e.,* the ``-c`` command line argument)
#. ``/etc/ceph/$cluster.conf``
#. ``~/.ceph/$cluster.conf``
#. ``./$cluster.conf`` (*i.e.,* in the current working directory)
#. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf``

where ``$cluster`` is the cluster's name (default ``ceph``).

The Ceph configuration file uses an *ini* style syntax. You can add comment
text after a pound sign (#) or a semi-colon (;).  For example:

.. code-block:: ini

	# <--A number (#) sign precedes a comment.
	; A comment may be anything.
	# Comments always follow a semi-colon (;) or a pound (#) on each line.
	# The end of the line terminates a comment.
	# We recommend that you provide comments in your configuration file(s).


.. _ceph-conf-settings:

Config file section names
-------------------------

The configuration file is divided into sections. Each section must begin with a
valid configuration section name (see `Configuration sections`_, above)
surrounded by square brackets. For example,

.. code-block:: ini

	[global]
	debug_ms = 0
	
	[osd]
	debug_ms = 1

	[osd.1]
	debug_ms = 10

	[osd.2]
	debug_ms = 10


Config file option values
-------------------------

The value of a configuration option is a string. If it is too long to
fit in a single line, you can put a backslash (``\``) at the end of line
as the line continuation marker, so the value of the option will be
the string after ``=`` in current line combined with the string in the next
line::

  [global]
  foo = long long ago\
  long ago

In the example above, the value of "``foo``" would be "``long long ago long ago``".

Normally, the option value ends with a new line, or a comment, like

.. code-block:: ini

    [global]
    obscure_one = difficult to explain # I will try harder in next release
    simpler_one = nothing to explain

In the example above, the value of "``obscure one``" would be "``difficult to explain``";
and the value of "``simpler one`` would be "``nothing to explain``".

If an option value contains spaces, and we want to make it explicit, we
could quote the value using single or double quotes, like

.. code-block:: ini

    [global]
    line = "to be, or not to be"

Certain characters are not allowed to be present in the option values directly.
They are ``=``, ``#``, ``;`` and ``[``. If we have to, we need to escape them,
like

.. code-block:: ini

    [global]
    secret = "i love \# and \["

Every configuration option is typed with one of the types below:

``int``

:Description: 64-bit signed integer, Some SI prefixes are supported, like "K", "M", "G",
              "T", "P", "E", meaning, respectively, 10\ :sup:`3`, 10\ :sup:`6`,
              10\ :sup:`9`, etc.  And "B" is the only supported unit. So, "1K", "1M", "128B" and "-1" are all valid
              option values. Some times, a negative value implies "unlimited" when it comes to
              an option for threshold or limit.
:Example: ``42``, ``-1``

``uint``

:Description: It is almost identical to ``integer``. But a negative value will be rejected.
:Example: ``256``, ``0``

``str``

:Description: Free style strings encoded in UTF-8, but some characters are not allowed. Please
              reference the above notes for the details.
:Example: ``"hello world"``, ``"i love \#"``, ``yet-another-name``

``boolean``

:Description: one of the two values ``true`` or ``false``. But an integer is also accepted,
              where "0" implies ``false``, and any non-zero values imply ``true``.
:Example: ``true``, ``false``, ``1``, ``0``

``addr``

:Description: a single address optionally prefixed with ``v1``, ``v2`` or ``any`` for the messenger
              protocol. If the prefix is not specified, ``v2`` protocol is used. Please see
              :ref:`address_formats` for more details.
:Example: ``v1:1.2.3.4:567``, ``v2:1.2.3.4:567``, ``1.2.3.4:567``, ``2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567``, ``[::1]:6789``

``addrvec``

:Description: a set of addresses separated by ",". The addresses can be optionally quoted with ``[`` and ``]``.
:Example: ``[v1:1.2.3.4:567,v2:1.2.3.4:568]``, ``v1:1.2.3.4:567,v1:1.2.3.14:567``  ``[2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567], [2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::568]``

``uuid``

:Description: the string format of a uuid defined by `RFC4122 <https://www.ietf.org/rfc/rfc4122.txt>`_.
              And some variants are also supported, for more details, see
              `Boost document <https://www.boost.org/doc/libs/1_74_0/libs/uuid/doc/uuid.html#String%20Generator>`_.
:Example: ``f81d4fae-7dec-11d0-a765-00a0c91e6bf6``

``size``

:Description: denotes a 64-bit unsigned integer. Both SI prefixes and IEC prefixes are
              supported. And "B" is the only supported unit. A negative value will be
              rejected.
:Example: ``1Ki``, ``1K``, ``1KiB`` and ``1B``.

``secs``

:Description: denotes a duration of time. By default the unit is second if not specified.
              Following units of time are supported:

              * second: "s", "sec", "second", "seconds"
              * minute: "m", "min", "minute", "minutes"
              * hour: "hs", "hr", "hour", "hours"
              * day: "d", "day", "days"
              * week: "w", "wk", "week", "weeks"
              * month: "mo", "month", "months"
              * year: "y", "yr", "year", "years"
:Example: ``1 m``, ``1m`` and ``1 week``

.. _ceph-conf-database:

Monitor configuration database
==============================

The monitor cluster manages a database of configuration options that
can be consumed by the entire cluster, enabling streamlined central
configuration management for the entire system.  The vast majority of
configuration options can and should be stored here for ease of
administration and transparency.

A handful of settings may still need to be stored in local
configuration files because they affect the ability to connect to the
monitors, authenticate, and fetch configuration information.  In most
cases this is limited to the ``mon_host`` option, although this can
also be avoided through the use of DNS SRV records.

Sections and masks
------------------

Configuration options stored by the monitor can live in a global
section, daemon type section, or specific daemon section, just like
options in a configuration file can.

In addition, options may also have a *mask* associated with them to
further restrict which daemons or clients the option applies to.
Masks take two forms:

#. ``type:location`` where *type* is a CRUSH property like `rack` or
   `host`, and *location* is a value for that property.  For example,
   ``host:foo`` would limit the option only to daemons or clients
   running on a particular host.
#. ``class:device-class`` where *device-class* is the name of a CRUSH
   device class (e.g., ``hdd`` or ``ssd``).  For example,
   ``class:ssd`` would limit the option only to OSDs backed by SSDs.
   (This mask has no effect for non-OSD daemons or clients.)

When setting a configuration option, the `who` may be a section name,
a mask, or a combination of both separated by a slash (``/``)
character.  For example, ``osd/rack:foo`` would mean all OSD daemons
in the ``foo`` rack.

When viewing configuration options, the section name and mask are
generally separated out into separate fields or columns to ease readability.


Commands
--------

The following CLI commands are used to configure the cluster:

* ``ceph config dump`` will dump the entire configuration database for
  the cluster.

* ``ceph config get <who>`` will dump the configuration for a specific
  daemon or client (e.g., ``mds.a``), as stored in the monitors'
  configuration database.

* ``ceph config set <who> <option> <value>`` will set a configuration
  option in the monitors' configuration database.

* ``ceph config show <who>`` will show the reported running
  configuration for a running daemon.  These settings may differ from
  those stored by the monitors if there are also local configuration
  files in use or options have been overridden on the command line or
  at run time.  The source of the option values is reported as part
  of the output.

* ``ceph config assimilate-conf -i <input file> -o <output file>``
  will ingest a configuration file from *input file* and move any
  valid options into the monitors' configuration database.  Any
  settings that are unrecognized, invalid, or cannot be controlled by
  the monitor will be returned in an abbreviated config file stored in
  *output file*.  This command is useful for transitioning from legacy
  configuration files to centralized monitor-based configuration.


Help
====

You can get help for a particular option with:

.. prompt:: bash $

   ceph config help <option>

Note that this will use the configuration schema that is compiled into the running monitors.  If you have a mixed-version cluster (e.g., during an upgrade), you might also want to query the option schema from a specific running daemon:

.. prompt:: bash $

   ceph daemon <name> config help [option]

For example:

.. prompt:: bash $

   ceph config help log_file

:: 

  log_file - path to log file
    (std::string, basic)
    Default (non-daemon):
    Default (daemon): /var/log/ceph/$cluster-$name.log
    Can update at runtime: false
    See also: [log_to_stderr,err_to_stderr,log_to_syslog,err_to_syslog]

or:

.. prompt:: bash $

   ceph config help log_file -f json-pretty

::

  {
      "name": "log_file",
      "type": "std::string",
      "level": "basic",
      "desc": "path to log file",
      "long_desc": "",
      "default": "",
      "daemon_default": "/var/log/ceph/$cluster-$name.log",
      "tags": [],
      "services": [],
      "see_also": [
          "log_to_stderr",
          "err_to_stderr",
          "log_to_syslog",
          "err_to_syslog"
      ],
      "enum_values": [],
      "min": "",
      "max": "",
      "can_update_at_runtime": false
  }

The ``level`` property can be any of `basic`, `advanced`, or `dev`.
The `dev` options are intended for use by developers, generally for
testing purposes, and are not recommended for use by operators.


Runtime Changes
===============

In most cases, Ceph allows you to make changes to the configuration of
a daemon at runtime. This capability is quite useful for
increasing/decreasing logging output, enabling/disabling debug
settings, and even for runtime optimization.

Generally speaking, configuration options can be updated in the usual
way via the ``ceph config set`` command.  For example, do enable the debug log level on a specific OSD:

.. prompt:: bash $

   ceph config set osd.123 debug_ms 20

Note that if the same option is also customized in a local
configuration file, the monitor setting will be ignored (it has a
lower priority than the local config file).

Override values
---------------

You can also temporarily set an option using the `tell` or `daemon`
interfaces on the Ceph CLI.  These *override* values are ephemeral in
that they only affect the running process and are discarded/lost if
the daemon or process restarts.

Override values can be set in two ways:

#. From any host, we can send a message to a daemon over the network with:
   
   .. prompt:: bash $

      ceph tell <name> config set <option> <value>

   For example:
   
   .. prompt:: bash $

      ceph tell osd.123 config set debug_osd 20

   The `tell` command can also accept a wildcard for the daemon
   identifier.  For example, to adjust the debug level on all OSD
   daemons:
   
   .. prompt:: bash $

      ceph tell osd.* config set debug_osd 20

#. From the host the process is running on, we can connect directly to
   the process via a socket in ``/var/run/ceph`` with:

   .. prompt:: bash $

      ceph daemon <name> config set <option> <value>

   For example:
   
   .. prompt:: bash $

      ceph daemon osd.4 config set debug_osd 20

Note that in the ``ceph config show`` command output these temporary
values will be shown with a source of ``override``.


Viewing runtime settings
========================

You can see the current options set for a running daemon with the ``ceph config show`` command.  For example:

.. prompt:: bash $

   ceph config show osd.0

will show you the (non-default) options for that daemon.  You can also look at a specific option with:

.. prompt:: bash $

   ceph config show osd.0 debug_osd

or view all options (even those with default values) with:

.. prompt:: bash $

   ceph config show-with-defaults osd.0

You can also observe settings for a running daemon by connecting to it from the local host via the admin socket.  For example:

.. prompt:: bash $

   ceph daemon osd.0 config show

will dump all current settings:

.. prompt:: bash $

   ceph daemon osd.0 config diff

will show only non-default settings (as well as where the value came from: a config file, the monitor, an override, etc.), and:

.. prompt:: bash $

   ceph daemon osd.0 config get debug_osd

will report the value of a single option.



Changes since Nautilus
======================

With the Octopus release We changed the way the configuration file is parsed.
These changes are as follows:

- Repeated configuration options are allowed, and no warnings will be printed.
  The value of the last one is used, which means that the setting last in the file
  is the one that takes effect. Before this change, we would print warning messages
  when lines with duplicated options were encountered, like::

    warning line 42: 'foo' in section 'bar' redefined

- Invalid UTF-8 options were ignored with warning messages. But since Octopus,
  they are treated as fatal errors.

- Backslash ``\`` is used as the line continuation marker to combine the next
  line with current one. Before Octopus, it was required to follow a backslash with
  a non-empty line. But in Octopus, an empty line following a backslash is now allowed.

- In the configuration file, each line specifies an individual configuration
  option. The option's name and its value are separated with ``=``, and the
  value may be quoted using single or double quotes. If an invalid
  configuration is specified, we will treat it as an invalid configuration
  file ::

    bad option ==== bad value

- Before Octopus, if no section name was specified in the configuration file,
  all options would be set as though they were within the ``global`` section. This is
  now discouraged. Since Octopus, only a single option is allowed for
  configuration files without a section name.