.. _configuring-ceph: ================== Configuring Ceph ================== When Ceph services start, the initialization process activates a series of daemons that run in the background. A :term:`Ceph Storage Cluster` runs at a minimum three types of daemons: - :term:`Ceph Monitor` (``ceph-mon``) - :term:`Ceph Manager` (``ceph-mgr``) - :term:`Ceph OSD Daemon` (``ceph-osd``) Ceph Storage Clusters that support the :term:`Ceph File System` also run at least one :term:`Ceph Metadata Server` (``ceph-mds``). Clusters that support :term:`Ceph Object Storage` run Ceph RADOS Gateway daemons (``radosgw``) as well. Each daemon has a number of configuration options, each of which has a default value. You may adjust the behavior of the system by changing these configuration options. Be careful to understand the consequences before overriding default values, as it is possible to significantly degrade the performance and stability of your cluster. Also note that default values sometimes change between releases, so it is best to review the version of this documentation that aligns with your Ceph release. Option names ============ All Ceph configuration options have a unique name consisting of words formed with lower-case characters and connected with underscore (``_``) characters. When option names are specified on the command line, either underscore (``_``) or dash (``-``) characters can be used interchangeable (e.g., ``--mon-host`` is equivalent to ``--mon_host``). When option names appear in configuration files, spaces can also be used in place of underscore or dash. We suggest, though, that for clarity and convenience you consistently use underscores, as we do throughout this documentation. Config sources ============== Each Ceph daemon, process, and library will pull its configuration from several sources, listed below. Sources later in the list will override those earlier in the list when both are present. - the compiled-in default value - the monitor cluster's centralized configuration database - a configuration file stored on the local host - environment variables - command line arguments - runtime overrides set by an administrator One of the first things a Ceph process does on startup is parse the configuration options provided via the command line, environment, and local configuration file. The process will then contact the monitor cluster to retrieve configuration stored centrally for the entire cluster. Once a complete view of the configuration is available, the daemon or process startup will proceed. .. _bootstrap-options: Bootstrap options ----------------- Because some configuration options affect the process's ability to contact the monitors, authenticate, and retrieve the cluster-stored configuration, they may need to be stored locally on the node and set in a local configuration file. These options include: - ``mon_host``, the list of monitors for the cluster - ``mon_host_override``, the list of monitors for the cluster to **initially** contact when beginning a new instance of communication with the Ceph cluster. This overrides the known monitor list derived from MonMap updates sent to older Ceph instances (like librados cluster handles). It is expected this option is primarily useful for debugging. - ``mon_dns_srv_name`` (default: `ceph-mon`), the name of the DNS SRV record to check to identify the cluster monitors via DNS - ``mon_data``, ``osd_data``, ``mds_data``, ``mgr_data``, and similar options that define which local directory the daemon stores its data in. - ``keyring``, ``keyfile``, and/or ``key``, which can be used to specify the authentication credential to use to authenticate with the monitor. Note that in most cases the default keyring location is in the data directory specified above. In the vast majority of cases the default values of these are appropriate, with the exception of the ``mon_host`` option that identifies the addresses of the cluster's monitors. When DNS is used to identify monitors a local ceph configuration file can be avoided entirely. Skipping monitor config ----------------------- Any process may be passed the option ``--no-mon-config`` to skip the step that retrieves configuration from the cluster monitors. This is useful in cases where configuration is managed entirely via configuration files or where the monitor cluster is currently down but some maintenance activity needs to be done. .. _ceph-conf-file: Configuration sections ====================== Any given process or daemon has a single value for each configuration option. However, values for an option may vary across different daemon types even daemons of the same type. Ceph options that are stored in the monitor configuration database or in local configuration files are grouped into sections to indicate which daemons or clients they apply to. These sections include: ``global`` :Description: Settings under ``global`` affect all daemons and clients in a Ceph Storage Cluster. :Example: ``log_file = /var/log/ceph/$cluster-$type.$id.log`` ``mon`` :Description: Settings under ``mon`` affect all ``ceph-mon`` daemons in the Ceph Storage Cluster, and override the same setting in ``global``. :Example: ``mon_cluster_log_to_syslog = true`` ``mgr`` :Description: Settings in the ``mgr`` section affect all ``ceph-mgr`` daemons in the Ceph Storage Cluster, and override the same setting in ``global``. :Example: ``mgr_stats_period = 10`` ``osd`` :Description: Settings under ``osd`` affect all ``ceph-osd`` daemons in the Ceph Storage Cluster, and override the same setting in ``global``. :Example: ``osd_op_queue = wpq`` ``mds`` :Description: Settings in the ``mds`` section affect all ``ceph-mds`` daemons in the Ceph Storage Cluster, and override the same setting in ``global``. :Example: ``mds_cache_memory_limit = 10G`` ``client`` :Description: Settings under ``client`` affect all Ceph Clients (e.g., mounted Ceph File Systems, mounted Ceph Block Devices, etc.) as well as Rados Gateway (RGW) daemons. :Example: ``objecter_inflight_ops = 512`` Sections may also specify an individual daemon or client name. For example, ``mon.foo``, ``osd.123``, and ``client.smith`` are all valid section names. Any given daemon will draw its settings from the global section, the daemon or client type section, and the section sharing its name. Settings in the most-specific section take precedence, so for example if the same option is specified in both ``global``, ``mon``, and ``mon.foo`` on the same source (i.e., in the same configurationfile), the ``mon.foo`` value will be used. If multiple values of the same configuration option are specified in the same section, the last value wins. Note that values from the local configuration file always take precedence over values from the monitor configuration database, regardless of which section they appear in. .. _ceph-metavariables: Metavariables ============= Metavariables simplify Ceph Storage Cluster configuration dramatically. When a metavariable is set in a configuration value, Ceph expands the metavariable into a concrete value at the time the configuration value is used. Ceph metavariables are similar to variable expansion in the Bash shell. Ceph supports the following metavariables: ``$cluster`` :Description: Expands to the Ceph Storage Cluster name. Useful when running multiple Ceph Storage Clusters on the same hardware. :Example: ``/etc/ceph/$cluster.keyring`` :Default: ``ceph`` ``$type`` :Description: Expands to a daemon or process type (e.g., ``mds``, ``osd``, or ``mon``) :Example: ``/var/lib/ceph/$type`` ``$id`` :Description: Expands to the daemon or client identifier. For ``osd.0``, this would be ``0``; for ``mds.a``, it would be ``a``. :Example: ``/var/lib/ceph/$type/$cluster-$id`` ``$host`` :Description: Expands to the host name where the process is running. ``$name`` :Description: Expands to ``$type.$id``. :Example: ``/var/run/ceph/$cluster-$name.asok`` ``$pid`` :Description: Expands to daemon pid. :Example: ``/var/run/ceph/$cluster-$name-$pid.asok`` The Configuration File ====================== On startup, Ceph processes search for a configuration file in the following locations: #. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` environment variable) #. ``-c path/path`` (*i.e.,* the ``-c`` command line argument) #. ``/etc/ceph/$cluster.conf`` #. ``~/.ceph/$cluster.conf`` #. ``./$cluster.conf`` (*i.e.,* in the current working directory) #. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf`` where ``$cluster`` is the cluster's name (default ``ceph``). The Ceph configuration file uses an *ini* style syntax. You can add comment text after a pound sign (#) or a semi-colon (;). For example: .. code-block:: ini # <--A number (#) sign precedes a comment. ; A comment may be anything. # Comments always follow a semi-colon (;) or a pound (#) on each line. # The end of the line terminates a comment. # We recommend that you provide comments in your configuration file(s). .. _ceph-conf-settings: Config file section names ------------------------- The configuration file is divided into sections. Each section must begin with a valid configuration section name (see `Configuration sections`_, above) surrounded by square brackets. For example, .. code-block:: ini [global] debug_ms = 0 [osd] debug_ms = 1 [osd.1] debug_ms = 10 [osd.2] debug_ms = 10 Config file option values ------------------------- The value of a configuration option is a string. If it is too long to fit in a single line, you can put a backslash (``\``) at the end of line as the line continuation marker, so the value of the option will be the string after ``=`` in current line combined with the string in the next line:: [global] foo = long long ago\ long ago In the example above, the value of "``foo``" would be "``long long ago long ago``". Normally, the option value ends with a new line, or a comment, like .. code-block:: ini [global] obscure_one = difficult to explain # I will try harder in next release simpler_one = nothing to explain In the example above, the value of "``obscure one``" would be "``difficult to explain``"; and the value of "``simpler one`` would be "``nothing to explain``". If an option value contains spaces, and we want to make it explicit, we could quote the value using single or double quotes, like .. code-block:: ini [global] line = "to be, or not to be" Certain characters are not allowed to be present in the option values directly. They are ``=``, ``#``, ``;`` and ``[``. If we have to, we need to escape them, like .. code-block:: ini [global] secret = "i love \# and \[" Every configuration option is typed with one of the types below: ``int`` :Description: 64-bit signed integer, Some SI prefixes are supported, like "K", "M", "G", "T", "P", "E", meaning, respectively, 10\ :sup:`3`, 10\ :sup:`6`, 10\ :sup:`9`, etc. And "B" is the only supported unit. So, "1K", "1M", "128B" and "-1" are all valid option values. Some times, a negative value implies "unlimited" when it comes to an option for threshold or limit. :Example: ``42``, ``-1`` ``uint`` :Description: It is almost identical to ``integer``. But a negative value will be rejected. :Example: ``256``, ``0`` ``str`` :Description: Free style strings encoded in UTF-8, but some characters are not allowed. Please reference the above notes for the details. :Example: ``"hello world"``, ``"i love \#"``, ``yet-another-name`` ``boolean`` :Description: one of the two values ``true`` or ``false``. But an integer is also accepted, where "0" implies ``false``, and any non-zero values imply ``true``. :Example: ``true``, ``false``, ``1``, ``0`` ``addr`` :Description: a single address optionally prefixed with ``v1``, ``v2`` or ``any`` for the messenger protocol. If the prefix is not specified, ``v2`` protocol is used. Please see :ref:`address_formats` for more details. :Example: ``v1:1.2.3.4:567``, ``v2:1.2.3.4:567``, ``1.2.3.4:567``, ``2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567``, ``[::1]:6789`` ``addrvec`` :Description: a set of addresses separated by ",". The addresses can be optionally quoted with ``[`` and ``]``. :Example: ``[v1:1.2.3.4:567,v2:1.2.3.4:568]``, ``v1:1.2.3.4:567,v1:1.2.3.14:567`` ``[2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567], [2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::568]`` ``uuid`` :Description: the string format of a uuid defined by `RFC4122 `_. And some variants are also supported, for more details, see `Boost document `_. :Example: ``f81d4fae-7dec-11d0-a765-00a0c91e6bf6`` ``size`` :Description: denotes a 64-bit unsigned integer. Both SI prefixes and IEC prefixes are supported. And "B" is the only supported unit. A negative value will be rejected. :Example: ``1Ki``, ``1K``, ``1KiB`` and ``1B``. ``secs`` :Description: denotes a duration of time. By default the unit is second if not specified. Following units of time are supported: * second: "s", "sec", "second", "seconds" * minute: "m", "min", "minute", "minutes" * hour: "hs", "hr", "hour", "hours" * day: "d", "day", "days" * week: "w", "wk", "week", "weeks" * month: "mo", "month", "months" * year: "y", "yr", "year", "years" :Example: ``1 m``, ``1m`` and ``1 week`` .. _ceph-conf-database: Monitor configuration database ============================== The monitor cluster manages a database of configuration options that can be consumed by the entire cluster, enabling streamlined central configuration management for the entire system. The vast majority of configuration options can and should be stored here for ease of administration and transparency. A handful of settings may still need to be stored in local configuration files because they affect the ability to connect to the monitors, authenticate, and fetch configuration information. In most cases this is limited to the ``mon_host`` option, although this can also be avoided through the use of DNS SRV records. Sections and masks ------------------ Configuration options stored by the monitor can live in a global section, daemon type section, or specific daemon section, just like options in a configuration file can. In addition, options may also have a *mask* associated with them to further restrict which daemons or clients the option applies to. Masks take two forms: #. ``type:location`` where *type* is a CRUSH property like `rack` or `host`, and *location* is a value for that property. For example, ``host:foo`` would limit the option only to daemons or clients running on a particular host. #. ``class:device-class`` where *device-class* is the name of a CRUSH device class (e.g., ``hdd`` or ``ssd``). For example, ``class:ssd`` would limit the option only to OSDs backed by SSDs. (This mask has no effect for non-OSD daemons or clients.) When setting a configuration option, the `who` may be a section name, a mask, or a combination of both separated by a slash (``/``) character. For example, ``osd/rack:foo`` would mean all OSD daemons in the ``foo`` rack. When viewing configuration options, the section name and mask are generally separated out into separate fields or columns to ease readability. Commands -------- The following CLI commands are used to configure the cluster: * ``ceph config dump`` will dump the entire configuration database for the cluster. * ``ceph config get `` will dump the configuration for a specific daemon or client (e.g., ``mds.a``), as stored in the monitors' configuration database. * ``ceph config set