# -*- mode: YAML -*- --- options: - name: host type: str level: basic desc: local hostname long_desc: if blank, ceph assumes the short hostname (hostname -s) tags: - network services: - common flags: - no_mon_update with_legacy: true - name: fsid type: uuid level: basic desc: cluster fsid (uuid) fmt_desc: The cluster ID. One per cluster. May be generated by a deployment tool if not specified. note: Do not set this value if you use a deployment tool that does it for you. tags: - service services: - common flags: - no_mon_update - startup - name: public_addr type: addr level: basic desc: public-facing address to bind to fmt_desc: The IP address for the public (front-side) network. Set for each daemon. services: - mon - mds - osd - mgr flags: - startup with_legacy: true - name: public_addrv type: addrvec level: basic desc: public-facing address to bind to services: - mon - mds - osd - mgr flags: - startup with_legacy: true - name: public_bind_addr type: addr level: advanced services: - mon flags: - startup fmt_desc: In some dynamic deployments the Ceph MON daemon might bind to an IP address locally that is different from the ``public_addr`` advertised to other peers in the network. The environment must ensure that routing rules are set correctly. If ``public_bind_addr`` is set the Ceph Monitor daemon will bind to it locally and use ``public_addr`` in the monmaps to advertise its address to peers. This behavior is limited to the Monitor daemon. with_legacy: true - name: cluster_addr type: addr level: basic desc: cluster-facing address to bind to fmt_desc: The IP address for the cluster (back-side) network. Set for each daemon. tags: - network services: - osd flags: - startup with_legacy: true - name: public_network type: str level: advanced desc: Network(s) from which to choose a public address to bind to fmt_desc: The IP address and netmask of the public (front-side) network (e.g., ``192.168.0.0/24``). Set in ``[global]``. You may specify comma-separated subnets. The format of it looks like ``{ip-address}/{netmask} [, {ip-address}/{netmask}]`` tags: - network services: - mon - mds - osd - mgr flags: - startup with_legacy: true - name: public_network_interface type: str level: advanced desc: Interface name(s) from which to choose an address from a public_network to bind to; public_network must also be specified. tags: - network services: - mon - mds - osd - mgr see_also: - public_network flags: - startup - name: cluster_network type: str level: advanced desc: Network(s) from which to choose a cluster address to bind to fmt_desc: The IP address and netmask of the cluster (back-side) network (e.g., ``10.0.0.0/24``). Set in ``[global]``. You may specify comma-separated subnets. The format of it looks like ``{ip-address}/{netmask} [, {ip-address}/{netmask}]`` tags: - network services: - osd flags: - startup with_legacy: true - name: cluster_network_interface type: str level: advanced desc: Interface name(s) from which to choose an address from a cluster_network to bind to; cluster_network must also be specified. tags: - network services: - mon - mds - osd - mgr see_also: - cluster_network flags: - startup - name: monmap type: str level: advanced desc: path to MonMap file long_desc: This option is normally used during mkfs, but can also be used to identify which monitors to connect to. services: - mon flags: - no_mon_update - create - name: mon_host type: str level: basic desc: list of hosts or addresses to search for a monitor long_desc: This is a list of IP addresses or hostnames that are separated by commas, whitespace, or semicolons. Hostnames are resolved via DNS. All A and AAAA records are included in the search list. services: - common flags: - no_mon_update - startup - name: mon_host_override type: str level: advanced desc: monitor(s) to use overriding the MonMap fmt_desc: This is the list of monitors that the Ceph process **initially** contacts when first establishing communication with the Ceph cluster. This overrides the known monitor list that is derived from MonMap updates sent to older Ceph instances (like librados cluster handles). This option is expected to be useful primarily for debugging. services: - common flags: - no_mon_update - startup - name: mon_dns_srv_name type: str level: advanced desc: name of DNS SRV record to check for monitor addresses fmt_desc: the service name used querying the DNS for the monitor hosts/addresses default: ceph-mon tags: - network services: - common see_also: - mon_host flags: - startup - name: container_image type: str level: basic desc: container image (used by cephadm orchestrator) default: docker.io/ceph/daemon-base:latest-master-devel flags: - startup - name: no_config_file type: bool level: advanced desc: signal that we don't require a config file to be present long_desc: When specified, we won't be looking for a configuration file, and will instead expect that whatever options or values are required for us to work will be passed as arguments. default: false tags: - config services: - common flags: - no_mon_update - startup - name: lockdep type: bool level: dev desc: enable lockdep lock dependency analyzer default: false services: - common flags: - no_mon_update - startup with_legacy: true - name: lockdep_force_backtrace type: bool level: dev desc: always gather current backtrace at every lock default: false services: - common see_also: - lockdep flags: - startup with_legacy: true - name: run_dir type: str level: advanced desc: path for the 'run' directory for storing pid and socket files default: /var/run/ceph services: - common see_also: - admin_socket flags: - startup with_legacy: true - name: admin_socket type: str level: advanced desc: path for the runtime control socket file, used by the 'ceph daemon' command fmt_desc: The socket for executing administrative commands on a daemon, irrespective of whether Ceph Monitors have established a quorum. daemon_default: $run_dir/$cluster-$name.asok services: - common flags: - startup # default changed by common_preinit() with_legacy: true - name: admin_socket_mode type: str level: advanced desc: file mode to set for the admin socket file, e.g, '0755' services: - common see_also: - admin_socket flags: - startup with_legacy: true - name: daemonize type: bool level: advanced desc: whether to daemonize (background) after startup default: false daemon_default: true tags: - service services: - mon - mgr - osd - mds see_also: - pid_file - chdir flags: - no_mon_update - startup # default changed by common_preinit() with_legacy: true - name: setuser type: str level: advanced desc: uid or user name to switch to on startup long_desc: This is normally specified by the systemd unit file. tags: - service services: - mon - mgr - osd - mds see_also: - setgroup flags: - startup with_legacy: true - name: setgroup type: str level: advanced desc: gid or group name to switch to on startup long_desc: This is normally specified by the systemd unit file. tags: - service services: - mon - mgr - osd - mds see_also: - setuser flags: - startup with_legacy: true - name: setuser_match_path type: str level: advanced desc: if set, setuser/setgroup is condition on this path matching ownership long_desc: If setuser or setgroup are specified, and this option is non-empty, then the uid/gid of the daemon will only be changed if the file or directory specified by this option has a matching uid and/or gid. This exists primarily to allow switching to user ceph for OSDs to be conditional on whether the osd data contents have also been chowned after an upgrade. This is normally specified by the systemd unit file. tags: - service services: - mon - mgr - osd - mds see_also: - setuser - setgroup flags: - startup with_legacy: true - name: pid_file type: str level: advanced desc: path to write a pid file (if any) fmt_desc: The file in which the mon, osd or mds will write its PID. For instance, ``/var/run/$cluster/$type.$id.pid`` will create /var/run/ceph/mon.a.pid for the ``mon`` with id ``a`` running in the ``ceph`` cluster. The ``pid file`` is removed when the daemon stops gracefully. If the process is not daemonized (i.e. runs with the ``-f`` or ``-d`` option), the ``pid file`` is not created. tags: - service services: - mon - mgr - osd - mds flags: - startup with_legacy: true - name: chdir type: str level: advanced desc: path to chdir(2) to after daemonizing fmt_desc: The directory Ceph daemons change to once they are up and running. Default ``/`` directory recommended. tags: - service services: - mon - mgr - osd - mds see_also: - daemonize flags: - no_mon_update - startup with_legacy: true - name: fatal_signal_handlers type: bool level: advanced desc: whether to register signal handlers for SIGABRT etc that dump a stack trace long_desc: This is normally true for daemons and values for libraries. fmt_desc: If set, we will install signal handlers for SEGV, ABRT, BUS, ILL, FPE, XCPU, XFSZ, SYS signals to generate a useful log message default: true tags: - service services: - mon - mgr - osd - mds flags: - startup with_legacy: true - name: crash_dir type: str level: advanced desc: Directory where crash reports are archived default: /var/lib/ceph/crash flags: - startup with_legacy: true - name: restapi_log_level type: str level: advanced desc: default set by python code with_legacy: true - name: restapi_base_url type: str level: advanced desc: default set by python code with_legacy: true - name: erasure_code_dir type: str level: advanced desc: directory where erasure-code plugins can be found default: @CEPH_INSTALL_FULL_PKGLIBDIR@/erasure-code services: - mon - osd flags: - startup with_legacy: true - name: log_file type: str level: basic desc: path to log file fmt_desc: The location of the logging file for your cluster. daemon_default: /var/log/ceph/$cluster-$name.log see_also: - log_to_file - log_to_stderr - err_to_stderr - log_to_syslog - err_to_syslog # default changed by common_preinit() with_legacy: true - name: log_max_new type: int level: advanced desc: max unwritten log entries to allow before waiting to flush to the log fmt_desc: The maximum number of new log files. default: 1000 see_also: - log_max_recent # default changed by common_preinit() with_legacy: true - name: log_max_recent type: int level: advanced desc: recent log entries to keep in memory to dump in the event of a crash long_desc: The purpose of this option is to log at a higher debug level only to the in-memory buffer, and write out the detailed log messages only if there is a crash. Only log entries below the lower log level will be written unconditionally to the log. For example, debug_osd=1/5 will write everything <= 1 to the log unconditionally but keep entries at levels 2-5 in memory. If there is a seg fault or assertion failure, all entries will be dumped to the log. min: 1 default: 500 daemon_default: 10000 # default changed by common_preinit() with_legacy: true - name: log_to_file type: bool level: basic desc: send log lines to a file fmt_desc: Determines if logging messages should appear in a file. default: true see_also: - log_file with_legacy: true - name: log_to_stderr type: bool level: basic desc: send log lines to stderr fmt_desc: Determines if logging messages should appear in ``stderr``. default: true daemon_default: false with_legacy: true - name: err_to_stderr type: bool level: basic desc: send critical error log lines to stderr fmt_desc: Determines if error messages should appear in ``stderr``. default: false daemon_default: true with_legacy: true - name: log_stderr_prefix type: str level: advanced desc: String to prefix log messages with when sent to stderr long_desc: This is useful in container environments when combined with mon_cluster_log_to_stderr. The mon log prefixes each line with the channel name (e.g., 'default', 'audit'), while log_stderr_prefix can be set to 'debug '. see_also: - mon_cluster_log_to_stderr - name: log_to_syslog type: bool level: basic desc: send log lines to syslog facility fmt_desc: Determines if logging messages should appear in ``syslog``. default: false with_legacy: true - name: err_to_syslog type: bool level: basic desc: send critical error log lines to syslog facility fmt_desc: Determines if error messages should appear in ``syslog``. default: false with_legacy: true - name: log_flush_on_exit type: bool level: advanced desc: set a process exit handler to ensure the log is flushed on exit fmt_desc: Determines if Ceph should flush the log files after exit. default: false with_legacy: true - name: log_stop_at_utilization type: float level: basic desc: stop writing to the log file when device utilization reaches this ratio default: 0.97 see_also: - log_file min: 0 max: 1 with_legacy: true - name: log_to_graylog type: bool level: basic desc: send log lines to remote graylog server default: false see_also: - err_to_graylog - log_graylog_host - log_graylog_port with_legacy: true - name: err_to_graylog type: bool level: basic desc: send critical error log lines to remote graylog server default: false see_also: - log_to_graylog - log_graylog_host - log_graylog_port with_legacy: true - name: log_graylog_host type: str level: basic desc: address or hostname of graylog server to log to default: 127.0.0.1 see_also: - log_to_graylog - err_to_graylog - log_graylog_port with_legacy: true - name: log_graylog_port type: int level: basic desc: port number for the remote graylog server default: 12201 see_also: - log_graylog_host with_legacy: true - name: log_to_journald type: bool level: basic desc: send log lines to journald default: false see_also: - err_to_journald - name: err_to_journald type: bool level: basic desc: send critical error log lines to journald default: false see_also: - log_to_journald - name: log_coarse_timestamps type: bool level: advanced desc: timestamp log entries from coarse system clock to improve performance default: true tags: - performance - service services: - common # options will take k/v pairs, or single-item that will be assumed as general # default for all, regardless of channel. # e.g., "info" would be taken as the same as "default=info" # also, "default=daemon audit=local0" would mean # "default all to 'daemon', override 'audit' with 'local0' - name: clog_to_monitors type: str level: advanced desc: Make daemons send cluster log messages to monitors fmt_desc: Determines if ``clog`` messages should be sent to monitors. default: default=true flags: - runtime with_legacy: true services: - mgr - osd - mds - name: clog_to_syslog type: str level: advanced desc: Make daemons send cluster log messages to syslog fmt_desc: Determines if ``clog`` messages should be sent to syslog. default: 'false' flags: - runtime with_legacy: true services: - mon - mgr - osd - mds - name: clog_to_syslog_level type: str level: advanced desc: Syslog level for cluster log messages default: info see_also: - clog_to_syslog flags: - runtime with_legacy: true services: - mon - mgr - osd - mds - name: clog_to_syslog_facility type: str level: advanced desc: Syslog facility for cluster log messages default: default=daemon audit=local0 see_also: - clog_to_syslog flags: - runtime with_legacy: true services: - mon - mgr - osd - mds - name: clog_to_graylog type: str level: advanced desc: Make daemons send cluster log to graylog default: 'false' flags: - runtime services: - mon - mgr - osd - mds - name: clog_to_graylog_host type: str level: advanced desc: Graylog host to cluster log messages default: 127.0.0.1 see_also: - clog_to_graylog flags: - runtime with_legacy: true services: - mon - mgr - osd - mds - name: clog_to_graylog_port type: str level: advanced desc: Graylog port number for cluster log messages default: '12201' see_also: - clog_to_graylog flags: - runtime with_legacy: true services: - mon - mgr - osd - mds - name: enable_experimental_unrecoverable_data_corrupting_features type: str level: advanced desc: Enable named (or all with '*') experimental features that may be untested, dangerous, and/or cause permanent data loss flags: - runtime with_legacy: true - name: plugin_dir type: str level: advanced desc: Base directory for dynamically loaded plugins default: @CEPH_INSTALL_FULL_PKGLIBDIR@ services: - mon - osd flags: - startup - name: compressor_zlib_isal type: bool level: advanced desc: Use Intel ISA-L accelerated zlib implementation if available default: false with_legacy: true # regular zlib compression level, not applicable to isa-l optimized version - name: compressor_zlib_level type: int level: advanced desc: Zlib compression level to use default: 5 with_legacy: true # regular zlib compression winsize, not applicable to isa-l optimized version - name: compressor_zlib_winsize type: int level: advanced desc: Zlib compression winsize to use default: -15 min: -15 max: 32 with_legacy: true # regular zstd compression level - name: compressor_zstd_level type: int level: advanced desc: Zstd compression level to use default: 1 with_legacy: true - name: qat_compressor_enabled type: bool level: advanced desc: Enable Intel QAT acceleration support for compression if available default: false with_legacy: true - name: qat_compressor_session_max_number type: uint level: advanced desc: Set the maximum number of session within Qatzip when using QAT compressor default: 256 - name: plugin_crypto_accelerator type: str level: advanced desc: Crypto accelerator library to use default: crypto_isal with_legacy: true - name: openssl_engine_opts type: str level: advanced desc: Use engine for specific openssl algorithm long_desc: 'Pass opts in this way: engine_id=engine1,dynamic_path=/some/path/engine1.so,default_algorithms=DIGESTS:engine_id=engine2,dynamic_path=/some/path/engine2.so,default_algorithms=CIPHERS,other_ctrl=other_value' flags: - startup with_legacy: true - name: mempool_debug type: bool level: dev default: false flags: - no_mon_update with_legacy: true - name: thp type: bool level: dev desc: enable transparent huge page (THP) support long_desc: Ceph is known to suffer from memory fragmentation due to THP use. This is indicated by RSS usage above configured memory targets. Enabling THP is currently discouraged until selective use of THP by Ceph is implemented. default: false flags: - startup - name: key type: str level: advanced desc: Authentication key long_desc: A CephX authentication key, base64 encoded. It normally looks something like 'AQAtut9ZdMbNJBAAHz6yBAWyJyz2yYRyeMWDag=='. fmt_desc: The key (i.e., the text string of the key itself). Not recommended. see_also: - keyfile - keyring flags: - no_mon_update - startup with_legacy: true - name: keyfile type: str level: advanced desc: Path to a file containing a key long_desc: The file should contain a CephX authentication key and optionally a trailing newline, but nothing else. fmt_desc: The path to a key file (i.e,. a file containing only the key). see_also: - key flags: - no_mon_update - startup with_legacy: true - name: keyring type: str level: advanced desc: Path to a keyring file. long_desc: A keyring file is an INI-style formatted file where the section names are client or daemon names (e.g., 'osd.0') and each section contains a 'key' property with CephX authentication key as the value. # please note, document are generated without accessing to the CMake # variables, so please update the document manually with a representive # default value using the ":default:" option of ".. confval::" directive. default: @keyring_paths@ see_also: - key - keyfile flags: - no_mon_update - startup with_legacy: true - name: heartbeat_interval type: int level: advanced desc: Frequency of internal heartbeat checks (seconds) default: 5 flags: - startup with_legacy: true - name: heartbeat_file type: str level: advanced desc: File to touch on successful internal heartbeat long_desc: If set, this file will be touched every time an internal heartbeat check succeeds. see_also: - heartbeat_interval flags: - startup with_legacy: true - name: heartbeat_inject_failure type: int level: dev default: 0 with_legacy: true - name: perf type: bool level: advanced desc: Enable internal performance metrics long_desc: If enabled, collect and expose internal health metrics default: true with_legacy: true - name: ms_type type: str level: advanced desc: Messenger implementation to use for network communication fmt_desc: Transport type used by Async Messenger. Can be ``async+posix``, ``async+dpdk`` or ``async+rdma``. Posix uses standard TCP/IP networking and is default. Other transports may be experimental and support may be limited. default: async+posix flags: - startup with_legacy: true - name: ms_public_type type: str level: advanced desc: Messenger implementation to use for the public network long_desc: If not specified, use ms_type see_also: - ms_type flags: - startup with_legacy: true - name: ms_cluster_type type: str level: advanced desc: Messenger implementation to use for the internal cluster network long_desc: If not specified, use ms_type see_also: - ms_type flags: - startup with_legacy: true - name: ms_mon_cluster_mode type: str level: basic desc: Connection modes (crc, secure) for intra-mon connections in order of preference fmt_desc: the connection mode (or permitted modes) to use between monitors. default: secure crc see_also: - ms_mon_service_mode - ms_mon_client_mode - ms_service_mode - ms_cluster_mode - ms_client_mode flags: - startup - name: ms_mon_service_mode type: str level: basic desc: Allowed connection modes (crc, secure) for connections to mons fmt_desc: a list of permitted modes for clients or other Ceph daemons to use when connecting to monitors. default: secure crc see_also: - ms_service_mode - ms_mon_cluster_mode - ms_mon_client_mode - ms_cluster_mode - ms_client_mode flags: - startup - name: ms_mon_client_mode type: str level: basic desc: Connection modes (crc, secure) for connections from clients to monitors in order of preference fmt_desc: a list of connection modes, in order of preference, for clients or non-monitor daemons to use when connecting to monitors. default: secure crc see_also: - ms_mon_service_mode - ms_mon_cluster_mode - ms_service_mode - ms_cluster_mode - ms_client_mode flags: - startup - name: ms_cluster_mode type: str level: basic desc: Connection modes (crc, secure) for intra-cluster connections in order of preference fmt_desc: connection mode (or permitted modes) used for intra-cluster communication between Ceph daemons. If multiple modes are listed, the modes listed first are preferred. default: crc secure see_also: - ms_service_mode - ms_client_mode flags: - startup - name: ms_service_mode type: str level: basic desc: Allowed connection modes (crc, secure) for connections to daemons fmt_desc: a list of permitted modes for clients to use when connecting to the cluster. default: crc secure see_also: - ms_cluster_mode - ms_client_mode flags: - startup - name: ms_client_mode type: str level: basic desc: Connection modes (crc, secure) for connections from clients in order of preference fmt_desc: a list of connection modes, in order of preference, for clients to use (or allow) when talking to a Ceph cluster. default: crc secure see_also: - ms_cluster_mode - ms_service_mode flags: - startup - name: ms_osd_compress_mode type: str level: advanced desc: Compression policy to use in Messenger for communicating with OSD default: none services: - osd enum_values: - none - force see_also: - ms_compress_secure flags: - runtime - name: ms_osd_compress_min_size type: uint level: advanced desc: Minimal message size eligable for on-wire compression default: 1_K services: - osd see_also: - ms_osd_compress_mode flags: - runtime - name: ms_osd_compression_algorithm type: str level: advanced desc: Compression algorithm to use in Messenger when communicating with OSD long_desc: Compression algorithm for connections with OSD in order of preference Although the default value is set to snappy, a list (like snappy zlib zstd etc.) is acceptable as well. default: snappy services: - osd see_also: - ms_osd_compress_mode flags: - runtime - name: ms_compress_secure type: bool level: advanced desc: Allowing compression when on-wire encryption is enabled long_desc: Combining encryption with compression reduces the level of security of messages between peers. In case both encryption and compression are enabled, compression setting will be ignored and message will not be compressed. This behaviour can be override using this setting. default: false see_also: - ms_osd_compress_mode flags: - runtime - name: ms_learn_addr_from_peer type: bool level: advanced desc: Learn address from what IP our first peer thinks we connect from long_desc: Use the IP address our first peer (usually a monitor) sees that we are connecting from. This is useful if a client is behind some sort of NAT and we want to see it identified by its local (not NATed) address. default: true with_legacy: true - name: ms_tcp_nodelay type: bool level: advanced desc: Disable Nagle's algorithm and send queued network traffic immediately fmt_desc: Ceph enables ``ms_tcp_nodelay`` so that each request is sent immediately (no buffering). Disabling `Nagle's algorithm`_ increases network traffic, which can introduce latency. If you experience large numbers of small packets, you may try disabling ``ms_tcp_nodelay``. default: true with_legacy: true - name: ms_tcp_rcvbuf type: size level: advanced desc: Size of TCP socket receive buffer fmt_desc: The size of the socket buffer on the receiving end of a network connection. Disable by default. default: 0 with_legacy: true - name: ms_tcp_prefetch_max_size type: size level: advanced desc: Maximum amount of data to prefetch out of the socket receive buffer default: 4_K with_legacy: true - name: ms_initial_backoff type: float level: advanced desc: Initial backoff after a network error is detected (seconds) fmt_desc: The initial time to wait before reconnecting on a fault. default: 0.2 with_legacy: true - name: ms_max_backoff type: float level: advanced desc: Maximum backoff after a network error before retrying (seconds) fmt_desc: The maximum time to wait before reconnecting on a fault. default: 15 see_also: - ms_initial_backoff with_legacy: true - name: ms_crc_data type: bool level: dev desc: Set and/or verify crc32c checksum on data payload sent over network default: true with_legacy: true - name: ms_crc_header type: bool level: dev desc: Set and/or verify crc32c checksum on header payload sent over network default: true with_legacy: true - name: ms_die_on_bad_msg type: bool level: dev desc: Induce a daemon crash/exit when a bad network message is received fmt_desc: Debug option; do not configure. default: false with_legacy: true - name: ms_die_on_unhandled_msg type: bool level: dev desc: Induce a daemon crash/exit when an unrecognized message is received default: false with_legacy: true - name: ms_die_on_old_message type: bool level: dev desc: Induce a daemon crash/exit when a old, undecodable message is received default: false with_legacy: true - name: ms_die_on_skipped_message type: bool level: dev desc: Induce a daemon crash/exit if sender skips a message sequence number default: false with_legacy: true - name: ms_die_on_bug type: bool level: dev desc: Induce a crash/exit on various bugs (for testing purposes) default: false with_legacy: true - name: ms_dispatch_throttle_bytes type: size level: advanced desc: Limit messages that are read off the network but still being processed fmt_desc: Throttles total size of messages waiting to be dispatched. default: 100_M with_legacy: true - name: ms_bind_ipv4 type: bool level: advanced desc: Bind servers to IPv4 address(es) fmt_desc: Enables Ceph daemons to bind to IPv4 addresses. default: true see_also: - ms_bind_ipv6 - name: ms_bind_ipv6 type: bool level: advanced desc: Bind servers to IPv6 address(es) fmt_desc: Enables Ceph daemons to bind to IPv6 addresses. default: false see_also: - ms_bind_ipv4 with_legacy: true - name: ms_bind_prefer_ipv4 type: bool level: advanced desc: Prefer IPV4 over IPV6 address(es) default: false - name: ms_bind_msgr1 type: bool level: advanced desc: Bind servers to msgr1 (legacy) protocol address(es) default: true see_also: - ms_bind_msgr2 - name: ms_bind_msgr2 type: bool level: advanced desc: Bind servers to msgr2 (nautilus+) protocol address(es) default: true see_also: - ms_bind_msgr1 - name: ms_bind_port_min type: int level: advanced desc: Lowest port number to bind daemon(s) to fmt_desc: The minimum port number to which an OSD or MDS daemon will bind. default: 6800 with_legacy: true - name: ms_bind_port_max type: int level: advanced desc: Highest port number to bind daemon(s) to fmt_desc: The maximum port number to which an OSD or MDS daemon will bind. default: 7568 with_legacy: true # FreeBSD does not use SO_REAUSEADDR so allow for a bit more time per default - name: ms_bind_retry_count type: int level: advanced desc: Number of attempts to make while bind(2)ing to a port default: @ms_bind_retry_count@ with_legacy: true # FreeBSD does not use SO_REAUSEADDR so allow for a bit more time per default - name: ms_bind_retry_delay type: int level: advanced desc: Delay between bind(2) attempts (seconds) default: @ms_bind_retry_delay@ with_legacy: true - name: ms_bind_before_connect type: bool level: advanced desc: Call bind(2) on client sockets default: false with_legacy: true - name: ms_tcp_listen_backlog type: int level: advanced desc: Size of queue of incoming connections for accept(2) default: 512 with_legacy: true - name: ms_connection_ready_timeout type: uint level: advanced desc: Time before we declare a not yet ready connection as dead (seconds) default: 10 with_legacy: true - name: ms_connection_idle_timeout type: uint level: advanced desc: Time before an idle connection is closed (seconds) default: 900 with_legacy: true - name: ms_pq_max_tokens_per_priority type: uint level: dev default: 16_M with_legacy: true - name: ms_pq_min_cost type: size level: dev default: 64_K with_legacy: true - name: ms_inject_socket_failures type: uint level: dev desc: Inject a socket failure every Nth socket operation fmt_desc: Debug option; do not configure. default: 0 with_legacy: true - name: ms_inject_delay_type type: str level: dev desc: Entity type to inject delays for flags: - runtime with_legacy: true - name: ms_inject_delay_max type: float level: dev desc: Max delay to inject default: 1 with_legacy: true - name: ms_inject_delay_probability type: float level: dev default: 0 with_legacy: true - name: ms_inject_internal_delays type: float level: dev desc: Inject various internal delays to induce races (seconds) default: 0 with_legacy: true - name: ms_inject_network_congestion type: uint level: dev desc: Inject a network congestions that stuck with N times operations default: 0 with_legacy: true - name: ms_blackhole_osd type: bool level: dev default: false with_legacy: true - name: ms_blackhole_mon type: bool level: dev default: false with_legacy: true - name: ms_blackhole_mds type: bool level: dev default: false with_legacy: true - name: ms_blackhole_mgr type: bool level: dev default: false with_legacy: true - name: ms_blackhole_client type: bool level: dev default: false with_legacy: true - name: ms_dump_on_send type: bool level: advanced desc: Hexdump message to debug log on message send default: false with_legacy: true - name: ms_dump_corrupt_message_level type: int level: advanced desc: Log level at which to hexdump corrupt messages we receive default: 1 with_legacy: true # number of worker processing threads for async messenger created on init - name: ms_async_op_threads type: uint level: advanced desc: Threadpool size for AsyncMessenger (ms_type=async) fmt_desc: Initial number of worker threads used by each Async Messenger instance. Should be at least equal to highest number of replicas, but you can decrease it if you are low on CPU core count and/or you host a lot of OSDs on single server. default: 3 min: 1 max: 24 with_legacy: true - name: ms_async_reap_threshold type: uint level: dev desc: number of deleted connections before we reap default: 5 min: 1 with_legacy: true - name: ms_async_rdma_device_name type: str level: advanced with_legacy: true - name: ms_async_rdma_enable_hugepage type: bool level: advanced default: false with_legacy: true - name: ms_async_rdma_buffer_size type: size level: advanced default: 128_K with_legacy: true - name: ms_async_rdma_send_buffers type: uint level: advanced default: 1_K with_legacy: true # size of the receive buffer pool, 0 is unlimited - name: ms_async_rdma_receive_buffers type: uint level: advanced default: 32_K with_legacy: true # max number of wr in srq - name: ms_async_rdma_receive_queue_len type: uint level: advanced default: 4_K with_legacy: true # support srq - name: ms_async_rdma_support_srq type: bool level: advanced default: true with_legacy: true - name: ms_async_rdma_port_num type: uint level: advanced default: 1 with_legacy: true - name: ms_async_rdma_polling_us type: uint level: advanced default: 1000 with_legacy: true - name: ms_async_rdma_gid_idx type: int level: advanced desc: use gid_idx to select GID for choosing RoCEv1 or RoCEv2 default: 0 with_legacy: true # GID format: "fe80:0000:0000:0000:7efe:90ff:fe72:6efe", no zero folding - name: ms_async_rdma_local_gid type: str level: advanced with_legacy: true # 0=RoCEv1, 1=RoCEv2, 2=RoCEv1.5 - name: ms_async_rdma_roce_ver type: int level: advanced default: 1 with_legacy: true # in RoCE, this means PCP - name: ms_async_rdma_sl type: int level: advanced default: 3 with_legacy: true # in RoCE, this means DSCP - name: ms_async_rdma_dscp type: int level: advanced default: 96 with_legacy: true # when there are enough accept failures, indicating there are unrecoverable failures, # just do ceph_abort() . Here we make it configurable. - name: ms_max_accept_failures type: int level: advanced desc: The maximum number of consecutive failed accept() calls before considering the daemon is misconfigured and abort it. default: 4 with_legacy: true # rdma connection management - name: ms_async_rdma_cm type: bool level: advanced default: false with_legacy: true - name: ms_async_rdma_type type: str level: advanced default: ib with_legacy: true - name: ms_dpdk_port_id type: int level: advanced default: 0 with_legacy: true # it is modified in unittest so that use SAFE_OPTION to declare - name: ms_dpdk_coremask type: str level: advanced default: '0xF' see_also: - ms_async_op_threads with_legacy: true - name: ms_dpdk_memory_channel type: str level: advanced default: '4' with_legacy: true - name: ms_dpdk_hugepages type: str level: advanced with_legacy: true - name: ms_dpdk_pmd type: str level: advanced with_legacy: true - name: ms_dpdk_devs_allowlist type: str level: advanced desc: NIC's PCIe address are allowed to use long_desc: for a single NIC use ms_dpdk_devs_allowlist=-a 0000:7d:010 or --allow=0000:7d:010; for a bond nics use ms_dpdk_devs_allowlist=--allow=0000:7d:01.0 --allow=0000:7d:02.6 --vdev=net_bonding0,mode=2,slave=0000:7d:01.0,slave=0000:7d:02.6. - name: ms_dpdk_host_ipv4_addr type: str level: advanced with_legacy: true - name: ms_dpdk_gateway_ipv4_addr type: str level: advanced with_legacy: true - name: ms_dpdk_netmask_ipv4_addr type: str level: advanced with_legacy: true - name: ms_dpdk_lro type: bool level: advanced default: true with_legacy: true - name: ms_dpdk_enable_tso type: bool level: advanced default: true - name: ms_dpdk_hw_flow_control type: bool level: advanced default: true with_legacy: true # Weighing of a hardware network queue relative to a software queue (0=no work, 1= equal share)") - name: ms_dpdk_hw_queue_weight type: float level: advanced default: 1 with_legacy: true - name: ms_dpdk_debug_allow_loopback type: bool level: dev default: false with_legacy: true - name: ms_dpdk_rx_buffer_count_per_core type: int level: advanced default: 8192 with_legacy: true - name: inject_early_sigterm type: bool level: dev desc: send ourselves a SIGTERM early during startup default: false with_legacy: true # list of initial cluster mon ids; if specified, need majority to form initial quorum and create new cluster - name: mon_initial_members type: str level: advanced fmt_desc: The IDs of initial monitors in a cluster during startup. If specified, Ceph requires an odd number of monitors to form an initial quorum (e.g., 3). note: A *majority* of monitors in your cluster must be able to reach each other in order to establish a quorum. You can decrease the initial number of monitors to establish a quorum with this setting. services: - mon flags: - no_mon_update - cluster_create with_legacy: true - name: mon_max_pg_per_osd type: uint level: advanced desc: Max number of PGs per OSD the cluster will allow long_desc: If the number of PGs per OSD exceeds this, a health warning will be visible in `ceph status`. This is also used in automated PG management, as the threshold at which some pools' pg_num may be shrunk in order to enable increasing the pg_num of others. default: 250 flags: - runtime services: - mgr - mon min: 1 - name: mon_osd_full_ratio type: float level: advanced desc: full ratio of OSDs to be set during initial creation of the cluster default: 0.95 flags: - no_mon_update - cluster_create with_legacy: true - name: mon_osd_backfillfull_ratio type: float level: advanced default: 0.9 flags: - no_mon_update - cluster_create with_legacy: true - name: mon_osd_nearfull_ratio type: float level: advanced desc: nearfull ratio for OSDs to be set during initial creation of cluster default: 0.85 flags: - no_mon_update - cluster_create with_legacy: true - name: mon_osd_initial_require_min_compat_client type: str level: advanced default: luminous flags: - no_mon_update - cluster_create with_legacy: true - name: mon_allow_pool_delete type: bool level: advanced desc: allow pool deletions fmt_desc: Should monitors allow pools to be removed, regardless of what the pool flags say? default: false services: - mon with_legacy: true - name: mon_fake_pool_delete type: bool level: advanced desc: fake pool deletions by renaming the rados pool default: false services: - mon with_legacy: true - name: mon_globalid_prealloc type: uint level: advanced desc: number of globalid values to preallocate long_desc: This setting caps how many new clients can authenticate with the cluster before the monitors have to perform a write to preallocate more. Large values burn through the 64-bit ID space more quickly. fmt_desc: The number of global IDs to pre-allocate for clients and daemons in the cluster. default: 10000 services: - mon with_legacy: true - name: mon_osd_report_timeout type: int level: advanced desc: time before OSDs who do not report to the mons are marked down (seconds) fmt_desc: The grace period in seconds before declaring unresponsive Ceph OSD Daemons ``down``. default: 15_min services: - mon with_legacy: true - name: mon_warn_on_insecure_global_id_reclaim type: bool level: advanced desc: issue AUTH_INSECURE_GLOBAL_ID_RECLAIM health warning if any connected clients are insecurely reclaiming global_id default: true services: - mon see_also: - mon_warn_on_insecure_global_id_reclaim_allowed - auth_allow_insecure_global_id_reclaim - auth_expose_insecure_global_id_reclaim - name: mon_warn_on_insecure_global_id_reclaim_allowed type: bool level: advanced desc: issue AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED health warning if insecure global_id reclaim is allowed default: true services: - mon see_also: - mon_warn_on_insecure_global_id_reclaim - auth_allow_insecure_global_id_reclaim - auth_expose_insecure_global_id_reclaim - name: mon_warn_on_msgr2_not_enabled type: bool level: advanced desc: issue MON_MSGR2_NOT_ENABLED health warning if monitors are all running Nautilus but not all binding to a msgr2 port default: true services: - mon see_also: - ms_bind_msgr2 - name: mon_warn_on_slow_ping_time type: float level: advanced desc: Override mon_warn_on_slow_ping_ratio with specified threshold in milliseconds fmt_desc: Override ``mon_warn_on_slow_ping_ratio`` with a specific value. Raise ``HEALTH_WARN`` if any heartbeat between OSDs exceeds ``mon_warn_on_slow_ping_time`` milliseconds. The default is 0 (disabled). default: 0 services: - mgr - osd see_also: - mon_warn_on_slow_ping_ratio - name: mon_warn_on_slow_ping_ratio type: float level: advanced desc: Issue a health warning if heartbeat ping longer than percentage of osd_heartbeat_grace fmt_desc: Raise ``HEALTH_WARN`` when any heartbeat between OSDs exceeds ``mon_warn_on_slow_ping_ratio`` of ``osd_heartbeat_grace``. default: 0.05 services: - mgr - osd see_also: - osd_heartbeat_grace - mon_warn_on_slow_ping_time - name: mon_max_snap_prune_per_epoch type: uint level: advanced desc: max number of pruned snaps we will process in a single OSDMap epoch default: 100 services: - mon - name: mon_min_osdmap_epochs type: int level: advanced desc: min number of OSDMaps to store fmt_desc: Minimum number of OSD map epochs to keep at all times. default: 500 services: - mon with_legacy: true - name: mon_max_log_epochs type: int level: advanced desc: max number of past cluster log epochs to store fmt_desc: Maximum number of Log epochs the monitor should keep. default: 500 services: - mon with_legacy: true - name: mon_max_mdsmap_epochs type: int level: advanced desc: max number of FSMaps/MDSMaps to store fmt_desc: The maximum number of mdsmap epochs to trim during a single proposal. default: 500 services: - mon with_legacy: true - name: mon_max_mgrmap_epochs type: int level: advanced desc: max number of MgrMaps to store default: 500 services: - mon - name: mon_max_osd type: int level: advanced desc: max number of OSDs in a cluster fmt_desc: The maximum number of OSDs allowed in the cluster. default: 10000 services: - mon with_legacy: true - name: mon_probe_timeout type: float level: advanced desc: timeout for querying other mons during bootstrap pre-election phase (seconds) fmt_desc: Number of seconds the monitor will wait to find peers before bootstrapping. default: 2 services: - mon with_legacy: true - name: mon_client_bytes type: size level: advanced desc: max bytes of outstanding client messages mon will read off the network fmt_desc: The amount of client message data allowed in memory (in bytes). default: 100_M services: - mon with_legacy: true - name: mon_warn_pg_not_scrubbed_ratio type: float level: advanced desc: Percentage of the scrub max interval past the scrub max interval to warn default: 0.5 see_also: - osd_scrub_max_interval min: 0 with_legacy: true - name: mon_warn_pg_not_deep_scrubbed_ratio type: float level: advanced desc: Percentage of the deep scrub interval past the deep scrub interval to warn default: 0.75 see_also: - osd_deep_scrub_interval min: 0 with_legacy: true - name: mon_scrub_interval type: secs level: advanced desc: frequency for scrubbing mon database fmt_desc: How often the monitor scrubs its store by comparing the stored checksums with the computed ones for all stored keys. (0 disables it. dangerous, use with care) default: 1_day services: - mon - name: mon_scrub_timeout type: int level: advanced desc: timeout to restart scrub of mon quorum participant does not respond for the latest chunk default: 5_min services: - mon with_legacy: true - name: mon_scrub_max_keys type: int level: advanced desc: max keys per on scrub chunk/step fmt_desc: The maximum number of keys to scrub each time. default: 100 services: - mon with_legacy: true # probability of injected crc mismatch [0.0, 1.0] - name: mon_scrub_inject_crc_mismatch type: float level: dev desc: probability for injecting crc mismatches into mon scrub default: 0 services: - mon with_legacy: true # probability of injected missing keys [0.0, 1.0] - name: mon_scrub_inject_missing_keys type: float level: dev desc: probability for injecting missing keys into mon scrub default: 0 services: - mon with_legacy: true - name: mon_config_key_max_entry_size type: size level: advanced desc: Defines the number of bytes allowed to be held in a single config-key entry fmt_desc: The maximum size of config-key entry (in bytes) default: 64_K services: - mon with_legacy: true - name: mon_sync_timeout type: float level: advanced desc: timeout before canceling sync if syncing mon does not respond fmt_desc: Number of seconds the monitor will wait for the next update message from its sync provider before it gives up and bootstrap again. default: 1_min services: - mon with_legacy: true - name: mon_sync_max_payload_size type: size level: advanced desc: target max message payload for mon sync fmt_desc: The maximum size for a sync payload (in bytes). default: 1_M services: - mon with_legacy: true - name: mon_sync_max_payload_keys type: int level: advanced desc: target max keys in message payload for mon sync default: 2000 services: - mon with_legacy: true - name: mon_sync_debug type: bool level: dev desc: enable extra debugging during mon sync default: false services: - mon with_legacy: true - name: mon_inject_sync_get_chunk_delay type: float level: dev desc: inject delay during sync (seconds) default: 0 services: - mon with_legacy: true - name: mon_osd_min_down_reporters type: uint level: advanced desc: number of OSDs from different subtrees who need to report a down OSD for it to count fmt_desc: The minimum number of Ceph OSD Daemons required to report a ``down`` Ceph OSD Daemon. default: 2 services: - mon see_also: - mon_osd_reporter_subtree_level - name: mon_osd_reporter_subtree_level type: str level: advanced desc: in which level of parent bucket the reporters are counted fmt_desc: In which level of parent bucket the reporters are counted. The OSDs send failure reports to monitors if they find a peer that is not responsive. Monitors mark the reported ``OSD`` out and then ``down`` after a grace period. default: host services: - mon flags: - runtime - name: mon_osd_snap_trim_queue_warn_on type: int level: advanced desc: Warn when snap trim queue is that large (or larger). long_desc: Warn when snap trim queue length for at least one PG crosses this value, as this is indicator of snap trimmer not keeping up, wasting disk space default: 32768 services: - mon with_legacy: true # force mon to trim maps to this point, regardless of min_last_epoch_clean (dangerous) - name: mon_osd_force_trim_to type: int level: dev desc: force mons to trim osdmaps through this epoch fmt_desc: Force monitor to trim osdmaps to this point, even if there is PGs not clean at the specified epoch (0 disables it. dangerous, use with care) default: 0 services: - mon with_legacy: true - name: mon_debug_extra_checks type: bool level: dev desc: Enable some additional monitor checks long_desc: Enable some additional monitor checks that would be too expensive to run on production systems, or would only be relevant while testing or debugging. default: false services: - mon - name: mon_debug_block_osdmap_trim type: bool level: dev desc: Block OSDMap trimming while the option is enabled. long_desc: Blocking OSDMap trimming may be quite helpful to easily reproduce states in which the monitor keeps (hundreds of) thousands of osdmaps. default: false services: - mon - name: mon_debug_deprecated_as_obsolete type: bool level: dev desc: treat deprecated mon commands as obsolete default: false services: - mon with_legacy: true - name: mon_debug_dump_transactions type: bool level: dev desc: dump paxos transactions to log default: false services: - mon see_also: - mon_debug_dump_location with_legacy: true - name: mon_debug_dump_json type: bool level: dev desc: dump paxos transasctions to log as json default: false services: - mon see_also: - mon_debug_dump_transactions with_legacy: true - name: mon_debug_dump_location type: str level: dev desc: file to dump paxos transactions to default: /var/log/ceph/$cluster-$name.tdump services: - mon see_also: - mon_debug_dump_transactions with_legacy: true - name: mon_debug_no_require_quincy type: bool level: dev desc: do not set quincy feature for new mon clusters default: false services: - mon flags: - cluster_create - name: mon_debug_no_require_reef type: bool level: dev desc: do not set reef feature for new mon clusters default: false services: - mon flags: - cluster_create - name: mon_debug_no_require_bluestore_for_ec_overwrites type: bool level: dev desc: do not require bluestore OSDs to enable EC overwrites on a rados pool default: false services: - mon with_legacy: true - name: mon_debug_no_initial_persistent_features type: bool level: dev desc: do not set any monmap features for new mon clusters default: false services: - mon flags: - cluster_create with_legacy: true - name: mon_inject_transaction_delay_max type: float level: dev desc: max duration of injected delay in paxos default: 10 services: - mon with_legacy: true # range [0, 1] - name: mon_inject_transaction_delay_probability type: float level: dev desc: probability of injecting a delay in paxos default: 0 services: - mon with_legacy: true - name: mon_inject_pg_merge_bounce_probability type: float level: dev desc: probability of failing and reverting a pg_num decrement default: 0 services: - mon # kill the sync provider at a specific point in the work flow - name: mon_sync_provider_kill_at type: int level: dev desc: kill mon sync requester at specific point default: 0 services: - mon with_legacy: true # kill the sync requester at a specific point in the work flow - name: mon_sync_requester_kill_at type: int level: dev desc: kill mon sync requestor at specific point default: 0 services: - mon with_legacy: true # force monitor to join quorum even if it has been previously removed from the map - name: mon_force_quorum_join type: bool level: advanced desc: force mon to rejoin quorum even though it was just removed fmt_desc: Force monitor to join quorum even if it has been previously removed from the map default: false services: - mon with_legacy: true # type of keyvaluedb backend - name: mon_keyvaluedb type: str level: advanced desc: database backend to use for the mon database default: rocksdb services: - mon enum_values: - leveldb - rocksdb flags: - create with_legacy: true # UNSAFE -- TESTING ONLY! Allows addition of a cache tier with preexisting snaps - name: mon_debug_unsafe_allow_tier_with_nonempty_snaps type: bool level: dev default: false services: - mon with_legacy: true # required of mon, mds, osd daemons - name: auth_cluster_required type: str level: advanced desc: authentication methods required by the cluster fmt_desc: If enabled, the Ceph Storage Cluster daemons (i.e., ``ceph-mon``, ``ceph-osd``, ``ceph-mds`` and ``ceph-mgr``) must authenticate with each other. Valid settings are ``cephx`` or ``none``. default: cephx with_legacy: true # required by daemons of clients - name: auth_service_required type: str level: advanced desc: authentication methods required by service daemons fmt_desc: If enabled, the Ceph Storage Cluster daemons require Ceph Clients to authenticate with the Ceph Storage Cluster in order to access Ceph services. Valid settings are ``cephx`` or ``none``. default: cephx with_legacy: true # what clients require of daemons - name: auth_client_required type: str level: advanced desc: authentication methods allowed by clients fmt_desc: If enabled, the Ceph Client requires the Ceph Storage Cluster to authenticate with the Ceph Client. Valid settings are ``cephx`` or ``none``. default: cephx, none with_legacy: true # deprecated; default value for above if they are not defined. - name: auth_supported type: str level: advanced desc: authentication methods required (deprecated) with_legacy: true - name: max_rotating_auth_attempts type: int level: advanced desc: number of attempts to initialize rotating keys before giving up default: 10 with_legacy: true - name: rotating_keys_bootstrap_timeout type: int level: advanced desc: timeout for obtaining rotating keys during bootstrap phase (seconds) default: 30 - name: rotating_keys_renewal_timeout type: int level: advanced desc: timeout for updating rotating keys (seconds) default: 10 - name: cephx_require_signatures type: bool level: advanced default: false fmt_desc: If set to ``true``, Ceph requires signatures on all message traffic between the Ceph Client and the Ceph Storage Cluster, and between daemons comprising the Ceph Storage Cluster. Ceph Argonaut and Linux kernel versions prior to 3.19 do not support signatures; if such clients are in use this option can be turned off to allow them to connect. with_legacy: true - name: cephx_require_version type: int level: advanced desc: Cephx version required (1 = pre-mimic, 2 = mimic+) default: 2 with_legacy: true - name: cephx_cluster_require_signatures type: bool level: advanced default: false fmt_desc: If set to ``true``, Ceph requires signatures on all message traffic between Ceph daemons comprising the Ceph Storage Cluster. with_legacy: true - name: cephx_cluster_require_version type: int level: advanced desc: Cephx version required by the cluster from clients (1 = pre-mimic, 2 = mimic+) default: 2 with_legacy: true - name: cephx_service_require_signatures type: bool level: advanced default: false fmt_desc: If set to ``true``, Ceph requires signatures on all message traffic between Ceph Clients and the Ceph Storage Cluster. with_legacy: true - name: cephx_service_require_version type: int level: advanced desc: Cephx version required from ceph services (1 = pre-mimic, 2 = mimic+) default: 2 with_legacy: true # Default to signing session messages if supported - name: cephx_sign_messages type: bool level: advanced default: true fmt_desc: If the Ceph version supports message signing, Ceph will sign all messages so they are more difficult to spoof. with_legacy: true - name: auth_mon_ticket_ttl type: float level: advanced default: 72_hr with_legacy: true - name: auth_service_ticket_ttl type: float level: advanced default: 1_hr fmt_desc: When the Ceph Storage Cluster sends a Ceph Client a ticket for authentication, the Ceph Storage Cluster assigns the ticket a time to live. with_legacy: true - name: auth_allow_insecure_global_id_reclaim type: bool level: advanced desc: Allow reclaiming global_id without presenting a valid ticket proving previous possession of that global_id long_desc: Allowing unauthorized global_id (re)use poses a security risk. Unfortunately, older clients may omit their ticket on reconnects and therefore rely on this being allowed for preserving their global_id for the lifetime of the client instance. Setting this value to false would immediately prevent new connections from those clients (assuming auth_expose_insecure_global_id_reclaim set to true) and eventually break existing sessions as well (regardless of auth_expose_insecure_global_id_reclaim setting). default: true see_also: - mon_warn_on_insecure_global_id_reclaim - mon_warn_on_insecure_global_id_reclaim_allowed - auth_expose_insecure_global_id_reclaim with_legacy: true - name: auth_expose_insecure_global_id_reclaim type: bool level: advanced desc: Force older clients that may omit their ticket on reconnects to reconnect as part of establishing a session long_desc: 'In permissive mode (auth_allow_insecure_global_id_reclaim set to true), this helps with identifying clients that are not patched. In enforcing mode (auth_allow_insecure_global_id_reclaim set to false), this is a fail-fast mechanism: don''t establish a session that will almost inevitably be broken later.' default: true see_also: - mon_warn_on_insecure_global_id_reclaim - mon_warn_on_insecure_global_id_reclaim_allowed - auth_allow_insecure_global_id_reclaim with_legacy: true # if true, assert when weird things happen - name: auth_debug type: bool level: dev default: false with_legacy: true # how many mons to try to connect to in parallel during hunt - name: mon_client_hunt_parallel type: uint level: advanced default: 3 with_legacy: true # try new mon every N seconds until we connect - name: mon_client_hunt_interval type: float level: advanced default: 3 fmt_desc: The client will try a new monitor every ``N`` seconds until it establishes a connection. with_legacy: true # send logs every N seconds - name: mon_client_log_interval type: float level: advanced desc: How frequently we send queued cluster log messages to mon default: 1 with_legacy: true # ping every N seconds - name: mon_client_ping_interval type: float level: advanced default: 10 fmt_desc: The client will ping the monitor every ``N`` seconds. with_legacy: true # fail if we don't hear back - name: mon_client_ping_timeout type: float level: advanced default: 30 with_legacy: true - name: mon_client_hunt_interval_backoff type: float level: advanced default: 1.5 with_legacy: true - name: mon_client_hunt_interval_min_multiple type: float level: advanced default: 1 with_legacy: true - name: mon_client_hunt_interval_max_multiple type: float level: advanced default: 10 with_legacy: true - name: mon_client_max_log_entries_per_message type: int level: advanced default: 1000 fmt_desc: The maximum number of log entries a monitor will generate per client message. with_legacy: true - name: mon_client_directed_command_retry type: int level: dev desc: Number of times to try sending a command directed at a specific monitor default: 2 with_legacy: true # whitespace-separated list of key=value pairs describing crush location - name: crush_location type: str level: advanced with_legacy: true - name: crush_location_hook type: str level: advanced with_legacy: true - name: crush_location_hook_timeout type: int level: advanced default: 10 with_legacy: true - name: objecter_tick_interval type: float level: dev default: 5 with_legacy: true # before we ask for a map - name: objecter_timeout type: float level: advanced desc: Seconds before in-flight op is considered 'laggy' and we query mon for the latest OSDMap default: 10 with_legacy: true - name: objecter_inflight_op_bytes type: size level: advanced desc: Max in-flight data in bytes (both directions) default: 100_M with_legacy: true - name: objecter_inflight_ops type: uint level: advanced desc: Max in-flight operations default: 1_K with_legacy: true # num of completion locks per each session, for serializing same object responses - name: objecter_completion_locks_per_session type: uint level: dev default: 32 with_legacy: true # suppress watch pings - name: objecter_inject_no_watch_ping type: bool level: dev default: false with_legacy: true # ignore the first reply for each write, and resend the osd op instead - name: objecter_retry_writes_after_first_reply type: bool level: dev default: false with_legacy: true - name: objecter_debug_inject_relock_delay type: bool level: dev default: false with_legacy: true - name: filer_max_purge_ops type: uint level: advanced desc: Max in-flight operations for purging a striped range (e.g., MDS journal) default: 10 with_legacy: true - name: filer_max_truncate_ops type: uint level: advanced desc: Max in-flight operations for truncating/deleting a striped sequence (e.g., MDS journal) default: 128 with_legacy: true - name: journaler_write_head_interval type: int level: advanced desc: Interval in seconds between journal header updates (to help bound replay time) default: 15 # * journal object size - name: journaler_prefetch_periods type: uint level: advanced desc: Number of striping periods to prefetch while reading MDS journal default: 10 # we need at least 2 periods to make progress. min: 2 # * journal object size - name: journaler_prezero_periods type: uint level: advanced desc: Number of striping periods to zero head of MDS journal write position default: 5 # we need to zero at least two periods, minimum, to ensure that we # have a full empty object/period in front of us. min: 2 - name: osd_calc_pg_upmaps_aggressively type: bool level: advanced desc: try to calculate PG upmaps more aggressively, e.g., by doing a fairly exhaustive search of existing PGs that can be unmapped or upmapped default: true flags: - runtime - name: osd_calc_pg_upmaps_aggressively_fast type: bool level: advanced desc: Prevent very long (>10 minutes) calculations in some extreme cases (applicable only to aggressive mode) default: true flags: - runtime - name: osd_calc_pg_upmaps_local_fallback_retries type: uint level: advanced desc: 'Maximum number of PGs we can attempt to unmap or upmap for a specific overfull or underfull osd per iteration ' default: 100 flags: - runtime # 1 = host - name: osd_crush_chooseleaf_type type: int level: dev desc: default chooseleaf type for osdmaptool --create fmt_desc: The bucket type to use for ``chooseleaf`` in a CRUSH rule. Uses ordinal rank rather than name. default: 1 flags: - cluster_create with_legacy: true # try to use gmt for hitset archive names if all osds in cluster support it - name: osd_pool_use_gmt_hitset type: bool level: dev desc: use UTC for hitset timestamps long_desc: This setting only exists for compatibility with hammer (and older) clusters. default: true with_legacy: true # whether turn on fast read on the pool or not - name: osd_pool_default_ec_fast_read type: bool level: advanced desc: set ec_fast_read for new erasure-coded pools fmt_desc: Whether to turn on fast read on the pool or not. It will be used as the default setting of newly created erasure coded pools if ``fast_read`` is not specified at create time. default: false services: - mon with_legacy: true - name: osd_pool_default_crush_rule type: int level: advanced desc: CRUSH rule for newly created pools fmt_desc: The default CRUSH rule to use when creating a replicated pool. The default value of ``-1`` means "pick the rule with the lowest numerical ID and use that". This is to make pool creation work in the absence of rule 0. default: -1 services: - mon - name: osd_pool_default_size type: uint level: advanced desc: the number of copies of an object for new replicated pools fmt_desc: Sets the number of replicas for objects in the pool. The default value is the same as ``ceph osd pool set {pool-name} size {size}``. default: 3 services: - mon min: 0 max: 10 flags: - runtime - name: osd_pool_default_min_size type: uint level: advanced desc: the minimal number of copies allowed to write to a degraded pool for new replicated pools long_desc: 0 means no specific default; ceph will use size-size/2 fmt_desc: Sets the minimum number of written replicas for objects in the pool in order to acknowledge an I/O operation to the client. If minimum is not met, Ceph will not acknowledge the I/O to the client, **which may result in data loss**. This setting ensures a minimum number of replicas when operating in ``degraded`` mode. The default value is ``0`` which means no particular minimum. If ``0``, minimum is ``size - (size / 2)``. default: 0 services: - mon see_also: - osd_pool_default_size min: 0 max: 255 flags: - runtime - name: osd_pool_default_pg_num type: uint level: advanced desc: number of PGs for new pools fmt_desc: The default number of placement groups for a pool. The default value is the same as ``pg_num`` with ``mkpool``. long_desc: With default value of `osd_pool_default_pg_autoscale_mode` being `on` the number of PGs for new pools will start out with 1 pg, unless the user specifies the pg_num. default: 32 services: - mon see_also: - osd_pool_default_pg_autoscale_mode flags: - runtime - name: osd_pool_default_pgp_num type: uint level: advanced desc: number of PGs for placement purposes (0 to match pg_num) fmt_desc: | The default number of placement groups for placement for a pool. The default value is the same as ``pgp_num`` with ``mkpool``. PG and PGP should be equal (for now). Note: should not be set unless autoscaling is disabled. default: 0 services: - mon see_also: - osd_pool_default_pg_num - osd_pool_default_pg_autoscale_mode flags: - runtime - name: osd_pool_default_type type: str level: advanced desc: default type of pool to create default: replicated services: - mon enum_values: - replicated - erasure flags: - runtime - name: osd_pool_default_erasure_code_profile type: str level: advanced desc: default erasure code profile for new erasure-coded pools default: plugin=jerasure technique=reed_sol_van k=2 m=2 services: - mon flags: - runtime - name: osd_erasure_code_plugins type: str level: advanced desc: erasure code plugins to load default: @osd_erasure_code_plugins@ services: - mon - osd flags: - startup with_legacy: true - name: osd_pool_default_flags type: int level: dev desc: (integer) flags to set on new pools fmt_desc: The default flags for new pools. default: 0 services: - mon with_legacy: true # use new pg hashing to prevent pool/pg overlap - name: osd_pool_default_flag_hashpspool type: bool level: advanced desc: set hashpspool (better hashing scheme) flag on new pools default: true services: - mon with_legacy: true # pool can't be deleted - name: osd_pool_default_flag_nodelete type: bool level: advanced desc: set nodelete flag on new pools fmt_desc: Set the ``nodelete`` flag on new pools, which prevents pool removal. default: false services: - mon with_legacy: true # pool's pg and pgp num can't be changed - name: osd_pool_default_flag_nopgchange type: bool level: advanced desc: set nopgchange flag on new pools fmt_desc: Set the ``nopgchange`` flag on new pools. Does not allow the number of PGs to be changed. default: false services: - mon with_legacy: true # pool's size and min size can't be changed - name: osd_pool_default_flag_nosizechange type: bool level: advanced desc: set nosizechange flag on new pools fmt_desc: Set the ``nosizechange`` flag on new pools. Does not allow the ``size`` to be changed. default: false services: - mon with_legacy: true - name: osd_pool_default_flag_bulk type: bool level: advanced desc: set bulk flag on new pools fmt_desc: Set the ``bulk`` flag on new pools. Allowing autoscaler to use scale-down mode. default: false services: - mon with_legacy: true - name: osd_pool_default_hit_set_bloom_fpp type: float level: advanced default: 0.05 services: - mon see_also: - osd_tier_default_cache_hit_set_type with_legacy: true - name: osd_pool_default_cache_target_dirty_ratio type: float level: advanced default: 0.4 with_legacy: true - name: osd_pool_default_cache_target_dirty_high_ratio type: float level: advanced default: 0.6 with_legacy: true - name: osd_pool_default_cache_target_full_ratio type: float level: advanced default: 0.8 with_legacy: true # seconds - name: osd_pool_default_cache_min_flush_age type: int level: advanced default: 0 with_legacy: true # seconds - name: osd_pool_default_cache_min_evict_age type: int level: advanced default: 0 with_legacy: true # max size to check for eviction - name: osd_pool_default_cache_max_evict_check_size type: int level: advanced default: 10 with_legacy: true - name: osd_pool_default_pg_autoscale_mode type: str level: advanced desc: Default PG autoscaling behavior for new pools long_desc: With default value `on`, the autoscaler starts a new pool with 1 pg, unless the user specifies the pg_num. default: 'on' enum_values: - 'off' - 'warn' - 'on' flags: - runtime - name: osd_pool_default_read_lease_ratio type: float level: dev desc: Default read_lease_ratio for a pool, as a multiple of osd_heartbeat_grace long_desc: This should be <= 1.0 so that the read lease will have expired by the time we decide to mark a peer OSD down. default: 0.8 see_also: - osd_heartbeat_grace flags: - runtime with_legacy: true # min target size for a HitSet - name: osd_hit_set_min_size type: int level: advanced default: 1000 with_legacy: true # max target size for a HitSet - name: osd_hit_set_max_size type: int level: advanced default: 100000 with_legacy: true # rados namespace for hit_set tracking - name: osd_hit_set_namespace type: str level: advanced default: .ceph-internal with_legacy: true # conservative default throttling values - name: osd_tier_promote_max_objects_sec type: uint level: advanced default: 25 with_legacy: true - name: osd_tier_promote_max_bytes_sec type: size level: advanced default: 5_M with_legacy: true - name: osd_tier_default_cache_mode type: str level: advanced default: writeback enum_values: - none - writeback - forward - readonly - readforward - readproxy - proxy flags: - runtime - name: osd_tier_default_cache_hit_set_count type: uint level: advanced default: 4 - name: osd_tier_default_cache_hit_set_period type: uint level: advanced default: 1200 - name: osd_tier_default_cache_hit_set_type type: str level: advanced default: bloom enum_values: - bloom - explicit_hash - explicit_object flags: - runtime - name: osd_tier_default_cache_min_read_recency_for_promote type: uint level: advanced desc: number of recent HitSets the object must appear in to be promoted (on read) default: 1 - name: osd_tier_default_cache_min_write_recency_for_promote type: uint level: advanced desc: number of recent HitSets the object must appear in to be promoted (on write) default: 1 - name: osd_tier_default_cache_hit_set_grade_decay_rate type: uint level: advanced default: 20 - name: osd_tier_default_cache_hit_set_search_last_n type: uint level: advanced default: 1 - name: osd_objecter_finishers type: int level: advanced default: 1 flags: - startup with_legacy: true - name: osd_map_dedup type: bool level: advanced default: true fmt_desc: Enable removing duplicates in the OSD map. with_legacy: true - name: osd_map_message_max type: int level: advanced desc: maximum number of OSDMaps to include in a single message fmt_desc: The maximum map entries allowed per MOSDMap message. default: 40 services: - osd - mon with_legacy: true - name: osd_map_message_max_bytes type: size level: advanced desc: maximum number of bytes worth of OSDMaps to include in a single message default: 10_M services: - osd - mon with_legacy: true # do not assert on divergent_prior entries which aren't in the log and whose on-disk objects are newer - name: osd_ignore_stale_divergent_priors type: bool level: advanced default: false with_legacy: true - name: osd_heartbeat_interval type: int level: dev desc: Interval (in seconds) between peer pings fmt_desc: How often an Ceph OSD Daemon pings its peers (in seconds). default: 6 min: 1 max: 1_min with_legacy: true # (seconds) how long before we decide a peer has failed # This setting is read by the MONs and OSDs and has to be set to a equal value in both settings of the configuration - name: osd_heartbeat_grace type: int level: advanced default: 20 fmt_desc: The elapsed time when a Ceph OSD Daemon hasn't shown a heartbeat that the Ceph Storage Cluster considers it ``down``. This setting must be set in both the [mon] and [osd] or [global] sections so that it is read by both monitor and OSD daemons. with_legacy: true - name: osd_heartbeat_stale type: int level: advanced desc: Interval (in seconds) we mark an unresponsive heartbeat peer as stale. long_desc: Automatically mark unresponsive heartbeat sessions as stale and tear them down. The primary benefit is that OSD doesn't need to keep a flood of blocked heartbeat messages around in memory. default: 10_min # prio the heartbeat tcp socket and set dscp as CS6 on it if true - name: osd_heartbeat_use_min_delay_socket type: bool level: advanced default: false with_legacy: true # the minimum size of OSD heartbeat messages to send - name: osd_heartbeat_min_size type: size level: advanced desc: Minimum heartbeat packet size in bytes. Will add dummy payload if heartbeat packet is smaller than this. default: 2000 with_legacy: true # max number of parallel snap trims/pg - name: osd_pg_max_concurrent_snap_trims type: uint level: advanced default: 2 min: 1 with_legacy: true # max number of trimming pgs - name: osd_max_trimming_pgs type: uint level: advanced default: 2 with_legacy: true # minimum number of peers that must be reachable to mark ourselves # back up after being wrongly marked down. - name: osd_heartbeat_min_healthy_ratio type: float level: advanced default: 0.33 with_legacy: true # (seconds) how often to ping monitor if no peers - name: osd_mon_heartbeat_interval type: int level: advanced default: 30 fmt_desc: How often the Ceph OSD Daemon pings a Ceph Monitor if it has no Ceph OSD Daemon peers. with_legacy: true - name: osd_mon_heartbeat_stat_stale type: int level: advanced desc: Stop reporting on heartbeat ping times not updated for this many seconds. long_desc: Stop reporting on old heartbeat information unless this is set to zero fmt_desc: Stop reporting on heartbeat ping times which haven't been updated for this many seconds. Set to zero to disable this action. default: 1_hr # failures, up_thru, boot. - name: osd_mon_report_interval type: int level: advanced desc: Frequency of OSD reports to mon for peer failures, fullness status changes fmt_desc: The number of seconds a Ceph OSD Daemon may wait from startup or another reportable event before reporting to a Ceph Monitor. default: 5 with_legacy: true # max updates in flight - name: osd_mon_report_max_in_flight type: int level: advanced default: 2 with_legacy: true # (second) how often to send beacon message to monitor - name: osd_beacon_report_interval type: int level: advanced default: 5_min with_legacy: true # report pg stats for any given pg at least this often - name: osd_pg_stat_report_interval_max type: int level: advanced default: 500 with_legacy: true # Max number of snap intervals to report to mgr in pg_stat_t - name: osd_max_snap_prune_intervals_per_epoch type: uint level: dev desc: Max number of snap intervals to report to mgr in pg_stat_t default: 512 with_legacy: true - name: osd_default_data_pool_replay_window type: int level: advanced default: 45 fmt_desc: The time (in seconds) for an OSD to wait for a client to replay a request. - name: osd_auto_mark_unfound_lost type: bool level: advanced default: false with_legacy: true - name: osd_check_for_log_corruption type: bool level: advanced default: false fmt_desc: Check log files for corruption. Can be computationally expensive. with_legacy: true - name: osd_use_stale_snap type: bool level: advanced default: false with_legacy: true - name: osd_rollback_to_cluster_snap type: str level: advanced with_legacy: true - name: osd_default_notify_timeout type: uint level: advanced desc: default number of seconds after which notify propagation times out. used if a client has not specified other value fmt_desc: The OSD default notification timeout (in seconds). default: 30 with_legacy: true - name: osd_kill_backfill_at type: int level: dev default: 0 with_legacy: true # Bounds how infrequently a new map epoch will be persisted for a pg # make this < map_cache_size! - name: osd_pg_epoch_persisted_max_stale type: uint level: advanced default: 40 with_legacy: true - name: osd_target_pg_log_entries_per_osd type: uint level: dev desc: target number of PG entries total on an OSD - limited per pg by the min and max options below default: 300000 see_also: - osd_max_pg_log_entries - osd_min_pg_log_entries with_legacy: true - name: osd_min_pg_log_entries type: uint level: dev desc: minimum number of entries to maintain in the PG log fmt_desc: The minimum number of placement group logs to maintain when trimming log files. default: 250 services: - osd see_also: - osd_max_pg_log_entries - osd_pg_log_dups_tracked - osd_target_pg_log_entries_per_osd with_legacy: true - name: osd_max_pg_log_entries type: uint level: dev desc: maximum number of entries to maintain in the PG log fmt_desc: The maximum number of placement group logs to maintain when trimming log files. default: 10000 services: - osd see_also: - osd_min_pg_log_entries - osd_pg_log_dups_tracked - osd_target_pg_log_entries_per_osd with_legacy: true - name: osd_pg_log_dups_tracked type: uint level: dev desc: how many versions back to track in order to detect duplicate ops; this is combined with both the regular pg log entries and additional minimal dup detection entries default: 3000 services: - osd see_also: - osd_min_pg_log_entries - osd_max_pg_log_entries with_legacy: true - name: osd_object_clean_region_max_num_intervals type: int level: dev desc: number of intervals in clean_offsets long_desc: partial recovery uses multiple intervals to record the clean part of the objectwhen the number of intervals is greater than osd_object_clean_region_max_num_intervals, minimum interval will be trimmed(0 will recovery the entire object data interval) default: 10 services: - osd with_legacy: true # max entries factor before force recovery - name: osd_force_recovery_pg_log_entries_factor type: float level: dev default: 1.3 with_legacy: true - name: osd_pg_log_trim_min type: uint level: dev desc: Minimum number of log entries to trim at once. This lets us trim in larger batches rather than with each write. default: 100 see_also: - osd_max_pg_log_entries - osd_min_pg_log_entries with_legacy: true - name: osd_force_auth_primary_missing_objects type: uint level: advanced desc: Approximate missing objects above which to force auth_log_shard to be primary temporarily default: 100 - name: osd_async_recovery_min_cost type: uint level: advanced desc: A mixture measure of number of current log entries difference and historical missing objects, above which we switch to use asynchronous recovery when appropriate default: 100 flags: - runtime - name: osd_max_pg_per_osd_hard_ratio type: float level: advanced desc: Maximum number of PG per OSD, a factor of 'mon_max_pg_per_osd' long_desc: OSD will refuse to instantiate PG if the number of PG it serves exceeds this number. fmt_desc: The ratio of number of PGs per OSD allowed by the cluster before the OSD refuses to create new PGs. An OSD stops creating new PGs if the number of PGs it serves exceeds ``osd_max_pg_per_osd_hard_ratio`` \* ``mon_max_pg_per_osd``. default: 3 see_also: - mon_max_pg_per_osd min: 1 - name: osd_pg_log_trim_max type: uint level: advanced desc: maximum number of entries to remove at once from the PG log default: 10000 services: - osd see_also: - osd_min_pg_log_entries - osd_max_pg_log_entries with_legacy: true # how many seconds old makes an op complaint-worthy - name: osd_op_complaint_time type: float level: advanced default: 30 fmt_desc: An operation becomes complaint worthy after the specified number of seconds have elapsed. with_legacy: true - name: osd_command_max_records type: int level: advanced default: 256 fmt_desc: Limits the number of lost objects to return. with_legacy: true # max peer osds to report that are blocking our progress - name: osd_max_pg_blocked_by type: uint level: advanced default: 16 with_legacy: true - name: osd_op_log_threshold type: int level: advanced default: 5 fmt_desc: How many operations logs to display at once. with_legacy: true - name: osd_backoff_on_unfound type: bool level: advanced default: true with_legacy: true # [mainly for debug?] object unreadable/writeable - name: osd_backoff_on_degraded type: bool level: advanced default: false with_legacy: true # [debug] pg peering - name: osd_backoff_on_peering type: bool level: advanced default: false with_legacy: true - name: osd_debug_shutdown type: bool level: dev desc: Turn up debug levels during shutdown default: false with_legacy: true # crash osd if client ignores a backoff; useful for debugging - name: osd_debug_crash_on_ignored_backoff type: bool level: dev default: false with_legacy: true - name: osd_debug_inject_dispatch_delay_probability type: float level: dev default: 0 with_legacy: true - name: osd_debug_inject_dispatch_delay_duration type: float level: dev default: 0.1 with_legacy: true - name: osd_debug_drop_ping_probability desc: N/A type: float level: dev default: 0 with_legacy: true - name: osd_debug_drop_ping_duration desc: N/A type: int level: dev default: 0 with_legacy: true - name: osd_debug_op_order type: bool level: dev default: false with_legacy: true - name: osd_debug_verify_missing_on_start type: bool level: dev default: false with_legacy: true - name: osd_debug_verify_snaps type: bool level: dev default: false with_legacy: true - name: osd_debug_verify_stray_on_activate type: bool level: dev default: false with_legacy: true - name: osd_debug_skip_full_check_in_backfill_reservation type: bool level: dev default: false with_legacy: true - name: osd_debug_reject_backfill_probability type: float level: dev default: 0 with_legacy: true # inject failure during copyfrom completion - name: osd_debug_inject_copyfrom_error type: bool level: dev default: false with_legacy: true - name: osd_debug_misdirected_ops type: bool level: dev default: false with_legacy: true - name: osd_debug_skip_full_check_in_recovery type: bool level: dev default: false with_legacy: true - name: osd_debug_random_push_read_error type: float level: dev default: 0 with_legacy: true - name: osd_debug_verify_cached_snaps type: bool level: dev default: false with_legacy: true - name: osd_debug_deep_scrub_sleep type: float level: dev desc: Inject an expensive sleep during deep scrub IO to make it easier to induce preemption default: 0 with_legacy: true - name: osd_debug_no_acting_change type: bool level: dev default: false with_legacy: true - name: osd_debug_no_purge_strays type: bool level: dev default: false with_legacy: true - name: osd_debug_pretend_recovery_active type: bool level: dev default: false with_legacy: true # enable/disable OSD op tracking - name: osd_enable_op_tracker type: bool level: advanced default: true with_legacy: true # The number of shards for holding the ops - name: osd_num_op_tracker_shard type: uint level: advanced default: 32 with_legacy: true # Max number of completed ops to track - name: osd_op_history_size type: uint level: advanced default: 20 fmt_desc: The maximum number of completed operations to track. with_legacy: true # Oldest completed op to track - name: osd_op_history_duration type: uint level: advanced default: 600 fmt_desc: The oldest completed operation to track. with_legacy: true # Max number of slow ops to track - name: osd_op_history_slow_op_size type: uint level: advanced default: 20 with_legacy: true # track the op if over this threshold - name: osd_op_history_slow_op_threshold type: float level: advanced default: 10 with_legacy: true # to adjust various transactions that batch smaller items - name: osd_target_transaction_size type: int level: advanced default: 30 with_legacy: true # what % full makes an OSD "full" (failsafe) - name: osd_failsafe_full_ratio type: float level: advanced default: 0.97 with_legacy: true - name: osd_fast_shutdown type: bool level: advanced desc: Fast, immediate shutdown long_desc: Setting this to false makes the OSD do a slower teardown of all state when it receives a SIGINT or SIGTERM or when shutting down for any other reason. That slow shutdown is primarilyy useful for doing memory leak checking with valgrind. default: true with_legacy: true - name: osd_fast_shutdown_timeout type: int level: advanced desc: timeout in seconds for osd fast-shutdown (0 is unlimited) default: 15 with_legacy: true min: 0 - name: osd_fast_shutdown_notify_mon type: bool level: advanced desc: Tell mon about OSD shutdown on immediate shutdown long_desc: Tell the monitor the OSD is shutting down on immediate shutdown. This helps with cluster log messages from other OSDs reporting it immediately failed. default: true see_also: - osd_fast_shutdown - osd_mon_shutdown_timeout with_legacy: true # immediately mark OSDs as down once they refuse to accept connections - name: osd_fast_fail_on_connection_refused type: bool level: advanced default: true fmt_desc: If this option is enabled, crashed OSDs are marked down immediately by connected peers and MONs (assuming that the crashed OSD host survives). Disable it to restore old behavior, at the expense of possible long I/O stalls when OSDs crash in the middle of I/O operations. with_legacy: true - name: osd_pg_object_context_cache_count type: int level: advanced default: 64 with_legacy: true # true if LTTng-UST tracepoints should be enabled - name: osd_tracing type: bool level: advanced default: false with_legacy: true # true if function instrumentation should use LTTng - name: osd_function_tracing type: bool level: advanced default: false with_legacy: true # use fast info attr, if we can - name: osd_fast_info type: bool level: advanced default: true with_legacy: true # determines whether PGLog::check() compares written out log to stored log - name: osd_debug_pg_log_writeout type: bool level: dev default: false with_legacy: true # Max number of loop before we reset thread-pool's handle - name: osd_loop_before_reset_tphandle type: uint level: advanced default: 64 with_legacy: true # default timeout while caling WaitInterval on an empty queue - name: threadpool_default_timeout type: int level: advanced default: 1_min with_legacy: true # default wait time for an empty queue before pinging the hb timeout - name: threadpool_empty_queue_max_wait type: int level: advanced default: 2 with_legacy: true - name: leveldb_log_to_ceph_log type: bool level: advanced default: true with_legacy: true - name: leveldb_write_buffer_size type: size level: advanced default: 8_M with_legacy: true - name: leveldb_cache_size type: size level: advanced default: 128_M with_legacy: true - name: leveldb_block_size type: size level: advanced default: 0 with_legacy: true - name: leveldb_bloom_size type: int level: advanced default: 0 with_legacy: true - name: leveldb_max_open_files type: int level: advanced default: 0 with_legacy: true - name: leveldb_compression type: bool level: advanced default: true with_legacy: true - name: leveldb_paranoid type: bool level: advanced default: false with_legacy: true - name: leveldb_log type: str level: advanced default: /dev/null with_legacy: true - name: leveldb_compact_on_mount type: bool level: advanced default: false with_legacy: true - name: rocksdb_log_to_ceph_log type: bool level: advanced default: true with_legacy: true - name: rocksdb_cache_size type: size level: advanced default: 512_M flags: - runtime with_legacy: true # ratio of cache for row (vs block) - name: rocksdb_cache_row_ratio type: float level: advanced default: 0 with_legacy: true # rocksdb block cache shard bits, 4 bit -> 16 shards - name: rocksdb_cache_shard_bits type: int level: advanced default: 4 with_legacy: true # 'lru' or 'clock' - name: rocksdb_cache_type type: str level: advanced default: binned_lru with_legacy: true - name: rocksdb_block_size type: size level: advanced default: 4_K with_legacy: true # Enabling this will have 5-10% impact on performance for the stats collection - name: rocksdb_perf type: bool level: advanced default: false with_legacy: true # For rocksdb, this behavior will be an overhead of 5%~10%, collected only rocksdb_perf is enabled. - name: rocksdb_collect_compaction_stats type: bool level: advanced default: false with_legacy: true # For rocksdb, this behavior will be an overhead of 5%~10%, collected only rocksdb_perf is enabled. - name: rocksdb_collect_extended_stats type: bool level: advanced default: false with_legacy: true # For rocksdb, this behavior will be an overhead of 5%~10%, collected only rocksdb_perf is enabled. - name: rocksdb_collect_memory_stats type: bool level: advanced default: false with_legacy: true - name: rocksdb_delete_range_threshold type: uint level: advanced desc: The number of keys required to invoke DeleteRange when deleting muliple keys. default: 1_M - name: rocksdb_bloom_bits_per_key type: uint level: advanced desc: Number of bits per key to use for RocksDB's bloom filters. long_desc: 'RocksDB bloom filters can be used to quickly answer the question of whether or not a key may exist or definitely does not exist in a given RocksDB SST file without having to read all keys into memory. Using a higher bit value decreases the likelihood of false positives at the expense of additional disk space and memory consumption when the filter is loaded into RAM. The current default value of 20 was found to provide significant performance gains when getattr calls are made (such as during new object creation in bluestore) without significant memory overhead or cache pollution when combined with rocksdb partitioned index filters. See: https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters for more information.' default: 20 - name: rocksdb_cache_index_and_filter_blocks type: bool level: dev desc: Whether to cache indices and filters in block cache long_desc: By default RocksDB will load an SST file's index and bloom filters into memory when it is opened and remove them from memory when an SST file is closed. Thus, memory consumption by indices and bloom filters is directly tied to the number of concurrent SST files allowed to be kept open. This option instead stores cached indicies and filters in the block cache where they directly compete with other cached data. By default we set this option to true to better account for and bound rocksdb memory usage and keep filters in memory even when an SST file is closed. default: true - name: rocksdb_cache_index_and_filter_blocks_with_high_priority type: bool level: dev desc: Whether to cache indices and filters in the block cache with high priority long_desc: A downside of setting rocksdb_cache_index_and_filter_blocks to true is that regular data can push indices and filters out of memory. Setting this option to true means they are cached with higher priority than other data and should typically stay in the block cache. default: false - name: rocksdb_pin_l0_filter_and_index_blocks_in_cache type: bool level: dev desc: Whether to pin Level 0 indices and bloom filters in the block cache long_desc: A downside of setting rocksdb_cache_index_and_filter_blocks to true is that regular data can push indices and filters out of memory. Setting this option to true means that level 0 SST files will always have their indices and filters pinned in the block cache. default: false - name: rocksdb_index_type type: str level: dev desc: 'Type of index for SST files: binary_search, hash_search, two_level' long_desc: 'This option controls the table index type. binary_search is a space efficient index block that is optimized for block-search-based index. hash_search may improve prefix lookup performance at the expense of higher disk and memory usage and potentially slower compactions. two_level is an experimental index type that uses two binary search indexes and works in conjunction with partition filters. See: http://rocksdb.org/blog/2017/05/12/partitioned-index-filter.html' default: binary_search - name: rocksdb_partition_filters type: bool level: dev desc: (experimental) partition SST index/filters into smaller blocks long_desc: 'This is an experimental option for rocksdb that works in conjunction with two_level indices to avoid having to keep the entire filter/index in cache when cache_index_and_filter_blocks is true. The idea is to keep a much smaller top-level index in heap/cache and then opportunistically cache the lower level indices. See: https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters' default: false - name: rocksdb_metadata_block_size type: size level: dev desc: The block size for index partitions. (0 = rocksdb default) default: 4_K # osd_*_priority adjust the relative priority of client io, recovery io, # snaptrim io, etc # # osd_*_priority determines the ratio of available io between client and # recovery. Each option may be set between # 1..63. - name: rocksdb_cf_compact_on_deletion type: bool level: dev desc: Compact the column family when a certain number of tombstones are observed within a given window. long_desc: 'This setting instructs RocksDB to compact a column family when a certain number of tombstones are observed during iteration within a certain sliding window. For instance if rocksdb_cf_compact_on_deletion_sliding_window is 8192 and rocksdb_cf_compact_on_deletion_trigger is 4096, then once 4096 tombstones are observed after iteration over 8192 entries, the column family will be compacted.' default: true with_legacy: true see_also: - rocksdb_cf_compact_on_deletion_sliding_window - rocksdb_cf_compact_on_deletion_trigger - name: rocksdb_cf_compact_on_deletion_sliding_window type: int level: dev desc: The sliding window to use when rocksdb_cf_compact_on_deletion is enabled. default: 32768 with_legacy: true see_also: - rocksdb_cf_compact_on_deletion - name: rocksdb_cf_compact_on_deletion_trigger type: int level: dev desc: The trigger to use when rocksdb_cf_compact_on_deletion is enabled. default: 16384 with_legacy: true see_also: - rocksdb_cf_compact_on_deletion - name: osd_client_op_priority type: uint level: advanced default: 63 fmt_desc: The priority set for client operations. This value is relative to that of ``osd_recovery_op_priority`` below. The default strongly favors client ops over recovery. with_legacy: true - name: osd_recovery_op_priority type: uint level: advanced desc: Priority to use for recovery operations if not specified for the pool fmt_desc: The priority of recovery operations vs client operations, if not specified by the pool's ``recovery_op_priority``. The default value prioritizes client ops (see above) over recovery ops. You may adjust the tradeoff of client impact against the time to restore cluster health by lowering this value for increased prioritization of client ops, or by increasing it to favor recovery. default: 3 with_legacy: true - name: osd_peering_op_priority type: uint level: dev default: 255 with_legacy: true - name: osd_snap_trim_priority type: uint level: advanced default: 5 fmt_desc: The priority set for the snap trim work queue. with_legacy: true - name: osd_snap_trim_cost type: size level: advanced default: 1_M with_legacy: true - name: osd_pg_delete_priority type: uint level: advanced default: 5 with_legacy: true - name: osd_pg_delete_cost type: size level: advanced default: 1_M with_legacy: true - name: osd_scrub_priority type: uint level: advanced desc: Priority for scrub operations in work queue fmt_desc: The default work queue priority for scheduled scrubs when the pool doesn't specify a value of ``scrub_priority``. This can be boosted to the value of ``osd_client_op_priority`` when scrubs are blocking client operations. default: 5 with_legacy: true - name: osd_scrub_cost type: size level: advanced desc: Cost for scrub operations in work queue default: 50_M with_legacy: true - name: osd_scrub_event_cost type: size level: advanced desc: Cost for each scrub operation, used when osd_op_queue=mclock_scheduler default: 4_K with_legacy: true # set requested scrub priority higher than scrub priority to make the # requested scrubs jump the queue of scheduled scrubs - name: osd_requested_scrub_priority type: uint level: advanced default: 120 fmt_desc: The priority set for user requested scrub on the work queue. If this value were to be smaller than ``osd_client_op_priority`` it can be boosted to the value of ``osd_client_op_priority`` when scrub is blocking client operations. with_legacy: true - name: osd_recovery_priority type: uint level: advanced desc: Priority of recovery in the work queue long_desc: Not related to a pool's recovery_priority fmt_desc: The default priority set for recovery work queue. Not related to a pool's ``recovery_priority``. default: 5 with_legacy: true # set default cost equal to 20MB io - name: osd_recovery_cost type: size level: advanced default: 20_M with_legacy: true # osd_recovery_op_warn_multiple scales the normal warning threshold, # osd_op_complaint_time, so that slow recovery ops won't cause noise - name: osd_recovery_op_warn_multiple type: uint level: advanced default: 16 with_legacy: true # Max time to wait between notifying mon of shutdown and shutting down - name: osd_mon_shutdown_timeout type: float level: advanced default: 5 with_legacy: true # crash if the OSD has stray PG refs on shutdown - name: osd_shutdown_pgref_assert type: bool level: advanced default: false with_legacy: true # OSD's maximum object size - name: osd_max_object_size type: size level: advanced default: 128_M fmt_desc: The maximum size of a RADOS object in bytes. with_legacy: true # max rados object name len - name: osd_max_object_name_len type: uint level: advanced default: 2_K with_legacy: true # max rados object namespace len - name: osd_max_object_namespace_len type: uint level: advanced default: 256 with_legacy: true # max rados attr name len; cannot go higher than 100 chars for file system backends - name: osd_max_attr_name_len type: uint level: advanced default: 100 with_legacy: true - name: osd_max_attr_size type: uint level: advanced default: 0 with_legacy: true - name: osd_max_omap_entries_per_request type: uint level: advanced default: 1_K with_legacy: true - name: osd_max_omap_bytes_per_request type: size level: advanced default: 1_G with_legacy: true # osd_recovery_op_warn_multiple scales the normal warning threshold, # osd_op_complaint_time, so that slow recovery ops won't cause noise - name: osd_max_write_op_reply_len type: size level: advanced desc: Max size of the per-op payload for requests with the RETURNVEC flag set long_desc: This value caps the amount of data (per op; a request may have many ops) that will be sent back to the client and recorded in the PG log. default: 64 with_legacy: true - name: osd_objectstore type: str level: advanced desc: backend type for an OSD (like filestore or bluestore) default: bluestore enum_values: - bluestore - filestore - memstore - kstore - seastore - cyanstore flags: - create with_legacy: true # true if LTTng-UST tracepoints should be enabled - name: osd_objectstore_tracing type: bool level: advanced default: false with_legacy: true - name: osd_objectstore_fuse type: bool level: advanced default: false with_legacy: true - name: osd_bench_small_size_max_iops type: uint level: advanced default: 100 with_legacy: true - name: osd_bench_large_size_max_throughput type: size level: advanced default: 100_M with_legacy: true - name: osd_bench_max_block_size type: size level: advanced default: 64_M with_legacy: true # duration of 'osd bench', capped at 30s to avoid triggering timeouts - name: osd_bench_duration type: uint level: advanced default: 30 with_legacy: true # create a blkin trace for all osd requests - name: osd_blkin_trace_all type: bool level: advanced default: false with_legacy: true # create a blkin trace for all objecter requests - name: osdc_blkin_trace_all type: bool level: advanced default: false with_legacy: true - name: osd_discard_disconnected_ops type: bool level: advanced default: true with_legacy: true - name: osd_memory_target type: size level: basic desc: When tcmalloc and cache autotuning is enabled, try to keep this many bytes mapped in memory. long_desc: The minimum value must be at least equal to osd_memory_base + osd_memory_cache_min. fmt_desc: | When TCMalloc is available and cache autotuning is enabled, try to keep this many bytes mapped in memory. Note: This may not exactly match the RSS memory usage of the process. While the total amount of heap memory mapped by the process should usually be close to this target, there is no guarantee that the kernel will actually reclaim memory that has been unmapped. During initial development, it was found that some kernels result in the OSD's RSS memory exceeding the mapped memory by up to 20%. It is hypothesised however, that the kernel generally may be more aggressive about reclaiming unmapped memory when there is a high amount of memory pressure. Your mileage may vary. default: 4_G see_also: - bluestore_cache_autotune - osd_memory_cache_min - osd_memory_base - osd_memory_target_autotune min: 896_M flags: - runtime - name: osd_memory_target_autotune type: bool default: false level: advanced desc: If enabled, allow orchestrator to automatically tune osd_memory_target see_also: - osd_memory_target - name: osd_memory_target_cgroup_limit_ratio type: float level: advanced desc: Set the default value for osd_memory_target to the cgroup memory limit (if set) times this value long_desc: A value of 0 disables this feature. default: 0.8 see_also: - osd_memory_target min: 0 max: 1 - name: osd_memory_base type: size level: dev desc: When tcmalloc and cache autotuning is enabled, estimate the minimum amount of memory in bytes the OSD will need. fmt_desc: When TCMalloc and cache autotuning are enabled, estimate the minimum amount of memory in bytes the OSD will need. This is used to help the autotuner estimate the expected aggregate memory consumption of the caches. default: 768_M see_also: - bluestore_cache_autotune flags: - runtime - name: osd_memory_expected_fragmentation type: float level: dev desc: When tcmalloc and cache autotuning is enabled, estimate the percent of memory fragmentation. fmt_desc: When TCMalloc and cache autotuning is enabled, estimate the percentage of memory fragmentation. This is used to help the autotuner estimate the expected aggregate memory consumption of the caches. default: 0.15 see_also: - bluestore_cache_autotune min: 0 max: 1 flags: - runtime - name: osd_memory_cache_min type: size level: dev desc: When tcmalloc and cache autotuning is enabled, set the minimum amount of memory used for caches. fmt_desc: | When TCMalloc and cache autotuning are enabled, set the minimum amount of memory used for caches. Note: Setting this value too low can result in significant cache thrashing. default: 128_M see_also: - bluestore_cache_autotune min: 128_M flags: - runtime - name: osd_memory_cache_resize_interval type: float level: dev desc: When tcmalloc and cache autotuning is enabled, wait this many seconds between resizing caches. fmt_desc: When TCMalloc and cache autotuning are enabled, wait this many seconds between resizing caches. This setting changes the total amount of memory available for BlueStore to use for caching. Note that setting this interval too small can result in memory allocator thrashing and lower performance. default: 1 see_also: - bluestore_cache_autotune - name: memstore_device_bytes type: size level: advanced default: 1_G with_legacy: true - name: memstore_page_set type: bool level: advanced default: false with_legacy: true - name: memstore_page_size type: size level: advanced default: 64_K with_legacy: true - name: memstore_debug_omit_block_device_write type: bool level: dev desc: write metadata only default: false see_also: - bluestore_debug_omit_block_device_write with_legacy: true - name: objectstore_blackhole type: bool level: advanced default: false with_legacy: true - name: bdev_debug_inflight_ios type: bool level: dev default: false with_legacy: true # if N>0, then ~ 1/N IOs will complete before we crash on flush - name: bdev_inject_crash type: int level: dev default: 0 with_legacy: true # wait N more seconds on flush - name: bdev_inject_crash_flush_delay type: int level: dev default: 2 with_legacy: true - name: bdev_aio type: bool level: advanced default: true with_legacy: true # milliseconds - name: bdev_aio_poll_ms type: int level: advanced default: 250 with_legacy: true - name: bdev_aio_max_queue_depth type: int level: advanced default: 1024 with_legacy: true - name: bdev_aio_reap_max type: int level: advanced default: 16 with_legacy: true - name: bdev_block_size type: size level: advanced default: 4_K with_legacy: true - name: bdev_read_buffer_alignment type: size level: advanced default: 4_K with_legacy: true - name: bdev_read_preallocated_huge_buffers type: str level: advanced desc: description of pools arrangement for huge page-based read buffers long_desc: Arrangement of preallocated, huge pages-based pools for reading from a KernelDevice. Applied to minimize size of scatter-gather lists sent to NICs. Targets really big buffers (>= 2 or 4 MBs). Keep in mind the system must be configured accordingly (see /proc/sys/vm/nr_hugepages). Otherwise the OSD wil fail early. Beware BlueStore, by default, stores large chunks across many smaller blobs. Increasing bluestore_max_blob_size changes that, and thus allows the data to be read back into small number of huge page-backed buffers. fmt_desc: List of key=value pairs delimited by comma, semicolon or tab. key specifies the targeted read size and must be expressed in bytes. value specifies the number of preallocated buffers. For instance, to preallocate 64 buffers that will be used to serve 2 MB-sized read requests and 128 for 4 MB, someone needs to set "2097152=64,4194304=128". see_also: - bluestore_max_blob_size - name: bdev_debug_aio type: bool level: dev default: false with_legacy: true - name: bdev_debug_aio_suicide_timeout type: float level: dev default: 1_min with_legacy: true - name: bdev_debug_aio_log_age type: float level: dev default: 5 with_legacy: true # if yes, osd will unbind all NVMe devices from kernel driver and bind them # to the uio_pci_generic driver. The purpose is to prevent the case where # NVMe driver is loaded while osd is running. - name: bdev_nvme_unbind_from_kernel type: bool level: advanced default: false with_legacy: true - name: bdev_enable_discard type: bool level: advanced default: false with_legacy: true - name: bdev_async_discard type: bool level: advanced default: false with_legacy: true - name: bdev_flock_retry_interval type: float level: advanced desc: interval to retry the flock default: 0.1 - name: bdev_flock_retry type: uint level: advanced desc: times to retry the flock long_desc: The number of times to retry on getting the block device lock. Programs such as systemd-udevd may compete with Ceph for this lock. 0 means 'unlimited'. default: 3 - name: bluefs_alloc_size type: size level: advanced desc: Allocation unit size for DB and WAL devices default: 1_M with_legacy: true - name: bluefs_shared_alloc_size type: size level: advanced desc: Allocation unit size for primary/shared device default: 64_K with_legacy: true - name: bluefs_failed_shared_alloc_cooldown type: float level: advanced desc: duration(in seconds) untill the next attempt to use 'bluefs_shared_alloc_size' after facing ENOSPC failure. long_desc: Cooldown period(in seconds) when BlueFS uses shared/slow device allocation size instead of "bluefs_shared_alloc_size' one after facing recoverable (via fallback to smaller chunk size) ENOSPC failure. Intended primarily to avoid repetitive unsuccessful allocations which might be expensive. default: 600 with_legacy: true - name: bluefs_max_prefetch type: size level: advanced default: 1_M with_legacy: true # alloc when we get this low - name: bluefs_min_log_runway type: size level: advanced default: 1_M with_legacy: true # alloc this much at a time - name: bluefs_max_log_runway type: size level: advanced default: 4_M with_legacy: true # before we consider - name: bluefs_log_compact_min_ratio type: float level: advanced default: 5 with_legacy: true # before we consider - name: bluefs_log_compact_min_size type: size level: advanced default: 16_M with_legacy: true # ignore flush until its this big - name: bluefs_min_flush_size type: size level: advanced default: 512_K with_legacy: true # sync or async log compaction - name: bluefs_compact_log_sync type: bool level: advanced default: false with_legacy: true - name: bluefs_buffered_io type: bool level: advanced desc: Enabled buffered IO for bluefs reads. long_desc: When this option is enabled, bluefs will in some cases perform buffered reads. This allows the kernel page cache to act as a secondary cache for things like RocksDB block reads. For example, if the rocksdb block cache isn't large enough to hold all blocks during OMAP iteration, it may be possible to read them from page cache instead of from the disk. This can dramatically improve performance when the osd_memory_target is too small to hold all entries in block cache but it does come with downsides. It has been reported to occasionally cause excessive kernel swapping (and associated stalls) under certain workloads. Currently the best and most consistent performing combination appears to be enabling bluefs_buffered_io and disabling system level swap. It is possible that this recommendation may change in the future however. default: true with_legacy: true - name: bluefs_sync_write type: bool level: advanced default: false with_legacy: true - name: bluefs_allocator type: str level: dev default: hybrid enum_values: - bitmap - stupid - avl - hybrid with_legacy: true - name: bluefs_log_replay_check_allocations type: bool level: advanced desc: Enables checks for allocations consistency during log replay default: true with_legacy: true - name: bluefs_replay_recovery type: bool level: dev desc: Attempt to read bluefs log so large that it became unreadable. long_desc: If BlueFS log grows to extreme sizes (200GB+) it is likely that it becames unreadable. This options enables heuristics that scans devices for missing data. DO NOT ENABLE BY DEFAULT default: false with_legacy: true - name: bluefs_replay_recovery_disable_compact type: bool level: advanced default: false with_legacy: true - name: bluefs_check_for_zeros type: bool level: dev desc: Check data read for suspicious pages long_desc: Looks into data read to check if there is a 4K block entirely filled with zeros. If this happens, we re-read data. If there is difference, we print error to log. default: false see_also: - bluestore_retry_disk_reads flags: - runtime with_legacy: true - name: bluefs_check_volume_selector_on_umount type: bool level: dev desc: Check validity of volume selector on umount long_desc: Checks if volume selector did not diverge from the state it should be in. Reference is constructed from bluefs inode table. Asserts on inconsistency. default: false flags: - runtime with_legacy: true - name: bluefs_check_volume_selector_often type: bool level: dev desc: Periodically check validity of volume selector long_desc: Periodically checks if current volume selector does not diverge from the valid state. Reference is constructed from bluefs inode table. Asserts on inconsistency. This is debug feature. default: false see_also: - bluefs_check_volume_selector_on_umount flags: - startup with_legacy: true - name: bluestore_bluefs type: bool level: dev desc: Use BlueFS to back rocksdb long_desc: BlueFS allows rocksdb to share the same physical device(s) as the rest of BlueStore. It should be used in all cases unless testing/developing an alternative metadata database for BlueStore. default: true flags: - create with_legacy: true # mirror to normal Env for debug - name: bluestore_bluefs_env_mirror type: bool level: dev desc: Mirror bluefs data to file system for testing/validation default: false flags: - create with_legacy: true - name: bluestore_bluefs_max_free type: size level: advanced default: 10_G desc: Maximum free space allocated to BlueFS - name: bluestore_bluefs_alloc_failure_dump_interval type: float level: advanced desc: How frequently (in seconds) to dump allocator on BlueFS space allocation failure default: 0 with_legacy: true - name: bluestore_spdk_mem type: size level: dev desc: Amount of dpdk memory size in MB long_desc: If running multiple SPDK instances per node, you must specify the amount of dpdk memory size in MB each instance will use, to make sure each instance uses its own dpdk memory default: 512 - name: bluestore_spdk_coremask type: str level: dev desc: A hexadecimal bit mask of the cores to run on. Note the core numbering can change between platforms and should be determined beforehand default: '0x1' - name: bluestore_spdk_max_io_completion type: uint level: dev desc: Maximal I/Os to be batched completed while checking queue pair completions, 0 means let spdk library determine it default: 0 - name: bluestore_spdk_io_sleep type: uint level: dev desc: Time period to wait if there is no completed I/O from polling default: 5 # If you want to use spdk driver, you need to specify NVMe serial number here # with "spdk:" prefix. # Users can use 'lspci -vvv -d 8086:0953 | grep "Device Serial Number"' to # get the serial number of Intel(R) Fultondale NVMe controllers. # Example: # bluestore_block_path = spdk:55cd2e404bd73932 - name: bluestore_block_path type: str level: dev desc: Path to block device/file flags: - create with_legacy: true - name: bluestore_block_size type: size level: dev desc: Size of file to create for backing bluestore default: 100_G flags: - create with_legacy: true - name: bluestore_block_create type: bool level: dev desc: Create bluestore_block_path if it doesn't exist default: true see_also: - bluestore_block_path - bluestore_block_size flags: - create with_legacy: true - name: bluestore_block_db_path type: str level: dev desc: Path for db block device flags: - create with_legacy: true # rocksdb ssts (hot/warm) - name: bluestore_block_db_size type: size level: dev desc: Size of file to create for bluestore_block_db_path default: 0 flags: - create with_legacy: true - name: bluestore_block_db_create type: bool level: dev desc: Create bluestore_block_db_path if it doesn't exist default: false see_also: - bluestore_block_db_path - bluestore_block_db_size flags: - create with_legacy: true - name: bluestore_block_wal_path type: str level: dev desc: Path to block device/file backing bluefs wal flags: - create with_legacy: true # rocksdb wal - name: bluestore_block_wal_size type: size level: dev desc: Size of file to create for bluestore_block_wal_path default: 96_M flags: - create with_legacy: true - name: bluestore_block_wal_create type: bool level: dev desc: Create bluestore_block_wal_path if it doesn't exist default: false see_also: - bluestore_block_wal_path - bluestore_block_wal_size flags: - create with_legacy: true # whether preallocate space if block/db_path/wal_path is file rather that block device. - name: bluestore_block_preallocate_file type: bool level: dev desc: Preallocate file created via bluestore_block*_create default: false flags: - create with_legacy: true - name: bluestore_ignore_data_csum type: bool level: dev desc: Ignore checksum errors on read and do not generate an EIO error default: false flags: - runtime with_legacy: true - name: bluestore_csum_type type: str level: advanced desc: Default checksum algorithm to use long_desc: crc32c, xxhash32, and xxhash64 are available. The _16 and _8 variants use only a subset of the bits for more compact (but less reliable) checksumming. fmt_desc: The default checksum algorithm to use. default: crc32c enum_values: - none - crc32c - crc32c_16 - crc32c_8 - xxhash32 - xxhash64 flags: - runtime with_legacy: true - name: bluestore_retry_disk_reads type: uint level: advanced desc: Number of read retries on checksum validation error long_desc: Retries to read data from the disk this many times when checksum validation fails to handle spurious read errors gracefully. default: 3 min: 0 max: 255 flags: - runtime with_legacy: true - name: bluestore_min_alloc_size type: uint level: advanced desc: Minimum allocation size to allocate for an object long_desc: A smaller allocation size generally means less data is read and then rewritten when a copy-on-write operation is triggered (e.g., when writing to something that was recently snapshotted). Similarly, less data is journaled before performing an overwrite (writes smaller than min_alloc_size must first pass through the BlueStore journal). Larger values of min_alloc_size reduce the amount of metadata required to describe the on-disk layout and reduce overall fragmentation. default: 0 flags: - create with_legacy: true - name: bluestore_min_alloc_size_hdd type: size level: advanced desc: Default min_alloc_size value for rotational media default: 4_K see_also: - bluestore_min_alloc_size flags: - create with_legacy: true - name: bluestore_min_alloc_size_ssd type: size level: advanced desc: Default min_alloc_size value for non-rotational (solid state) media default: 4_K see_also: - bluestore_min_alloc_size flags: - create with_legacy: true - name: bluestore_use_optimal_io_size_for_min_alloc_size type: bool level: advanced desc: Discover media optimal IO Size and use for min_alloc_size default: false see_also: - bluestore_min_alloc_size flags: - create with_legacy: true - name: bluestore_max_alloc_size type: size level: advanced desc: Maximum size of a single allocation (0 for no max) default: 0 flags: - create with_legacy: true - name: bluestore_prefer_deferred_size type: size level: advanced desc: Writes smaller than this size will be written to the journal and then asynchronously written to the device. This can be beneficial when using rotational media where seeks are expensive, and is helpful both with and without solid state journal/wal devices. default: 0 flags: - runtime with_legacy: true - name: bluestore_prefer_deferred_size_hdd type: size level: advanced desc: Default bluestore_prefer_deferred_size for rotational media default: 64_K see_also: - bluestore_prefer_deferred_size flags: - runtime with_legacy: true - name: bluestore_prefer_deferred_size_ssd type: size level: advanced desc: Default bluestore_prefer_deferred_size for non-rotational (solid state) media default: 0 see_also: - bluestore_prefer_deferred_size flags: - runtime with_legacy: true - name: bluestore_compression_mode type: str level: advanced desc: Default policy for using compression when pool does not specify long_desc: '''none'' means never use compression. ''passive'' means use compression when clients hint that data is compressible. ''aggressive'' means use compression unless clients hint that data is not compressible. This option is used when the per-pool property for the compression mode is not present.' fmt_desc: The default policy for using compression if the per-pool property ``compression_mode`` is not set. ``none`` means never use compression. ``passive`` means use compression when :c:func:`clients hint ` that data is compressible. ``aggressive`` means use compression unless clients hint that data is not compressible. ``force`` means use compression under all circumstances even if the clients hint that the data is not compressible. default: none enum_values: - none - passive - aggressive - force flags: - runtime with_legacy: true - name: bluestore_compression_algorithm type: str level: advanced desc: Default compression algorithm to use when writing object data long_desc: This controls the default compressor to use (if any) if the per-pool property is not set. Note that zstd is *not* recommended for bluestore due to high CPU overhead when compressing small amounts of data. fmt_desc: The default compressor to use (if any) if the per-pool property ``compression_algorithm`` is not set. Note that ``zstd`` is *not* recommended for BlueStore due to high CPU overhead when compressing small amounts of data. default: snappy enum_values: - '' - snappy - zlib - zstd - lz4 flags: - runtime with_legacy: true - name: bluestore_compression_min_blob_size type: size level: advanced desc: Maximum chunk size to apply compression to when random access is expected for an object. long_desc: Chunks larger than this are broken into smaller chunks before being compressed fmt_desc: Chunks smaller than this are never compressed. The per-pool property ``compression_min_blob_size`` overrides this setting. default: 0 flags: - runtime with_legacy: true - name: bluestore_compression_min_blob_size_hdd type: size level: advanced desc: Default value of bluestore_compression_min_blob_size for rotational media fmt_desc: Default value of ``bluestore compression min blob size`` for rotational media. default: 8_K see_also: - bluestore_compression_min_blob_size flags: - runtime with_legacy: true - name: bluestore_compression_min_blob_size_ssd type: size level: advanced desc: Default value of bluestore_compression_min_blob_size for non-rotational (solid state) media fmt_desc: Default value of ``bluestore compression min blob size`` for non-rotational (solid state) media. default: 64_K see_also: - bluestore_compression_min_blob_size flags: - runtime with_legacy: true - name: bluestore_compression_max_blob_size type: size level: advanced desc: Maximum chunk size to apply compression to when non-random access is expected for an object. long_desc: Chunks larger than this are broken into smaller chunks before being compressed fmt_desc: Chunks larger than this value are broken into smaller blobs of at most ``bluestore_compression_max_blob_size`` bytes before being compressed. The per-pool property ``compression_max_blob_size`` overrides this setting. default: 0 flags: - runtime with_legacy: true - name: bluestore_compression_max_blob_size_hdd type: size level: advanced desc: Default value of bluestore_compression_max_blob_size for rotational media fmt_desc: Default value of ``bluestore compression max blob size`` for rotational media. default: 64_K see_also: - bluestore_compression_max_blob_size flags: - runtime with_legacy: true - name: bluestore_compression_max_blob_size_ssd type: size level: advanced desc: Default value of bluestore_compression_max_blob_size for non-rotational (solid state) media fmt_desc: Default value of ``bluestore compression max blob size`` for non-rotational (SSD, NVMe) media. default: 64_K see_also: - bluestore_compression_max_blob_size flags: - runtime with_legacy: true # Specifies minimum expected amount of saved allocation units # per single blob to enable compressed blobs garbage collection - name: bluestore_gc_enable_blob_threshold type: int level: dev default: 0 flags: - runtime with_legacy: true # Specifies minimum expected amount of saved allocation units # per all blobsb to enable compressed blobs garbage collection - name: bluestore_gc_enable_total_threshold type: int level: dev default: 0 flags: - runtime with_legacy: true - name: bluestore_max_blob_size type: size level: dev long_desc: Bluestore blobs are collections of extents (ie on-disk data) originating from one or more objects. Blobs can be compressed, typically have checksum data, may be overwritten, may be shared (with an extent ref map), or split. This setting controls the maximum size a blob is allowed to be. default: 0 flags: - runtime with_legacy: true - name: bluestore_max_blob_size_hdd type: size level: dev default: 64_K see_also: - bluestore_max_blob_size flags: - runtime with_legacy: true - name: bluestore_max_blob_size_ssd type: size level: dev default: 64_K see_also: - bluestore_max_blob_size flags: - runtime with_legacy: true # Require the net gain of compression at least to be at this ratio, # otherwise we don't compress. # And ask for compressing at least 12.5%(1/8) off, by default. - name: bluestore_compression_required_ratio type: float level: advanced desc: Compression ratio required to store compressed data long_desc: If we compress data and get less than this we discard the result and store the original uncompressed data. fmt_desc: The ratio of the size of the data chunk after compression relative to the original size must be at least this small in order to store the compressed version. default: 0.875 flags: - runtime with_legacy: true - name: bluestore_extent_map_shard_max_size type: size level: dev desc: Max size (bytes) for a single extent map shard before splitting default: 1200 with_legacy: true - name: bluestore_extent_map_shard_target_size type: size level: dev desc: Target size (bytes) for a single extent map shard default: 500 with_legacy: true - name: bluestore_extent_map_shard_min_size type: size level: dev desc: Min size (bytes) for a single extent map shard before merging default: 150 with_legacy: true - name: bluestore_extent_map_shard_target_size_slop type: float level: dev desc: Ratio above/below target for a shard when trying to align to an existing extent or blob boundary default: 0.2 with_legacy: true - name: bluestore_extent_map_inline_shard_prealloc_size type: size level: dev desc: Preallocated buffer for inline shards default: 256 with_legacy: true - name: bluestore_cache_trim_interval type: float level: advanced desc: How frequently we trim the bluestore cache default: 0.05 with_legacy: true - name: bluestore_cache_trim_max_skip_pinned type: uint level: dev desc: Max pinned cache entries we consider before giving up default: 1000 with_legacy: true - name: bluestore_cache_type type: str level: dev desc: Cache replacement algorithm default: 2q enum_values: - 2q - lru with_legacy: true - name: bluestore_2q_cache_kin_ratio type: float level: dev desc: 2Q paper suggests .5 default: 0.5 with_legacy: true - name: bluestore_2q_cache_kout_ratio type: float level: dev desc: 2Q paper suggests .5 default: 0.5 with_legacy: true - name: bluestore_cache_size type: size level: dev desc: Cache size (in bytes) for BlueStore long_desc: This includes data and metadata cached by BlueStore as well as memory devoted to rocksdb's cache(s). fmt_desc: The amount of memory BlueStore will use for its cache. If zero, ``bluestore_cache_size_hdd`` or ``bluestore_cache_size_ssd`` will be used instead. default: 0 with_legacy: true - name: bluestore_cache_size_hdd type: size level: dev desc: Default bluestore_cache_size for rotational media fmt_desc: The default amount of memory BlueStore will use for its cache when backed by an HDD. default: 1_G see_also: - bluestore_cache_size with_legacy: true - name: bluestore_cache_size_ssd type: size level: dev desc: Default bluestore_cache_size for non-rotational (solid state) media fmt_desc: The default amount of memory BlueStore will use for its cache when backed by an SSD. default: 3_G see_also: - bluestore_cache_size with_legacy: true - name: bluestore_cache_meta_ratio type: float level: dev desc: Ratio of bluestore cache to devote to metadata default: 0.45 see_also: - bluestore_cache_size with_legacy: true - name: bluestore_cache_kv_ratio type: float level: dev desc: Ratio of bluestore cache to devote to key/value database (RocksDB) default: 0.45 see_also: - bluestore_cache_size with_legacy: true - name: bluestore_cache_kv_onode_ratio type: float level: dev desc: Ratio of bluestore cache to devote to kv onode column family (rocksdb) default: 0.04 see_also: - bluestore_cache_size - name: bluestore_cache_autotune type: bool level: dev desc: Automatically tune the ratio of caches while respecting min values. fmt_desc: Automatically tune the space ratios assigned to various BlueStore caches while respecting minimum values. default: true see_also: - bluestore_cache_size - bluestore_cache_meta_ratio - name: bluestore_cache_autotune_interval type: float level: dev desc: The number of seconds to wait between rebalances when cache autotune is enabled. fmt_desc: | The number of seconds to wait between rebalances when cache autotune is enabled. `bluestore_cache_autotune_interval` sets the speed at which Ceph recomputes the allocation ratios of various caches. Note: Setting this interval too small can result in high CPU usage and lower performance. default: 5 see_also: - bluestore_cache_autotune - name: bluestore_cache_age_bin_interval type: float level: dev desc: The duration (in seconds) represented by a single cache age bin. fmt_desc: | The caches used by bluestore will assign cache entries to an 'age bin' that represents a period of time during which that cache entry was most recently updated. By binning the caches in this way, Ceph's priority cache balancing code can make better decisions about which caches should receive priority based on the relative ages of items in the caches. By default, a single cache age bin represents 1 second of time. Note: Setting this interval too small can result in high CPU usage and lower performance. default: 1 see_also: - bluestore_cache_age_bins_kv - bluestore_cache_age_bins_kv_onode - bluestore_cache_age_bins_meta - bluestore_cache_age_bins_data - name: bluestore_cache_age_bins_kv type: str level: dev desc: A 10 element, space separated list of age bins for kv cache fmt_desc: | A 10 element, space separated list of cache age bins grouped by priority such that PRI1=[0,n), PRI2=[n,n+1), PRI3=[n+1,n+2) ... PRI10=[n+8,n+9). Values represent the starting and ending bin for each priority level. A 0 in the 2nd term will prevent any items from being associated with that priority. bin duration is based on the bluestore_cache_age_bin_interval value. For example, "1 5 0 0 0 0 0 0 0 0" defines bin ranges for two priority levels. PRI1 contains 1 age bin. Assuming the default age bin interval of 1 second, PRI1 represents cache items that are less than 1 second old. PRI2 has 4 bins representing cache items that are 1 to less than 5 seconds old. All other cache items in this example are associated with the lowest priority level as PRI3-PRI10 all have 0s in their second term. default: "1 2 6 24 120 720 0 0 0 0" see_also: - bluestore_cache_age_bin_interval - name: bluestore_cache_age_bins_kv_onode type: str level: dev desc: A 10 element, space separated list of age bins for kv onode cache fmt_desc: | A 10 element, space separated list of cache age bins grouped by priority such that PRI1=[0,n), PRI2=[n,n+1), PRI3=[n+1,n+2) ... PRI10=[n+8,n+9). Values represent the starting and ending bin for each priority level. A 0 in the 2nd term will prevent any items from being associated with that priority. bin duration is based on the bluestore_cache_age_bin_interval value. For example, "1 5 0 0 0 0 0 0 0 0" defines bin ranges for two priority levels. PRI1 contains 1 age bin. Assuming the default age bin interval of 1 second, PRI1 represents cache items that are less than 1 second old. PRI2 has 4 bins representing cache items that are 1 to less than 5 seconds old. All other cache items in this example are associated with the lowest priority level as PRI3-PRI10 all have 0s in their second term. default: "0 0 0 0 0 0 0 0 0 720" see_also: - bluestore_cache_age_bin_interval - name: bluestore_cache_age_bins_meta type: str level: dev desc: A 10 element, space separated list of age bins for onode cache fmt_desc: | A 10 element, space separated list of cache age bins grouped by priority such that PRI1=[0,n), PRI2=[n,n+1), PRI3=[n+1,n+2) ... PRI10=[n+8,n+9). Values represent the starting and ending bin for each priority level. A 0 in the 2nd term will prevent any items from being associated with that priority. bin duration is based on the bluestore_cache_age_bin_interval value. For example, "1 5 0 0 0 0 0 0 0 0" defines bin ranges for two priority levels. PRI1 contains 1 age bin. Assuming the default age bin interval of 1 second, PRI1 represents cache items that are less than 1 second old. PRI2 has 4 bins representing cache items that are 1 to less than 5 seconds old. All other cache items in this example are associated with the lowest priority level as PRI3-PRI10 all have 0s in their second term. default: "1 2 6 24 120 720 0 0 0 0" see_also: - bluestore_cache_age_bin_interval - name: bluestore_cache_age_bins_data type: str level: dev desc: A 10 element, space separated list of age bins for data cache fmt_desc: | A 10 element, space separated list of cache age bins grouped by priority such that PRI1=[0,n), PRI2=[n,n+1), PRI3=[n+1,n+2) ... PRI10=[n+8,n+9). Values represent the starting and ending bin for each priority level. A 0 in the 2nd term will prevent any items from being associated with that priority. bin duration is based on the bluestore_cache_age_bin_interval value. For example, "1 5 0 0 0 0 0 0 0 0" defines bin ranges for two priority levels. PRI1 contains 1 age bin. Assuming the default age bin interval of 1 second, PRI1 represents cache items that are less than 1 second old. PRI2 has 4 bins representing cache items that are 1 to less than 5 seconds old. All other cache items in this example are associated with the lowest priority level as PRI3-PRI10 all have 0s in their second term. default: "1 2 6 24 120 720 0 0 0 0" see_also: - bluestore_cache_age_bin_interval - name: bluestore_alloc_stats_dump_interval type: float level: dev desc: The period (in second) for logging allocation statistics. default: 1_day with_legacy: true - name: bluestore_kvbackend type: str level: dev desc: Key value database to use for bluestore default: rocksdb flags: - create with_legacy: true - name: bluestore_allocator type: str level: advanced desc: Allocator policy long_desc: Allocator to use for bluestore. Stupid should only be used for testing. default: hybrid enum_values: - bitmap - stupid - avl - hybrid - zoned with_legacy: true - name: bluestore_freelist_blocks_per_key type: size level: dev desc: Block (and bits) per database key default: 128 with_legacy: true - name: bluestore_bitmapallocator_blocks_per_zone type: size level: dev default: 1_K with_legacy: true - name: bluestore_bitmapallocator_span_size type: size level: dev default: 1_K with_legacy: true - name: bluestore_max_deferred_txc type: uint level: advanced desc: Max transactions with deferred writes that can accumulate before we force flush deferred writes default: 32 with_legacy: true - name: bluestore_max_defer_interval type: float level: advanced desc: max duration to force deferred submit default: 3 with_legacy: true - name: bluestore_rocksdb_options type: str level: advanced desc: Full set of rocksdb settings to override default: compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 with_legacy: true - name: bluestore_rocksdb_options_annex type: str level: advanced desc: An addition to bluestore_rocksdb_options. Allows setting rocksdb options without repeating the existing defaults. with_legacy: true - name: bluestore_rocksdb_cf type: bool level: advanced desc: Enable use of rocksdb column families for bluestore metadata fmt_desc: Enables sharding of BlueStore's RocksDB. When ``true``, ``bluestore_rocksdb_cfs`` is used. Only applied when OSD is doing ``--mkfs``. default: true verbatim: | #ifdef WITH_SEASTAR // This is necessary as the Seastar's allocator imposes restrictions // on the number of threads that entered malloc/free/*. Unfortunately, // RocksDB sharding in BlueStore dramatically lifted the number of // threads spawn during RocksDB's init. .set_validator([](std::string *value, std::string *error_message) { if (const bool parsed_value = strict_strtob(value->c_str(), error_message); error_message->empty() && parsed_value) { *error_message = "invalid BlueStore sharding configuration." " Be aware any change takes effect only on mkfs!"; return -EINVAL; } else { return 0; } }) #endif - name: bluestore_rocksdb_cfs type: str level: dev desc: Definition of column families and their sharding long_desc: 'Space separated list of elements: column_def [ ''='' rocksdb_options ]. column_def := column_name [ ''('' shard_count [ '','' hash_begin ''-'' [ hash_end ] ] '')'' ]. Example: ''I=write_buffer_size=1048576 O(6) m(7,10-)''. Interval [hash_begin..hash_end) defines characters to use for hash calculation. Recommended hash ranges: O(0-13) P(0-8) m(0-16). Sharding of S,T,C,M,B prefixes is inadvised' fmt_desc: Definition of BlueStore's RocksDB sharding. The optimal value depends on multiple factors, and modification is invadvisable. This setting is used only when OSD is doing ``--mkfs``. Next runs of OSD retrieve sharding from disk. default: m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 - name: bluestore_qfsck_on_mount type: bool level: dev desc: Run quick-fsck at mount comparing allocation-file to RocksDB allocation state default: true with_legacy: true - name: bluestore_fsck_on_mount type: bool level: dev desc: Run fsck at mount default: false with_legacy: true - name: bluestore_fsck_on_mount_deep type: bool level: dev desc: Run deep fsck at mount when bluestore_fsck_on_mount is set to true default: false with_legacy: true - name: bluestore_fsck_quick_fix_on_mount type: bool level: dev desc: Do quick-fix for the store at mount default: false with_legacy: true - name: bluestore_fsck_on_umount type: bool level: dev desc: Run fsck at umount default: false with_legacy: true - name: bluestore_allocation_from_file type: bool level: dev desc: Remove allocation info from RocksDB and store the info in a new allocation file default: true with_legacy: true - name: bluestore_debug_inject_allocation_from_file_failure type: float level: dev desc: Enables random error injections when restoring allocation map from file. long_desc: Specifies error injection probability for restoring allocation map from file hence causing full recovery. Intended primarily for testing. default: 0 with_legacy: true - name: bluestore_fsck_on_umount_deep type: bool level: dev desc: Run deep fsck at umount when bluestore_fsck_on_umount is set to true default: false with_legacy: true - name: bluestore_fsck_on_mkfs type: bool level: dev desc: Run fsck after mkfs default: true with_legacy: true - name: bluestore_fsck_on_mkfs_deep type: bool level: dev desc: Run deep fsck after mkfs default: false with_legacy: true - name: bluestore_sync_submit_transaction type: bool level: dev desc: Try to submit metadata transaction to rocksdb in queuing thread context default: false with_legacy: true - name: bluestore_fsck_read_bytes_cap type: size level: advanced desc: Maximum bytes read at once by deep fsck default: 64_M flags: - runtime with_legacy: true - name: bluestore_fsck_quick_fix_threads type: int level: advanced desc: Number of additional threads to perform quick-fix (shallow fsck) command default: 2 with_legacy: true - name: bluestore_fsck_shared_blob_tracker_size type: float level: dev desc: Size(a fraction of osd_memory_target, defaults to 128MB) of a hash table to track shared blobs ref counts. Higher the size, more precise is the tracker -> less overhead during the repair. default: 0.03125 see_also: - osd_memory_target flags: - runtime - name: bluestore_throttle_bytes type: size level: advanced desc: Maximum bytes in flight before we throttle IO submission default: 64_M flags: - runtime with_legacy: true - name: bluestore_throttle_deferred_bytes type: size level: advanced desc: Maximum bytes for deferred writes before we throttle IO submission default: 128_M flags: - runtime with_legacy: true - name: bluestore_throttle_cost_per_io type: size level: advanced desc: Overhead added to transaction cost (in bytes) for each IO default: 0 flags: - runtime with_legacy: true - name: bluestore_throttle_cost_per_io_hdd type: uint level: advanced desc: Default bluestore_throttle_cost_per_io for rotational media default: 670000 see_also: - bluestore_throttle_cost_per_io flags: - runtime with_legacy: true - name: bluestore_throttle_cost_per_io_ssd type: uint level: advanced desc: Default bluestore_throttle_cost_per_io for non-rotation (solid state) media default: 4000 see_also: - bluestore_throttle_cost_per_io flags: - runtime with_legacy: true - name: bluestore_deferred_batch_ops type: uint level: advanced desc: Max number of deferred writes before we flush the deferred write queue default: 0 min: 0 max: 65535 flags: - runtime with_legacy: true - name: bluestore_deferred_batch_ops_hdd type: uint level: advanced desc: Default bluestore_deferred_batch_ops for rotational media default: 64 see_also: - bluestore_deferred_batch_ops min: 0 max: 65535 flags: - runtime with_legacy: true - name: bluestore_deferred_batch_ops_ssd type: uint level: advanced desc: Default bluestore_deferred_batch_ops for non-rotational (solid state) media default: 16 see_also: - bluestore_deferred_batch_ops min: 0 max: 65535 flags: - runtime with_legacy: true - name: bluestore_nid_prealloc type: int level: dev desc: Number of unique object ids to preallocate at a time default: 1024 with_legacy: true - name: bluestore_blobid_prealloc type: uint level: dev desc: Number of unique blob ids to preallocate at a time default: 10_K with_legacy: true - name: bluestore_clone_cow type: bool level: advanced desc: Use copy-on-write when cloning objects (versus reading and rewriting them at clone time) default: true flags: - runtime with_legacy: true - name: bluestore_default_buffered_read type: bool level: advanced desc: Cache read results by default (unless hinted NOCACHE or WONTNEED) default: true flags: - runtime with_legacy: true - name: bluestore_default_buffered_write type: bool level: advanced desc: Cache writes by default (unless hinted NOCACHE or WONTNEED) default: false flags: - runtime with_legacy: true - name: bluestore_debug_no_reuse_blocks type: bool level: dev default: false with_legacy: true - name: bluestore_debug_small_allocations type: int level: dev default: 0 with_legacy: true - name: bluestore_debug_too_many_blobs_threshold type: int level: dev default: 24576 with_legacy: true - name: bluestore_debug_freelist type: bool level: dev default: false with_legacy: true - name: bluestore_debug_prefill type: float level: dev desc: simulate fragmentation default: 0 with_legacy: true - name: bluestore_debug_prefragment_max type: size level: dev default: 1_M with_legacy: true - name: bluestore_debug_inject_read_err type: bool level: dev default: false with_legacy: true - name: bluestore_debug_randomize_serial_transaction type: int level: dev default: 0 with_legacy: true - name: bluestore_debug_omit_block_device_write type: bool level: dev default: false with_legacy: true - name: bluestore_debug_fsck_abort type: bool level: dev default: false with_legacy: true - name: bluestore_debug_omit_kv_commit type: bool level: dev default: false with_legacy: true - name: bluestore_debug_permit_any_bdev_label type: bool level: dev default: false with_legacy: true - name: bluestore_debug_random_read_err type: float level: dev default: 0 with_legacy: true - name: bluestore_debug_inject_bug21040 type: bool level: dev default: false with_legacy: true - name: bluestore_debug_inject_csum_err_probability type: float level: dev desc: inject crc verification errors into bluestore device reads default: 0 with_legacy: true - name: bluestore_debug_legacy_omap type: bool level: dev desc: Allows mkfs to create OSD in legacy OMAP naming mode (neither per-pool nor per-pg). This is intended primarily for developers' purposes. The resulting OSD might/would be transformed to the currrently default 'per-pg' format when BlueStore's quick-fix or repair are applied. default: false with_legacy: true - name: bluestore_fsck_error_on_no_per_pool_stats type: bool level: advanced desc: Make fsck error (instead of warn) when bluestore lacks per-pool stats, e.g., after an upgrade default: false with_legacy: true - name: bluestore_warn_on_bluefs_spillover type: bool level: advanced desc: Enable health indication on bluefs slow device usage default: true with_legacy: true - name: bluestore_warn_on_legacy_statfs type: bool level: advanced desc: Enable health indication on lack of per-pool statfs reporting from bluestore default: true with_legacy: true - name: bluestore_warn_on_spurious_read_errors type: bool level: advanced desc: Enable health indication when spurious read errors are observed by OSD default: true with_legacy: true - name: bluestore_fsck_error_on_no_per_pool_omap type: bool level: advanced desc: Make fsck error (instead of warn) when objects without per-pool omap are found default: false with_legacy: true - name: bluestore_fsck_error_on_no_per_pg_omap type: bool level: advanced desc: Make fsck error (instead of warn) when objects without per-pg omap are found default: false with_legacy: true - name: bluestore_warn_on_no_per_pool_omap type: bool level: advanced desc: Enable health indication on lack of per-pool omap default: true with_legacy: true - name: bluestore_warn_on_no_per_pg_omap type: bool level: advanced desc: Enable health indication on lack of per-pg omap default: false with_legacy: true - name: bluestore_log_op_age type: float level: advanced desc: log operation if it's slower than this age (seconds) default: 5 with_legacy: true - name: bluestore_log_omap_iterator_age type: float level: advanced desc: log omap iteration operation if it's slower than this age (seconds) default: 5 with_legacy: true - name: bluestore_log_collection_list_age type: float level: advanced desc: log collection list operation if it's slower than this age (seconds) default: 1_min with_legacy: true - name: bluestore_debug_enforce_settings type: str level: dev desc: Enforces specific hw profile settings long_desc: '''hdd'' enforces settings intended for BlueStore above a rotational drive. ''ssd'' enforces settings intended for BlueStore above a solid drive. ''default'' - using settings for the actual hardware.' default: default enum_values: - default - hdd - ssd with_legacy: true - name: bluestore_avl_alloc_ff_max_search_count type: uint level: dev desc: Search for this many ranges in first-fit mode before switching over to to best-fit mode. 0 to iterate through all ranges for required chunk. default: 100 - name: bluestore_avl_alloc_ff_max_search_bytes type: size level: dev desc: Maximum distance to search in first-fit mode before switching over to to best-fit mode. 0 to iterate through all ranges for required chunk. default: 16_M - name: bluestore_avl_alloc_bf_threshold type: uint level: dev desc: Sets threshold at which shrinking max free chunk size triggers enabling best-fit mode. long_desc: 'AVL allocator works in two modes: near-fit and best-fit. By default, it uses very fast near-fit mode, in which it tries to fit a new block near the last allocated block of similar size. The second mode is much slower best-fit mode, in which it tries to find an exact match for the requested allocation. This mode is used when either the device gets fragmented or when it is low on free space. When the largest free block is smaller than ''bluestore_avl_alloc_bf_threshold'', best-fit mode is used.' default: 128_K see_also: - bluestore_avl_alloc_bf_free_pct - name: bluestore_avl_alloc_bf_free_pct type: uint level: dev desc: Sets threshold at which shrinking free space (in %, integer) triggers enabling best-fit mode. long_desc: 'AVL allocator works in two modes: near-fit and best-fit. By default, it uses very fast near-fit mode, in which it tries to fit a new block near the last allocated block of similar size. The second mode is much slower best-fit mode, in which it tries to find an exact match for the requested allocation. This mode is used when either the device gets fragmented or when it is low on free space. When free space is smaller than ''bluestore_avl_alloc_bf_free_pct'', best-fit mode is used.' default: 4 see_also: - bluestore_avl_alloc_bf_threshold - name: bluestore_hybrid_alloc_mem_cap type: uint level: dev desc: Maximum RAM hybrid allocator should use before enabling bitmap supplement default: 64_M - name: bluestore_volume_selection_policy type: str level: dev desc: Determines bluefs volume selection policy long_desc: Determines bluefs volume selection policy. 'use_some_extra*' policy allows to override RocksDB level granularity and put high level's data to faster device even when the level doesn't completely fit there. 'fit_to_fast' policy enables using 100% of faster disk capacity and allows the user to turn on 'level_compaction_dynamic_level_bytes' option in RocksDB options. default: use_some_extra enum_values: - rocksdb_original - use_some_extra - use_some_extra_enforced - fit_to_fast with_legacy: true - name: bluestore_volume_selection_reserved_factor type: float level: advanced desc: DB level size multiplier. Determines amount of space at DB device to bar from the usage when 'use some extra' policy is in action. Reserved size is determined as sum(L_max_size[0], L_max_size[L-1]) + L_max_size[L] * this_factor default: 2 flags: - startup with_legacy: true - name: bluestore_volume_selection_reserved type: int level: advanced desc: Space reserved at DB device and not allowed for 'use some extra' policy usage. Overrides 'bluestore_volume_selection_reserved_factor' setting and introduces straightforward limit. default: 0 flags: - startup with_legacy: true - name: bdev_ioring type: bool level: advanced desc: Enables Linux io_uring API instead of libaio default: false - name: bdev_ioring_hipri type: bool level: advanced desc: Enables Linux io_uring API Use polled IO completions default: false - name: bdev_ioring_sqthread_poll type: bool level: advanced desc: Enables Linux io_uring API Offload submission/completion to kernel thread default: false - name: bluestore_kv_sync_util_logging_s type: float level: advanced desc: KV sync thread utilization logging period long_desc: How often (in seconds) to print KV sync thread utilization, not logged when set to 0 or when utilization is 0% default: 10 flags: - runtime with_legacy: true - name: bluestore_fail_eio type: bool level: dev desc: fail/crash on EIO long_desc: whether bluestore osd fails on eio default: false flags: - runtime with_legacy: true - name: bluestore_zero_block_detection type: bool level: dev desc: punch holes instead of writing zeros long_desc: Intended for large-scale synthetic testing. Currently this is implemented with punch hole semantics, affecting the logical extent map of the object. This does not interact well with some RBD and CephFS features. default: false flags: - runtime with_legacy: true - name: kstore_max_ops type: uint level: advanced default: 512 with_legacy: true - name: kstore_max_bytes type: size level: advanced default: 64_M with_legacy: true - name: kstore_backend type: str level: advanced default: rocksdb with_legacy: true - name: kstore_rocksdb_options type: str level: advanced desc: Options to pass through when RocksDB is used as the KeyValueDB for kstore. default: compression=kNoCompression with_legacy: true - name: kstore_fsck_on_mount type: bool level: advanced desc: Whether or not to run fsck on mount for kstore. default: false with_legacy: true - name: kstore_fsck_on_mount_deep type: bool level: advanced desc: Whether or not to run deep fsck on mount for kstore default: true with_legacy: true - name: kstore_nid_prealloc type: uint level: advanced default: 1_K with_legacy: true - name: kstore_sync_transaction type: bool level: advanced default: false with_legacy: true - name: kstore_sync_submit_transaction type: bool level: advanced default: false with_legacy: true - name: kstore_onode_map_size type: uint level: advanced default: 1_K with_legacy: true - name: kstore_default_stripe_size type: size level: advanced default: 64_K with_legacy: true # rocksdb options that will be used for omap(if omap_backend is rocksdb) - name: filestore_rocksdb_options type: str level: dev desc: Options to pass through when RocksDB is used as the KeyValueDB for filestore. default: max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression with_legacy: true - name: filestore_omap_backend type: str level: dev desc: The KeyValueDB to use for filestore metadata (ie omap). default: rocksdb enum_values: - leveldb - rocksdb with_legacy: true - name: filestore_omap_backend_path type: str level: dev desc: The path where the filestore KeyValueDB should store it's database(s). with_legacy: true # filestore wb throttle limits - name: filestore_wbthrottle_enable type: bool level: advanced desc: Enabling throttling of operations to backing file system default: true with_legacy: true - name: filestore_wbthrottle_btrfs_bytes_start_flusher type: size level: advanced desc: Start flushing (fsyncing) when this many bytes are written(btrfs) default: 40_M with_legacy: true - name: filestore_wbthrottle_btrfs_bytes_hard_limit type: size level: advanced desc: Block writes when this many bytes haven't been flushed (fsynced) (btrfs) default: 400_M with_legacy: true - name: filestore_wbthrottle_btrfs_ios_start_flusher type: uint level: advanced desc: Start flushing (fsyncing) when this many IOs are written (brtrfs) default: 500 with_legacy: true - name: filestore_wbthrottle_btrfs_ios_hard_limit type: uint level: advanced desc: Block writes when this many IOs haven't been flushed (fsynced) (btrfs) default: 5000 with_legacy: true - name: filestore_wbthrottle_btrfs_inodes_start_flusher type: uint level: advanced desc: Start flushing (fsyncing) when this many distinct inodes have been modified (btrfs) default: 500 with_legacy: true - name: filestore_wbthrottle_xfs_bytes_start_flusher type: size level: advanced desc: Start flushing (fsyncing) when this many bytes are written(xfs) default: 40_M with_legacy: true - name: filestore_wbthrottle_xfs_bytes_hard_limit type: size level: advanced desc: Block writes when this many bytes haven't been flushed (fsynced) (xfs) default: 400_M with_legacy: true - name: filestore_wbthrottle_xfs_ios_start_flusher type: uint level: advanced desc: Start flushing (fsyncing) when this many IOs are written (xfs) default: 500 with_legacy: true - name: filestore_wbthrottle_xfs_ios_hard_limit type: uint level: advanced desc: Block writes when this many IOs haven't been flushed (fsynced) (xfs) default: 5000 with_legacy: true - name: filestore_wbthrottle_xfs_inodes_start_flusher type: uint level: advanced desc: Start flushing (fsyncing) when this many distinct inodes have been modified (xfs) default: 500 with_legacy: true # These must be less than the fd limit - name: filestore_wbthrottle_btrfs_inodes_hard_limit type: uint level: advanced desc: Block writing when this many inodes have outstanding writes (btrfs) default: 5000 with_legacy: true - name: filestore_wbthrottle_xfs_inodes_hard_limit type: uint level: advanced desc: Block writing when this many inodes have outstanding writes (xfs) default: 5000 with_legacy: true # Introduce a O_DSYNC write in the filestore - name: filestore_odsync_write type: bool level: dev desc: Write with O_DSYNC default: false with_legacy: true # Tests index failure paths - name: filestore_index_retry_probability type: float level: dev default: 0 with_legacy: true # Allow object read error injection - name: filestore_debug_inject_read_err type: bool level: dev default: false with_legacy: true - name: filestore_debug_random_read_err type: float level: dev default: 0 with_legacy: true # Expensive debugging check on sync - name: filestore_debug_omap_check type: bool level: dev default: false fmt_desc: Debugging check on synchronization. This is an expensive operation. with_legacy: true - name: filestore_omap_header_cache_size type: size level: dev default: 1_K with_legacy: true # Use omap for xattrs for attrs over # filestore_max_inline_xattr_size or - name: filestore_max_inline_xattr_size type: size level: dev default: 0 with_legacy: true - name: filestore_max_inline_xattr_size_xfs type: size level: dev default: 64_K with_legacy: true - name: filestore_max_inline_xattr_size_btrfs type: size level: dev default: 2_K with_legacy: true - name: filestore_max_inline_xattr_size_other type: size level: dev default: 512 with_legacy: true # for more than filestore_max_inline_xattrs attrs - name: filestore_max_inline_xattrs type: uint level: dev default: 0 with_legacy: true - name: filestore_max_inline_xattrs_xfs type: uint level: dev default: 10 with_legacy: true - name: filestore_max_inline_xattrs_btrfs type: uint level: dev default: 10 with_legacy: true - name: filestore_max_inline_xattrs_other type: uint level: dev default: 2 with_legacy: true - name: filestore_max_xattr_value_size type: size level: dev default: 0 with_legacy: true - name: filestore_max_xattr_value_size_xfs type: size level: dev default: 64_K with_legacy: true - name: filestore_max_xattr_value_size_btrfs type: size level: dev default: 64_K with_legacy: true # ext4 allows 4k xattrs total including some smallish extra fields and the # keys. We're allowing 2 512 inline attrs in addition some some filestore # replay attrs. After accounting for those, we still need to fit up to # two attrs of this value. That means we need this value to be around 1k # to be safe. This is hacky, but it's not worth complicating the code # to work around ext4's total xattr limit. - name: filestore_max_xattr_value_size_other type: size level: dev default: 1_K with_legacy: true # track sloppy crcs - name: filestore_sloppy_crc type: bool level: dev default: false with_legacy: true - name: filestore_sloppy_crc_block_size type: size level: dev default: 64_K with_legacy: true - name: filestore_max_alloc_hint_size type: size level: dev default: 1_M with_legacy: true # seconds - name: filestore_max_sync_interval type: float level: advanced desc: Period between calls to syncfs(2) and journal trims (seconds) default: 5 with_legacy: true # seconds - name: filestore_min_sync_interval type: float level: dev desc: Minimum period between calls to syncfs(2) default: 0.01 with_legacy: true - name: filestore_btrfs_snap type: bool level: dev default: true with_legacy: true - name: filestore_btrfs_clone_range type: bool level: advanced desc: Use btrfs clone_range ioctl to efficiently duplicate objects default: true with_legacy: true # zfsonlinux is still unstable - name: filestore_zfs_snap type: bool level: dev default: false with_legacy: true - name: filestore_fsync_flushes_journal_data type: bool level: dev default: false with_legacy: true # (try to) use fiemap - name: filestore_fiemap type: bool level: advanced desc: Use fiemap ioctl(2) to determine which parts of objects are sparse default: false with_legacy: true - name: filestore_punch_hole type: bool level: advanced desc: Use fallocate(2) FALLOC_FL_PUNCH_HOLE to efficiently zero ranges of objects default: false with_legacy: true # (try to) use seek_data/hole - name: filestore_seek_data_hole type: bool level: advanced desc: Use lseek(2) SEEK_HOLE and SEEK_DATA to determine which parts of objects are sparse default: false with_legacy: true - name: filestore_splice type: bool level: advanced desc: Use splice(2) to more efficiently copy data between files default: false with_legacy: true - name: filestore_fadvise type: bool level: advanced desc: Use posix_fadvise(2) to pass hints to file system default: true with_legacy: true # collect device partition information for management application to use - name: filestore_collect_device_partition_information type: bool level: advanced desc: Collect metadata about the backing file system on OSD startup default: true with_legacy: true # (try to) use extsize for alloc hint NOTE: extsize seems to trigger # data corruption in xfs prior to kernel 3.5. filestore will # implicitly disable this if it cannot confirm the kernel is newer # than that. # NOTE: This option involves a tradeoff: When disabled, fragmentation is # worse, but large sequential writes are faster. When enabled, large # sequential writes are slower, but fragmentation is reduced. - name: filestore_xfs_extsize type: bool level: advanced desc: Use XFS extsize ioctl(2) to hint allocator about expected write sizes default: false with_legacy: true - name: filestore_journal_parallel type: bool level: dev default: false with_legacy: true - name: filestore_journal_writeahead type: bool level: dev default: false with_legacy: true - name: filestore_journal_trailing type: bool level: dev default: false with_legacy: true - name: filestore_queue_max_ops type: uint level: advanced desc: Max IO operations in flight default: 50 with_legacy: true - name: filestore_queue_max_bytes type: size level: advanced desc: Max (written) bytes in flight default: 100_M with_legacy: true - name: filestore_caller_concurrency type: int level: dev default: 10 with_legacy: true # Expected filestore throughput in B/s - name: filestore_expected_throughput_bytes type: float level: advanced desc: Expected throughput of backend device (aids throttling calculations) default: 209715200 with_legacy: true # Expected filestore throughput in ops/s - name: filestore_expected_throughput_ops type: float level: advanced desc: Expected through of backend device in IOPS (aids throttling calculations) default: 200 with_legacy: true # Filestore max delay multiple. Defaults to 0 (disabled) - name: filestore_queue_max_delay_multiple type: float level: dev default: 0 with_legacy: true # Filestore high delay multiple. Defaults to 0 (disabled) - name: filestore_queue_high_delay_multiple type: float level: dev default: 0 with_legacy: true # Filestore max delay multiple ops. Defaults to 0 (disabled) - name: filestore_queue_max_delay_multiple_bytes type: float level: dev default: 0 with_legacy: true # Filestore high delay multiple bytes. Defaults to 0 (disabled) - name: filestore_queue_high_delay_multiple_bytes type: float level: dev default: 0 with_legacy: true # Filestore max delay multiple ops. Defaults to 0 (disabled) - name: filestore_queue_max_delay_multiple_ops type: float level: dev default: 0 with_legacy: true # Filestore high delay multiple ops. Defaults to 0 (disabled) - name: filestore_queue_high_delay_multiple_ops type: float level: dev default: 0 with_legacy: true - name: filestore_queue_low_threshhold type: float level: dev default: 0.3 with_legacy: true - name: filestore_queue_high_threshhold type: float level: dev with_legacy: true default: 0.9 - name: filestore_op_threads type: int level: advanced desc: Threads used to apply changes to backing file system default: 2 with_legacy: true - name: filestore_op_thread_timeout type: int level: advanced desc: Seconds before a worker thread is considered stalled default: 1_min with_legacy: true - name: filestore_op_thread_suicide_timeout type: int level: advanced desc: Seconds before a worker thread is considered dead default: 3_min with_legacy: true - name: filestore_commit_timeout type: float level: advanced desc: Seconds before backing file system is considered hung default: 10_min with_legacy: true - name: filestore_fiemap_threshold type: size level: dev default: 4_K with_legacy: true - name: filestore_merge_threshold type: int level: dev default: -10 with_legacy: true - name: filestore_split_multiple type: int level: dev default: 2 with_legacy: true - name: filestore_split_rand_factor type: uint level: dev default: 20 with_legacy: true - name: filestore_update_to type: int level: dev default: 1000 with_legacy: true - name: filestore_blackhole type: bool level: dev default: false with_legacy: true - name: filestore_fd_cache_size type: int level: dev default: 128 with_legacy: true - name: filestore_fd_cache_shards type: int level: dev default: 16 with_legacy: true - name: filestore_ondisk_finisher_threads type: int level: dev default: 1 with_legacy: true - name: filestore_apply_finisher_threads type: int level: dev default: 1 with_legacy: true # file onto which store transaction dumps - name: filestore_dump_file type: str level: dev with_legacy: true # inject a failure at the n'th opportunity - name: filestore_kill_at type: int level: dev default: 0 with_legacy: true # artificially stall for N seconds in op queue thread - name: filestore_inject_stall type: int level: dev default: 0 with_legacy: true # fail/crash on EIO - name: filestore_fail_eio type: bool level: dev default: true with_legacy: true - name: filestore_debug_verify_split type: bool level: dev default: false with_legacy: true - name: journal_dio type: bool level: dev default: true fmt_desc: Enables direct i/o to the journal. Requires ``journal block align`` set to ``true``. with_legacy: true - name: journal_aio type: bool level: dev default: true fmt_desc: Enables using ``libaio`` for asynchronous writes to the journal. Requires ``journal dio`` set to ``true``. Version 0.61 and later, ``true``. Version 0.60 and earlier, ``false``. with_legacy: true - name: journal_force_aio type: bool level: dev default: false with_legacy: true - name: journal_block_size type: size level: dev default: 4_K with_legacy: true - name: journal_block_align type: bool level: dev default: true fmt_desc: Block aligns write operations. Required for ``dio`` and ``aio``. with_legacy: true - name: journal_write_header_frequency type: uint level: dev default: 0 with_legacy: true - name: journal_max_write_bytes type: size level: advanced desc: Max bytes in flight to journal fmt_desc: The maximum number of bytes the journal will write at any one time. default: 10_M with_legacy: true - name: journal_max_write_entries type: int level: advanced desc: Max IOs in flight to journal fmt_desc: The maximum number of entries the journal will write at any one time. default: 100 with_legacy: true # Target range for journal fullness - name: journal_throttle_low_threshhold type: float level: dev default: 0.6 with_legacy: true - name: journal_throttle_high_threshhold type: float level: dev default: 0.9 with_legacy: true # Multiple over expected at high_threshhold. Defaults to 0 (disabled). - name: journal_throttle_high_multiple type: float level: dev default: 0 with_legacy: true # Multiple over expected at max. Defaults to 0 (disabled). - name: journal_throttle_max_multiple type: float level: dev default: 0 with_legacy: true # align data payloads >= this. - name: journal_align_min_size type: size level: dev default: 64_K fmt_desc: Align data payloads greater than the specified minimum. with_legacy: true - name: journal_replay_from type: int level: dev default: 0 with_legacy: true - name: journal_zero_on_create type: bool level: dev default: false fmt_desc: | Causes the file store to overwrite the entire journal with ``0``'s during ``mkfs``. with_legacy: true # assume journal is not corrupt - name: journal_ignore_corruption type: bool level: dev default: false with_legacy: true # using ssd disk as journal, whether support discard nouse journal-data. - name: journal_discard type: bool level: dev default: false with_legacy: true # fio data directory for fio-objectstore - name: fio_dir type: str level: advanced default: /tmp/fio with_legacy: true - name: rados_mon_op_timeout type: secs level: advanced desc: timeout for operations handled by monitors such as statfs (0 is unlimited) default: 0 min: 0 flags: - runtime - name: rados_osd_op_timeout type: secs level: advanced desc: timeout for operations handled by osds such as write (0 is unlimited) default: 0 min: 0 flags: - runtime # true if LTTng-UST tracepoints should be enabled - name: rados_tracing type: bool level: advanced default: false with_legacy: true - name: mgr_connect_retry_interval type: float level: dev default: 1 services: - common - name: mgr_client_service_daemon_unregister_timeout type: float level: dev desc: Time to wait during shutdown to deregister service with mgr default: 1 - name: throttler_perf_counter type: bool level: advanced default: true with_legacy: true - name: event_tracing type: bool level: advanced default: false with_legacy: true - name: bluestore_tracing type: bool level: advanced desc: Enable bluestore event tracing. default: false - name: bluestore_throttle_trace_rate type: float level: advanced desc: Rate at which to sample bluestore transactions (per second) default: 0 - name: debug_deliberately_leak_memory type: bool level: dev default: false with_legacy: true - name: debug_asserts_on_shutdown type: bool level: dev desc: Enable certain asserts to check for refcounting bugs on shutdown; see http://tracker.ceph.com/issues/21738 default: false - name: debug_asok_assert_abort type: bool level: dev desc: allow commands 'assert' and 'abort' via asok for testing crash dumps etc default: false with_legacy: true - name: target_max_misplaced_ratio type: float level: basic desc: Max ratio of misplaced objects to target when throttling data rebalancing activity default: 0.05 - name: device_failure_prediction_mode type: str level: basic desc: Method used to predict device failures long_desc: To disable prediction, use 'none', 'local' uses a prediction model that runs inside the mgr daemon. 'cloud' will share metrics with a cloud service and query the service for devicelife expectancy. default: none enum_values: - none - local - cloud flags: - runtime - name: gss_ktab_client_file type: str level: advanced desc: GSS/KRB5 Keytab file for client authentication long_desc: This sets the full path for the GSS/Kerberos client keytab file location. default: /var/lib/ceph/$name/gss_client_$name.ktab services: - mon - osd - name: gss_target_name type: str level: advanced long_desc: This sets the gss target service name. default: ceph services: - mon - osd - name: debug_disable_randomized_ping type: bool level: dev desc: Disable heartbeat ping randomization for testing purposes default: false - name: debug_heartbeat_testing_span type: int level: dev desc: Override 60 second periods for testing only default: 0 - name: librados_thread_count type: uint level: advanced desc: Size of thread pool for Objecter default: 2 tags: - client min: 1 - name: osd_asio_thread_count type: uint level: advanced desc: Size of thread pool for ASIO completions default: 2 tags: - osd min: 1 - name: cephsqlite_lock_renewal_interval type: millisecs level: advanced desc: number of milliseconds before lock is renewed default: 2000 tags: - client see_also: - cephsqlite_lock_renewal_timeout min: 100 - name: cephsqlite_lock_renewal_timeout type: millisecs level: advanced desc: number of milliseconds before transaction lock times out long_desc: The amount of time before a running libcephsqlite VFS connection has to renew a lock on the database before the lock is automatically lost. If the lock is lost, the VFS will abort the process to prevent database corruption. default: 30000 tags: - client see_also: - cephsqlite_lock_renewal_interval min: 100 - name: cephsqlite_blocklist_dead_locker type: bool level: advanced desc: blocklist the last dead owner of the database lock long_desc: Require that the Ceph SQLite VFS blocklist the last dead owner of the database when cleanup was incomplete. DO NOT CHANGE THIS UNLESS YOU UNDERSTAND THE RAMIFICATIONS. CORRUPTION MAY RESULT. default: true tags: - client - name: bdev_type type: str level: advanced desc: Explicitly set the device type to select the driver if it's needed enum_values: - aio - spdk - pmem - hm_smr - name: bluestore_cleaner_sleep_interval type: float level: advanced desc: How long cleaner should sleep before re-checking utilization default: 5 with_legacy: true - name: jaeger_tracing_enable type: bool level: advanced desc: Ceph should use jaeger tracing system default: false services: - rgw - osd with_legacy: true - name: jaeger_agent_port type: int level: advanced desc: port number of the jaeger agent default: 6799 services: - rgw - osd - name: mgr_ttl_cache_expire_seconds type: uint level: dev desc: Set the time to live in seconds - set to 0 to disable the cache. default: 0 services: - mgr