================== Service Management ================== A service is a group of daemons configured together. See these chapters for details on individual services: .. toctree:: :maxdepth: 1 mon mgr osd rgw mds nfs iscsi custom-container monitoring snmp-gateway Service Status ============== To see the status of one of the services running in the Ceph cluster, do the following: #. Use the command line to print a list of services. #. Locate the service whose status you want to check. #. Print the status of the service. The following command prints a list of services known to the orchestrator. To limit the output to services only on a specified host, use the optional ``--host`` parameter. To limit the output to services of only a particular type, use the optional ``--type`` parameter (mon, osd, mgr, mds, rgw): .. prompt:: bash # ceph orch ls [--service_type type] [--service_name name] [--export] [--format f] [--refresh] Discover the status of a particular service or daemon: .. prompt:: bash # ceph orch ls --service_type type --service_name [--refresh] To export the service specifications knows to the orchestrator, run the following command. .. prompt:: bash # ceph orch ls --export The service specifications exported with this command will be exported as yaml and that yaml can be used with the ``ceph orch apply -i`` command. For information about retrieving the specifications of single services (including examples of commands), see :ref:`orchestrator-cli-service-spec-retrieve`. Daemon Status ============= A daemon is a systemd unit that is running and part of a service. To see the status of a daemon, do the following: #. Print a list of all daemons known to the orchestrator. #. Query the status of the target daemon. First, print a list of all daemons known to the orchestrator: .. prompt:: bash # ceph orch ps [--hostname host] [--daemon_type type] [--service_name name] [--daemon_id id] [--format f] [--refresh] Then query the status of a particular service instance (mon, osd, mds, rgw). For OSDs the id is the numeric OSD ID. For MDS services the id is the file system name: .. prompt:: bash # ceph orch ps --daemon_type osd --daemon_id 0 .. _orchestrator-cli-service-spec: Service Specification ===================== A *Service Specification* is a data structure that is used to specify the deployment of services. In addition to parameters such as `placement` or `networks`, the user can set initial values of service configuration parameters by means of the `config` section. For each param/value configuration pair, cephadm calls the following command to set its value: .. prompt:: bash # ceph config set cephadm raises health warnings in case invalid configuration parameters are found in the spec (`CEPHADM_INVALID_CONFIG_OPTION`) or if any error while trying to apply the new configuration option(s) (`CEPHADM_FAILED_SET_OPTION`). Here is an example of a service specification in YAML: .. code-block:: yaml service_type: rgw service_id: realm.zone placement: hosts: - host1 - host2 - host3 config: param_1: val_1 ... param_N: val_N unmanaged: false networks: - 192.169.142.0/24 spec: # Additional service specific attributes. In this example, the properties of this service specification are: .. py:currentmodule:: ceph.deployment.service_spec .. autoclass:: ServiceSpec :members: Each service type can have additional service-specific properties. Service specifications of type ``mon``, ``mgr``, and the monitoring types do not require a ``service_id``. A service of type ``osd`` is described in :ref:`drivegroups` Many service specifications can be applied at once using ``ceph orch apply -i`` by submitting a multi-document YAML file:: cat <. --export > rgw...yaml ceph orch ls --service-type mgr --export > mgr.yaml ceph orch ls --export > cluster.yaml The Specification can then be changed and re-applied as above. Updating Service Specifications ------------------------------- The Ceph Orchestrator maintains a declarative state of each service in a ``ServiceSpec``. For certain operations, like updating the RGW HTTP port, we need to update the existing specification. 1. List the current ``ServiceSpec``: .. prompt:: bash # ceph orch ls --service_name= --export > myservice.yaml 2. Update the yaml file: .. prompt:: bash # vi myservice.yaml 3. Apply the new ``ServiceSpec``: .. prompt:: bash # ceph orch apply -i myservice.yaml [--dry-run] .. _orchestrator-cli-placement-spec: Daemon Placement ================ For the orchestrator to deploy a *service*, it needs to know where to deploy *daemons*, and how many to deploy. This is the role of a placement specification. Placement specifications can either be passed as command line arguments or in a YAML files. .. note:: cephadm will not deploy daemons on hosts with the ``_no_schedule`` label; see :ref:`cephadm-special-host-labels`. .. note:: The **apply** command can be confusing. For this reason, we recommend using YAML specifications. Each ``ceph orch apply `` command supersedes the one before it. If you do not use the proper syntax, you will clobber your work as you go. For example: .. prompt:: bash # ceph orch apply mon host1 ceph orch apply mon host2 ceph orch apply mon host3 This results in only one host having a monitor applied to it: host 3. (The first command creates a monitor on host1. Then the second command clobbers the monitor on host1 and creates a monitor on host2. Then the third command clobbers the monitor on host2 and creates a monitor on host3. In this scenario, at this point, there is a monitor ONLY on host3.) To make certain that a monitor is applied to each of these three hosts, run a command like this: .. prompt:: bash # ceph orch apply mon "host1,host2,host3" There is another way to apply monitors to multiple hosts: a ``yaml`` file can be used. Instead of using the "ceph orch apply mon" commands, run a command of this form: .. prompt:: bash # ceph orch apply -i file.yaml Here is a sample **file.yaml** file .. code-block:: yaml service_type: mon placement: hosts: - host1 - host2 - host3 Explicit placements ------------------- Daemons can be explicitly placed on hosts by simply specifying them: .. prompt:: bash # orch apply prometheus --placement="host1 host2 host3" Or in YAML: .. code-block:: yaml service_type: prometheus placement: hosts: - host1 - host2 - host3 MONs and other services may require some enhanced network specifications: .. prompt:: bash # orch daemon add mon --placement="myhost:[v2:1.2.3.4:3300,v1:1.2.3.4:6789]=name" where ``[v2:1.2.3.4:3300,v1:1.2.3.4:6789]`` is the network address of the monitor and ``=name`` specifies the name of the new monitor. .. _orch-placement-by-labels: Placement by labels ------------------- Daemon placement can be limited to hosts that match a specific label. To set a label ``mylabel`` to the appropriate hosts, run this command: .. prompt:: bash # ceph orch host label add ** mylabel To view the current hosts and labels, run this command: .. prompt:: bash # ceph orch host ls For example: .. prompt:: bash # ceph orch host label add host1 mylabel ceph orch host label add host2 mylabel ceph orch host label add host3 mylabel ceph orch host ls .. code-block:: bash HOST ADDR LABELS STATUS host1 mylabel host2 mylabel host3 mylabel host4 host5 Now, Tell cephadm to deploy daemons based on the label by running this command: .. prompt:: bash # orch apply prometheus --placement="label:mylabel" Or in YAML: .. code-block:: yaml service_type: prometheus placement: label: "mylabel" * See :ref:`orchestrator-host-labels` Placement by pattern matching ----------------------------- Daemons can be placed on hosts as well: .. prompt:: bash # orch apply prometheus --placement='myhost[1-3]' Or in YAML: .. code-block:: yaml service_type: prometheus placement: host_pattern: "myhost[1-3]" To place a service on *all* hosts, use ``"*"``: .. prompt:: bash # orch apply node-exporter --placement='*' Or in YAML: .. code-block:: yaml service_type: node-exporter placement: host_pattern: "*" Changing the number of daemons ------------------------------ By specifying ``count``, only the number of daemons specified will be created: .. prompt:: bash # orch apply prometheus --placement=3 To deploy *daemons* on a subset of hosts, specify the count: .. prompt:: bash # orch apply prometheus --placement="2 host1 host2 host3" If the count is bigger than the amount of hosts, cephadm deploys one per host: .. prompt:: bash # orch apply prometheus --placement="3 host1 host2" The command immediately above results in two Prometheus daemons. YAML can also be used to specify limits, in the following way: .. code-block:: yaml service_type: prometheus placement: count: 3 YAML can also be used to specify limits on hosts: .. code-block:: yaml service_type: prometheus placement: count: 2 hosts: - host1 - host2 - host3 .. _cephadm_co_location: Co-location of daemons ---------------------- Cephadm supports the deployment of multiple daemons on the same host: .. code-block:: yaml service_type: rgw placement: label: rgw count_per_host: 2 The main reason for deploying multiple daemons per host is an additional performance benefit for running multiple RGW and MDS daemons on the same host. See also: * :ref:`cephadm_mgr_co_location`. * :ref:`cephadm-rgw-designated_gateways`. This feature was introduced in Pacific. Algorithm description --------------------- Cephadm's declarative state consists of a list of service specifications containing placement specifications. Cephadm continually compares a list of daemons actually running in the cluster against the list in the service specifications. Cephadm adds new daemons and removes old daemons as necessary in order to conform to the service specifications. Cephadm does the following to maintain compliance with the service specifications. Cephadm first selects a list of candidate hosts. Cephadm seeks explicit host names and selects them. If cephadm finds no explicit host names, it looks for label specifications. If no label is defined in the specification, cephadm selects hosts based on a host pattern. If no host pattern is defined, as a last resort, cephadm selects all known hosts as candidates. Cephadm is aware of existing daemons running services and tries to avoid moving them. Cephadm supports the deployment of a specific amount of services. Consider the following service specification: .. code-block:: yaml service_type: mds service_name: myfs placement: count: 3 label: myfs This service specifcation instructs cephadm to deploy three daemons on hosts labeled ``myfs`` across the cluster. If there are fewer than three daemons deployed on the candidate hosts, cephadm randomly chooses hosts on which to deploy new daemons. If there are more than three daemons deployed on the candidate hosts, cephadm removes existing daemons. Finally, cephadm removes daemons on hosts that are outside of the list of candidate hosts. .. note:: There is a special case that cephadm must consider. If there are fewer hosts selected by the placement specification than demanded by ``count``, cephadm will deploy only on the selected hosts. Extra Container Arguments ========================= .. warning:: The arguments provided for extra container args are limited to whatever arguments are available for a `run` command from whichever container engine you are using. Providing any arguments the `run` command does not support (or invalid values for arguments) will cause the daemon to fail to start. Cephadm supports providing extra miscellaneous container arguments for specific cases when they may be necessary. For example, if a user needed to limit the amount of cpus their mon daemons make use of they could apply a spec like .. code-block:: yaml service_type: mon service_name: mon placement: hosts: - host1 - host2 - host3 extra_container_args: - "--cpus=2" which would cause each mon daemon to be deployed with `--cpus=2`. Mounting Files with Extra Container Arguments --------------------------------------------- A common use case for extra container arguments is to mount additional files within the container. However, some intuitive formats for doing so can cause deployment to fail (see https://tracker.ceph.com/issues/57338). The recommended syntax for mounting a file with extra container arguments is: .. code-block:: yaml extra_container_args: - "-v" - "/absolute/file/path/on/host:/absolute/file/path/in/container" For example: .. code-block:: yaml extra_container_args: - "-v" - "/opt/ceph_cert/host.cert:/etc/grafana/certs/cert_file:ro" .. _orch-rm: Removing a Service ================== In order to remove a service including the removal of all daemons of that service, run .. prompt:: bash ceph orch rm For example: .. prompt:: bash ceph orch rm rgw.myrgw .. _cephadm-spec-unmanaged: Disabling automatic deployment of daemons ========================================= Cephadm supports disabling the automated deployment and removal of daemons on a per service basis. The CLI supports two commands for this. In order to fully remove a service, see :ref:`orch-rm`. Disabling automatic management of daemons ----------------------------------------- To disable the automatic management of dameons, set ``unmanaged=True`` in the :ref:`orchestrator-cli-service-spec` (``mgr.yaml``). ``mgr.yaml``: .. code-block:: yaml service_type: mgr unmanaged: true placement: label: mgr .. prompt:: bash # ceph orch apply -i mgr.yaml .. note:: After you apply this change in the Service Specification, cephadm will no longer deploy any new daemons (even if the placement specification matches additional hosts). Deploying a daemon on a host manually ------------------------------------- .. note:: This workflow has a very limited use case and should only be used in rare circumstances. To manually deploy a daemon on a host, follow these steps: Modify the service spec for a service by getting the existing spec, adding ``unmanaged: true``, and applying the modified spec. Then manually deploy the daemon using the following: .. prompt:: bash # ceph orch daemon add --placement= For example : .. prompt:: bash # ceph orch daemon add mgr --placement=my_host .. note:: Removing ``unmanaged: true`` from the service spec will enable the reconciliation loop for this service and will potentially lead to the removal of the daemon, depending on the placement spec. Removing a daemon from a host manually -------------------------------------- To manually remove a daemon, run a command of the following form: .. prompt:: bash # ceph orch daemon rm ... [--force] For example: .. prompt:: bash # ceph orch daemon rm mgr.my_host.xyzxyz .. note:: For managed services (``unmanaged=False``), cephadm will automatically deploy a new daemon a few seconds later. See also -------- * See :ref:`cephadm-osd-declarative` for special handling of unmanaged OSDs. * See also :ref:`cephadm-pause`