diff options
Diffstat (limited to 'doc/mgr/orchestrator_modules.rst')
| -rw-r--r-- | doc/mgr/orchestrator_modules.rst | 285 |
1 files changed, 285 insertions, 0 deletions
diff --git a/doc/mgr/orchestrator_modules.rst b/doc/mgr/orchestrator_modules.rst new file mode 100644 index 00000000..6e600571 --- /dev/null +++ b/doc/mgr/orchestrator_modules.rst @@ -0,0 +1,285 @@ + + +.. _orchestrator-modules: + +.. py:currentmodule:: orchestrator + +ceph-mgr orchestrator modules +============================= + +.. warning:: + + This is developer documentation, describing Ceph internals that + are only relevant to people writing ceph-mgr orchestrator modules. + +In this context, *orchestrator* refers to some external service that +provides the ability to discover devices and create Ceph services. This +includes external projects such as ceph-ansible, DeepSea, and Rook. + +An *orchestrator module* is a ceph-mgr module (:ref:`mgr-module-dev`) +which implements common management operations using a particular +orchestrator. + +Orchestrator modules subclass the ``Orchestrator`` class: this class is +an interface, it only provides method definitions to be implemented +by subclasses. The purpose of defining this common interface +for different orchestrators is to enable common UI code, such as +the dashboard, to work with various different backends. + + +.. graphviz:: + + digraph G { + subgraph cluster_1 { + volumes [label="mgr/volumes"] + rook [label="mgr/rook"] + dashboard [label="mgr/dashboard"] + orchestrator_cli [label="mgr/orchestrator_cli"] + orchestrator [label="Orchestrator Interface"] + ansible [label="mgr/ansible"] + ssh [label="mgr/ssh"] + deepsea [label="mgr/deepsea"] + + label = "ceph-mgr"; + } + + volumes -> orchestrator + dashboard -> orchestrator + orchestrator_cli -> orchestrator + orchestrator -> rook -> rook_io + orchestrator -> ansible -> ceph_ansible + orchestrator -> deepsea -> suse_deepsea + orchestrator -> ssh + + + rook_io [label="Rook"] + ceph_ansible [label="ceph-ansible"] + suse_deepsea [label="DeepSea"] + + rankdir="TB"; + } + +Behind all the abstraction, the purpose of orchestrator modules is simple: +enable Ceph to do things like discover available hardware, create and +destroy OSDs, and run MDS and RGW services. + +A tutorial is not included here: for full and concrete examples, see +the existing implemented orchestrator modules in the Ceph source tree. + +Glossary +-------- + +Stateful service + a daemon that uses local storage, such as OSD or mon. + +Stateless service + a daemon that doesn't use any local storage, such + as an MDS, RGW, nfs-ganesha, iSCSI gateway. + +Label + arbitrary string tags that may be applied by administrators + to nodes. Typically administrators use labels to indicate + which nodes should run which kinds of service. Labels are + advisory (from human input) and do not guarantee that nodes + have particular physical capabilities. + +Drive group + collection of block devices with common/shared OSD + formatting (typically one or more SSDs acting as + journals/dbs for a group of HDDs). + +Placement + choice of which node is used to run a service. + +Key Concepts +------------ + +The underlying orchestrator remains the source of truth for information +about whether a service is running, what is running where, which +nodes are available, etc. Orchestrator modules should avoid taking +any internal copies of this information, and read it directly from +the orchestrator backend as much as possible. + +Bootstrapping nodes and adding them to the underlying orchestration +system is outside the scope of Ceph's orchestrator interface. Ceph +can only work on nodes when the orchestrator is already aware of them. + +Calls to orchestrator modules are all asynchronous, and return *completion* +objects (see below) rather than returning values immediately. + +Where possible, placement of stateless services should be left up to the +orchestrator. + +Completions and batching +------------------------ + +All methods that read or modify the state of the system can potentially +be long running. To handle that, all such methods return a *completion* +object (a *ReadCompletion* or a *WriteCompletion*). Orchestrator modules +must implement the *wait* method: this takes a list of completions, and +is responsible for checking if they're finished, and advancing the underlying +operations as needed. + +Each orchestrator module implements its own underlying mechanisms +for completions. This might involve running the underlying operations +in threads, or batching the operations up before later executing +in one go in the background. If implementing such a batching pattern, the +module would do no work on any operation until it appeared in a list +of completions passed into *wait*. + +*WriteCompletion* objects have a two-stage execution. First they become +*persistent*, meaning that the write has made it to the orchestrator +itself, and been persisted there (e.g. a manifest file has been updated). +If ceph-mgr crashed at this point, the operation would still eventually take +effect. Second, the completion becomes *effective*, meaning that the operation has really happened (e.g. a service has actually been started). + +.. automethod:: Orchestrator.wait + +.. autoclass:: _Completion + :members: + +.. autoclass:: ReadCompletion + :members: + +.. autoclass:: WriteCompletion + :members: + +Placement +--------- + +In general, stateless services do not require any specific placement +rules, as they can run anywhere that sufficient system resources +are available. However, some orchestrators may not include the +functionality to choose a location in this way, so we can optionally +specify a location when creating a stateless service. + +OSD services generally require a specific placement choice, as this +will determine which storage devices are used. + +Error Handling +-------------- + +The main goal of error handling within orchestrator modules is to provide debug information to +assist users when dealing with deployment errors. + +.. autoclass:: OrchestratorError +.. autoclass:: NoOrchestrator +.. autoclass:: OrchestratorValidationError + + +In detail, orchestrators need to explicitly deal with different kinds of errors: + +1. No orchestrator configured + + See :class:`NoOrchestrator`. + +2. An orchestrator doesn't implement a specific method. + + For example, an Orchestrator doesn't support ``add_host``. + + In this case, a ``NotImplementedError`` is raised. + +3. Missing features within implemented methods. + + E.g. optional parameters to a command that are not supported by the + backend (e.g. the hosts field in :func:`Orchestrator.update_mons` command with the rook backend). + + See :class:`OrchestratorValidationError`. + +4. Input validation errors + + The ``orchestrator_cli`` module and other calling modules are supposed to + provide meaningful error messages. + + See :class:`OrchestratorValidationError`. + +5. Errors when actually executing commands + + The resulting Completion should contain an error string that assists in understanding the + problem. In addition, :func:`_Completion.is_errored` is set to ``True`` + +6. Invalid configuration in the orchestrator modules + + This can be tackled similar to 5. + + +All other errors are unexpected orchestrator issues and thus should raise an exception that are then +logged into the mgr log file. If there is a completion object at that point, +:func:`_Completion.result` may contain an error message. + + +Excluded functionality +---------------------- + +- Ceph's orchestrator interface is not a general purpose framework for + managing linux servers -- it is deliberately constrained to manage + the Ceph cluster's services only. +- Multipathed storage is not handled (multipathing is unnecessary for + Ceph clusters). Each drive is assumed to be visible only on + a single node. + +Host management +--------------- + +.. automethod:: Orchestrator.add_host +.. automethod:: Orchestrator.remove_host +.. automethod:: Orchestrator.get_hosts + +Inventory and status +-------------------- + +.. automethod:: Orchestrator.get_inventory +.. autoclass:: InventoryFilter +.. autoclass:: InventoryNode + +.. autoclass:: InventoryDevice + :members: + +.. automethod:: Orchestrator.describe_service +.. autoclass:: ServiceDescription + +Service Actions +--------------- + +.. automethod:: Orchestrator.service_action + +OSD management +-------------- + +.. automethod:: Orchestrator.create_osds +.. automethod:: Orchestrator.replace_osds +.. automethod:: Orchestrator.remove_osds + +.. autoclass:: DeviceSelection + :members: + +.. autoclass:: DriveGroupSpec + :members: + :exclude-members: from_json + +Stateless Services +------------------ + +.. automethod:: Orchestrator.add_stateless_service +.. automethod:: Orchestrator.update_stateless_service +.. automethod:: Orchestrator.remove_stateless_service + +Upgrades +-------- + +.. automethod:: Orchestrator.upgrade_available +.. automethod:: Orchestrator.upgrade_start +.. automethod:: Orchestrator.upgrade_status +.. autoclass:: UpgradeSpec +.. autoclass:: UpgradeStatusSpec + +Utility +------- + +.. automethod:: Orchestrator.available + +Client Modules +-------------- + +.. autoclass:: OrchestratorClientMixin + :members: |