summaryrefslogtreecommitdiffstats
path: root/doc/architecture.rst
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--doc/architecture.rst1688
1 files changed, 1688 insertions, 0 deletions
diff --git a/doc/architecture.rst b/doc/architecture.rst
new file mode 100644
index 000000000..852225ce6
--- /dev/null
+++ b/doc/architecture.rst
@@ -0,0 +1,1688 @@
+==============
+ Architecture
+==============
+
+:term:`Ceph` uniquely delivers **object, block, and file storage** in one
+unified system. Ceph is highly reliable, easy to manage, and free. The power of
+Ceph can transform your company's IT infrastructure and your ability to manage
+vast amounts of data. Ceph delivers extraordinary scalability–thousands of
+clients accessing petabytes to exabytes of data. A :term:`Ceph Node` leverages
+commodity hardware and intelligent daemons, and a :term:`Ceph Storage Cluster`
+accommodates large numbers of nodes, which communicate with each other to
+replicate and redistribute data dynamically.
+
+.. image:: images/stack.png
+
+.. _arch-ceph-storage-cluster:
+
+The Ceph Storage Cluster
+========================
+
+Ceph provides an infinitely scalable :term:`Ceph Storage Cluster` based upon
+:abbr:`RADOS (Reliable Autonomic Distributed Object Store)`, which you can read
+about in `RADOS - A Scalable, Reliable Storage Service for Petabyte-scale
+Storage Clusters`_.
+
+A Ceph Storage Cluster consists of multiple types of daemons:
+
+- :term:`Ceph Monitor`
+- :term:`Ceph OSD Daemon`
+- :term:`Ceph Manager`
+- :term:`Ceph Metadata Server`
+
+.. _arch_monitor:
+
+Ceph Monitors maintain the master copy of the cluster map, which they provide
+to Ceph clients. Provisioning multiple monitors within the Ceph cluster ensures
+availability in the event that one of the monitor daemons or its host fails.
+The Ceph monitor provides copies of the cluster map to storage cluster clients.
+
+A Ceph OSD Daemon checks its own state and the state of other OSDs and reports
+back to monitors.
+
+A Ceph Manager serves as an endpoint for monitoring, orchestration, and plug-in
+modules.
+
+A Ceph Metadata Server (MDS) manages file metadata when CephFS is used to
+provide file services.
+
+Storage cluster clients and :term:`Ceph OSD Daemon`\s use the CRUSH algorithm
+to compute information about data location. This means that clients and OSDs
+are not bottlenecked by a central lookup table. Ceph's high-level features
+include a native interface to the Ceph Storage Cluster via ``librados``, and a
+number of service interfaces built on top of ``librados``.
+
+Storing Data
+------------
+
+The Ceph Storage Cluster receives data from :term:`Ceph Client`\s--whether it
+comes through a :term:`Ceph Block Device`, :term:`Ceph Object Storage`, the
+:term:`Ceph File System`, or a custom implementation that you create by using
+``librados``. The data received by the Ceph Storage Cluster is stored as RADOS
+objects. Each object is stored on an :term:`Object Storage Device` (this is
+also called an "OSD"). Ceph OSDs control read, write, and replication
+operations on storage drives. The default BlueStore back end stores objects
+in a monolithic, database-like fashion.
+
+.. ditaa::
+
+ /------\ +-----+ +-----+
+ | obj |------>| {d} |------>| {s} |
+ \------/ +-----+ +-----+
+
+ Object OSD Drive
+
+Ceph OSD Daemons store data as objects in a flat namespace. This means that
+objects are not stored in a hierarchy of directories. An object has an
+identifier, binary data, and metadata consisting of name/value pairs.
+:term:`Ceph Client`\s determine the semantics of the object data. For example,
+CephFS uses metadata to store file attributes such as the file owner, the
+created date, and the last modified date.
+
+
+.. ditaa::
+
+ /------+------------------------------+----------------\
+ | ID | Binary Data | Metadata |
+ +------+------------------------------+----------------+
+ | 1234 | 0101010101010100110101010010 | name1 = value1 |
+ | | 0101100001010100110101010010 | name2 = value2 |
+ | | 0101100001010100110101010010 | nameN = valueN |
+ \------+------------------------------+----------------/
+
+.. note:: An object ID is unique across the entire cluster, not just the local
+ filesystem.
+
+
+.. index:: architecture; high availability, scalability
+
+.. _arch_scalability_and_high_availability:
+
+Scalability and High Availability
+---------------------------------
+
+In traditional architectures, clients talk to a centralized component. This
+centralized component might be a gateway, a broker, an API, or a facade. A
+centralized component of this kind acts as a single point of entry to a complex
+subsystem. Architectures that rely upon such a centralized component have a
+single point of failure and incur limits to performance and scalability. If
+the centralized component goes down, the whole system becomes unavailable.
+
+Ceph eliminates this centralized component. This enables clients to interact
+with Ceph OSDs directly. Ceph OSDs create object replicas on other Ceph Nodes
+to ensure data safety and high availability. Ceph also uses a cluster of
+monitors to ensure high availability. To eliminate centralization, Ceph uses an
+algorithm called :abbr:`CRUSH (Controlled Replication Under Scalable Hashing)`.
+
+
+.. index:: CRUSH; architecture
+
+CRUSH Introduction
+~~~~~~~~~~~~~~~~~~
+
+Ceph Clients and Ceph OSD Daemons both use the :abbr:`CRUSH (Controlled
+Replication Under Scalable Hashing)` algorithm to compute information about
+object location instead of relying upon a central lookup table. CRUSH provides
+a better data management mechanism than do older approaches, and CRUSH enables
+massive scale by distributing the work to all the OSD daemons in the cluster
+and all the clients that communicate with them. CRUSH uses intelligent data
+replication to ensure resiliency, which is better suited to hyper-scale
+storage. The following sections provide additional details on how CRUSH works.
+For a detailed discussion of CRUSH, see `CRUSH - Controlled, Scalable,
+Decentralized Placement of Replicated Data`_.
+
+.. index:: architecture; cluster map
+
+.. _architecture_cluster_map:
+
+Cluster Map
+~~~~~~~~~~~
+
+In order for a Ceph cluster to function properly, Ceph Clients and Ceph OSDs
+must have current information about the cluster's topology. Current information
+is stored in the "Cluster Map", which is in fact a collection of five maps. The
+five maps that constitute the cluster map are:
+
+#. **The Monitor Map:** Contains the cluster ``fsid``, the position, the name,
+ the address, and the TCP port of each monitor. The monitor map specifies the
+ current epoch, the time of the monitor map's creation, and the time of the
+ monitor map's last modification. To view a monitor map, run ``ceph mon
+ dump``.
+
+#. **The OSD Map:** Contains the cluster ``fsid``, the time of the OSD map's
+ creation, the time of the OSD map's last modification, a list of pools, a
+ list of replica sizes, a list of PG numbers, and a list of OSDs and their
+ statuses (for example, ``up``, ``in``). To view an OSD map, run ``ceph
+ osd dump``.
+
+#. **The PG Map:** Contains the PG version, its time stamp, the last OSD map
+ epoch, the full ratios, and the details of each placement group. This
+ includes the PG ID, the `Up Set`, the `Acting Set`, the state of the PG (for
+ example, ``active + clean``), and data usage statistics for each pool.
+
+#. **The CRUSH Map:** Contains a list of storage devices, the failure domain
+ hierarchy (for example, ``device``, ``host``, ``rack``, ``row``, ``room``),
+ and rules for traversing the hierarchy when storing data. To view a CRUSH
+ map, run ``ceph osd getcrushmap -o {filename}`` and then decompile it by
+ running ``crushtool -d {comp-crushmap-filename} -o
+ {decomp-crushmap-filename}``. Use a text editor or ``cat`` to view the
+ decompiled map.
+
+#. **The MDS Map:** Contains the current MDS map epoch, when the map was
+ created, and the last time it changed. It also contains the pool for
+ storing metadata, a list of metadata servers, and which metadata servers
+ are ``up`` and ``in``. To view an MDS map, execute ``ceph fs dump``.
+
+Each map maintains a history of changes to its operating state. Ceph Monitors
+maintain a master copy of the cluster map. This master copy includes the
+cluster members, the state of the cluster, changes to the cluster, and
+information recording the overall health of the Ceph Storage Cluster.
+
+.. index:: high availability; monitor architecture
+
+High Availability Monitors
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A Ceph Client must contact a Ceph Monitor and obtain a current copy of the
+cluster map in order to read data from or to write data to the Ceph cluster.
+
+It is possible for a Ceph cluster to function properly with only a single
+monitor, but a Ceph cluster that has only a single monitor has a single point
+of failure: if the monitor goes down, Ceph clients will be unable to read data
+from or write data to the cluster.
+
+Ceph leverages a cluster of monitors in order to increase reliability and fault
+tolerance. When a cluster of monitors is used, however, one or more of the
+monitors in the cluster can fall behind due to latency or other faults. Ceph
+mitigates these negative effects by requiring multiple monitor instances to
+agree about the state of the cluster. To establish consensus among the monitors
+regarding the state of the cluster, Ceph uses the `Paxos`_ algorithm and a
+majority of monitors (for example, one in a cluster that contains only one
+monitor, two in a cluster that contains three monitors, three in a cluster that
+contains five monitors, four in a cluster that contains six monitors, and so
+on).
+
+See the `Monitor Config Reference`_ for more detail on configuring monitors.
+
+.. index:: architecture; high availability authentication
+
+.. _arch_high_availability_authentication:
+
+High Availability Authentication
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``cephx`` authentication system is used by Ceph to authenticate users and
+daemons and to protect against man-in-the-middle attacks.
+
+.. note:: The ``cephx`` protocol does not address data encryption in transport
+ (for example, SSL/TLS) or encryption at rest.
+
+``cephx`` uses shared secret keys for authentication. This means that both the
+client and the monitor cluster keep a copy of the client's secret key.
+
+The ``cephx`` protocol makes it possible for each party to prove to the other
+that it has a copy of the key without revealing it. This provides mutual
+authentication and allows the cluster to confirm (1) that the user has the
+secret key and (2) that the user can be confident that the cluster has a copy
+of the secret key.
+
+As stated in :ref:`Scalability and High Availability
+<arch_scalability_and_high_availability>`, Ceph does not have any centralized
+interface between clients and the Ceph object store. By avoiding such a
+centralized interface, Ceph avoids the bottlenecks that attend such centralized
+interfaces. However, this means that clients must interact directly with OSDs.
+Direct interactions between Ceph clients and OSDs require authenticated
+connections. The ``cephx`` authentication system establishes and sustains these
+authenticated connections.
+
+The ``cephx`` protocol operates in a manner similar to `Kerberos`_.
+
+A user invokes a Ceph client to contact a monitor. Unlike Kerberos, each
+monitor can authenticate users and distribute keys, which means that there is
+no single point of failure and no bottleneck when using ``cephx``. The monitor
+returns an authentication data structure that is similar to a Kerberos ticket.
+This authentication data structure contains a session key for use in obtaining
+Ceph services. The session key is itself encrypted with the user's permanent
+secret key, which means that only the user can request services from the Ceph
+Monitors. The client then uses the session key to request services from the
+monitors, and the monitors provide the client with a ticket that authenticates
+the client against the OSDs that actually handle data. Ceph Monitors and OSDs
+share a secret, which means that the clients can use the ticket provided by the
+monitors to authenticate against any OSD or metadata server in the cluster.
+
+Like Kerberos tickets, ``cephx`` tickets expire. An attacker cannot use an
+expired ticket or session key that has been obtained surreptitiously. This form
+of authentication prevents attackers who have access to the communications
+medium from creating bogus messages under another user's identity and prevents
+attackers from altering another user's legitimate messages, as long as the
+user's secret key is not divulged before it expires.
+
+An administrator must set up users before using ``cephx``. In the following
+diagram, the ``client.admin`` user invokes ``ceph auth get-or-create-key`` from
+the command line to generate a username and secret key. Ceph's ``auth``
+subsystem generates the username and key, stores a copy on the monitor(s), and
+transmits the user's secret back to the ``client.admin`` user. This means that
+the client and the monitor share a secret key.
+
+.. note:: The ``client.admin`` user must provide the user ID and
+ secret key to the user in a secure manner.
+
+.. ditaa::
+
+ +---------+ +---------+
+ | Client | | Monitor |
+ +---------+ +---------+
+ | request to |
+ | create a user |
+ |-------------->|----------+ create user
+ | | | and
+ |<--------------|<---------+ store key
+ | transmit key |
+ | |
+
+Here is how a client authenticates with a monitor. The client passes the user
+name to the monitor. The monitor generates a session key that is encrypted with
+the secret key associated with the ``username``. The monitor transmits the
+encrypted ticket to the client. The client uses the shared secret key to
+decrypt the payload. The session key identifies the user, and this act of
+identification will last for the duration of the session. The client requests
+a ticket for the user, and the ticket is signed with the session key. The
+monitor generates a ticket and uses the user's secret key to encrypt it. The
+encrypted ticket is transmitted to the client. The client decrypts the ticket
+and uses it to sign requests to OSDs and to metadata servers in the cluster.
+
+.. ditaa::
+
+ +---------+ +---------+
+ | Client | | Monitor |
+ +---------+ +---------+
+ | authenticate |
+ |-------------->|----------+ generate and
+ | | | encrypt
+ |<--------------|<---------+ session key
+ | transmit |
+ | encrypted |
+ | session key |
+ | |
+ |-----+ decrypt |
+ | | session |
+ |<----+ key |
+ | |
+ | req. ticket |
+ |-------------->|----------+ generate and
+ | | | encrypt
+ |<--------------|<---------+ ticket
+ | recv. ticket |
+ | |
+ |-----+ decrypt |
+ | | ticket |
+ |<----+ |
+
+
+The ``cephx`` protocol authenticates ongoing communications between the clients
+and Ceph daemons. After initial authentication, each message sent between a
+client and a daemon is signed using a ticket that can be verified by monitors,
+OSDs, and metadata daemons. This ticket is verified by using the secret shared
+between the client and the daemon.
+
+.. ditaa::
+
+ +---------+ +---------+ +-------+ +-------+
+ | Client | | Monitor | | MDS | | OSD |
+ +---------+ +---------+ +-------+ +-------+
+ | request to | | |
+ | create a user | | |
+ |-------------->| mon and | |
+ |<--------------| client share | |
+ | receive | a secret. | |
+ | shared secret | | |
+ | |<------------>| |
+ | |<-------------+------------>|
+ | | mon, mds, | |
+ | authenticate | and osd | |
+ |-------------->| share | |
+ |<--------------| a secret | |
+ | session key | | |
+ | | | |
+ | req. ticket | | |
+ |-------------->| | |
+ |<--------------| | |
+ | recv. ticket | | |
+ | | | |
+ | make request (CephFS only) | |
+ |----------------------------->| |
+ |<-----------------------------| |
+ | receive response (CephFS only) |
+ | |
+ | make request |
+ |------------------------------------------->|
+ |<-------------------------------------------|
+ receive response
+
+This authentication protects only the connections between Ceph clients and Ceph
+daemons. The authentication is not extended beyond the Ceph client. If a user
+accesses the Ceph client from a remote host, cephx authentication will not be
+applied to the connection between the user's host and the client host.
+
+See `Cephx Config Guide`_ for more on configuration details.
+
+See `User Management`_ for more on user management.
+
+See :ref:`A Detailed Description of the Cephx Authentication Protocol
+<cephx_2012_peter>` for more on the distinction between authorization and
+authentication and for a step-by-step explanation of the setup of ``cephx``
+tickets and session keys.
+
+.. index:: architecture; smart daemons and scalability
+
+Smart Daemons Enable Hyperscale
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A feature of many storage clusters is a centralized interface that keeps track
+of the nodes that clients are permitted to access. Such centralized
+architectures provide services to clients by means of a double dispatch. At the
+petabyte-to-exabyte scale, such double dispatches are a significant
+bottleneck.
+
+Ceph obviates this bottleneck: Ceph's OSD Daemons AND Ceph clients are
+cluster-aware. Like Ceph clients, each Ceph OSD Daemon is aware of other Ceph
+OSD Daemons in the cluster. This enables Ceph OSD Daemons to interact directly
+with other Ceph OSD Daemons and to interact directly with Ceph Monitors. Being
+cluster-aware makes it possible for Ceph clients to interact directly with Ceph
+OSD Daemons.
+
+Because Ceph clients, Ceph monitors, and Ceph OSD daemons interact with one
+another directly, Ceph OSD daemons can make use of the aggregate CPU and RAM
+resources of the nodes in the Ceph cluster. This means that a Ceph cluster can
+easily perform tasks that a cluster with a centralized interface would struggle
+to perform. The ability of Ceph nodes to make use of the computing power of
+the greater cluster provides several benefits:
+
+#. **OSDs Service Clients Directly:** Network devices can support only a
+ limited number of concurrent connections. Because Ceph clients contact
+ Ceph OSD daemons directly without first connecting to a central interface,
+ Ceph enjoys improved perfomance and increased system capacity relative to
+ storage redundancy strategies that include a central interface. Ceph clients
+ maintain sessions only when needed, and maintain those sessions with only
+ particular Ceph OSD daemons, not with a centralized interface.
+
+#. **OSD Membership and Status**: When Ceph OSD Daemons join a cluster, they
+ report their status. At the lowest level, the Ceph OSD Daemon status is
+ ``up`` or ``down``: this reflects whether the Ceph OSD daemon is running and
+ able to service Ceph Client requests. If a Ceph OSD Daemon is ``down`` and
+ ``in`` the Ceph Storage Cluster, this status may indicate the failure of the
+ Ceph OSD Daemon. If a Ceph OSD Daemon is not running because it has crashed,
+ the Ceph OSD Daemon cannot notify the Ceph Monitor that it is ``down``. The
+ OSDs periodically send messages to the Ceph Monitor (in releases prior to
+ Luminous, this was done by means of ``MPGStats``, and beginning with the
+ Luminous release, this has been done with ``MOSDBeacon``). If the Ceph
+ Monitors receive no such message after a configurable period of time,
+ then they mark the OSD ``down``. This mechanism is a failsafe, however.
+ Normally, Ceph OSD Daemons determine if a neighboring OSD is ``down`` and
+ report it to the Ceph Monitors. This contributes to making Ceph Monitors
+ lightweight processes. See `Monitoring OSDs`_ and `Heartbeats`_ for
+ additional details.
+
+#. **Data Scrubbing:** To maintain data consistency, Ceph OSD Daemons scrub
+ RADOS objects. Ceph OSD Daemons compare the metadata of their own local
+ objects against the metadata of the replicas of those objects, which are
+ stored on other OSDs. Scrubbing occurs on a per-Placement-Group basis, finds
+ mismatches in object size and finds metadata mismatches, and is usually
+ performed daily. Ceph OSD Daemons perform deeper scrubbing by comparing the
+ data in objects, bit-for-bit, against their checksums. Deep scrubbing finds
+ bad sectors on drives that are not detectable with light scrubs. See `Data
+ Scrubbing`_ for details on configuring scrubbing.
+
+#. **Replication:** Data replication involves a collaboration between Ceph
+ Clients and Ceph OSD Daemons. Ceph OSD Daemons use the CRUSH algorithm to
+ determine the storage location of object replicas. Ceph clients use the
+ CRUSH algorithm to determine the storage location of an object, then the
+ object is mapped to a pool and to a placement group, and then the client
+ consults the CRUSH map to identify the placement group's primary OSD.
+
+ After identifying the target placement group, the client writes the object
+ to the identified placement group's primary OSD. The primary OSD then
+ consults its own copy of the CRUSH map to identify secondary and tertiary
+ OSDS, replicates the object to the placement groups in those secondary and
+ tertiary OSDs, confirms that the object was stored successfully in the
+ secondary and tertiary OSDs, and reports to the client that the object
+ was stored successfully.
+
+.. ditaa::
+
+ +----------+
+ | Client |
+ | |
+ +----------+
+ * ^
+ Write (1) | | Ack (6)
+ | |
+ v *
+ +-------------+
+ | Primary OSD |
+ | |
+ +-------------+
+ * ^ ^ *
+ Write (2) | | | | Write (3)
+ +------+ | | +------+
+ | +------+ +------+ |
+ | | Ack (4) Ack (5)| |
+ v * * v
+ +---------------+ +---------------+
+ | Secondary OSD | | Tertiary OSD |
+ | | | |
+ +---------------+ +---------------+
+
+By performing this act of data replication, Ceph OSD Daemons relieve Ceph
+clients of the burden of replicating data.
+
+Dynamic Cluster Management
+--------------------------
+
+In the `Scalability and High Availability`_ section, we explained how Ceph uses
+CRUSH, cluster topology, and intelligent daemons to scale and maintain high
+availability. Key to Ceph's design is the autonomous, self-healing, and
+intelligent Ceph OSD Daemon. Let's take a deeper look at how CRUSH works to
+enable modern cloud storage infrastructures to place data, rebalance the
+cluster, and adaptively place and balance data and recover from faults.
+
+.. index:: architecture; pools
+
+About Pools
+~~~~~~~~~~~
+
+The Ceph storage system supports the notion of 'Pools', which are logical
+partitions for storing objects.
+
+Ceph Clients retrieve a `Cluster Map`_ from a Ceph Monitor, and write RADOS
+objects to pools. The way that Ceph places the data in the pools is determined
+by the pool's ``size`` or number of replicas, the CRUSH rule, and the number of
+placement groups in the pool.
+
+.. ditaa::
+
+ +--------+ Retrieves +---------------+
+ | Client |------------>| Cluster Map |
+ +--------+ +---------------+
+ |
+ v Writes
+ /-----\
+ | obj |
+ \-----/
+ | To
+ v
+ +--------+ +---------------+
+ | Pool |---------->| CRUSH Rule |
+ +--------+ Selects +---------------+
+
+
+Pools set at least the following parameters:
+
+- Ownership/Access to Objects
+- The Number of Placement Groups, and
+- The CRUSH Rule to Use.
+
+See `Set Pool Values`_ for details.
+
+
+.. index: architecture; placement group mapping
+
+Mapping PGs to OSDs
+~~~~~~~~~~~~~~~~~~~
+
+Each pool has a number of placement groups (PGs) within it. CRUSH dynamically
+maps PGs to OSDs. When a Ceph Client stores objects, CRUSH maps each RADOS
+object to a PG.
+
+This mapping of RADOS objects to PGs implements an abstraction and indirection
+layer between Ceph OSD Daemons and Ceph Clients. The Ceph Storage Cluster must
+be able to grow (or shrink) and redistribute data adaptively when the internal
+topology changes.
+
+If the Ceph Client "knew" which Ceph OSD Daemons were storing which objects, a
+tight coupling would exist between the Ceph Client and the Ceph OSD Daemon.
+But Ceph avoids any such tight coupling. Instead, the CRUSH algorithm maps each
+RADOS object to a placement group and then maps each placement group to one or
+more Ceph OSD Daemons. This "layer of indirection" allows Ceph to rebalance
+dynamically when new Ceph OSD Daemons and their underlying OSD devices come
+online. The following diagram shows how the CRUSH algorithm maps objects to
+placement groups, and how it maps placement groups to OSDs.
+
+.. ditaa::
+
+ /-----\ /-----\ /-----\ /-----\ /-----\
+ | obj | | obj | | obj | | obj | | obj |
+ \-----/ \-----/ \-----/ \-----/ \-----/
+ | | | | |
+ +--------+--------+ +---+----+
+ | |
+ v v
+ +-----------------------+ +-----------------------+
+ | Placement Group #1 | | Placement Group #2 |
+ | | | |
+ +-----------------------+ +-----------------------+
+ | |
+ | +-----------------------+---+
+ +------+------+-------------+ |
+ | | | |
+ v v v v
+ /----------\ /----------\ /----------\ /----------\
+ | | | | | | | |
+ | OSD #1 | | OSD #2 | | OSD #3 | | OSD #4 |
+ | | | | | | | |
+ \----------/ \----------/ \----------/ \----------/
+
+The client uses its copy of the cluster map and the CRUSH algorithm to compute
+precisely which OSD it will use when reading or writing a particular object.
+
+.. index:: architecture; calculating PG IDs
+
+Calculating PG IDs
+~~~~~~~~~~~~~~~~~~
+
+When a Ceph Client binds to a Ceph Monitor, it retrieves the latest version of
+the `Cluster Map`_. When a client has been equipped with a copy of the cluster
+map, it is aware of all the monitors, OSDs, and metadata servers in the
+cluster. **However, even equipped with a copy of the latest version of the
+cluster map, the client doesn't know anything about object locations.**
+
+**Object locations must be computed.**
+
+The client requies only the object ID and the name of the pool in order to
+compute the object location.
+
+Ceph stores data in named pools (for example, "liverpool"). When a client
+stores a named object (for example, "john", "paul", "george", or "ringo") it
+calculates a placement group by using the object name, a hash code, the number
+of PGs in the pool, and the pool name. Ceph clients use the following steps to
+compute PG IDs.
+
+#. The client inputs the pool name and the object ID. (for example: pool =
+ "liverpool" and object-id = "john")
+#. Ceph hashes the object ID.
+#. Ceph calculates the hash, modulo the number of PGs (for example: ``58``), to
+ get a PG ID.
+#. Ceph uses the pool name to retrieve the pool ID: (for example: "liverpool" =
+ ``4``)
+#. Ceph prepends the pool ID to the PG ID (for example: ``4.58``).
+
+It is much faster to compute object locations than to perform object location
+query over a chatty session. The :abbr:`CRUSH (Controlled Replication Under
+Scalable Hashing)` algorithm allows a client to compute where objects are
+expected to be stored, and enables the client to contact the primary OSD to
+store or retrieve the objects.
+
+.. index:: architecture; PG Peering
+
+Peering and Sets
+~~~~~~~~~~~~~~~~
+
+In previous sections, we noted that Ceph OSD Daemons check each other's
+heartbeats and report back to Ceph Monitors. Ceph OSD daemons also 'peer',
+which is the process of bringing all of the OSDs that store a Placement Group
+(PG) into agreement about the state of all of the RADOS objects (and their
+metadata) in that PG. Ceph OSD Daemons `Report Peering Failure`_ to the Ceph
+Monitors. Peering issues usually resolve themselves; however, if the problem
+persists, you may need to refer to the `Troubleshooting Peering Failure`_
+section.
+
+.. Note:: PGs that agree on the state of the cluster do not necessarily have
+ the current data yet.
+
+The Ceph Storage Cluster was designed to store at least two copies of an object
+(that is, ``size = 2``), which is the minimum requirement for data safety. For
+high availability, a Ceph Storage Cluster should store more than two copies of
+an object (that is, ``size = 3`` and ``min size = 2``) so that it can continue
+to run in a ``degraded`` state while maintaining data safety.
+
+.. warning:: Although we say here that R2 (replication with two copies) is the
+ minimum requirement for data safety, R3 (replication with three copies) is
+ recommended. On a long enough timeline, data stored with an R2 strategy will
+ be lost.
+
+As explained in the diagram in `Smart Daemons Enable Hyperscale`_, we do not
+name the Ceph OSD Daemons specifically (for example, ``osd.0``, ``osd.1``,
+etc.), but rather refer to them as *Primary*, *Secondary*, and so forth. By
+convention, the *Primary* is the first OSD in the *Acting Set*, and is
+responsible for orchestrating the peering process for each placement group
+where it acts as the *Primary*. The *Primary* is the **ONLY** OSD in a given
+placement group that accepts client-initiated writes to objects.
+
+The set of OSDs that is responsible for a placement group is called the
+*Acting Set*. The term "*Acting Set*" can refer either to the Ceph OSD Daemons
+that are currently responsible for the placement group, or to the Ceph OSD
+Daemons that were responsible for a particular placement group as of some
+epoch.
+
+The Ceph OSD daemons that are part of an *Acting Set* might not always be
+``up``. When an OSD in the *Acting Set* is ``up``, it is part of the *Up Set*.
+The *Up Set* is an important distinction, because Ceph can remap PGs to other
+Ceph OSD Daemons when an OSD fails.
+
+.. note:: Consider a hypothetical *Acting Set* for a PG that contains
+ ``osd.25``, ``osd.32`` and ``osd.61``. The first OSD (``osd.25``), is the
+ *Primary*. If that OSD fails, the Secondary (``osd.32``), becomes the
+ *Primary*, and ``osd.25`` is removed from the *Up Set*.
+
+.. index:: architecture; Rebalancing
+
+Rebalancing
+~~~~~~~~~~~
+
+When you add a Ceph OSD Daemon to a Ceph Storage Cluster, the cluster map gets
+updated with the new OSD. Referring back to `Calculating PG IDs`_, this changes
+the cluster map. Consequently, it changes object placement, because it changes
+an input for the calculations. The following diagram depicts the rebalancing
+process (albeit rather crudely, since it is substantially less impactful with
+large clusters) where some, but not all of the PGs migrate from existing OSDs
+(OSD 1, and OSD 2) to the new OSD (OSD 3). Even when rebalancing, CRUSH is
+stable. Many of the placement groups remain in their original configuration,
+and each OSD gets some added capacity, so there are no load spikes on the
+new OSD after rebalancing is complete.
+
+
+.. ditaa::
+
+ +--------+ +--------+
+ Before | OSD 1 | | OSD 2 |
+ +--------+ +--------+
+ | PG #1 | | PG #6 |
+ | PG #2 | | PG #7 |
+ | PG #3 | | PG #8 |
+ | PG #4 | | PG #9 |
+ | PG #5 | | PG #10 |
+ +--------+ +--------+
+
+ +--------+ +--------+ +--------+
+ After | OSD 1 | | OSD 2 | | OSD 3 |
+ +--------+ +--------+ +--------+
+ | PG #1 | | PG #7 | | PG #3 |
+ | PG #2 | | PG #8 | | PG #6 |
+ | PG #4 | | PG #10 | | PG #9 |
+ | PG #5 | | | | |
+ | | | | | |
+ +--------+ +--------+ +--------+
+
+
+.. index:: architecture; Data Scrubbing
+
+Data Consistency
+~~~~~~~~~~~~~~~~
+
+As part of maintaining data consistency and cleanliness, Ceph OSDs also scrub
+objects within placement groups. That is, Ceph OSDs compare object metadata in
+one placement group with its replicas in placement groups stored in other
+OSDs. Scrubbing (usually performed daily) catches OSD bugs or filesystem
+errors, often as a result of hardware issues. OSDs also perform deeper
+scrubbing by comparing data in objects bit-for-bit. Deep scrubbing (by default
+performed weekly) finds bad blocks on a drive that weren't apparent in a light
+scrub.
+
+See `Data Scrubbing`_ for details on configuring scrubbing.
+
+
+
+
+
+.. index:: erasure coding
+
+Erasure Coding
+--------------
+
+An erasure coded pool stores each object as ``K+M`` chunks. It is divided into
+``K`` data chunks and ``M`` coding chunks. The pool is configured to have a size
+of ``K+M`` so that each chunk is stored in an OSD in the acting set. The rank of
+the chunk is stored as an attribute of the object.
+
+For instance an erasure coded pool can be created to use five OSDs (``K+M = 5``) and
+sustain the loss of two of them (``M = 2``).
+
+Reading and Writing Encoded Chunks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When the object **NYAN** containing ``ABCDEFGHI`` is written to the pool, the erasure
+encoding function splits the content into three data chunks simply by dividing
+the content in three: the first contains ``ABC``, the second ``DEF`` and the
+last ``GHI``. The content will be padded if the content length is not a multiple
+of ``K``. The function also creates two coding chunks: the fourth with ``YXY``
+and the fifth with ``QGC``. Each chunk is stored in an OSD in the acting set.
+The chunks are stored in objects that have the same name (**NYAN**) but reside
+on different OSDs. The order in which the chunks were created must be preserved
+and is stored as an attribute of the object (``shard_t``), in addition to its
+name. Chunk 1 contains ``ABC`` and is stored on **OSD5** while chunk 4 contains
+``YXY`` and is stored on **OSD3**.
+
+
+.. ditaa::
+
+ +-------------------+
+ name | NYAN |
+ +-------------------+
+ content | ABCDEFGHI |
+ +--------+----------+
+ |
+ |
+ v
+ +------+------+
+ +---------------+ encode(3,2) +-----------+
+ | +--+--+---+---+ |
+ | | | | |
+ | +-------+ | +-----+ |
+ | | | | |
+ +--v---+ +--v---+ +--v---+ +--v---+ +--v---+
+ name | NYAN | | NYAN | | NYAN | | NYAN | | NYAN |
+ +------+ +------+ +------+ +------+ +------+
+ shard | 1 | | 2 | | 3 | | 4 | | 5 |
+ +------+ +------+ +------+ +------+ +------+
+ content | ABC | | DEF | | GHI | | YXY | | QGC |
+ +--+---+ +--+---+ +--+---+ +--+---+ +--+---+
+ | | | | |
+ | | v | |
+ | | +--+---+ | |
+ | | | OSD1 | | |
+ | | +------+ | |
+ | | | |
+ | | +------+ | |
+ | +------>| OSD2 | | |
+ | +------+ | |
+ | | |
+ | +------+ | |
+ | | OSD3 |<----+ |
+ | +------+ |
+ | |
+ | +------+ |
+ | | OSD4 |<--------------+
+ | +------+
+ |
+ | +------+
+ +----------------->| OSD5 |
+ +------+
+
+
+When the object **NYAN** is read from the erasure coded pool, the decoding
+function reads three chunks: chunk 1 containing ``ABC``, chunk 3 containing
+``GHI`` and chunk 4 containing ``YXY``. Then, it rebuilds the original content
+of the object ``ABCDEFGHI``. The decoding function is informed that the chunks 2
+and 5 are missing (they are called 'erasures'). The chunk 5 could not be read
+because the **OSD4** is out. The decoding function can be called as soon as
+three chunks are read: **OSD2** was the slowest and its chunk was not taken into
+account.
+
+.. ditaa::
+
+ +-------------------+
+ name | NYAN |
+ +-------------------+
+ content | ABCDEFGHI |
+ +---------+---------+
+ ^
+ |
+ |
+ +-------+-------+
+ | decode(3,2) |
+ +------------->+ erasures 2,5 +<-+
+ | | | |
+ | +-------+-------+ |
+ | ^ |
+ | | |
+ | | |
+ +--+---+ +------+ +---+--+ +---+--+
+ name | NYAN | | NYAN | | NYAN | | NYAN |
+ +------+ +------+ +------+ +------+
+ shard | 1 | | 2 | | 3 | | 4 |
+ +------+ +------+ +------+ +------+
+ content | ABC | | DEF | | GHI | | YXY |
+ +--+---+ +--+---+ +--+---+ +--+---+
+ ^ . ^ ^
+ | TOO . | |
+ | SLOW . +--+---+ |
+ | ^ | OSD1 | |
+ | | +------+ |
+ | | |
+ | | +------+ |
+ | +-------| OSD2 | |
+ | +------+ |
+ | |
+ | +------+ |
+ | | OSD3 |------+
+ | +------+
+ |
+ | +------+
+ | | OSD4 | OUT
+ | +------+
+ |
+ | +------+
+ +------------------| OSD5 |
+ +------+
+
+
+Interrupted Full Writes
+~~~~~~~~~~~~~~~~~~~~~~~
+
+In an erasure coded pool, the primary OSD in the up set receives all write
+operations. It is responsible for encoding the payload into ``K+M`` chunks and
+sends them to the other OSDs. It is also responsible for maintaining an
+authoritative version of the placement group logs.
+
+In the following diagram, an erasure coded placement group has been created with
+``K = 2, M = 1`` and is supported by three OSDs, two for ``K`` and one for
+``M``. The acting set of the placement group is made of **OSD 1**, **OSD 2** and
+**OSD 3**. An object has been encoded and stored in the OSDs : the chunk
+``D1v1`` (i.e. Data chunk number 1, version 1) is on **OSD 1**, ``D2v1`` on
+**OSD 2** and ``C1v1`` (i.e. Coding chunk number 1, version 1) on **OSD 3**. The
+placement group logs on each OSD are identical (i.e. ``1,1`` for epoch 1,
+version 1).
+
+
+.. ditaa::
+
+ Primary OSD
+
+ +-------------+
+ | OSD 1 | +-------------+
+ | log | Write Full | |
+ | +----+ |<------------+ Ceph Client |
+ | |D1v1| 1,1 | v1 | |
+ | +----+ | +-------------+
+ +------+------+
+ |
+ |
+ | +-------------+
+ | | OSD 2 |
+ | | log |
+ +--------->+ +----+ |
+ | | |D2v1| 1,1 |
+ | | +----+ |
+ | +-------------+
+ |
+ | +-------------+
+ | | OSD 3 |
+ | | log |
+ +--------->| +----+ |
+ | |C1v1| 1,1 |
+ | +----+ |
+ +-------------+
+
+**OSD 1** is the primary and receives a **WRITE FULL** from a client, which
+means the payload is to replace the object entirely instead of overwriting a
+portion of it. Version 2 (v2) of the object is created to override version 1
+(v1). **OSD 1** encodes the payload into three chunks: ``D1v2`` (i.e. Data
+chunk number 1 version 2) will be on **OSD 1**, ``D2v2`` on **OSD 2** and
+``C1v2`` (i.e. Coding chunk number 1 version 2) on **OSD 3**. Each chunk is sent
+to the target OSD, including the primary OSD which is responsible for storing
+chunks in addition to handling write operations and maintaining an authoritative
+version of the placement group logs. When an OSD receives the message
+instructing it to write the chunk, it also creates a new entry in the placement
+group logs to reflect the change. For instance, as soon as **OSD 3** stores
+``C1v2``, it adds the entry ``1,2`` ( i.e. epoch 1, version 2 ) to its logs.
+Because the OSDs work asynchronously, some chunks may still be in flight ( such
+as ``D2v2`` ) while others are acknowledged and persisted to storage drives
+(such as ``C1v1`` and ``D1v1``).
+
+.. ditaa::
+
+ Primary OSD
+
+ +-------------+
+ | OSD 1 |
+ | log |
+ | +----+ | +-------------+
+ | |D1v2| 1,2 | Write Full | |
+ | +----+ +<------------+ Ceph Client |
+ | | v2 | |
+ | +----+ | +-------------+
+ | |D1v1| 1,1 |
+ | +----+ |
+ +------+------+
+ |
+ |
+ | +------+------+
+ | | OSD 2 |
+ | +------+ | log |
+ +->| D2v2 | | +----+ |
+ | +------+ | |D2v1| 1,1 |
+ | | +----+ |
+ | +-------------+
+ |
+ | +-------------+
+ | | OSD 3 |
+ | | log |
+ | | +----+ |
+ | | |C1v2| 1,2 |
+ +---------->+ +----+ |
+ | |
+ | +----+ |
+ | |C1v1| 1,1 |
+ | +----+ |
+ +-------------+
+
+
+If all goes well, the chunks are acknowledged on each OSD in the acting set and
+the logs' ``last_complete`` pointer can move from ``1,1`` to ``1,2``.
+
+.. ditaa::
+
+ Primary OSD
+
+ +-------------+
+ | OSD 1 |
+ | log |
+ | +----+ | +-------------+
+ | |D1v2| 1,2 | Write Full | |
+ | +----+ +<------------+ Ceph Client |
+ | | v2 | |
+ | +----+ | +-------------+
+ | |D1v1| 1,1 |
+ | +----+ |
+ +------+------+
+ |
+ | +-------------+
+ | | OSD 2 |
+ | | log |
+ | | +----+ |
+ | | |D2v2| 1,2 |
+ +---------->+ +----+ |
+ | | |
+ | | +----+ |
+ | | |D2v1| 1,1 |
+ | | +----+ |
+ | +-------------+
+ |
+ | +-------------+
+ | | OSD 3 |
+ | | log |
+ | | +----+ |
+ | | |C1v2| 1,2 |
+ +---------->+ +----+ |
+ | |
+ | +----+ |
+ | |C1v1| 1,1 |
+ | +----+ |
+ +-------------+
+
+
+Finally, the files used to store the chunks of the previous version of the
+object can be removed: ``D1v1`` on **OSD 1**, ``D2v1`` on **OSD 2** and ``C1v1``
+on **OSD 3**.
+
+.. ditaa::
+
+ Primary OSD
+
+ +-------------+
+ | OSD 1 |
+ | log |
+ | +----+ |
+ | |D1v2| 1,2 |
+ | +----+ |
+ +------+------+
+ |
+ |
+ | +-------------+
+ | | OSD 2 |
+ | | log |
+ +--------->+ +----+ |
+ | | |D2v2| 1,2 |
+ | | +----+ |
+ | +-------------+
+ |
+ | +-------------+
+ | | OSD 3 |
+ | | log |
+ +--------->| +----+ |
+ | |C1v2| 1,2 |
+ | +----+ |
+ +-------------+
+
+
+But accidents happen. If **OSD 1** goes down while ``D2v2`` is still in flight,
+the object's version 2 is partially written: **OSD 3** has one chunk but that is
+not enough to recover. It lost two chunks: ``D1v2`` and ``D2v2`` and the
+erasure coding parameters ``K = 2``, ``M = 1`` require that at least two chunks are
+available to rebuild the third. **OSD 4** becomes the new primary and finds that
+the ``last_complete`` log entry (i.e., all objects before this entry were known
+to be available on all OSDs in the previous acting set ) is ``1,1`` and that
+will be the head of the new authoritative log.
+
+.. ditaa::
+
+ +-------------+
+ | OSD 1 |
+ | (down) |
+ | c333 |
+ +------+------+
+ |
+ | +-------------+
+ | | OSD 2 |
+ | | log |
+ | | +----+ |
+ +---------->+ |D2v1| 1,1 |
+ | | +----+ |
+ | | |
+ | +-------------+
+ |
+ | +-------------+
+ | | OSD 3 |
+ | | log |
+ | | +----+ |
+ | | |C1v2| 1,2 |
+ +---------->+ +----+ |
+ | |
+ | +----+ |
+ | |C1v1| 1,1 |
+ | +----+ |
+ +-------------+
+ Primary OSD
+ +-------------+
+ | OSD 4 |
+ | log |
+ | |
+ | 1,1 |
+ | |
+ +------+------+
+
+
+
+The log entry 1,2 found on **OSD 3** is divergent from the new authoritative log
+provided by **OSD 4**: it is discarded and the file containing the ``C1v2``
+chunk is removed. The ``D1v1`` chunk is rebuilt with the ``decode`` function of
+the erasure coding library during scrubbing and stored on the new primary
+**OSD 4**.
+
+
+.. ditaa::
+
+ Primary OSD
+
+ +-------------+
+ | OSD 4 |
+ | log |
+ | +----+ |
+ | |D1v1| 1,1 |
+ | +----+ |
+ +------+------+
+ ^
+ |
+ | +-------------+
+ | | OSD 2 |
+ | | log |
+ +----------+ +----+ |
+ | | |D2v1| 1,1 |
+ | | +----+ |
+ | +-------------+
+ |
+ | +-------------+
+ | | OSD 3 |
+ | | log |
+ +----------| +----+ |
+ | |C1v1| 1,1 |
+ | +----+ |
+ +-------------+
+
+ +-------------+
+ | OSD 1 |
+ | (down) |
+ | c333 |
+ +-------------+
+
+See `Erasure Code Notes`_ for additional details.
+
+
+
+Cache Tiering
+-------------
+
+.. note:: Cache tiering is deprecated in Reef.
+
+A cache tier provides Ceph Clients with better I/O performance for a subset of
+the data stored in a backing storage tier. Cache tiering involves creating a
+pool of relatively fast/expensive storage devices (e.g., solid state drives)
+configured to act as a cache tier, and a backing pool of either erasure-coded
+or relatively slower/cheaper devices configured to act as an economical storage
+tier. The Ceph objecter handles where to place the objects and the tiering
+agent determines when to flush objects from the cache to the backing storage
+tier. So the cache tier and the backing storage tier are completely transparent
+to Ceph clients.
+
+
+.. ditaa::
+
+ +-------------+
+ | Ceph Client |
+ +------+------+
+ ^
+ Tiering is |
+ Transparent | Faster I/O
+ to Ceph | +---------------+
+ Client Ops | | |
+ | +----->+ Cache Tier |
+ | | | |
+ | | +-----+---+-----+
+ | | | ^
+ v v | | Active Data in Cache Tier
+ +------+----+--+ | |
+ | Objecter | | |
+ +-----------+--+ | |
+ ^ | | Inactive Data in Storage Tier
+ | v |
+ | +-----+---+-----+
+ | | |
+ +----->| Storage Tier |
+ | |
+ +---------------+
+ Slower I/O
+
+See `Cache Tiering`_ for additional details. Note that Cache Tiers can be
+tricky and their use is now discouraged.
+
+
+.. index:: Extensibility, Ceph Classes
+
+Extending Ceph
+--------------
+
+You can extend Ceph by creating shared object classes called 'Ceph Classes'.
+Ceph loads ``.so`` classes stored in the ``osd class dir`` directory dynamically
+(i.e., ``$libdir/rados-classes`` by default). When you implement a class, you
+can create new object methods that have the ability to call the native methods
+in the Ceph Object Store, or other class methods you incorporate via libraries
+or create yourself.
+
+On writes, Ceph Classes can call native or class methods, perform any series of
+operations on the inbound data and generate a resulting write transaction that
+Ceph will apply atomically.
+
+On reads, Ceph Classes can call native or class methods, perform any series of
+operations on the outbound data and return the data to the client.
+
+.. topic:: Ceph Class Example
+
+ A Ceph class for a content management system that presents pictures of a
+ particular size and aspect ratio could take an inbound bitmap image, crop it
+ to a particular aspect ratio, resize it and embed an invisible copyright or
+ watermark to help protect the intellectual property; then, save the
+ resulting bitmap image to the object store.
+
+See ``src/objclass/objclass.h``, ``src/fooclass.cc`` and ``src/barclass`` for
+exemplary implementations.
+
+
+Summary
+-------
+
+Ceph Storage Clusters are dynamic--like a living organism. Whereas, many storage
+appliances do not fully utilize the CPU and RAM of a typical commodity server,
+Ceph does. From heartbeats, to peering, to rebalancing the cluster or
+recovering from faults, Ceph offloads work from clients (and from a centralized
+gateway which doesn't exist in the Ceph architecture) and uses the computing
+power of the OSDs to perform the work. When referring to `Hardware
+Recommendations`_ and the `Network Config Reference`_, be cognizant of the
+foregoing concepts to understand how Ceph utilizes computing resources.
+
+.. index:: Ceph Protocol, librados
+
+Ceph Protocol
+=============
+
+Ceph Clients use the native protocol for interacting with the Ceph Storage
+Cluster. Ceph packages this functionality into the ``librados`` library so that
+you can create your own custom Ceph Clients. The following diagram depicts the
+basic architecture.
+
+.. ditaa::
+
+ +---------------------------------+
+ | Ceph Storage Cluster Protocol |
+ | (librados) |
+ +---------------------------------+
+ +---------------+ +---------------+
+ | OSDs | | Monitors |
+ +---------------+ +---------------+
+
+
+Native Protocol and ``librados``
+--------------------------------
+
+Modern applications need a simple object storage interface with asynchronous
+communication capability. The Ceph Storage Cluster provides a simple object
+storage interface with asynchronous communication capability. The interface
+provides direct, parallel access to objects throughout the cluster.
+
+
+- Pool Operations
+- Snapshots and Copy-on-write Cloning
+- Read/Write Objects
+ - Create or Remove
+ - Entire Object or Byte Range
+ - Append or Truncate
+- Create/Set/Get/Remove XATTRs
+- Create/Set/Get/Remove Key/Value Pairs
+- Compound operations and dual-ack semantics
+- Object Classes
+
+
+.. index:: architecture; watch/notify
+
+Object Watch/Notify
+-------------------
+
+A client can register a persistent interest with an object and keep a session to
+the primary OSD open. The client can send a notification message and a payload to
+all watchers and receive notification when the watchers receive the
+notification. This enables a client to use any object as a
+synchronization/communication channel.
+
+
+.. ditaa::
+
+ +----------+ +----------+ +----------+ +---------------+
+ | Client 1 | | Client 2 | | Client 3 | | OSD:Object ID |
+ +----------+ +----------+ +----------+ +---------------+
+ | | | |
+ | | | |
+ | | Watch Object | |
+ |--------------------------------------------------->|
+ | | | |
+ |<---------------------------------------------------|
+ | | Ack/Commit | |
+ | | | |
+ | | Watch Object | |
+ | |---------------------------------->|
+ | | | |
+ | |<----------------------------------|
+ | | Ack/Commit | |
+ | | | Watch Object |
+ | | |----------------->|
+ | | | |
+ | | |<-----------------|
+ | | | Ack/Commit |
+ | | Notify | |
+ |--------------------------------------------------->|
+ | | | |
+ |<---------------------------------------------------|
+ | | Notify | |
+ | | | |
+ | |<----------------------------------|
+ | | Notify | |
+ | | |<-----------------|
+ | | | Notify |
+ | | Ack | |
+ |----------------+---------------------------------->|
+ | | | |
+ | | Ack | |
+ | +---------------------------------->|
+ | | | |
+ | | | Ack |
+ | | |----------------->|
+ | | | |
+ |<---------------+----------------+------------------|
+ | Complete
+
+.. index:: architecture; Striping
+
+Data Striping
+-------------
+
+Storage devices have throughput limitations, which impact performance and
+scalability. So storage systems often support `striping`_--storing sequential
+pieces of information across multiple storage devices--to increase throughput
+and performance. The most common form of data striping comes from `RAID`_.
+The RAID type most similar to Ceph's striping is `RAID 0`_, or a 'striped
+volume'. Ceph's striping offers the throughput of RAID 0 striping, the
+reliability of n-way RAID mirroring and faster recovery.
+
+Ceph provides three types of clients: Ceph Block Device, Ceph File System, and
+Ceph Object Storage. A Ceph Client converts its data from the representation
+format it provides to its users (a block device image, RESTful objects, CephFS
+filesystem directories) into objects for storage in the Ceph Storage Cluster.
+
+.. tip:: The objects Ceph stores in the Ceph Storage Cluster are not striped.
+ Ceph Object Storage, Ceph Block Device, and the Ceph File System stripe their
+ data over multiple Ceph Storage Cluster objects. Ceph Clients that write
+ directly to the Ceph Storage Cluster via ``librados`` must perform the
+ striping (and parallel I/O) for themselves to obtain these benefits.
+
+The simplest Ceph striping format involves a stripe count of 1 object. Ceph
+Clients write stripe units to a Ceph Storage Cluster object until the object is
+at its maximum capacity, and then create another object for additional stripes
+of data. The simplest form of striping may be sufficient for small block device
+images, S3 or Swift objects and CephFS files. However, this simple form doesn't
+take maximum advantage of Ceph's ability to distribute data across placement
+groups, and consequently doesn't improve performance very much. The following
+diagram depicts the simplest form of striping:
+
+.. ditaa::
+
+ +---------------+
+ | Client Data |
+ | Format |
+ | cCCC |
+ +---------------+
+ |
+ +--------+-------+
+ | |
+ v v
+ /-----------\ /-----------\
+ | Begin cCCC| | Begin cCCC|
+ | Object 0 | | Object 1 |
+ +-----------+ +-----------+
+ | stripe | | stripe |
+ | unit 1 | | unit 5 |
+ +-----------+ +-----------+
+ | stripe | | stripe |
+ | unit 2 | | unit 6 |
+ +-----------+ +-----------+
+ | stripe | | stripe |
+ | unit 3 | | unit 7 |
+ +-----------+ +-----------+
+ | stripe | | stripe |
+ | unit 4 | | unit 8 |
+ +-----------+ +-----------+
+ | End cCCC | | End cCCC |
+ | Object 0 | | Object 1 |
+ \-----------/ \-----------/
+
+
+If you anticipate large images sizes, large S3 or Swift objects (e.g., video),
+or large CephFS directories, you may see considerable read/write performance
+improvements by striping client data over multiple objects within an object set.
+Significant write performance occurs when the client writes the stripe units to
+their corresponding objects in parallel. Since objects get mapped to different
+placement groups and further mapped to different OSDs, each write occurs in
+parallel at the maximum write speed. A write to a single drive would be limited
+by the head movement (e.g. 6ms per seek) and bandwidth of that one device (e.g.
+100MB/s). By spreading that write over multiple objects (which map to different
+placement groups and OSDs) Ceph can reduce the number of seeks per drive and
+combine the throughput of multiple drives to achieve much faster write (or read)
+speeds.
+
+.. note:: Striping is independent of object replicas. Since CRUSH
+ replicates objects across OSDs, stripes get replicated automatically.
+
+In the following diagram, client data gets striped across an object set
+(``object set 1`` in the following diagram) consisting of 4 objects, where the
+first stripe unit is ``stripe unit 0`` in ``object 0``, and the fourth stripe
+unit is ``stripe unit 3`` in ``object 3``. After writing the fourth stripe, the
+client determines if the object set is full. If the object set is not full, the
+client begins writing a stripe to the first object again (``object 0`` in the
+following diagram). If the object set is full, the client creates a new object
+set (``object set 2`` in the following diagram), and begins writing to the first
+stripe (``stripe unit 16``) in the first object in the new object set (``object
+4`` in the diagram below).
+
+.. ditaa::
+
+ +---------------+
+ | Client Data |
+ | Format |
+ | cCCC |
+ +---------------+
+ |
+ +-----------------+--------+--------+-----------------+
+ | | | | +--\
+ v v v v |
+ /-----------\ /-----------\ /-----------\ /-----------\ |
+ | Begin cCCC| | Begin cCCC| | Begin cCCC| | Begin cCCC| |
+ | Object 0 | | Object 1 | | Object 2 | | Object 3 | |
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | stripe | | stripe | | stripe | | stripe | |
+ | unit 0 | | unit 1 | | unit 2 | | unit 3 | |
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | stripe | | stripe | | stripe | | stripe | +-\
+ | unit 4 | | unit 5 | | unit 6 | | unit 7 | | Object
+ +-----------+ +-----------+ +-----------+ +-----------+ +- Set
+ | stripe | | stripe | | stripe | | stripe | | 1
+ | unit 8 | | unit 9 | | unit 10 | | unit 11 | +-/
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | stripe | | stripe | | stripe | | stripe | |
+ | unit 12 | | unit 13 | | unit 14 | | unit 15 | |
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | End cCCC | | End cCCC | | End cCCC | | End cCCC | |
+ | Object 0 | | Object 1 | | Object 2 | | Object 3 | |
+ \-----------/ \-----------/ \-----------/ \-----------/ |
+ |
+ +--/
+
+ +--\
+ |
+ /-----------\ /-----------\ /-----------\ /-----------\ |
+ | Begin cCCC| | Begin cCCC| | Begin cCCC| | Begin cCCC| |
+ | Object 4 | | Object 5 | | Object 6 | | Object 7 | |
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | stripe | | stripe | | stripe | | stripe | |
+ | unit 16 | | unit 17 | | unit 18 | | unit 19 | |
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | stripe | | stripe | | stripe | | stripe | +-\
+ | unit 20 | | unit 21 | | unit 22 | | unit 23 | | Object
+ +-----------+ +-----------+ +-----------+ +-----------+ +- Set
+ | stripe | | stripe | | stripe | | stripe | | 2
+ | unit 24 | | unit 25 | | unit 26 | | unit 27 | +-/
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | stripe | | stripe | | stripe | | stripe | |
+ | unit 28 | | unit 29 | | unit 30 | | unit 31 | |
+ +-----------+ +-----------+ +-----------+ +-----------+ |
+ | End cCCC | | End cCCC | | End cCCC | | End cCCC | |
+ | Object 4 | | Object 5 | | Object 6 | | Object 7 | |
+ \-----------/ \-----------/ \-----------/ \-----------/ |
+ |
+ +--/
+
+Three important variables determine how Ceph stripes data:
+
+- **Object Size:** Objects in the Ceph Storage Cluster have a maximum
+ configurable size (e.g., 2MB, 4MB, etc.). The object size should be large
+ enough to accommodate many stripe units, and should be a multiple of
+ the stripe unit.
+
+- **Stripe Width:** Stripes have a configurable unit size (e.g., 64kb).
+ The Ceph Client divides the data it will write to objects into equally
+ sized stripe units, except for the last stripe unit. A stripe width,
+ should be a fraction of the Object Size so that an object may contain
+ many stripe units.
+
+- **Stripe Count:** The Ceph Client writes a sequence of stripe units
+ over a series of objects determined by the stripe count. The series
+ of objects is called an object set. After the Ceph Client writes to
+ the last object in the object set, it returns to the first object in
+ the object set.
+
+.. important:: Test the performance of your striping configuration before
+ putting your cluster into production. You CANNOT change these striping
+ parameters after you stripe the data and write it to objects.
+
+Once the Ceph Client has striped data to stripe units and mapped the stripe
+units to objects, Ceph's CRUSH algorithm maps the objects to placement groups,
+and the placement groups to Ceph OSD Daemons before the objects are stored as
+files on a storage drive.
+
+.. note:: Since a client writes to a single pool, all data striped into objects
+ get mapped to placement groups in the same pool. So they use the same CRUSH
+ map and the same access controls.
+
+
+.. index:: architecture; Ceph Clients
+
+Ceph Clients
+============
+
+Ceph Clients include a number of service interfaces. These include:
+
+- **Block Devices:** The :term:`Ceph Block Device` (a.k.a., RBD) service
+ provides resizable, thin-provisioned block devices that can be snapshotted
+ and cloned. Ceph stripes a block device across the cluster for high
+ performance. Ceph supports both kernel objects (KO) and a QEMU hypervisor
+ that uses ``librbd`` directly--avoiding the kernel object overhead for
+ virtualized systems.
+
+- **Object Storage:** The :term:`Ceph Object Storage` (a.k.a., RGW) service
+ provides RESTful APIs with interfaces that are compatible with Amazon S3
+ and OpenStack Swift.
+
+- **Filesystem**: The :term:`Ceph File System` (CephFS) service provides
+ a POSIX compliant filesystem usable with ``mount`` or as
+ a filesystem in user space (FUSE).
+
+Ceph can run additional instances of OSDs, MDSs, and monitors for scalability
+and high availability. The following diagram depicts the high-level
+architecture.
+
+.. ditaa::
+
+ +--------------+ +----------------+ +-------------+
+ | Block Device | | Object Storage | | CephFS |
+ +--------------+ +----------------+ +-------------+
+
+ +--------------+ +----------------+ +-------------+
+ | librbd | | librgw | | libcephfs |
+ +--------------+ +----------------+ +-------------+
+
+ +---------------------------------------------------+
+ | Ceph Storage Cluster Protocol (librados) |
+ +---------------------------------------------------+
+
+ +---------------+ +---------------+ +---------------+
+ | OSDs | | MDSs | | Monitors |
+ +---------------+ +---------------+ +---------------+
+
+
+.. index:: architecture; Ceph Object Storage
+
+Ceph Object Storage
+-------------------
+
+The Ceph Object Storage daemon, ``radosgw``, is a FastCGI service that provides
+a RESTful_ HTTP API to store objects and metadata. It layers on top of the Ceph
+Storage Cluster with its own data formats, and maintains its own user database,
+authentication, and access control. The RADOS Gateway uses a unified namespace,
+which means you can use either the OpenStack Swift-compatible API or the Amazon
+S3-compatible API. For example, you can write data using the S3-compatible API
+with one application and then read data using the Swift-compatible API with
+another application.
+
+.. topic:: S3/Swift Objects and Store Cluster Objects Compared
+
+ Ceph's Object Storage uses the term *object* to describe the data it stores.
+ S3 and Swift objects are not the same as the objects that Ceph writes to the
+ Ceph Storage Cluster. Ceph Object Storage objects are mapped to Ceph Storage
+ Cluster objects. The S3 and Swift objects do not necessarily
+ correspond in a 1:1 manner with an object stored in the storage cluster. It
+ is possible for an S3 or Swift object to map to multiple Ceph objects.
+
+See `Ceph Object Storage`_ for details.
+
+
+.. index:: Ceph Block Device; block device; RBD; Rados Block Device
+
+Ceph Block Device
+-----------------
+
+A Ceph Block Device stripes a block device image over multiple objects in the
+Ceph Storage Cluster, where each object gets mapped to a placement group and
+distributed, and the placement groups are spread across separate ``ceph-osd``
+daemons throughout the cluster.
+
+.. important:: Striping allows RBD block devices to perform better than a single
+ server could!
+
+Thin-provisioned snapshottable Ceph Block Devices are an attractive option for
+virtualization and cloud computing. In virtual machine scenarios, people
+typically deploy a Ceph Block Device with the ``rbd`` network storage driver in
+QEMU/KVM, where the host machine uses ``librbd`` to provide a block device
+service to the guest. Many cloud computing stacks use ``libvirt`` to integrate
+with hypervisors. You can use thin-provisioned Ceph Block Devices with QEMU and
+``libvirt`` to support OpenStack and CloudStack among other solutions.
+
+While we do not provide ``librbd`` support with other hypervisors at this time,
+you may also use Ceph Block Device kernel objects to provide a block device to a
+client. Other virtualization technologies such as Xen can access the Ceph Block
+Device kernel object(s). This is done with the command-line tool ``rbd``.
+
+
+.. index:: CephFS; Ceph File System; libcephfs; MDS; metadata server; ceph-mds
+
+.. _arch-cephfs:
+
+Ceph File System
+----------------
+
+The Ceph File System (CephFS) provides a POSIX-compliant filesystem as a
+service that is layered on top of the object-based Ceph Storage Cluster.
+CephFS files get mapped to objects that Ceph stores in the Ceph Storage
+Cluster. Ceph Clients mount a CephFS filesystem as a kernel object or as
+a Filesystem in User Space (FUSE).
+
+.. ditaa::
+
+ +-----------------------+ +------------------------+
+ | CephFS Kernel Object | | CephFS FUSE |
+ +-----------------------+ +------------------------+
+
+ +---------------------------------------------------+
+ | CephFS Library (libcephfs) |
+ +---------------------------------------------------+
+
+ +---------------------------------------------------+
+ | Ceph Storage Cluster Protocol (librados) |
+ +---------------------------------------------------+
+
+ +---------------+ +---------------+ +---------------+
+ | OSDs | | MDSs | | Monitors |
+ +---------------+ +---------------+ +---------------+
+
+
+The Ceph File System service includes the Ceph Metadata Server (MDS) deployed
+with the Ceph Storage cluster. The purpose of the MDS is to store all the
+filesystem metadata (directories, file ownership, access modes, etc) in
+high-availability Ceph Metadata Servers where the metadata resides in memory.
+The reason for the MDS (a daemon called ``ceph-mds``) is that simple filesystem
+operations like listing a directory or changing a directory (``ls``, ``cd``)
+would tax the Ceph OSD Daemons unnecessarily. So separating the metadata from
+the data means that the Ceph File System can provide high performance services
+without taxing the Ceph Storage Cluster.
+
+CephFS separates the metadata from the data, storing the metadata in the MDS,
+and storing the file data in one or more objects in the Ceph Storage Cluster.
+The Ceph filesystem aims for POSIX compatibility. ``ceph-mds`` can run as a
+single process, or it can be distributed out to multiple physical machines,
+either for high availability or for scalability.
+
+- **High Availability**: The extra ``ceph-mds`` instances can be `standby`,
+ ready to take over the duties of any failed ``ceph-mds`` that was
+ `active`. This is easy because all the data, including the journal, is
+ stored on RADOS. The transition is triggered automatically by ``ceph-mon``.
+
+- **Scalability**: Multiple ``ceph-mds`` instances can be `active`, and they
+ will split the directory tree into subtrees (and shards of a single
+ busy directory), effectively balancing the load amongst all `active`
+ servers.
+
+Combinations of `standby` and `active` etc are possible, for example
+running 3 `active` ``ceph-mds`` instances for scaling, and one `standby`
+instance for high availability.
+
+
+
+.. _RADOS - A Scalable, Reliable Storage Service for Petabyte-scale Storage Clusters: https://ceph.io/assets/pdfs/weil-rados-pdsw07.pdf
+.. _Paxos: https://en.wikipedia.org/wiki/Paxos_(computer_science)
+.. _Monitor Config Reference: ../rados/configuration/mon-config-ref
+.. _Monitoring OSDs and PGs: ../rados/operations/monitoring-osd-pg
+.. _Heartbeats: ../rados/configuration/mon-osd-interaction
+.. _Monitoring OSDs: ../rados/operations/monitoring-osd-pg/#monitoring-osds
+.. _CRUSH - Controlled, Scalable, Decentralized Placement of Replicated Data: https://ceph.io/assets/pdfs/weil-crush-sc06.pdf
+.. _Data Scrubbing: ../rados/configuration/osd-config-ref#scrubbing
+.. _Report Peering Failure: ../rados/configuration/mon-osd-interaction#osds-report-peering-failure
+.. _Troubleshooting Peering Failure: ../rados/troubleshooting/troubleshooting-pg#placement-group-down-peering-failure
+.. _Ceph Authentication and Authorization: ../rados/operations/auth-intro/
+.. _Hardware Recommendations: ../start/hardware-recommendations
+.. _Network Config Reference: ../rados/configuration/network-config-ref
+.. _Data Scrubbing: ../rados/configuration/osd-config-ref#scrubbing
+.. _striping: https://en.wikipedia.org/wiki/Data_striping
+.. _RAID: https://en.wikipedia.org/wiki/RAID
+.. _RAID 0: https://en.wikipedia.org/wiki/RAID_0#RAID_0
+.. _Ceph Object Storage: ../radosgw/
+.. _RESTful: https://en.wikipedia.org/wiki/RESTful
+.. _Erasure Code Notes: https://github.com/ceph/ceph/blob/40059e12af88267d0da67d8fd8d9cd81244d8f93/doc/dev/osd_internals/erasure_coding/developer_notes.rst
+.. _Cache Tiering: ../rados/operations/cache-tiering
+.. _Set Pool Values: ../rados/operations/pools#set-pool-values
+.. _Kerberos: https://en.wikipedia.org/wiki/Kerberos_(protocol)
+.. _Cephx Config Guide: ../rados/configuration/auth-config-ref
+.. _User Management: ../rados/operations/user-management