Adding upstream version 18.2.2.upstream/18.2.2

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 364 insertions, 0 deletions

diff --git a/doc/rbd/rbd-kubernetes.rst b/doc/rbd/rbd-kubernetes.rst
new file mode 100644
index 000000000..ccec4813a
--- /dev/null
+++ b/doc/rbd/rbd-kubernetes.rst
@@ -0,0 +1,364 @@
+==============================
+ Block Devices and Kubernetes
+==============================
+
+You may use Ceph Block Device images with Kubernetes v1.13 and later through
+`ceph-csi`_, which dynamically provisions RBD images to back Kubernetes
+`volumes`_ and maps these RBD images as block devices (optionally mounting
+a file system contained within the image) on worker nodes running
+`pods`_ that reference an RBD-backed volume. Ceph stripes block device images as
+objects across the cluster, which means that large Ceph Block Device images have
+better performance than a standalone server!
+
+To use Ceph Block Devices with Kubernetes v1.13 and higher, you must install
+and configure ``ceph-csi`` within your Kubernetes environment. The following
+diagram depicts the Kubernetes/Ceph technology stack.
+
+.. ditaa::
+            +---------------------------------------------------+
+            |                   Kubernetes                      |
+            +---------------------------------------------------+
+            |                    ceph--csi                      |
+            +------------------------+--------------------------+
+                                     |
+                                     | configures
+                                     v
+            +------------------------+ +------------------------+
+            |                        | |        rbd--nbd        |
+            |     Kernel Modules     | +------------------------+
+            |                        | |         librbd         |
+            +------------------------+-+------------------------+
+            |                   RADOS Protocol                  |
+            +------------------------+-+------------------------+
+            |          OSDs          | |        Monitors        |
+            +------------------------+ +------------------------+
+
+
+.. important::
+   ``ceph-csi`` uses the RBD kernel modules by default which may not support all
+   Ceph `CRUSH tunables`_ or `RBD image features`_.
+
+Create a Pool
+=============
+
+By default, Ceph block devices use the ``rbd`` pool. Create a pool for
+Kubernetes volume storage. Ensure your Ceph cluster is running, then create
+the pool. ::
+
+        $ ceph osd pool create kubernetes
+
+See `Create a Pool`_ for details on specifying the number of placement groups
+for your pools, and `Placement Groups`_ for details on the number of placement
+groups you should set for your pools.
+
+A newly created pool must be initialized prior to use. Use the ``rbd`` tool
+to initialize the pool::
+
+        $ rbd pool init kubernetes
+
+Configure ceph-csi
+==================
+
+Setup Ceph Client Authentication
+--------------------------------
+
+Create a new user for Kubernetes and `ceph-csi`. Execute the following and
+record the generated key::
+
+    $ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
+    [client.kubernetes]
+        key = AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg==
+
+Generate `ceph-csi` `ConfigMap`
+-------------------------------
+
+The `ceph-csi` requires a `ConfigMap` object stored in Kubernetes to define the
+the Ceph monitor addresses for the Ceph cluster. Collect both the Ceph cluster
+unique `fsid` and the monitor addresses::
+
+        $ ceph mon dump
+        <...>
+        fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
+        <...>
+        0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
+        1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
+        2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c
+
+.. note::
+   ``ceph-csi`` currently only supports the `legacy V1 protocol`_.
+
+Generate a `csi-config-map.yaml` file similar to the example below, substituting
+the `fsid` for "clusterID", and the monitor addresses for "monitors"::
+
+        $ cat <<EOF > csi-config-map.yaml
+        ---
+        apiVersion: v1
+        kind: ConfigMap
+        data:
+          config.json: |-
+            [
+              {
+                "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+                "monitors": [
+                  "192.168.1.1:6789",
+                  "192.168.1.2:6789",
+                  "192.168.1.3:6789"
+                ]
+              }
+            ]
+        metadata:
+          name: ceph-csi-config
+        EOF
+
+Once generated, store the new `ConfigMap` object in Kubernetes::
+
+        $ kubectl apply -f csi-config-map.yaml
+
+Recent versions of `ceph-csi` also require an additional `ConfigMap` object to
+define Key Management Service (KMS) provider details.  If KMS isn't set up, put
+an empty configuration in a `csi-kms-config-map.yaml` file or refer to examples
+at https://github.com/ceph/ceph-csi/tree/master/examples/kms::
+
+        $ cat <<EOF > csi-kms-config-map.yaml
+        ---
+        apiVersion: v1
+        kind: ConfigMap
+        data:
+          config.json: |-
+            {}
+        metadata:
+          name: ceph-csi-encryption-kms-config
+        EOF
+
+Once generated, store the new `ConfigMap` object in Kubernetes::
+
+        $ kubectl apply -f csi-kms-config-map.yaml
+
+Recent versions of `ceph-csi` also require yet another `ConfigMap` object
+to define Ceph configuration to add to ceph.conf file inside CSI containers::
+
+        $ cat <<EOF > ceph-config-map.yaml
+        ---
+        apiVersion: v1
+        kind: ConfigMap
+        data:
+          ceph.conf: |
+            [global]
+            auth_cluster_required = cephx
+            auth_service_required = cephx
+            auth_client_required = cephx
+          # keyring is a required key and its value should be empty
+          keyring: |
+        metadata:
+          name: ceph-config
+        EOF
+
+Once generated, store the new `ConfigMap` object in Kubernetes::
+
+        $ kubectl apply -f ceph-config-map.yaml
+
+Generate `ceph-csi` cephx `Secret`
+----------------------------------
+
+`ceph-csi` requires the cephx credentials for communicating with the Ceph
+cluster. Generate a `csi-rbd-secret.yaml` file similar to the example below,
+using the newly created Kubernetes user id and cephx key::
+
+        $ cat <<EOF > csi-rbd-secret.yaml
+        ---
+        apiVersion: v1
+        kind: Secret
+        metadata:
+          name: csi-rbd-secret
+          namespace: default
+        stringData:
+          userID: kubernetes
+          userKey: AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg==
+        EOF
+
+Once generated, store the new `Secret` object in Kubernetes::
+
+        $ kubectl apply -f csi-rbd-secret.yaml
+
+Configure `ceph-csi` Plugins
+----------------------------
+
+Create the required `ServiceAccount` and RBAC `ClusterRole`/`ClusterRoleBinding`
+Kubernetes objects. These objects do not necessarily need to be customized for
+your Kubernetes environment and therefore can be used as-is from the `ceph-csi`
+deployment YAMLs::
+
+        $ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
+        $ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
+
+Finally, create the `ceph-csi` provisioner and node plugins. With the
+possible exception of the `ceph-csi` container release version, these objects do
+not necessarily need to be customized for your Kubernetes environment and
+therefore can be used as-is from the `ceph-csi` deployment YAMLs::
+
+        $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
+        $ kubectl apply -f csi-rbdplugin-provisioner.yaml
+        $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
+        $ kubectl apply -f csi-rbdplugin.yaml
+
+.. important::
+   The provisioner and node plugin YAMLs will, by default, pull the development
+   release of the `ceph-csi` container (quay.io/cephcsi/cephcsi:canary).
+   The YAMLs should be updated to use a release version container for
+   production workloads.
+
+Using Ceph Block Devices
+========================
+
+Create a `StorageClass`
+-----------------------
+
+The Kubernetes `StorageClass` defines a class of storage. Multiple `StorageClass`
+objects can be created to map to different quality-of-service levels (i.e. NVMe
+vs HDD-based pools) and features.
+
+For example, to create a `ceph-csi` `StorageClass` that maps to the `kubernetes`
+pool created above, the following YAML file can be used after ensuring that the
+"clusterID" property matches your Ceph cluster's `fsid`::
+
+        $ cat <<EOF > csi-rbd-sc.yaml
+        ---
+        apiVersion: storage.k8s.io/v1
+        kind: StorageClass
+        metadata:
+           name: csi-rbd-sc
+        provisioner: rbd.csi.ceph.com
+        parameters:
+           clusterID: b9127830-b0cc-4e34-aa47-9d1a2e9949a8
+           pool: kubernetes
+           imageFeatures: layering
+           csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
+           csi.storage.k8s.io/provisioner-secret-namespace: default
+           csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
+           csi.storage.k8s.io/controller-expand-secret-namespace: default
+           csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
+           csi.storage.k8s.io/node-stage-secret-namespace: default
+        reclaimPolicy: Delete
+        allowVolumeExpansion: true
+        mountOptions:
+           - discard
+        EOF
+        $ kubectl apply -f csi-rbd-sc.yaml
+
+Note that in Kubernetes v1.14 and v1.15 volume expansion feature was in alpha
+status and required enabling `ExpandCSIVolumes` feature gate.
+
+Create a `PersistentVolumeClaim`
+--------------------------------
+
+A `PersistentVolumeClaim` is a request for abstract storage resources by a user.
+The `PersistentVolumeClaim` would then be associated to a `Pod` resource to
+provision a `PersistentVolume`, which would be backed by a Ceph block image.
+An optional `volumeMode` can be included to select between a mounted file system
+(default) or raw block device-based volume.
+
+Using `ceph-csi`, specifying `Filesystem` for `volumeMode` can support both
+`ReadWriteOnce` and `ReadOnlyMany` `accessMode` claims, and specifying `Block`
+for `volumeMode` can support `ReadWriteOnce`, `ReadWriteMany`, and
+`ReadOnlyMany` `accessMode` claims.
+
+For example, to create a block-based `PersistentVolumeClaim` that utilizes
+the `ceph-csi`-based `StorageClass` created above, the following YAML can be
+used to request raw block storage from the `csi-rbd-sc` `StorageClass`::
+
+        $ cat <<EOF > raw-block-pvc.yaml
+        ---
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: raw-block-pvc
+        spec:
+          accessModes:
+            - ReadWriteOnce
+          volumeMode: Block
+          resources:
+            requests:
+              storage: 1Gi
+          storageClassName: csi-rbd-sc
+        EOF
+        $ kubectl apply -f raw-block-pvc.yaml
+
+The following demonstrates and example of binding the above
+`PersistentVolumeClaim` to a `Pod` resource as a raw block device::
+
+        $ cat <<EOF > raw-block-pod.yaml
+        ---
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: pod-with-raw-block-volume
+        spec:
+          containers:
+            - name: fc-container
+              image: fedora:26
+              command: ["/bin/sh", "-c"]
+              args: ["tail -f /dev/null"]
+              volumeDevices:
+                - name: data
+                  devicePath: /dev/xvda
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: raw-block-pvc
+        EOF
+        $ kubectl apply -f raw-block-pod.yaml
+
+To create a file-system-based `PersistentVolumeClaim` that utilizes the
+`ceph-csi`-based `StorageClass` created above, the following YAML can be used to
+request a mounted file system (backed by an RBD image) from the `csi-rbd-sc`
+`StorageClass`::
+
+        $ cat <<EOF > pvc.yaml
+        ---
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: rbd-pvc
+        spec:
+          accessModes:
+            - ReadWriteOnce
+          volumeMode: Filesystem
+          resources:
+            requests:
+              storage: 1Gi
+          storageClassName: csi-rbd-sc
+        EOF
+        $ kubectl apply -f pvc.yaml
+
+The following demonstrates and example of binding the above
+`PersistentVolumeClaim` to a `Pod` resource as a mounted file system::
+
+        $ cat <<EOF > pod.yaml
+        ---
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: csi-rbd-demo-pod
+        spec:
+          containers:
+            - name: web-server
+              image: nginx
+              volumeMounts:
+                - name: mypvc
+                  mountPath: /var/lib/www/html
+          volumes:
+            - name: mypvc
+              persistentVolumeClaim:
+                claimName: rbd-pvc
+                readOnly: false
+        EOF
+        $ kubectl apply -f pod.yaml
+
+.. _ceph-csi: https://github.com/ceph/ceph-csi/
+.. _volumes: https://kubernetes.io/docs/concepts/storage/volumes/
+.. _pods: https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/
+.. _Create a Pool: ../../rados/operations/pools#createpool
+.. _Placement Groups: ../../rados/operations/placement-groups
+.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
+.. _RBD image features: ../rbd-config-ref/#image-features
+.. _legacy V1 protocol: ../../rados/configuration/msgr2/#address-formats