Adding upstream version 18.2.2.upstream/18.2.2

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 475 insertions, 0 deletions

diff --git a/doc/rbd/rbd-nomad.rst b/doc/rbd/rbd-nomad.rst
new file mode 100644
index 000000000..66d87d6ce
--- /dev/null
+++ b/doc/rbd/rbd-nomad.rst
@@ -0,0 +1,475 @@
+=========================
+ Block Devices and Nomad
+=========================
+
+Like Kubernetes, Nomad can use Ceph Block Device. This is made possible by
+`ceph-csi`_, which allows you to dynamically provision RBD images or import
+existing RBD images.
+
+Every version of Nomad is compatible with `ceph-csi`_, but the reference
+version of Nomad that was used to generate the procedures and guidance in this
+document is Nomad v1.1.2, the latest version available at the time of the
+writing of the document.
+
+To use Ceph Block Devices with Nomad, you must install
+and configure ``ceph-csi`` within your Nomad environment. The following
+diagram shows the Nomad/Ceph technology stack.
+
+.. ditaa::
+            +-------------------------+-------------------------+
+            |      Container          |          ceph--csi      |
+            |                         |            node         |
+            |          ^              |                 ^       |
+            |          |              |                 |       |
+            +----------+--------------+-------------------------+
+            |          |                                |       |
+            |          v                                |       |
+            |                       Nomad               |       |
+            |                                           |       |
+            +---------------------------------------------------+
+            |                       ceph--csi                   |
+            |                       controller                  |
+            +--------+------------------------------------------+
+                     |                                  |
+                     | configures       maps            |
+                     +---------------+ +----------------+
+                                     | | 
+                                     v v
+            +------------------------+ +------------------------+
+            |                        | |        rbd--nbd        |
+            |     Kernel Modules     | +------------------------+
+            |                        | |         librbd         |
+            +------------------------+-+------------------------+
+            |                   RADOS Protocol                  |
+            +------------------------+-+------------------------+
+            |          OSDs          | |        Monitors        |
+            +------------------------+ +------------------------+
+
+.. note::
+    Nomad has many possible task drivers, but this example uses only a Docker container.
+
+.. important::
+   ``ceph-csi`` uses the RBD kernel modules by default, which may not support
+   all Ceph `CRUSH tunables`_ or `RBD image features`_.
+
+Create a Pool
+=============
+
+By default, Ceph block devices use the ``rbd`` pool. Ensure that your Ceph
+cluster is running, then create a pool for Nomad persistent storage:
+
+.. prompt:: bash $
+
+  ceph osd pool create nomad
+
+See `Create a Pool`_ for details on specifying the number of placement groups
+for your pools. See `Placement Groups`_ for details on the number of placement
+groups you should set for your pools.
+
+A newly created pool must be initialized prior to use. Use the ``rbd`` tool
+to initialize the pool:
+
+.. prompt:: bash $
+
+  rbd pool init nomad
+
+Configure ceph-csi
+==================
+
+Ceph Client Authentication Setup
+--------------------------------
+
+Create a new user for Nomad and `ceph-csi`. Execute the following command and
+record the generated key:
+
+.. code-block:: console
+
+  $ ceph auth get-or-create client.nomad mon 'profile rbd' osd 'profile rbd pool=nomad' mgr 'profile rbd pool=nomad'
+  [client.nomad]
+          key = AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ==
+
+
+Configure Nomad
+---------------
+
+Configuring Nomad to Allow Containers to Use Privileged Mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, Nomad doesn't allow containers to use privileged mode. We must
+configure Nomad so that it allows containers to use privileged mode. Edit the
+Nomad configuration file by adding the following configuration block to
+`/etc/nomad.d/nomad.hcl`::
+
+    plugin "docker" {
+      config {
+        allow_privileged = true
+      }
+    }
+
+Loading the rbd module
+~~~~~~~~~~~~~~~~~~~~~~
+
+Nomad must have the `rbd` module loaded. Run the following command to confirm that the `rbd` module is loaded:
+
+.. code-block:: console
+
+  $ lsmod | grep rbd
+  rbd                    94208  2
+  libceph               364544  1 rbd
+
+If the `rbd` module is not loaded, load it:
+
+.. prompt:: bash $
+
+  sudo modprobe rbd
+
+Restarting Nomad
+~~~~~~~~~~~~~~~~
+
+Restart Nomad:
+
+.. prompt:: bash $
+
+  sudo systemctl restart nomad
+
+
+Create ceph-csi controller and plugin nodes
+===========================================
+
+The `ceph-csi`_ plugin requires two components:
+
+- **Controller plugin**: communicates with the provider's API.
+- **Node plugin**: executes tasks on the client.
+
+.. note::
+    We'll set the ceph-csi's version in those files. See `ceph-csi release`_
+    for information about ceph-csi's compatibility with other versions.
+
+Configure controller plugin
+---------------------------
+
+The controller plugin requires the Ceph monitor addresses of the Ceph
+cluster. Collect both (1) the Ceph cluster unique `fsid` and (2) the monitor
+addresses:
+
+.. code-block:: console
+
+  $ ceph mon dump
+  <...>
+  fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
+  <...>
+  0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
+  1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
+  2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c
+
+Generate a ``ceph-csi-plugin-controller.nomad`` file similar to the example
+below. Substitute the `fsid` for "clusterID", and the monitor addresses for
+"monitors"::
+
+    job "ceph-csi-plugin-controller" {
+      datacenters = ["dc1"]
+      group "controller" {
+        network {
+          port "metrics" {}
+        }
+        task "ceph-controller" {
+          template {
+            data        = <<EOF
+    [{
+        "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+        "monitors": [
+            "192.168.1.1",
+            "192.168.1.2",
+            "192.168.1.3"
+        ]
+    }]
+    EOF
+            destination = "local/config.json"
+            change_mode = "restart"
+          }
+          driver = "docker"
+          config {
+            image = "quay.io/cephcsi/cephcsi:v3.3.1"
+            volumes = [
+              "./local/config.json:/etc/ceph-csi-config/config.json"
+            ]
+            mounts = [
+              {
+                type     = "tmpfs"
+                target   = "/tmp/csi/keys"
+                readonly = false
+                tmpfs_options = {
+                  size = 1000000 # size in bytes
+                }
+              }
+            ]
+            args = [
+              "--type=rbd",
+              "--controllerserver=true",
+              "--drivername=rbd.csi.ceph.com",
+              "--endpoint=unix://csi/csi.sock",
+              "--nodeid=${node.unique.name}",
+              "--instanceid=${node.unique.name}-controller",
+              "--pidlimit=-1",
+              "--logtostderr=true",
+              "--v=5",
+              "--metricsport=$${NOMAD_PORT_metrics}"
+            ]
+          }
+          resources {
+            cpu    = 500
+            memory = 256
+          }
+          service {
+            name = "ceph-csi-controller"
+            port = "metrics"
+            tags = [ "prometheus" ]
+          }
+          csi_plugin {
+            id        = "ceph-csi"
+            type      = "controller"
+            mount_dir = "/csi"
+          }
+        }
+      }
+    }
+
+Configure plugin node
+---------------------
+
+Generate a ``ceph-csi-plugin-nodes.nomad`` file similar to the example below.
+Substitute the `fsid` for "clusterID" and the monitor addresses for
+"monitors"::
+
+    job "ceph-csi-plugin-nodes" {
+      datacenters = ["dc1"]
+      type        = "system"
+      group "nodes" {
+        network {
+          port "metrics" {}
+        }
+        task "ceph-node" {
+          driver = "docker"
+          template {
+            data        = <<EOF
+    [{
+        "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+        "monitors": [
+            "192.168.1.1",
+            "192.168.1.2",
+            "192.168.1.3"
+        ]
+    }]
+    EOF
+            destination = "local/config.json"
+            change_mode = "restart"
+          }
+          config {
+            image = "quay.io/cephcsi/cephcsi:v3.3.1"
+            volumes = [
+              "./local/config.json:/etc/ceph-csi-config/config.json"
+            ]
+            mounts = [
+              {
+                type     = "tmpfs"
+                target   = "/tmp/csi/keys"
+                readonly = false
+                tmpfs_options = {
+                  size = 1000000 # size in bytes
+                }
+              }
+            ]
+            args = [
+              "--type=rbd",
+              "--drivername=rbd.csi.ceph.com",
+              "--nodeserver=true",
+              "--endpoint=unix://csi/csi.sock",
+              "--nodeid=${node.unique.name}",
+              "--instanceid=${node.unique.name}-nodes",
+              "--pidlimit=-1",
+              "--logtostderr=true",
+              "--v=5",
+              "--metricsport=$${NOMAD_PORT_metrics}"
+            ]
+            privileged = true
+          }
+          resources {
+            cpu    = 500
+            memory = 256
+          }
+          service {
+            name = "ceph-csi-nodes"
+            port = "metrics"
+            tags = [ "prometheus" ]
+          }
+          csi_plugin {
+            id        = "ceph-csi"
+            type      = "node"
+            mount_dir = "/csi"
+          }
+        }
+      }
+    }
+
+Start plugin controller and node
+--------------------------------
+
+To start the plugin controller and the Nomad node, run the following commands:
+
+.. prompt:: bash $
+
+  nomad job run ceph-csi-plugin-controller.nomad
+  nomad job run ceph-csi-plugin-nodes.nomad
+
+The `ceph-csi`_ image will be downloaded.
+
+Check the plugin status after a few minutes:
+
+.. code-block:: console
+
+  $ nomad plugin status ceph-csi
+  ID                   = ceph-csi
+  Provider             = rbd.csi.ceph.com
+  Version              = 3.3.1
+  Controllers Healthy  = 1
+  Controllers Expected = 1
+  Nodes Healthy        = 1
+  Nodes Expected       = 1
+
+  Allocations
+  ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
+  23b4db0c  a61ef171  nodes       4        run      running  3h26m ago  3h25m ago
+  fee74115  a61ef171  controller  6        run      running  3h26m ago  3h25m ago
+
+Using Ceph Block Devices
+========================
+
+Create rbd image
+----------------
+
+``ceph-csi`` requires the cephx credentials for communicating with the Ceph
+cluster. Generate a ``ceph-volume.hcl`` file similar to the example below,
+using the newly created nomad user id and cephx key::
+
+    id = "ceph-mysql"
+    name = "ceph-mysql"
+    type = "csi"
+    plugin_id = "ceph-csi"
+    capacity_max = "200G"
+    capacity_min = "100G"
+
+    capability {
+      access_mode     = "single-node-writer"
+      attachment_mode = "file-system"
+    }
+
+    secrets {
+      userID  = "admin"
+      userKey = "AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ=="
+    }
+
+    parameters {
+      clusterID = "b9127830-b0cc-4e34-aa47-9d1a2e9949a8"
+      pool = "nomad"
+      imageFeatures = "layering"
+    }
+
+After the ``ceph-volume.hcl`` file has been generated, create the volume:
+
+.. prompt:: bash $
+
+  nomad volume create ceph-volume.hcl
+
+Use rbd image with a container
+------------------------------
+
+As an exercise in using an rbd image with a container, modify the Hashicorp
+`nomad stateful`_ example.
+
+Generate a ``mysql.nomad`` file similar to the example below::
+
+    job "mysql-server" {
+      datacenters = ["dc1"]
+      type        = "service"
+      group "mysql-server" {
+        count = 1
+        volume "ceph-mysql" {
+          type      = "csi"
+          attachment_mode = "file-system"
+          access_mode     = "single-node-writer"
+          read_only = false
+          source    = "ceph-mysql"
+        }
+        network {
+          port "db" {
+            static = 3306
+          }
+        }
+        restart {
+          attempts = 10
+          interval = "5m"
+          delay    = "25s"
+          mode     = "delay"
+        }
+        task "mysql-server" {
+          driver = "docker"
+          volume_mount {
+            volume      = "ceph-mysql"
+            destination = "/srv"
+            read_only   = false
+          }
+          env {
+            MYSQL_ROOT_PASSWORD = "password"
+          }
+          config {
+            image = "hashicorp/mysql-portworx-demo:latest"
+            args  = ["--datadir", "/srv/mysql"]
+            ports = ["db"]
+          }
+          resources {
+            cpu    = 500
+            memory = 1024
+          }
+          service {
+            name = "mysql-server"
+            port = "db"
+            check {
+              type     = "tcp"
+              interval = "10s"
+              timeout  = "2s"
+            }
+          }
+        }
+      }
+    }
+
+Start the job:
+
+.. prompt:: bash $
+
+  nomad job run mysql.nomad
+
+Check the status of the job:
+
+.. code-block:: console
+
+  $ nomad job status mysql-server
+  ...
+  Status        = running
+  ...
+  Allocations
+  ID        Node ID   Task Group    Version  Desired  Status   Created  Modified
+  38070da7  9ad01c63  mysql-server  0        run      running  6s ago   3s ago
+
+To check that data are persistent, modify the database, purge the job, then
+create it using the same file. The same RBD image will be used (re-used,
+really).
+
+.. _ceph-csi: https://github.com/ceph/ceph-csi/
+.. _csi: https://www.nomadproject.io/docs/internals/plugins/csi
+.. _Create a Pool: ../../rados/operations/pools#createpool
+.. _Placement Groups: ../../rados/operations/placement-groups
+.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
+.. _RBD image features: ../rbd-config-ref/#image-features
+.. _nomad stateful: https://learn.hashicorp.com/tutorials/nomad/stateful-workloads-csi-volumes?in=nomad/stateful-workloads#create-the-job-file
+.. _ceph-csi release: https://github.com/ceph/ceph-csi#ceph-csi-container-images-and-release-compatibility