1 files changed, 228 insertions, 0 deletions
diff --git a/doc/dev/kubernetes.rst b/doc/dev/kubernetes.rst
new file mode 100644
index 000000000..75b100b24
--- /dev/null
+++ b/doc/dev/kubernetes.rst
@@ -0,0 +1,228 @@
+
+.. _kubernetes-dev:
+
+=======================================
+Hacking on Ceph in Kubernetes with Rook
+=======================================
+
+.. warning::
+
+    This is *not* official user documentation for setting up production
+    Ceph clusters with Kubernetes.  It is aimed at developers who want
+    to hack on Ceph in Kubernetes.
+
+This guide is aimed at Ceph developers getting started with running
+in a Kubernetes environment. It assumes that you may be hacking on Rook,
+Ceph or both, so everything is built from source.
+
+TL;DR for hacking on MGR modules
+================================
+
+Make your changes to the Python code base and then from Ceph's
+``build`` directory, run::
+
+    ../src/script/kubejacker/kubejacker.sh '192.168.122.1:5000'
+
+where ``'192.168.122.1:5000'`` is a local docker registry and
+Rook's ``CephCluster`` CR uses ``image: 192.168.122.1:5000/ceph/ceph:latest``.
+
+1. Build a kubernetes cluster
+=============================
+
+Before installing Ceph/Rook, make sure you've got a working kubernetes
+cluster with some nodes added (i.e. ``kubectl get nodes`` shows you something).
+The rest of this guide assumes that your development workstation has network
+access to your kubernetes cluster, such that ``kubectl`` works from your
+workstation.
+
+`There are many ways <https://kubernetes.io/docs/setup/>`_
+to build a kubernetes cluster: here we include some tips/pointers on where
+to get started.
+
+`kubic-terraform-kvm <https://github.com/kubic-project/kubic-terraform-kvm>`_
+might also be an option.
+
+Or `Host your own <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/>`_ with
+``kubeadm``.
+
+Some Tips
+---------
+
+Here are some tips for a smoother ride with
+
+``kubeadm``:
+
+- If you have previously added any yum/deb repos for kubernetes packages,
+  disable them before trying to use the packages.cloud.google.com repository.
+  If you don't, you'll get quite confusing conflicts.
+- Even if your distro already has docker, make sure you're installing it
+  a version from docker.com that is within the range mentioned in the
+  kubeadm install instructions. Especially, note that the docker in CentOS 7, 8
+  will *not* work.
+
+``minikube``:
+
+- Start up minikube by passing local docker registry address::
+  ``minikube start --driver=docker --insecure-registry='192.168.122.1:5000'``
+
+Hosted elsewhere
+----------------
+
+If you do not have any servers to hand, you might try a pure
+container provider such as Google Compute Engine. Your mileage may
+vary when it comes to what kinds of storage devices are visible
+to your kubernetes cluster.
+
+Make sure you check how much it's costing you before you spin up a big cluster!
+
+
+2. Run a docker registry
+========================
+
+Run this somewhere accessible from both your workstation and your
+kubernetes cluster (i.e. so that ``docker push/pull`` just works everywhere).
+This is likely to be the same host you're using as your kubernetes master.
+
+1. Install the ``docker-distribution`` package.
+2. If you want to configure the port, edit ``/etc/docker-distribution/registry/config.yml``
+3. Enable the registry service:
+
+::
+
+    systemctl enable docker-distribution
+    systemctl start docker-distribution
+
+You may need to mark the registry as **insecure**.
+
+3. Build Rook
+=============
+
+.. note::
+
+    Building Rook is **not required** to make changes to Ceph.
+
+Install Go if you don't already have it.
+
+Download the Rook source code:
+
+::
+
+    go get github.com/rook/rook
+
+    # Ignore this warning, as Rook is not a conventional go package
+    can't load package: package github.com/rook/rook: no Go files in /home/jspray/go/src/github.com/rook/rook
+
+You will now have a Rook source tree in ~/go/src/github.com/rook/rook -- you may
+be tempted to clone it elsewhere, but your life will be easier if you
+leave it in your GOPATH.
+
+Run ``make`` in the root of your Rook tree to build its binaries and containers:
+
+::
+
+    make
+    ...
+    === saving image build-9204c79b/ceph-amd64
+    === docker build build-9204c79b/ceph-toolbox-base-amd64
+    sha256:653bb4f8d26d6178570f146fe637278957e9371014ea9fce79d8935d108f1eaa
+    === docker build build-9204c79b/ceph-toolbox-amd64
+    sha256:445d97b71e6f8de68ca1c40793058db0b7dd1ebb5d05789694307fd567e13863
+    === caching image build-9204c79b/ceph-toolbox-base-amd64
+
+You can use ``docker image ls`` to see the resulting built images.  The
+images you care about are the ones with tags ending "ceph-amd64" (used
+for the Rook operator and Ceph daemons) and "ceph-toolbox-amd64" (used
+for the "toolbox" container where the CLI is run).
+
+4. Build Ceph
+=============
+
+.. note::
+
+    Building Ceph is **not required** to make changes to MGR modules
+    written in Python.
+
+
+The Rook containers and the Ceph containers are independent now. Note that
+Rook's Ceph client libraries need to communicate with the Ceph cluster,
+therefore a compatible major version is required.
+
+You can run a Registry docker container with access to your Ceph source
+tree using a command like:
+
+::
+
+    docker run -i -v /my/ceph/src:/my/ceph/src -p 192.168.122.1:5000:5000 -t --name registry registry:2
+
+
+Once you have built Ceph, you can inject the resulting binaries into
+the Rook container image using the ``kubejacker.sh`` script (run from
+your build directory but from *outside* your build container).
+
+5. Run Kubejacker
+=================
+
+``kubejacker`` needs access to your docker registry. Execute the script
+to build a docker image containing your latest Ceph binaries:
+
+::
+
+    build$ ../src/script/kubejacker/kubejacker.sh "<host>:<port>"
+
+
+Now you've got your freshly built Rook and freshly built Ceph into
+a single container image, ready to run.  Next time you change something
+in Ceph, you can re-run this to update your image and restart your
+kubernetes containers. If you change something in Rook, then re-run the Rook
+build, and the Ceph build too.
+
+5. Run a Rook cluster
+=====================
+
+Please refer to `Rook's documentation <https://rook.io/docs/rook/master/ceph-quickstart.html>`_
+for setting up a Rook operator, a Ceph cluster and the toolbox.
+
+The Rook source tree includes example .yaml files in
+``cluster/examples/kubernetes/ceph/``. Copy these into
+a working directory, and edit as necessary to configure
+the setup you want:
+
+- Ensure that ``spec.cephVersion.image`` points to your docker registry::
+
+    spec:
+      cephVersion:
+        allowUnsupported: true
+        image: 192.168.122.1:5000/ceph/ceph:latest
+
+Then, load the configuration into the kubernetes API using ``kubectl``:
+
+::
+
+    kubectl apply -f ./cluster-test.yaml
+
+Use ``kubectl -n rook-ceph get pods`` to check the operator
+pod the Ceph daemons and toolbox are is coming up.
+
+Once everything is up and running,
+you should be able to open a shell in the toolbox container and
+run ``ceph status``.
+
+If your mon services start but the rest don't, it could be that they're
+unable to form a quorum due to a Kubernetes networking issue: check that
+containers in your Kubernetes cluster can ping containers on other nodes.
+
+Cheat sheet
+===========
+
+Open a shell in your toolbox container::
+
+    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath="{.items[0].metadata.name}") -- bash
+
+Inspect the Rook operator container's logs::
+
+    kubectl -n rook-ceph logs -l app=rook-ceph-operator
+
+Inspect the ceph-mgr container's logs::
+
+    kubectl -n rook-ceph logs -l app=rook-ceph-mgr
+