From 19fcec84d8d7d21e796c7624e521b60d28ee21ed Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sun, 7 Apr 2024 20:45:59 +0200 Subject: Adding upstream version 16.2.11+ds. Signed-off-by: Daniel Baumann --- doc/dev/kubernetes.rst | 228 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 doc/dev/kubernetes.rst (limited to 'doc/dev/kubernetes.rst') diff --git a/doc/dev/kubernetes.rst b/doc/dev/kubernetes.rst new file mode 100644 index 000000000..75b100b24 --- /dev/null +++ b/doc/dev/kubernetes.rst @@ -0,0 +1,228 @@ + +.. _kubernetes-dev: + +======================================= +Hacking on Ceph in Kubernetes with Rook +======================================= + +.. warning:: + + This is *not* official user documentation for setting up production + Ceph clusters with Kubernetes. It is aimed at developers who want + to hack on Ceph in Kubernetes. + +This guide is aimed at Ceph developers getting started with running +in a Kubernetes environment. It assumes that you may be hacking on Rook, +Ceph or both, so everything is built from source. + +TL;DR for hacking on MGR modules +================================ + +Make your changes to the Python code base and then from Ceph's +``build`` directory, run:: + + ../src/script/kubejacker/kubejacker.sh '192.168.122.1:5000' + +where ``'192.168.122.1:5000'`` is a local docker registry and +Rook's ``CephCluster`` CR uses ``image: 192.168.122.1:5000/ceph/ceph:latest``. + +1. Build a kubernetes cluster +============================= + +Before installing Ceph/Rook, make sure you've got a working kubernetes +cluster with some nodes added (i.e. ``kubectl get nodes`` shows you something). +The rest of this guide assumes that your development workstation has network +access to your kubernetes cluster, such that ``kubectl`` works from your +workstation. + +`There are many ways `_ +to build a kubernetes cluster: here we include some tips/pointers on where +to get started. + +`kubic-terraform-kvm `_ +might also be an option. + +Or `Host your own `_ with +``kubeadm``. + +Some Tips +--------- + +Here are some tips for a smoother ride with + +``kubeadm``: + +- If you have previously added any yum/deb repos for kubernetes packages, + disable them before trying to use the packages.cloud.google.com repository. + If you don't, you'll get quite confusing conflicts. +- Even if your distro already has docker, make sure you're installing it + a version from docker.com that is within the range mentioned in the + kubeadm install instructions. Especially, note that the docker in CentOS 7, 8 + will *not* work. + +``minikube``: + +- Start up minikube by passing local docker registry address:: + ``minikube start --driver=docker --insecure-registry='192.168.122.1:5000'`` + +Hosted elsewhere +---------------- + +If you do not have any servers to hand, you might try a pure +container provider such as Google Compute Engine. Your mileage may +vary when it comes to what kinds of storage devices are visible +to your kubernetes cluster. + +Make sure you check how much it's costing you before you spin up a big cluster! + + +2. Run a docker registry +======================== + +Run this somewhere accessible from both your workstation and your +kubernetes cluster (i.e. so that ``docker push/pull`` just works everywhere). +This is likely to be the same host you're using as your kubernetes master. + +1. Install the ``docker-distribution`` package. +2. If you want to configure the port, edit ``/etc/docker-distribution/registry/config.yml`` +3. Enable the registry service: + +:: + + systemctl enable docker-distribution + systemctl start docker-distribution + +You may need to mark the registry as **insecure**. + +3. Build Rook +============= + +.. note:: + + Building Rook is **not required** to make changes to Ceph. + +Install Go if you don't already have it. + +Download the Rook source code: + +:: + + go get github.com/rook/rook + + # Ignore this warning, as Rook is not a conventional go package + can't load package: package github.com/rook/rook: no Go files in /home/jspray/go/src/github.com/rook/rook + +You will now have a Rook source tree in ~/go/src/github.com/rook/rook -- you may +be tempted to clone it elsewhere, but your life will be easier if you +leave it in your GOPATH. + +Run ``make`` in the root of your Rook tree to build its binaries and containers: + +:: + + make + ... + === saving image build-9204c79b/ceph-amd64 + === docker build build-9204c79b/ceph-toolbox-base-amd64 + sha256:653bb4f8d26d6178570f146fe637278957e9371014ea9fce79d8935d108f1eaa + === docker build build-9204c79b/ceph-toolbox-amd64 + sha256:445d97b71e6f8de68ca1c40793058db0b7dd1ebb5d05789694307fd567e13863 + === caching image build-9204c79b/ceph-toolbox-base-amd64 + +You can use ``docker image ls`` to see the resulting built images. The +images you care about are the ones with tags ending "ceph-amd64" (used +for the Rook operator and Ceph daemons) and "ceph-toolbox-amd64" (used +for the "toolbox" container where the CLI is run). + +4. Build Ceph +============= + +.. note:: + + Building Ceph is **not required** to make changes to MGR modules + written in Python. + + +The Rook containers and the Ceph containers are independent now. Note that +Rook's Ceph client libraries need to communicate with the Ceph cluster, +therefore a compatible major version is required. + +You can run a Registry docker container with access to your Ceph source +tree using a command like: + +:: + + docker run -i -v /my/ceph/src:/my/ceph/src -p 192.168.122.1:5000:5000 -t --name registry registry:2 + + +Once you have built Ceph, you can inject the resulting binaries into +the Rook container image using the ``kubejacker.sh`` script (run from +your build directory but from *outside* your build container). + +5. Run Kubejacker +================= + +``kubejacker`` needs access to your docker registry. Execute the script +to build a docker image containing your latest Ceph binaries: + +:: + + build$ ../src/script/kubejacker/kubejacker.sh ":" + + +Now you've got your freshly built Rook and freshly built Ceph into +a single container image, ready to run. Next time you change something +in Ceph, you can re-run this to update your image and restart your +kubernetes containers. If you change something in Rook, then re-run the Rook +build, and the Ceph build too. + +5. Run a Rook cluster +===================== + +Please refer to `Rook's documentation `_ +for setting up a Rook operator, a Ceph cluster and the toolbox. + +The Rook source tree includes example .yaml files in +``cluster/examples/kubernetes/ceph/``. Copy these into +a working directory, and edit as necessary to configure +the setup you want: + +- Ensure that ``spec.cephVersion.image`` points to your docker registry:: + + spec: + cephVersion: + allowUnsupported: true + image: 192.168.122.1:5000/ceph/ceph:latest + +Then, load the configuration into the kubernetes API using ``kubectl``: + +:: + + kubectl apply -f ./cluster-test.yaml + +Use ``kubectl -n rook-ceph get pods`` to check the operator +pod the Ceph daemons and toolbox are is coming up. + +Once everything is up and running, +you should be able to open a shell in the toolbox container and +run ``ceph status``. + +If your mon services start but the rest don't, it could be that they're +unable to form a quorum due to a Kubernetes networking issue: check that +containers in your Kubernetes cluster can ping containers on other nodes. + +Cheat sheet +=========== + +Open a shell in your toolbox container:: + + kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath="{.items[0].metadata.name}") -- bash + +Inspect the Rook operator container's logs:: + + kubectl -n rook-ceph logs -l app=rook-ceph-operator + +Inspect the ceph-mgr container's logs:: + + kubectl -n rook-ceph logs -l app=rook-ceph-mgr + -- cgit v1.2.3