diff options
Diffstat (limited to 'doc/dev/continuous-integration.rst')
| -rw-r--r-- | doc/dev/continuous-integration.rst | 285 |
1 files changed, 285 insertions, 0 deletions
diff --git a/doc/dev/continuous-integration.rst b/doc/dev/continuous-integration.rst new file mode 100644 index 000000000..5c2f15823 --- /dev/null +++ b/doc/dev/continuous-integration.rst @@ -0,0 +1,285 @@ +Continuous Integration Architecture +=================================== + +In Ceph, we rely on multiple CI pipelines in our development. Most of these pipelines +are centered around Jenkins. And their configurations are generated using `Jenkins Job Builder`_. + +.. _Jenkins Job Builder: https://docs.openstack.org/infra/jenkins-job-builder/ + +Let's take the ``make check`` performed by Jenkins as an example. + +ceph-pull-requests +------------------ + +``ceph-pull-requests`` is a jenkins job which gets triggered by a GitHub pull +request or a trigger phrase like:: + + jenkins test make check + +There are multiple parties involved in this jenkins job: + +.. graphviz:: + + digraph { + rankdir="LR"; + github [ + label="<git> git_repo | <webhooks> webhooks | <api> api"; + shape=record; + href="https://github.com/ceph/ceph"; + ]; + subgraph cluster_lab { + label="Sepia Lab"; + href="https://wiki.sepia.ceph.com/doku.php"; + shape=circle; + apt_mirror [ + href="http://apt-mirror.front.sepia.ceph.com"; + ]; + shaman [ + href="https://shaman.ceph.com"; + ]; + chacra [ + peripheries=3; + href="https://chacra.ceph.com"; + ]; + subgraph cluster_jenkins { + label="jenkins"; + href="https://jenkins.ceph.com"; + jenkins_controller [ label = "controller" ]; + jenkins_agents [ label = "agents", peripheries=3 ]; + }; + }; + { + rank=same; + package_repos [ peripheries=3 ]; + pypi; + npm; + } + github:webhooks -> jenkins_controller [ label = "notify", color = "crimson" ]; + jenkins_controller -> jenkins_agents [ label = "schedule jobs" ]; + jenkins_agents -> github:git [ label = "git pull" ]; + jenkins_agents -> shaman [ label = "query for chacra repo URL" ]; + jenkins_agents -> chacra [ label = "pull build dependencies" ]; + jenkins_agents -> package_repos [ label = "pull build dependencies" ]; + jenkins_agents -> pypi [ label = "pull Python packages" ]; + jenkins_agents -> npm [ label = "pull JavaScript packages" ]; + jenkins_agents -> apt_mirror [ label = "pull build dependencies" ]; + jenkins_agents -> github:api [ label = "update", color = "crimson" ]; + } + +Where + +Sepia Lab + `Sepia Lab`_ is a test lab used by the Ceph project. This lab offers + the storage and computing resources required by our CI infra. + +Jenkins agents + are a set of machines which perform the CI jobs. In this case, they + + #. pull the git repo from GitHub and + #. rebase the pull request against the latest master + #. set necessary environment variables + #. run ``run-make-check.sh`` + +Chacra + is a server offering RESTful API allowing the clients to store and + retrieve binary packages. It also creates the repo for uploaded + packages automatically. Once a certain repo is created on chacra, the + configured shaman server is updated as well, then we can query shaman + for the corresponding repo address. Chacra not only hosts Ceph packages, + it also hosts quite a few other packages like various build dependencies. + +Shaman + is a server offering RESTful API allowing the clients to query the + information of repos hosted by chacra nodes. Shaman is also known + for its `Web UI`_. But please note, shaman does not build the + packages, it just offers information on the builds. + +As the following shows, `chacra`_ manages multiple projects whose metadata +are stored in a database. These metadata are exposed via Shaman as a web +service. `chacractl`_ is a utility to interact with the `chacra`_ service. + +.. graphviz:: + + digraph { + libboost [ + shape=cylinder; + ]; + libzbd [ + shape=cylinder; + ]; + other_repos [ + label="..."; + shape=cylinder; + ]; + postgresql [ + shape=cylinder; + style=filled; + ] + shaman -> postgresql; + chacra -> postgresql; + chacractl -> chacra; + chacra -> libboost; + chacra -> libzbd; + chacra -> other_repos; + } + +.. _Sepia Lab: https://wiki.sepia.ceph.com/doku.php +.. _Web UI: https://shaman.ceph.com + +build dependencies +------------------ + +Just like lots of other software projects, Ceph has both build-time and +run-time dependencies. Most of time, we are inclined to use the packages +prebuilt by the distro. But there are cases where + +- the necessary dependencies are either missing in the distro, or +- their versions are too old, or +- they are packaged without some important feature enabled. +- we want to ensure that the version of a certain runtime dependency is + identical to the one we tested in our lab. + +No matter what the reason is, we either need to build them from source, or +to package them as binary packages instead of using the ones shipped by the +distro. Quite a few build-time dependencies are included as git submodules, +but in order to avoid rebuilding these dependencies repeatedly, we pre-built +some of them and uploaded them to our own repos. So, when performing +``make check``, the building hosts in our CI just pull them from our internal +repos hosting these packages instead of building them. + +So far, following packages are prebuilt for ubuntu focal, and then uploaded to +`chacra`_: + +libboost + packages `boost`_. The packages' names are changed from ``libboost-*`` to + ``ceph-libboost-*``, and they are instead installed into ``/opt/ceph``, so + they don't interfere with the official ``libboost`` packages shipped by + distro. Its build scripts are hosted at https://github.com/ceph/ceph-boost. + See https://github.com/ceph/ceph-boost/commit/2a8ae02932b2a1fd6a68072da8ca0df2b99b805c + for an example of how to bump the version number. The commands used to + build 1.79 on a vanilla Ubuntu Focal OS are below. + + .. prompt:: bash $ + + sudo apt install debhelper dctrl-tools chrpath libbz2-dev libicu-dev bison \ + flex docbook-to-man help2man xsltproc doxygen dh-python python3-all-dev graphviz + wget http://download.ceph.com/qa/boost_1_79_0.tar.bz2 + git clone https://github.com/ceph/ceph-boost + tar xjf boost_1_79_0.tar.bz2 + cp -ra ceph-boost/debian boost_1_79_0/ + pushd boost_1_79_0 + export DEB_BUILD_OPTIONS='parallel=6 nodoc' + dpkg-buildpackage -us -uc -b + popd + BOOST_SHA=$(git ls-remote https://github.com/ceph/ceph-boost main | awk '{ print $1 }') + ls *.deb | chacractl binary create \ + libboost/master/$BOOST_SHA/ubuntu/focal/amd64/flavors/default + +libzbd + packages `libzbd`_ . The upstream libzbd includes debian packaging already. + +libpmem + packages `pmdk`_ . Please note, ``ndctl`` is one of the build dependencies of + pmdk, for an updated debian packaging, please see + https://github.com/ceph/ceph-ndctl . + +.. note:: + + please ensure that the package version and the release number of the + packaging are properly updated when updating/upgrading the packaging, + otherwise it would be difficult to tell which version of the package + is installed. We check the package version before trying to upgrade + it in ``install-deps.sh``. + +.. _boost: https://www.boost.org +.. _libzbd: https://github.com/westerndigitalcorporation/libzbd +.. _pmdk: https://github.com/pmem/pmdk + +But in addition to these libraries, ``ceph-mgr-dashboard``'s frontend uses lots of +JavaScript packages. Quite a few of them are not packaged by distros. Not to +mention the trouble of testing different combination of versions of these +packages. So we decided to include these JavaScript packages in our dist tarball +using ``make-dist``. + +Also, because our downstream might not want to use the prepackaged binaries when +redistributing the precompiled Ceph packages, we also need to include these +libraries in our dist tarball. They are + +- boost +- liburing +- pmdk + +``make-dist`` is a script used by our CI pipeline to create dist tarball so the +tarball can be used to build the Ceph packages in a clean room environment. When +we need to upgrade these third party libraries, we should + +- update the CMake script +- rebuild the prebuilt packages and +- update this script to reflect the change. + +Uploading Dependencies +---------------------- + +To ensure that prebuilt packages are available by the jenkins agents, we need to +upload them to either ``apt-mirror.front.sepia.ceph.com`` or `chacra`_. To upload +packages to the former would require the help of our lab administrator, so if we +want to maintain the package repositories on regular basis, a better choice would be +to manage them using `chacractl`_. `chacra`_ represents packages repositories using +a resource hierarchy, like:: + + <project>/<branch>/<ref>/<distro>/<distro-version>/<arch> + +In which: + +project + in general, it is used for denoting a set of related packages. For instance, + ``libboost``. + +branch + branch of project. This mirrors the concept of a Git repo. + +ref + a unique id of a given version of a set packages. This id is used to reference + the set packages under the ``<project>/<branch>``. It is a good practice to + version the packaging recipes, like the ``debian`` directory for building DEB + packages and the ``spec`` for building RPM packages, and use the SHA1 of the + packaging recipe for the ``ref``. But you could also use a random string for + ``ref``, like the tag name of the built source tree. + +distro + the distro name for which the packages are built. Currently, following distros are + supported: + + - centos + - debian + - fedora + - rhel + - ubuntu + +distro-version + the version of the distro. For instance, if a package is built on ubuntu focal, + the ``distro-version`` should be ``20.04``. + +arch + the architecture of the packages. It could be: + + - arm64 + - amd64 + - noarch + +So, for example, we can upload the prebuilt boost packages to chacra like + +.. prompt:: bash $ + + ls *.deb | chacractl binary create \ + libboost/master/099c0fd56b4a54457e288a2eff8fffdc0d416f7a/ubuntu/focal/amd64/flavors/default + +.. _chacra: https://github.com/ceph/chacra +.. _chacractl: https://github.com/ceph/chacractl + +Update ``install-deps.sh`` +-------------------------- + +We also need to update ``install-deps.sh`` to point the built script to the new +repo. Please refer to the `script <https://github.com/ceph/ceph/blob/master/install-deps.sh>`_, +for more details. |