diff options
Diffstat (limited to 'doc/dev/index.rst')
| -rw-r--r-- | doc/dev/index.rst | 1608 |
1 files changed, 1608 insertions, 0 deletions
diff --git a/doc/dev/index.rst b/doc/dev/index.rst new file mode 100644 index 00000000..bc4e887a --- /dev/null +++ b/doc/dev/index.rst @@ -0,0 +1,1608 @@ +============================================ +Contributing to Ceph: A Guide for Developers +============================================ + +:Author: Loic Dachary +:Author: Nathan Cutler +:License: Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0) + +.. note:: You may also be interested in the :doc:`/dev/internals` documentation. + +.. contents:: + :depth: 3 + +Introduction +============ + +This guide has two aims. First, it should lower the barrier to entry for +software developers who wish to get involved in the Ceph project. Second, +it should serve as a reference for Ceph developers. + +We assume that readers are already familiar with Ceph (the distributed +object store and file system designed to provide excellent performance, +reliability and scalability). If not, please refer to the `project website`_ +and especially the `publications list`_. + +.. _`project website`: https://ceph.com +.. _`publications list`: https://ceph.com/publications/ + +Since this document is to be consumed by developers, who are assumed to +have Internet access, topics covered elsewhere, either within the Ceph +documentation or elsewhere on the web, are treated by linking. If you +notice that a link is broken or if you know of a better link, please +`report it as a bug`_. + +.. _`report it as a bug`: http://tracker.ceph.com/projects/ceph/issues/new + +Essentials (tl;dr) +================== + +This chapter presents essential information that every Ceph developer needs +to know. + +Leads +----- + +The Ceph project is led by Sage Weil. In addition, each major project +component has its own lead. The following table shows all the leads and +their nicks on `GitHub`_: + +.. _github: https://github.com/ + +========= ================ ============= +Scope Lead GitHub nick +========= ================ ============= +Ceph Sage Weil liewegas +RADOS Neha Ojha neha-ojha +RGW Yehuda Sadeh yehudasa +RGW Matt Benjamin mattbenjamin +RBD Jason Dillaman dillaman +CephFS Patrick Donnelly batrick +Dashboard Lenz Grimmer LenzGr +MON Joao Luis jecluis +Build/Ops Ken Dreyer ktdreyer +========= ================ ============= + +The Ceph-specific acronyms in the table are explained in +:doc:`/architecture`. + +History +------- + +See the `History chapter of the Wikipedia article`_. + +.. _`History chapter of the Wikipedia article`: https://en.wikipedia.org/wiki/Ceph_%28software%29#History + +Licensing +--------- + +Ceph is free software. + +Unless stated otherwise, the Ceph source code is distributed under the terms of +the LGPL2.1. For full details, see the file `COPYING`_ in the top-level +directory of the source-code tree. + +.. _`COPYING`: + https://github.com/ceph/ceph/blob/master/COPYING + +Source code repositories +------------------------ + +The source code of Ceph lives on `GitHub`_ in a number of repositories below +the `Ceph "organization"`_. + +.. _`Ceph "organization"`: https://github.com/ceph + +To make a meaningful contribution to the project as a developer, a working +knowledge of git_ is essential. + +.. _git: https://git-scm.com/doc + +Although the `Ceph "organization"`_ includes several software repositories, +this document covers only one: https://github.com/ceph/ceph. + +Redmine issue tracker +--------------------- + +Although `GitHub`_ is used for code, Ceph-related issues (Bugs, Features, +Backports, Documentation, etc.) are tracked at http://tracker.ceph.com, +which is powered by `Redmine`_. + +.. _Redmine: http://www.redmine.org + +The tracker has a Ceph project with a number of subprojects loosely +corresponding to the various architectural components (see +:doc:`/architecture`). + +Mere `registration`_ in the tracker automatically grants permissions +sufficient to open new issues and comment on existing ones. + +.. _registration: http://tracker.ceph.com/account/register + +To report a bug or propose a new feature, `jump to the Ceph project`_ and +click on `New issue`_. + +.. _`jump to the Ceph project`: http://tracker.ceph.com/projects/ceph +.. _`New issue`: http://tracker.ceph.com/projects/ceph/issues/new + +Mailing list +------------ + +Ceph development email discussions take place on the mailing list +``ceph-devel@vger.kernel.org``. The list is open to all. Subscribe by +sending a message to ``majordomo@vger.kernel.org`` with the line: :: + + subscribe ceph-devel + +in the body of the message. + +There are also `other Ceph-related mailing lists`_. + +.. _`other Ceph-related mailing lists`: https://ceph.com/irc/ + +IRC +--- + +In addition to mailing lists, the Ceph community also communicates in real +time using `Internet Relay Chat`_. + +.. _`Internet Relay Chat`: http://www.irchelp.org/ + +See ``https://ceph.com/irc/`` for how to set up your IRC +client and a list of channels. + +Submitting patches +------------------ + +The canonical instructions for submitting patches are contained in the +file `CONTRIBUTING.rst`_ in the top-level directory of the source-code +tree. There may be some overlap between this guide and that file. + +.. _`CONTRIBUTING.rst`: + https://github.com/ceph/ceph/blob/master/CONTRIBUTING.rst + +All newcomers are encouraged to read that file carefully. + +Building from source +-------------------- + +See instructions at :doc:`/install/build-ceph`. + +Using ccache to speed up local builds +------------------------------------- + +Rebuilds of the ceph source tree can benefit significantly from use of `ccache`_. +Many a times while switching branches and such, one might see build failures for +certain older branches mostly due to older build artifacts. These rebuilds can +significantly benefit the use of ccache. For a full clean source tree, one could +do :: + + $ make clean + + # note the following will nuke everything in the source tree that + # isn't tracked by git, so make sure to backup any log files /conf options + + $ git clean -fdx; git submodule foreach git clean -fdx + +ccache is available as a package in most distros. To build ceph with ccache one +can:: + + $ cmake -DWITH_CCACHE=ON .. + +ccache can also be used for speeding up all builds in the system. for more +details refer to the `run modes`_ of the ccache manual. The default settings of +``ccache`` can be displayed with ``ccache -s``. + +.. note:: It is recommended to override the ``max_size``, which is the size of + cache, defaulting to 10G, to a larger size like 25G or so. Refer to the + `configuration`_ section of ccache manual. + +To further increase the cache hit rate and reduce compile times in a development +environment, it is possible to set version information and build timestamps to +fixed values, which avoids frequent rebuilds of binaries that contain this +information. + +This can be achieved by adding the following settings to the ``ccache`` +configuration file ``ccache.conf``:: + + sloppiness = time_macros + run_second_cpp = true + +Now, set the environment variable ``SOURCE_DATE_EPOCH`` to a fixed value (a UNIX +timestamp) and set ``ENABLE_GIT_VERSION`` to ``OFF`` when running ``cmake``:: + + $ export SOURCE_DATE_EPOCH=946684800 + $ cmake -DWITH_CCACHE=ON -DENABLE_GIT_VERSION=OFF .. + +.. note:: Binaries produced with these build options are not suitable for + production or debugging purposes, as they do not contain the correct build + time and git version information. + +.. _`ccache`: https://ccache.samba.org/ +.. _`run modes`: https://ccache.samba.org/manual.html#_run_modes +.. _`configuration`: https://ccache.samba.org/manual.html#_configuration + +Development-mode cluster +------------------------ + +See :doc:`/dev/quick_guide`. + +Kubernetes/Rook development cluster +----------------------------------- + +See :ref:`kubernetes-dev` + +Backporting +----------- + +All bugfixes should be merged to the ``master`` branch before being backported. +To flag a bugfix for backporting, make sure it has a `tracker issue`_ +associated with it and set the ``Backport`` field to a comma-separated list of +previous releases (e.g. "hammer,jewel") that you think need the backport. +The rest (including the actual backporting) will be taken care of by the +`Stable Releases and Backports`_ team. + +.. _`tracker issue`: http://tracker.ceph.com/ +.. _`Stable Releases and Backports`: http://tracker.ceph.com/projects/ceph-releases/wiki + +Guidance for use of cluster log +------------------------------- + +If your patches emit messages to the Ceph cluster log, please consult +this guidance: :doc:`/dev/logging`. + + +What is merged where and when ? +=============================== + +Commits are merged into branches according to criteria that change +during the lifecycle of a Ceph release. This chapter is the inventory +of what can be merged in which branch at a given point in time. + +Development releases (i.e. x.0.z) +--------------------------------- + +What ? +^^^^^^ + +* features +* bug fixes + +Where ? +^^^^^^^ + +Features are merged to the master branch. Bug fixes should be merged +to the corresponding named branch (e.g. "jewel" for 10.0.z, "kraken" +for 11.0.z, etc.). However, this is not mandatory - bug fixes can be +merged to the master branch as well, since the master branch is +periodically merged to the named branch during the development +releases phase. In either case, if the bugfix is important it can also +be flagged for backport to one or more previous stable releases. + +When ? +^^^^^^ + +After the stable release candidates of the previous release enters +phase 2 (see below). For example: the "jewel" named branch was +created when the infernalis release candidates entered phase 2. From +this point on, master was no longer associated with infernalis. As +soon as the named branch of the next stable release is created, master +starts getting periodically merged into it. + +Branch merges +^^^^^^^^^^^^^ + +* The branch of the stable release is merged periodically into master. +* The master branch is merged periodically into the branch of the + stable release. +* The master is merged into the branch of the stable release + immediately after each development x.0.z release. + +Stable release candidates (i.e. x.1.z) phase 1 +---------------------------------------------- + +What ? +^^^^^^ + +* bug fixes only + +Where ? +^^^^^^^ + +The branch of the stable release (e.g. "jewel" for 10.0.z, "kraken" +for 11.0.z, etc.) or master. Bug fixes should be merged to the named +branch corresponding to the stable release candidate (e.g. "jewel" for +10.1.z) or to master. During this phase, all commits to master will be +merged to the named branch, and vice versa. In other words, it makes +no difference whether a commit is merged to the named branch or to +master - it will make it into the next release candidate either way. + +When ? +^^^^^^ + +After the first stable release candidate is published, i.e. after the +x.1.0 tag is set in the release branch. + +Branch merges +^^^^^^^^^^^^^ + +* The branch of the stable release is merged periodically into master. +* The master branch is merged periodically into the branch of the + stable release. +* The master is merged into the branch of the stable release + immediately after each x.1.z release candidate. + +Stable release candidates (i.e. x.1.z) phase 2 +---------------------------------------------- + +What ? +^^^^^^ + +* bug fixes only + +Where ? +^^^^^^^ + +The branch of the stable release (e.g. "jewel" for 10.0.z, "kraken" +for 11.0.z, etc.). During this phase, all commits to the named branch +will be merged into master. Cherry-picking to the named branch during +release candidate phase 2 is done manually since the official +backporting process only begins when the release is pronounced +"stable". + +When ? +^^^^^^ + +After Sage Weil decides it is time for phase 2 to happen. + +Branch merges +^^^^^^^^^^^^^ + +* The branch of the stable release is merged periodically into master. + +Stable releases (i.e. x.2.z) +---------------------------- + +What ? +^^^^^^ + +* bug fixes +* features are sometime accepted +* commits should be cherry-picked from master when possible +* commits that are not cherry-picked from master must be about a bug unique to the stable release +* see also `the backport HOWTO`_ + +.. _`the backport HOWTO`: + http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO#HOWTO + +Where ? +^^^^^^^ + +The branch of the stable release (hammer for 0.94.x, infernalis for 9.2.x, etc.) + +When ? +^^^^^^ + +After the stable release is published, i.e. after the "vx.2.0" tag is +set in the release branch. + +Branch merges +^^^^^^^^^^^^^ + +Never + +Issue tracker +============= + +See `Redmine issue tracker`_ for a brief introduction to the Ceph Issue Tracker. + +Ceph developers use the issue tracker to + +1. keep track of issues - bugs, fix requests, feature requests, backport +requests, etc. + +2. communicate with other developers and keep them informed as work +on the issues progresses. + +Issue tracker conventions +------------------------- + +When you start working on an existing issue, it's nice to let the other +developers know this - to avoid duplication of labor. Typically, this is +done by changing the :code:`Assignee` field (to yourself) and changing the +:code:`Status` to *In progress*. Newcomers to the Ceph community typically do not +have sufficient privileges to update these fields, however: they can +simply update the issue with a brief note. + +.. table:: Meanings of some commonly used statuses + + ================ =========================================== + Status Meaning + ================ =========================================== + New Initial status + In Progress Somebody is working on it + Need Review Pull request is open with a fix + Pending Backport Fix has been merged, backport(s) pending + Resolved Fix and backports (if any) have been merged + ================ =========================================== + +Basic workflow +============== + +The following chart illustrates basic development workflow: + +.. ditaa:: + + Upstream Code Your Local Environment + + /----------\ git clone /-------------\ + | Ceph | -------------------------> | ceph/master | + \----------/ \-------------/ + ^ | + | | git branch fix_1 + | git merge | + | v + /----------------\ git commit --amend /-------------\ + | make check |---------------------> | ceph/fix_1 | + | ceph--qa--suite| \-------------/ + \----------------/ | + ^ | fix changes + | | test changes + | review | git commit + | | + | v + /--------------\ /-------------\ + | github |<---------------------- | ceph/fix_1 | + | pull request | git push \-------------/ + \--------------/ + +Below we present an explanation of this chart. The explanation is written +with the assumption that you, the reader, are a beginning developer who +has an idea for a bugfix, but do not know exactly how to proceed. + +Update the tracker +------------------ + +Before you start, you should know the `Issue tracker`_ number of the bug +you intend to fix. If there is no tracker issue, now is the time to create +one. + +The tracker is there to explain the issue (bug) to your fellow Ceph +developers and keep them informed as you make progress toward resolution. +To this end, then, provide a descriptive title as well as sufficient +information and details in the description. + +If you have sufficient tracker permissions, assign the bug to yourself by +changing the ``Assignee`` field. If your tracker permissions have not yet +been elevated, simply add a comment to the issue with a short message like +"I am working on this issue". + +Upstream code +------------- + +This section, and the ones that follow, correspond to the nodes in the +above chart. + +The upstream code lives in https://github.com/ceph/ceph.git, which is +sometimes referred to as the "upstream repo", or simply "upstream". As the +chart illustrates, we will make a local copy of this code, modify it, test +our modifications, and submit the modifications back to the upstream repo +for review. + +A local copy of the upstream code is made by + +1. forking the upstream repo on GitHub, and +2. cloning your fork to make a local working copy + +See the `the GitHub documentation +<https://help.github.com/articles/fork-a-repo/#platform-linux>`_ for +detailed instructions on forking. In short, if your GitHub username is +"mygithubaccount", your fork of the upstream repo will show up at +https://github.com/mygithubaccount/ceph. Once you have created your fork, +you clone it by doing: + +.. code:: + + $ git clone https://github.com/mygithubaccount/ceph + +While it is possible to clone the upstream repo directly, in this case you +must fork it first. Forking is what enables us to open a `GitHub pull +request`_. + +For more information on using GitHub, refer to `GitHub Help +<https://help.github.com/>`_. + +Local environment +----------------- + +In the local environment created in the previous step, you now have a +copy of the ``master`` branch in ``remotes/origin/master``. Since the fork +(https://github.com/mygithubaccount/ceph.git) is frozen in time and the +upstream repo (https://github.com/ceph/ceph.git, typically abbreviated to +``ceph/ceph.git``) is updated frequently by other developers, you will need +to sync your fork periodically. To do this, first add the upstream repo as +a "remote" and fetch it:: + + $ git remote add ceph https://github.com/ceph/ceph.git + $ git fetch ceph + +Fetching downloads all objects (commits, branches) that were added since +the last sync. After running these commands, all the branches from +``ceph/ceph.git`` are downloaded to the local git repo as +``remotes/ceph/$BRANCH_NAME`` and can be referenced as +``ceph/$BRANCH_NAME`` in certain git commands. + +For example, your local ``master`` branch can be reset to the upstream Ceph +``master`` branch by doing:: + + $ git fetch ceph + $ git checkout master + $ git reset --hard ceph/master + +Finally, the ``master`` branch of your fork can then be synced to upstream +master by:: + + $ git push -u origin master + +Bugfix branch +------------- + +Next, create a branch for the bugfix: + +.. code:: + + $ git checkout master + $ git checkout -b fix_1 + $ git push -u origin fix_1 + +This creates a ``fix_1`` branch locally and in our GitHub fork. At this +point, the ``fix_1`` branch is identical to the ``master`` branch, but not +for long! You are now ready to modify the code. + +Fix bug locally +--------------- + +At this point, change the status of the tracker issue to "In progress" to +communicate to the other Ceph developers that you have begun working on a +fix. If you don't have permission to change that field, your comment that +you are working on the issue is sufficient. + +Possibly, your fix is very simple and requires only minimal testing. +More likely, it will be an iterative process involving trial and error, not +to mention skill. An explanation of how to fix bugs is beyond the +scope of this document. Instead, we focus on the mechanics of the process +in the context of the Ceph project. + +A detailed discussion of the tools available for validating your bugfixes, +see the `Testing`_ chapter. + +For now, let us just assume that you have finished work on the bugfix and +that you have tested it and believe it works. Commit the changes to your local +branch using the ``--signoff`` option:: + + $ git commit -as + +and push the changes to your fork:: + + $ git push origin fix_1 + +GitHub pull request +------------------- + +The next step is to open a GitHub pull request. The purpose of this step is +to make your bugfix available to the community of Ceph developers. They +will review it and may do additional testing on it. + +In short, this is the point where you "go public" with your modifications. +Psychologically, you should be prepared to receive suggestions and +constructive criticism. Don't worry! In our experience, the Ceph project is +a friendly place! + +If you are uncertain how to use pull requests, you may read +`this GitHub pull request tutorial`_. + +.. _`this GitHub pull request tutorial`: + https://help.github.com/articles/using-pull-requests/ + +For some ideas on what constitutes a "good" pull request, see +the `Git Commit Good Practice`_ article at the `OpenStack Project Wiki`_. + +.. _`Git Commit Good Practice`: https://wiki.openstack.org/wiki/GitCommitMessages +.. _`OpenStack Project Wiki`: https://wiki.openstack.org/wiki/Main_Page + +Once your pull request (PR) is opened, update the `Issue tracker`_ by +adding a comment to the bug pointing the other developers to your PR. The +update can be as simple as:: + + *PR*: https://github.com/ceph/ceph/pull/$NUMBER_OF_YOUR_PULL_REQUEST + +Automated PR validation +----------------------- + +When your PR hits GitHub, the Ceph project's `Continuous Integration (CI) +<https://en.wikipedia.org/wiki/Continuous_integration>`_ +infrastructure will test it automatically. At the time of this writing +(March 2016), the automated CI testing included a test to check that the +commits in the PR are properly signed (see `Submitting patches`_) and a +`make check`_ test. + +The latter, `make check`_, builds the PR and runs it through a battery of +tests. These tests run on machines operated by the Ceph Continuous +Integration (CI) team. When the tests complete, the result will be shown +on GitHub in the pull request itself. + +You can (and should) also test your modifications before you open a PR. +Refer to the `Testing`_ chapter for details. + +Notes on PR make check test +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The GitHub `make check`_ test is driven by a Jenkins instance. + +Jenkins merges the PR branch into the latest version of the base branch before +starting the build, so you don't have to rebase the PR to pick up any fixes. + +You can trigger the PR tests at any time by adding a comment to the PR - the +comment should contain the string "test this please". Since a human subscribed +to the PR might interpret that as a request for him or her to test the PR, it's +good to write the request as "Jenkins, test this please". + +The `make check`_ log is the place to go if there is a failure and you're not +sure what caused it. To reach it, first click on "details" (next to the `make +check`_ test in the PR) to get into the Jenkins web GUI, and then click on +"Console Output" (on the left). + +Jenkins is set up to grep the log for strings known to have been associated +with `make check`_ failures in the past. However, there is no guarantee that +the strings are associated with any given `make check`_ failure. You have to +dig into the log to be sure. + +Integration tests AKA ceph-qa-suite +----------------------------------- + +Since Ceph is a complex beast, it may also be necessary to test your fix to +see how it behaves on real clusters running either on real or virtual +hardware. Tests designed for this purpose live in the `ceph/qa +sub-directory`_ and are run via the `teuthology framework`_. + +.. _`ceph/qa sub-directory`: https://github.com/ceph/ceph/tree/master/qa/ +.. _`teuthology repository`: https://github.com/ceph/teuthology +.. _`teuthology framework`: https://github.com/ceph/teuthology + +If you have access to an OpenStack tenant, you are encouraged to run the +integration tests yourself using `ceph-workbench ceph-qa-suite`_, +and to post the test results to the PR. + +.. _`ceph-workbench ceph-qa-suite`: http://ceph-workbench.readthedocs.org/ + +The Ceph community has access to the `Sepia lab +<http://ceph.github.io/sepia/>`_ where integration tests can be run on +real hardware. Other developers may add tags like "needs-qa" to your PR. +This allows PRs that need testing to be merged into a single branch and +tested all at the same time. Since teuthology suites can take hours +(even days in some cases) to run, this can save a lot of time. + +Integration testing is discussed in more detail in the `Testing`_ chapter. + +Code review +----------- + +Once your bugfix has been thoroughly tested, or even during this process, +it will be subjected to code review by other developers. This typically +takes the form of correspondence in the PR itself, but can be supplemented +by discussions on `IRC`_ and the `Mailing list`_. + +Amending your PR +---------------- + +While your PR is going through `Testing`_ and `Code review`_, you can +modify it at any time by editing files in your local branch. + +After the changes are committed locally (to the ``fix_1`` branch in our +example), they need to be pushed to GitHub so they appear in the PR. + +Modifying the PR is done by adding commits to the ``fix_1`` branch upon +which it is based, often followed by rebasing to modify the branch's git +history. See `this tutorial +<https://www.atlassian.com/git/tutorials/rewriting-history>`_ for a good +introduction to rebasing. When you are done with your modifications, you +will need to force push your branch with: + +.. code:: + + $ git push --force origin fix_1 + +Merge +----- + +The bugfixing process culminates when one of the project leads decides to +merge your PR. + +When this happens, it is a signal for you (or the lead who merged the PR) +to change the `Issue tracker`_ status to "Resolved". Some issues may be +flagged for backporting, in which case the status should be changed to +"Pending Backport" (see the `Backporting`_ chapter for details). + + +Testing +======= + +Ceph has two types of tests: `make check`_ tests and integration tests. +The former are run via `GNU Make <https://www.gnu.org/software/make/>`, +and the latter are run via the `teuthology framework`_. The following two +chapters examine the `make check`_ and integration tests in detail. + +.. _`make check`: + +Unit tests - make check +----------------------- + +After compiling Ceph, the code can be run through a battery of tests covering +various aspects of Ceph. For historical reasons, this battery of tests is often +referred to as `make check`_ even though the actual command used to run the +tests is now ``ctest``. For inclusion in this battery of tests, a test must: + +* bind ports that do not conflict with other tests +* not require root access +* not require more than one machine to run +* complete within a few minutes + +For simplicity, we will refer to this class of tests as "make check tests" or +"unit tests", to distinguish them from the more complex "integration tests" +that are run via the `teuthology framework`_. + +While it is possible to run ``ctest`` directly, it can be tricky to correctly +set up your environment. Fortunately, a script is provided to make it easier +run the unit tests on your code. It can be run from the top-level directory of +the Ceph source tree by doing:: + + $ ./run-make-check.sh + +You will need a minimum of 8GB of RAM and 32GB of free disk space for this +command to complete successfully on x86_64 (other architectures may have +different constraints). Depending on your hardware, it can take from 20 +minutes to three hours to complete, but it's worth the wait. + +How unit tests are declared +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Unit tests are declared in the ``CMakeLists.txt`` files (multiple files under +``./src``) using the ``add_ceph_test`` or ``add_ceph_unittest`` CMake functions, +which are themselves defined in ``./cmake/modules/AddCephTest.cmake``. Some +unit tests are scripts, while others are binaries that are compiled during the +build process. The ``add_ceph_test`` function is used to declare unit test +scripts, while ``add_ceph_unittest`` is used for unit test binaries. + +Unit testing of CLI tools +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Some of the CLI tools are tested using special files ending with the extension +``.t`` and stored under ``./src/test/cli``. These tests are run using a tool +called `cram`_ via a shell script ``./src/test/run-cli-tests``. `cram`_ tests +that are not suitable for `make check`_ may also be run by teuthology using +the `cram task`_. + +.. _`cram`: https://bitheap.org/cram/ +.. _`cram task`: https://github.com/ceph/ceph/blob/master/qa/tasks/cram.py + +Caveats +^^^^^^^ + +1. Unlike the various Ceph daemons and ``ceph-fuse``, the unit tests + are linked against the default memory allocator (glibc) unless explicitly + linked against something else. This enables tools like valgrind to be used + in the tests. + +Integration tests +----------------- + +When a test requires multiple machines, root access or lasts for a +longer time (for example, to simulate a realistic Ceph deployment), it +is deemed to be an integration test. Integration tests are organized into +"suites", which are defined in the `ceph/qa sub-directory`_ and run with +the ``teuthology-suite`` command. + +The ``teuthology-suite`` command is part of the `teuthology framework`_. +In the sections that follow we attempt to provide a detailed introduction +to that framework from the perspective of a beginning Ceph developer. + +Teuthology consumes packages +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It may take some time to understand the significance of this fact, but it +is `very` significant. It means that automated tests can be conducted on +multiple platforms using the same packages (RPM, DEB) that can be +installed on any machine running those platforms. + +Teuthology has a `list of platforms that it supports +<https://github.com/ceph/ceph/tree/master/qa/distros/supported>`_ (as +of December 2017 the list consisted of "CentOS 7.2" and "Ubuntu 16.04"). It +expects to be provided pre-built Ceph packages for these platforms. +Teuthology deploys these platforms on machines (bare-metal or +cloud-provisioned), installs the packages on them, and deploys Ceph +clusters on them - all as called for by the test. + +The nightlies +^^^^^^^^^^^^^ + +A number of integration tests are run on a regular basis in the `Sepia +lab`_ against the official Ceph repositories (on the ``master`` development +branch and the stable branches). Traditionally, these tests are called "the +nightlies" because the Ceph core developers used to live and work in +the same time zone and from their perspective the tests were run overnight. + +The results of the nightlies are published at http://pulpito.ceph.com/ and +http://pulpito.ovh.sepia.ceph.com:8081/. The developer nick shows in the +test results URL and in the first column of the Pulpito dashboard. The +results are also reported on the `ceph-qa mailing list +<https://ceph.com/irc/>`_ for analysis. + +Suites inventory +^^^^^^^^^^^^^^^^ + +The ``suites`` directory of the `ceph/qa sub-directory`_ contains +all the integration tests, for all the Ceph components. + +`ceph-deploy <https://github.com/ceph/ceph/tree/master/qa/suites/ceph-deploy>`_ + install a Ceph cluster with ``ceph-deploy`` (:ref:`ceph-deploy man page <ceph-deploy>`) + +`dummy <https://github.com/ceph/ceph/tree/master/qa/suites/dummy>`_ + get a machine, do nothing and return success (commonly used to + verify the integration testing infrastructure works as expected) + +`fs <https://github.com/ceph/ceph/tree/master/qa/suites/fs>`_ + test CephFS + +`kcephfs <https://github.com/ceph/ceph/tree/master/qa/suites/kcephfs>`_ + test the CephFS kernel module + +`krbd <https://github.com/ceph/ceph/tree/master/qa/suites/krbd>`_ + test the RBD kernel module + +`powercycle <https://github.com/ceph/ceph/tree/master/qa/suites/powercycle>`_ + verify the Ceph cluster behaves when machines are powered off + and on again + +`rados <https://github.com/ceph/ceph/tree/master/qa/suites/rados>`_ + run Ceph clusters including OSDs and MONs, under various conditions of + stress + +`rbd <https://github.com/ceph/ceph/tree/master/qa/suites/rbd>`_ + run RBD tests using actual Ceph clusters, with and without qemu + +`rgw <https://github.com/ceph/ceph/tree/master/qa/suites/rgw>`_ + run RGW tests using actual Ceph clusters + +`smoke <https://github.com/ceph/ceph/tree/master/qa/suites/smoke>`_ + run tests that exercise the Ceph API with an actual Ceph cluster + +`teuthology <https://github.com/ceph/ceph/tree/master/qa/suites/teuthology>`_ + verify that teuthology can run integration tests, with and without OpenStack + +`upgrade <https://github.com/ceph/ceph/tree/master/qa/suites/upgrade>`_ + for various versions of Ceph, verify that upgrades can happen + without disrupting an ongoing workload + +.. _`ceph-deploy man page`: ../../man/8/ceph-deploy + +teuthology-describe-tests +^^^^^^^^^^^^^^^^^^^^^^^^^ + +In February 2016, a new feature called ``teuthology-describe-tests`` was +added to the `teuthology framework`_ to facilitate documentation and better +understanding of integration tests (`feature announcement +<http://article.gmane.org/gmane.comp.file-systems.ceph.devel/29287>`_). + +The upshot is that tests can be documented by embedding ``meta:`` +annotations in the yaml files used to define the tests. The results can be +seen in the `ceph-qa-suite wiki +<http://tracker.ceph.com/projects/ceph-qa-suite/wiki/>`_. + +Since this is a new feature, many yaml files have yet to be annotated. +Developers are encouraged to improve the documentation, in terms of both +coverage and quality. + +How integration tests are run +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given that - as a new Ceph developer - you will typically not have access +to the `Sepia lab`_, you may rightly ask how you can run the integration +tests in your own environment. + +One option is to set up a teuthology cluster on bare metal. Though this is +a non-trivial task, it `is` possible. Here are `some notes +<http://docs.ceph.com/teuthology/docs/LAB_SETUP.html>`_ to get you started +if you decide to go this route. + +If you have access to an OpenStack tenant, you have another option: the +`teuthology framework`_ has an OpenStack backend, which is documented `here +<https://github.com/dachary/teuthology/tree/openstack#openstack-backend>`__. +This OpenStack backend can build packages from a given git commit or +branch, provision VMs, install the packages and run integration tests +on those VMs. This process is controlled using a tool called +`ceph-workbench ceph-qa-suite`_. This tool also automates publishing of +test results at http://teuthology-logs.public.ceph.com. + +Running integration tests on your code contributions and publishing the +results allows reviewers to verify that changes to the code base do not +cause regressions, or to analyze test failures when they do occur. + +Every teuthology cluster, whether bare-metal or cloud-provisioned, has a +so-called "teuthology machine" from which tests suites are triggered using the +``teuthology-suite`` command. + +A detailed and up-to-date description of each `teuthology-suite`_ option is +available by running the following command on the teuthology machine:: + + $ teuthology-suite --help + +.. _teuthology-suite: http://docs.ceph.com/teuthology/docs/teuthology.suite.html + +How integration tests are defined +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Integration tests are defined by yaml files found in the ``suites`` +subdirectory of the `ceph/qa sub-directory`_ and implemented by python +code found in the ``tasks`` subdirectory. Some tests ("standalone tests") +are defined in a single yaml file, while other tests are defined by a +directory tree containing yaml files that are combined, at runtime, into a +larger yaml file. + +Reading a standalone test +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Let us first examine a standalone test, or "singleton". + +Here is a commented example using the integration test +`rados/singleton/all/admin-socket.yaml +<https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/admin-socket.yaml>`_ +:: + + roles: + - - mon.a + - osd.0 + - osd.1 + tasks: + - install: + - ceph: + - admin_socket: + osd.0: + version: + git_version: + help: + config show: + config set filestore_dump_file /tmp/foo: + perf dump: + perf schema: + +The ``roles`` array determines the composition of the cluster (how +many MONs, OSDs, etc.) on which this test is designed to run, as well +as how these roles will be distributed over the machines in the +testing cluster. In this case, there is only one element in the +top-level array: therefore, only one machine is allocated to the +test. The nested array declares that this machine shall run a MON with +id ``a`` (that is the ``mon.a`` in the list of roles) and two OSDs +(``osd.0`` and ``osd.1``). + +The body of the test is in the ``tasks`` array: each element is +evaluated in order, causing the corresponding python file found in the +``tasks`` subdirectory of the `teuthology repository`_ or +`ceph/qa sub-directory`_ to be run. "Running" in this case means calling +the ``task()`` function defined in that file. + +In this case, the `install +<https://github.com/ceph/teuthology/blob/master/teuthology/task/install/__init__.py>`_ +task comes first. It installs the Ceph packages on each machine (as +defined by the ``roles`` array). A full description of the ``install`` +task is `found in the python file +<https://github.com/ceph/teuthology/blob/master/teuthology/task/install/__init__.py>`_ +(search for "def task"). + +The ``ceph`` task, which is documented `here +<https://github.com/ceph/ceph/blob/master/qa/tasks/ceph.py>`__ (again, +search for "def task"), starts OSDs and MONs (and possibly MDSs as well) +as required by the ``roles`` array. In this example, it will start one MON +(``mon.a``) and two OSDs (``osd.0`` and ``osd.1``), all on the same +machine. Control moves to the next task when the Ceph cluster reaches +``HEALTH_OK`` state. + +The next task is ``admin_socket`` (`source code +<https://github.com/ceph/ceph/blob/master/qa/tasks/admin_socket.py>`_). +The parameter of the ``admin_socket`` task (and any other task) is a +structure which is interpreted as documented in the task. In this example +the parameter is a set of commands to be sent to the admin socket of +``osd.0``. The task verifies that each of them returns on success (i.e. +exit code zero). + +This test can be run with:: + + $ teuthology-suite --suite rados/singleton/all/admin-socket.yaml fs/ext4.yaml + +Test descriptions +^^^^^^^^^^^^^^^^^ + +Each test has a "test description", which is similar to a directory path, +but not the same. In the case of a standalone test, like the one in +`Reading a standalone test`_, the test description is identical to the +relative path (starting from the ``suites/`` directory of the +`ceph/qa sub-directory`_) of the yaml file defining the test. + +Much more commonly, tests are defined not by a single yaml file, but by a +`directory tree of yaml files`. At runtime, the tree is walked and all yaml +files (facets) are combined into larger yaml "programs" that define the +tests. A full listing of the yaml defining the test is included at the +beginning of every test log. + +In these cases, the description of each test consists of the +subdirectory under `suites/ +<https://github.com/ceph/ceph/tree/master/qa/suites>`_ containing the +yaml facets, followed by an expression in curly braces (``{}``) consisting of +a list of yaml facets in order of concatenation. For instance the +test description:: + + ceph-deploy/basic/{distros/centos_7.0.yaml tasks/ceph-deploy.yaml} + +signifies the concatenation of two files: + +* ceph-deploy/basic/distros/centos_7.0.yaml +* ceph-deploy/basic/tasks/ceph-deploy.yaml + +How are tests built from directories? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As noted in the previous section, most tests are not defined in a single +yaml file, but rather as a `combination` of files collected from a +directory tree within the ``suites/`` subdirectory of the `ceph/qa sub-directory`_. + +The set of all tests defined by a given subdirectory of ``suites/`` is +called an "integration test suite", or a "teuthology suite". + +Combination of yaml facets is controlled by special files (``%`` and +``+``) that are placed within the directory tree and can be thought of as +operators. The ``%`` file is the "convolution" operator and ``+`` +signifies concatenation. + +Convolution operator +^^^^^^^^^^^^^^^^^^^^ + +The convolution operator, implemented as an empty file called ``%``, tells +teuthology to construct a test matrix from yaml facets found in +subdirectories below the directory containing the operator. + +For example, the `ceph-deploy suite +<https://github.com/ceph/ceph/tree/jewel/qa/suites/ceph-deploy/>`_ is +defined by the ``suites/ceph-deploy/`` tree, which consists of the files and +subdirectories in the following structure:: + + directory: ceph-deploy/basic + file: % + directory: distros + file: centos_7.0.yaml + file: ubuntu_16.04.yaml + directory: tasks + file: ceph-deploy.yaml + +This is interpreted as a 2x1 matrix consisting of two tests: + +1. ceph-deploy/basic/{distros/centos_7.0.yaml tasks/ceph-deploy.yaml} +2. ceph-deploy/basic/{distros/ubuntu_16.04.yaml tasks/ceph-deploy.yaml} + +i.e. the concatenation of centos_7.0.yaml and ceph-deploy.yaml and +the concatenation of ubuntu_16.04.yaml and ceph-deploy.yaml, respectively. +In human terms, this means that the task found in ``ceph-deploy.yaml`` is +intended to run on both CentOS 7.0 and Ubuntu 16.04. + +Without the file percent, the ``ceph-deploy`` tree would be interpreted as +three standalone tests: + +* ceph-deploy/basic/distros/centos_7.0.yaml +* ceph-deploy/basic/distros/ubuntu_16.04.yaml +* ceph-deploy/basic/tasks/ceph-deploy.yaml + +(which would of course be wrong in this case). + +Referring to the `ceph/qa sub-directory`_, you will notice that the +``centos_7.0.yaml`` and ``ubuntu_16.04.yaml`` files in the +``suites/ceph-deploy/basic/distros/`` directory are implemented as symlinks. +By using symlinks instead of copying, a single file can appear in multiple +suites. This eases the maintenance of the test framework as a whole. + +All the tests generated from the ``suites/ceph-deploy/`` directory tree +(also known as the "ceph-deploy suite") can be run with:: + + $ teuthology-suite --suite ceph-deploy + +An individual test from the `ceph-deploy suite`_ can be run by adding the +``--filter`` option:: + + $ teuthology-suite \ + --suite ceph-deploy/basic \ + --filter 'ceph-deploy/basic/{distros/ubuntu_16.04.yaml tasks/ceph-deploy.yaml}' + +.. note:: To run a standalone test like the one in `Reading a standalone + test`_, ``--suite`` alone is sufficient. If you want to run a single + test from a suite that is defined as a directory tree, ``--suite`` must + be combined with ``--filter``. This is because the ``--suite`` option + understands POSIX relative paths only. + +Concatenation operator +^^^^^^^^^^^^^^^^^^^^^^ + +For even greater flexibility in sharing yaml files between suites, the +special file plus (``+``) can be used to concatenate files within a +directory. For instance, consider the `suites/rbd/thrash +<https://github.com/ceph/ceph/tree/master/qa/suites/rbd/thrash>`_ +tree:: + + directory: rbd/thrash + file: % + directory: clusters + file: + + file: fixed-2.yaml + file: openstack.yaml + directory: workloads + file: rbd_api_tests_copy_on_read.yaml + file: rbd_api_tests.yaml + +This creates two tests: + +* rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml} +* rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests.yaml} + +Because the ``clusters/`` subdirectory contains the special file plus +(``+``), all the other files in that subdirectory (``fixed-2.yaml`` and +``openstack.yaml`` in this case) are concatenated together +and treated as a single file. Without the special file plus, they would +have been convolved with the files from the workloads directory to create +a 2x2 matrix: + +* rbd/thrash/{clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml} +* rbd/thrash/{clusters/openstack.yaml workloads/rbd_api_tests.yaml} +* rbd/thrash/{clusters/fixed-2.yaml workloads/rbd_api_tests_copy_on_read.yaml} +* rbd/thrash/{clusters/fixed-2.yaml workloads/rbd_api_tests.yaml} + +The ``clusters/fixed-2.yaml`` file is shared among many suites to +define the following ``roles``:: + + roles: + - [mon.a, mon.c, osd.0, osd.1, osd.2, client.0] + - [mon.b, osd.3, osd.4, osd.5, client.1] + +The ``rbd/thrash`` suite as defined above, consisting of two tests, +can be run with:: + + $ teuthology-suite --suite rbd/thrash + +A single test from the rbd/thrash suite can be run by adding the +``--filter`` option:: + + $ teuthology-suite \ + --suite rbd/thrash \ + --filter 'rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml}' + +Filtering tests by their description +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a few jobs fail and need to be run again, the ``--filter`` option +can be used to select tests with a matching description. For instance, if the +``rados`` suite fails the `all/peer.yaml <https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/peer.yaml>`_ test, the following will only run the tests that contain this file:: + + teuthology-suite --suite rados --filter all/peer.yaml + +The ``--filter-out`` option does the opposite (it matches tests that do +`not` contain a given string), and can be combined with the ``--filter`` +option. + +Both ``--filter`` and ``--filter-out`` take a comma-separated list of strings (which +means the comma character is implicitly forbidden in filenames found in the +`ceph/qa sub-directory`_). For instance:: + + teuthology-suite --suite rados --filter all/peer.yaml,all/rest-api.yaml + +will run tests that contain either +`all/peer.yaml <https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/peer.yaml>`_ +or +`all/rest-api.yaml <https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/rest-api.yaml>`_ + +Each string is looked up anywhere in the test description and has to +be an exact match: they are not regular expressions. + +Reducing the number of tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``rados`` suite generates thousands of tests out of a few hundred +files. This happens because teuthology constructs test matrices from +subdirectories wherever it encounters a file named ``%``. For instance, +all tests in the `rados/basic suite +<https://github.com/ceph/ceph/tree/master/qa/suites/rados/basic>`_ +run with different messenger types: ``simple``, ``async`` and +``random``, because they are combined (via the special file ``%``) with +the `msgr directory +<https://github.com/ceph/ceph/tree/master/qa/suites/rados/basic/msgr>`_ + +All integration tests are required to be run before a Ceph release is published. +When merely verifying whether a contribution can be merged without +risking a trivial regression, it is enough to run a subset. The ``--subset`` option can be used to +reduce the number of tests that are triggered. For instance:: + + teuthology-suite --suite rados --subset 0/4000 + +will run as few tests as possible. The tradeoff in this case is that +not all combinations of test variations will together, +but no matter how small a ratio is provided in the ``--subset``, +teuthology will still ensure that all files in the suite are in at +least one test. Understanding the actual logic that drives this +requires reading the teuthology source code. + +The ``--limit`` option only runs the first ``N`` tests in the suite: +this is rarely useful, however, because there is no way to control which +test will be first. + +Testing in the cloud +==================== + +In this chapter, we will explain in detail how use an OpenStack +tenant as an environment for Ceph integration testing. + +Assumptions and caveat +---------------------- + +We assume that: + +1. you are the only person using the tenant +2. you have the credentials +3. the tenant supports the ``nova`` and ``cinder`` APIs + +Caveat: be aware that, as of this writing (July 2016), testing in +OpenStack clouds is a new feature. Things may not work as advertised. +If you run into trouble, ask for help on `IRC`_ or the `Mailing list`_, or +open a bug report at the `ceph-workbench bug tracker`_. + +.. _`ceph-workbench bug tracker`: http://ceph-workbench.dachary.org/root/ceph-workbench/issues + +Prepare tenant +-------------- + +If you have not tried to use ``ceph-workbench`` with this tenant before, +proceed to the next step. + +To start with a clean slate, login to your tenant via the Horizon dashboard and: + +* terminate the ``teuthology`` and ``packages-repository`` instances, if any +* delete the ``teuthology`` and ``teuthology-worker`` security groups, if any +* delete the ``teuthology`` and ``teuthology-myself`` key pairs, if any + +Also do the above if you ever get key-related errors ("invalid key", etc.) when +trying to schedule suites. + +Getting ceph-workbench +---------------------- + +Since testing in the cloud is done using the `ceph-workbench +ceph-qa-suite`_ tool, you will need to install that first. It is designed +to be installed via Docker, so if you don't have Docker running on your +development machine, take care of that first. You can follow `the official +tutorial <https://docs.docker.com/engine/installation/>`_ to install if +you have not installed yet. + +Once Docker is up and running, install ``ceph-workbench`` by following the +`Installation instructions in the ceph-workbench documentation +<http://ceph-workbench.readthedocs.io/en/latest/#installation>`_. + +Linking ceph-workbench with your OpenStack tenant +------------------------------------------------- + +Before you can trigger your first teuthology suite, you will need to link +``ceph-workbench`` with your OpenStack account. + +First, download a ``openrc.sh`` file by clicking on the "Download OpenStack +RC File" button, which can be found in the "API Access" tab of the "Access +& Security" dialog of the OpenStack Horizon dashboard. + +Second, create a ``~/.ceph-workbench`` directory, set its permissions to +700, and move the ``openrc.sh`` file into it. Make sure that the filename +is exactly ``~/.ceph-workbench/openrc.sh``. + +Third, edit the file so it does not ask for your OpenStack password +interactively. Comment out the relevant lines and replace them with +something like:: + + export OS_PASSWORD="aiVeth0aejee3eep8rogho3eep7Pha6ek" + +When `ceph-workbench ceph-qa-suite`_ connects to your OpenStack tenant for +the first time, it will generate two keypairs: ``teuthology-myself`` and +``teuthology``. + +.. If this is not the first time you have tried to use +.. `ceph-workbench ceph-qa-suite`_ with this tenant, make sure to delete any +.. stale keypairs with these names! + +Run the dummy suite +------------------- + +You are now ready to take your OpenStack teuthology setup for a test +drive:: + + $ ceph-workbench ceph-qa-suite --suite dummy + +Be forewarned that the first run of `ceph-workbench ceph-qa-suite`_ on a +pristine tenant will take a long time to complete because it downloads a VM +image and during this time the command may not produce any output. + +The images are cached in OpenStack, so they are only downloaded once. +Subsequent runs of the same command will complete faster. + +Although ``dummy`` suite does not run any tests, in all other respects it +behaves just like a teuthology suite and produces some of the same +artifacts. + +The last bit of output should look something like this:: + + pulpito web interface: http://149.202.168.201:8081/ + ssh access : ssh -i /home/smithfarm/.ceph-workbench/teuthology-myself.pem ubuntu@149.202.168.201 # logs in /usr/share/nginx/html + +What this means is that `ceph-workbench ceph-qa-suite`_ triggered the test +suite run. It does not mean that the suite run has completed. To monitor +progress of the run, check the Pulpito web interface URL periodically, or +if you are impatient, ssh to the teuthology machine using the ssh command +shown and do:: + + $ tail -f /var/log/teuthology.* + +The `/usr/share/nginx/html` directory contains the complete logs of the +test suite. If we had provided the ``--upload`` option to the +`ceph-workbench ceph-qa-suite`_ command, these logs would have been +uploaded to http://teuthology-logs.public.ceph.com. + +Run a standalone test +--------------------- + +The standalone test explained in `Reading a standalone test`_ can be run +with the following command:: + + $ ceph-workbench ceph-qa-suite --suite rados/singleton/all/admin-socket.yaml + +This will run the suite shown on the current ``master`` branch of +``ceph/ceph.git``. You can specify a different branch with the ``--ceph`` +option, and even a different git repo with the ``--ceph-git-url`` option. (Run +``ceph-workbench ceph-qa-suite --help`` for an up-to-date list of available +options.) + +The first run of a suite will also take a long time, because ceph packages +have to be built, first. Again, the packages so built are cached and +`ceph-workbench ceph-qa-suite`_ will not build identical packages a second +time. + +Interrupt a running suite +------------------------- + +Teuthology suites take time to run. From time to time one may wish to +interrupt a running suite. One obvious way to do this is:: + + ceph-workbench ceph-qa-suite --teardown + +This destroys all VMs created by `ceph-workbench ceph-qa-suite`_ and +returns the OpenStack tenant to a "clean slate". + +Sometimes you may wish to interrupt the running suite, but keep the logs, +the teuthology VM, the packages-repository VM, etc. To do this, you can +``ssh`` to the teuthology VM (using the ``ssh access`` command reported +when you triggered the suite -- see `Run the dummy suite`_) and, once +there:: + + sudo /etc/init.d/teuthology restart + +This will keep the teuthology machine, the logs and the packages-repository +instance but nuke everything else. + +Upload logs to archive server +----------------------------- + +Since the teuthology instance in OpenStack is only semi-permanent, with limited +space for storing logs, ``teuthology-openstack`` provides an ``--upload`` +option which, if included in the ``ceph-workbench ceph-qa-suite`` command, +will cause logs from all failed jobs to be uploaded to the log archive server +maintained by the Ceph project. The logs will appear at the URL:: + + http://teuthology-logs.public.ceph.com/$RUN + +where ``$RUN`` is the name of the run. It will be a string like this:: + + ubuntu-2016-07-23_16:08:12-rados-hammer-backports---basic-openstack + +Even if you don't providing the ``--upload`` option, however, all the logs can +still be found on the teuthology machine in the directory +``/usr/share/nginx/html``. + +Provision VMs ad hoc +-------------------- + +From the teuthology VM, it is possible to provision machines on an "ad hoc" +basis, to use however you like. The magic incantation is:: + + teuthology-lock --lock-many $NUMBER_OF_MACHINES \ + --os-type $OPERATING_SYSTEM \ + --os-version $OS_VERSION \ + --machine-type openstack \ + --owner $EMAIL_ADDRESS + +The command must be issued from the ``~/teuthology`` directory. The possible +values for ``OPERATING_SYSTEM`` AND ``OS_VERSION`` can be found by examining +the contents of the directory ``teuthology/openstack/``. For example:: + + teuthology-lock --lock-many 1 --os-type ubuntu --os-version 16.04 \ + --machine-type openstack --owner foo@example.com + +When you are finished with the machine, find it in the list of machines:: + + openstack server list + +to determine the name or ID, and then terminate it with:: + + openstack server delete $NAME_OR_ID + +Deploy a cluster for manual testing +----------------------------------- + +The `teuthology framework`_ and `ceph-workbench ceph-qa-suite`_ are +versatile tools that automatically provision Ceph clusters in the cloud and +run various tests on them in an automated fashion. This enables a single +engineer, in a matter of hours, to perform thousands of tests that would +keep dozens of human testers occupied for days or weeks if conducted +manually. + +However, there are times when the automated tests do not cover a particular +scenario and manual testing is desired. It turns out that it is simple to +adapt a test to stop and wait after the Ceph installation phase, and the +engineer can then ssh into the running cluster. Simply add the following +snippet in the desired place within the test YAML and schedule a run with the +test:: + + tasks: + - exec: + client.0: + - sleep 1000000000 # forever + +(Make sure you have a ``client.0`` defined in your ``roles`` stanza or adapt +accordingly.) + +The same effect can be achieved using the ``interactive`` task:: + + tasks: + - interactive + +By following the test log, you can determine when the test cluster has entered +the "sleep forever" condition. At that point, you can ssh to the teuthology +machine and from there to one of the target VMs (OpenStack) or teuthology +worker machines machine (Sepia) where the test cluster is running. + +The VMs (or "instances" in OpenStack terminology) created by +`ceph-workbench ceph-qa-suite`_ are named as follows: + +``teuthology`` - the teuthology machine + +``packages-repository`` - VM where packages are stored + +``ceph-*`` - VM where packages are built + +``target*`` - machines where tests are run + +The VMs named ``target*`` are used by tests. If you are monitoring the +teuthology log for a given test, the hostnames of these target machines can +be found out by searching for the string ``Locked targets``:: + + 2016-03-20T11:39:06.166 INFO:teuthology.task.internal:Locked targets: + target149202171058.teuthology: null + target149202171059.teuthology: null + +The IP addresses of the target machines can be found by running ``openstack +server list`` on the teuthology machine, but the target VM hostnames (e.g. +``target149202171058.teuthology``) are resolvable within the teuthology +cluster. + + +Testing - how to run s3-tests locally +===================================== + +RGW code can be tested by building Ceph locally from source, starting a vstart +cluster, and running the "s3-tests" suite against it. + +The following instructions should work on jewel and above. + +Step 1 - build Ceph +------------------- + +Refer to :doc:`/install/build-ceph`. + +You can do step 2 separately while it is building. + +Step 2 - vstart +--------------- + +When the build completes, and still in the top-level directory of the git +clone where you built Ceph, do the following, for cmake builds:: + + cd build/ + RGW=1 ../src/vstart.sh -n + +This will produce a lot of output as the vstart cluster is started up. At the +end you should see a message like:: + + started. stop.sh to stop. see out/* (e.g. 'tail -f out/????') for debug output. + +This means the cluster is running. + + +Step 3 - run s3-tests +--------------------- + +To run the s3tests suite do the following:: + + $ ../qa/workunits/rgw/run-s3tests.sh + +.. WIP +.. === +.. +.. Building RPM packages +.. --------------------- +.. +.. Ceph is regularly built and packaged for a number of major Linux +.. distributions. At the time of this writing, these included CentOS, Debian, +.. Fedora, openSUSE, and Ubuntu. +.. +.. Architecture +.. ============ +.. +.. Ceph is a collection of components built on top of RADOS and provide +.. services (RBD, RGW, CephFS) and APIs (S3, Swift, POSIX) for the user to +.. store and retrieve data. +.. +.. See :doc:`/architecture` for an overview of Ceph architecture. The +.. following sections treat each of the major architectural components +.. in more detail, with links to code and tests. +.. +.. FIXME The following are just stubs. These need to be developed into +.. detailed descriptions of the various high-level components (RADOS, RGW, +.. etc.) with breakdowns of their respective subcomponents. +.. +.. FIXME Later, in the Testing chapter I would like to take another look +.. at these components/subcomponents with a focus on how they are tested. +.. +.. RADOS +.. ----- +.. +.. RADOS stands for "Reliable, Autonomic Distributed Object Store". In a Ceph +.. cluster, all data are stored in objects, and RADOS is the component responsible +.. for that. +.. +.. RADOS itself can be further broken down into Monitors, Object Storage Daemons +.. (OSDs), and client APIs (librados). Monitors and OSDs are introduced at +.. :doc:`/start/intro`. The client library is explained at +.. :doc:`/rados/api/index`. +.. +.. RGW +.. --- +.. +.. RGW stands for RADOS Gateway. Using the embedded HTTP server civetweb_ or +.. Apache FastCGI, RGW provides a REST interface to RADOS objects. +.. +.. .. _civetweb: https://github.com/civetweb/civetweb +.. +.. A more thorough introduction to RGW can be found at :doc:`/radosgw/index`. +.. +.. RBD +.. --- +.. +.. RBD stands for RADOS Block Device. It enables a Ceph cluster to store disk +.. images, and includes in-kernel code enabling RBD images to be mounted. +.. +.. To delve further into RBD, see :doc:`/rbd/rbd`. +.. +.. CephFS +.. ------ +.. +.. CephFS is a distributed file system that enables a Ceph cluster to be used as a NAS. +.. +.. File system metadata is managed by Meta Data Server (MDS) daemons. The Ceph +.. file system is explained in more detail at :doc:`/cephfs/index`. +.. |