diff options
Diffstat (limited to 'doc/dev/corpus.rst')
| -rw-r--r-- | doc/dev/corpus.rst | 100 |
1 files changed, 100 insertions, 0 deletions
diff --git a/doc/dev/corpus.rst b/doc/dev/corpus.rst new file mode 100644 index 000000000..4005f70c0 --- /dev/null +++ b/doc/dev/corpus.rst @@ -0,0 +1,100 @@ + +Corpus structure +================ + +ceph.git/ceph-object-corpus is a submodule.:: + + bin/ # misc scripts + archive/$version/objects/$type/$hash # a sample of encoded objects from a specific version + +You can also mark known or deliberate incompatibilities between versions with:: + + archive/$version/forward_incompat/$type + +The presence of a file indicates that new versions of code cannot +decode old objects across that ``$version`` (this is normally the case). + + +How to generate an object corpus +-------------------------------- + +.. highlight:: shell + +We can generate an object corpus for a particular version of ceph using the +script of ``script/gen-corpus.sh``, or by following the instructions below: + +#. Checkout a clean repo (best not to do this where you normally work):: + + git clone ceph.git + cd ceph + git submodule update --init --recursive + +#. Build with flag to dump objects to ``/tmp/foo``:: + + rm -rf /tmp/foo ; mkdir /tmp/foo + do_cmake.sh -DCMAKE_CXX_FLAGS="-DENCODE_DUMP_PATH=/tmp/foo" + cd build + make + +#. Start via vstart:: + + cd build + MON=3 MGR=2 OSD=3 MDS=3 RGW=1 ../src/vstart.sh -n -x + +#. Use as much functionality of the cluster as you can, to exercise as many object encoder methods as possible:: + + bin/ceph osd pool create mypool + bin/rados -p mypool bench 10 write -b 123 + bin/ceph osd out 0 + bin/ceph osd in 0 + bin/init-ceph restart osd.1 + for f in ../qa/workunits/cls/*.sh ; do PATH="bin:$PATH" $f ; done + PATH="bin:$PATH" ../qa/workunits/rados/test.sh + bin/ceph_test_librbd + bin/ceph_test_libcephfs + bin/init-ceph restart mds.a + ../qa/workunits/rgw/run-s3tests.sh + +#. Stop:: + + ../src/stop.sh + +#. Import the corpus (this will take a few minutes):: + + ../src/test/encoding/import.sh /tmp/foo `bin/ceph-dencoder version` ../ceph-object-corpus/archive + ../src/test/encoding/import-generated.sh ../ceph-object-corpus/archive + +#. Prune it! There will be a bazillion copies of various objects, and we only want a representative sample.:: + + pushd ../ceph-object-corpus + bin/prune-archive.sh + popd + +#. Verify the tests pass:: + + ctest -R readable.sh + +#. Commit it to the corpus repo and push:: + + pushd ../ceph-object-corpus + git checkout -b wip-new + git add archive/`../build/bin/ceph-dencoder version` + git commit -m `../build/bin/ceph-dencoder version` + git remote add cc git@github.com:ceph/ceph-object-corpus.git + git push cc wip-new + popd + +#. Go test it out:: + + cd my/regular/tree + cd ceph-object-corpus + git fetch origin + git checkout wip-new + cd ../build + ctest -R readable.sh + +#. If everything looks good, update the submodule master branch, and commit the submodule in ceph.git. + + + + |