commit cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba Author: Jenkins Build Slave User Date: Thu Nov 30 14:59:27 2017 +0000 12.2.2 commit 83684b91a3c6b31419114b83fc22106146885fb6 Merge: 6d9f2161b2 b5d02ac0fd Author: Kefu Chai Date: Thu Nov 30 22:26:46 2017 +0800 Merge pull request #19240 from tchaikov/wip-22266-luminous luminous: tools/ceph_monstore_tool: rebuild initial mgrmap also Reviewed-by: Abhishek Lekshmanan commit 6d9f2161b269b19b0301348dd9b17660420624d5 Merge: 0b4c3b8f1e b02075a963 Author: Gregory Farnum Date: Wed Nov 29 16:25:14 2017 -0500 Merge pull request #19230 from tchaikov/wip-22247-luminous luminous: mon/PGMap: Fix %USED calculation Reviewed-by: Greg Farnum commit 0b4c3b8f1ed72c3a0053ed372e1c5c9b0d48de59 Merge: a57edd4f1a eb99a7ee36 Author: Andrew Schoen Date: Wed Nov 29 10:33:34 2017 -0600 Merge pull request #19239 from ceph/luminous-bz1518264 luminous: ceph-volume correctly fallback to bluestore when no objectstore is specified Reviewed-by: Andrew Schoen commit b5d02ac0fd60c26417ad33dc6efcf80bff503fdd Author: Kefu Chai Date: Wed Nov 29 20:21:33 2017 +0800 tools/ceph_monstore_tool: rebuild initial mgrmap also Fixes: http://tracker.ceph.com/issues/22266 Signed-off-by: Kefu Chai (cherry picked from commit f63d1da4afa3463730ecbc0be29df6375b79fa8f) commit eb99a7ee36d541939f589a674d9890b14e017dbe Author: Alfredo Deza Date: Tue Nov 28 09:22:19 2017 -0500 ceph-volume lvm.activate correct detection and fallback of objectstore flags Signed-off-by: Alfredo Deza (cherry picked from commit f0ac2dfdbc0a05cff0a5edbedca91274885b7870) commit 10dbbb0a6e09c6181e81b04765cd368963753bb8 Author: Alfredo Deza Date: Tue Nov 28 09:21:34 2017 -0500 ceph-volume tests.devices.lvm verify objectstore flags in activate Signed-off-by: Alfredo Deza (cherry picked from commit 8d22856d6c95ac3b45918765b382cae507a8d1a4) commit a57edd4f1a47f08c0f93d7a3ff57c376b72c9a8a Merge: 1b0a5660f7 ee02b5c037 Author: Sage Weil Date: Wed Nov 29 08:43:07 2017 -0600 Merge pull request #19217 from liewegas/wip-p2p-app-warning-luminous upgrade/jewel-x/parallel: debug fuse commit b02075a963c24c6e1b24d1b076275dcce5350a03 Author: Xiaoxi Chen Date: Sun Nov 26 22:51:58 2017 -0700 mon/PGMap: Fix %USED calculation bug. Previous code forgot to multiple raw_used_ratio to calculate used byte. Fixes: http://tracker.ceph.com/issues/22247 Signed-off-by: Xiaoxi Chen (cherry picked from commit d10c6c26f9ef7372e2c95da79d23b07ce5f4e0e5) commit ee02b5c037c67c708335745898328e990ecacc90 Author: Sage Weil Date: Tue Nov 28 16:48:36 2017 -0600 qa/suites/upgrade/jewel-x/point-to-point: whitelist more stuff Signed-off-by: Sage Weil commit d2a284c782fed3bec4ed38c898351826bca0a8bc Author: Sage Weil Date: Tue Nov 28 15:02:18 2017 -0600 qa/suites/upgrade/jewel-x/parallel: only mount ceph-fuse once Otherwise we do 4 mounts for no good reason. Signed-off-by: Sage Weil commit b7c6d9799cbfba3cff7bee5fd1de309a8299d911 Author: Sage Weil Date: Tue Nov 28 15:00:05 2017 -0600 upgrade/jewel-x/parallel: debug fuse Signed-off-by: Sage Weil commit 1b0a5660f793714e8cb470c35e084a141135cd8c Merge: 4ed747dccf e50af8bb04 Author: Sage Weil Date: Tue Nov 28 14:48:08 2017 -0600 Merge pull request #19216 from liewegas/wip-p2p-app-warning-luminous qa/suites/upgrade/jewel-x/point-to-point: fix suppression of pool app warning commit e50af8bb04bd37bd0e28ddc7935e29bcdc106b05 Author: Sage Weil Date: Tue Nov 28 14:46:02 2017 -0600 qa/suites/upgrade/jewel-x/point-to-point: fix suppression of pool app warning It's generated on the mgr. Signed-off-by: Sage Weil commit 4ed747dccfaae15a9a807d3f3db2fc44b8b21fda Merge: acb02717f6 6e1e33a88d Author: Kefu Chai Date: Tue Nov 28 23:47:17 2017 +0800 Merge pull request #19205 from tchaikov/wip-22136-luminous luminous: qa/ceph-disk: enlarge the simulated SCSI disk Reviewed-by: Alfredo Deza commit acb02717f6e96f96d4128bbebd946238d3c79291 Merge: 55f5d7d1f1 bb03f6b20b Author: Nathan Cutler Date: Tue Nov 28 14:03:01 2017 +0100 Merge pull request #19152 from smithfarm/wip-22235-luminous luminous: ceph-disk flake8 test fails on very old, and very new, versions of flake8 Reviewed-by: Kefu Chai commit 6e1e33a88d038d5b6a469d7fb4e52182b888010f Author: Kefu Chai Date: Tue Nov 28 14:42:31 2017 +0800 qa/ceph-disk: enlarge the simulated SCSI disk 100MB will be allocated for journal, and the remaining 100MB is for data device. taking the inode into consideration, there will be approximately 87988 kB available for the activated OSD. and it will complain with a "nearfull" state. Fixes: http://tracker.ceph.com/issues/22136 Signed-off-by: Kefu Chai (cherry picked from commit b3c159e9fb8c31d0acd75f0702080f18959f672e) commit 55f5d7d1f19696af981007c63a7503b2e5d0a5a8 Merge: 3bd0007d5f ad2393b8b8 Author: Nathan Cutler Date: Tue Nov 28 03:23:09 2017 +0100 Merge pull request #19173 from smithfarm/wip-22251-luminous luminous: build/ops: macros expanding in spec file comment Reviewed-by: David Disseldorp Reviewed-by: Ken Dreyer commit 3bd0007d5f1c991a80ff80fc2b7bb983ceee971e Merge: 7ce2013dfd 5633a58824 Author: Yuri Weinstein Date: Mon Nov 27 10:23:57 2017 -0800 Merge pull request #19179 from ceph/wip-yuriw-BP-19175-luminous qa/suites/rados/rest: move rest_test from qa/suites/rest/ Reviewed-by: Yuri Weinstein commit 5633a58824f6248b79af74780411746cb7518813 Author: Sage Weil Date: Mon Nov 27 10:11:51 2017 -0600 qa/suites/rados/rest: move rest_test from qa/suites/rest/ ...and add more health whitelists. Signed-off-by: Sage Weil (cherry picked from commit ddb274669ca16b95c452b256e58a2830af10c73d) Signed-off-by: Yuri Weinstein commit 7ce2013dfda6f06ac47878fd4635ac0862bfbbbd Merge: ecec65906a 61e4de133d Author: Yuri Weinstein Date: Mon Nov 27 09:20:18 2017 -0800 Merge pull request #19177 from liewegas/wip-jewel-x-p2p qa/suites/upgrade/jewel-x/point-to-point: skip ec tests when mons may be old Reviewed-by: Yuri Weinstein commit 61e4de133d9fa1eddcd858152b7828515e0b280f Author: Sage Weil Date: Mon Nov 27 10:28:16 2017 -0600 qa/suites/upgrade/jewel-x/point-to-point: skip ec tests when mons may be old Early point release mons don't handle legacy ruleset-* ec profiles, new ones do. Skip the ec tests that may trigger this when we are doing a workload that races with mon upgrades. Signed-off-by: Sage Weil commit ad2393b8b8dc71a8099fa87924dcb6936a27fd84 Author: Ken Dreyer Date: Thu Aug 17 09:19:38 2017 -0600 rpm: rm macros in comments rpm expands all macros in a .spec file, even those in comments. Drop the percent signs so rpm will not expand these. This change silences rpmlint's warning about macros in comments. Signed-off-by: Ken Dreyer (cherry picked from commit 5c1e9f55d3a144c93c9c18b031db3cdc320561ef) commit bb03f6b20b4963ad5d51c1c6f45f19bc7eaf220b Author: Nathan Cutler Date: Tue Nov 21 11:36:02 2017 +0100 tests: ceph-disk: ignore E722 in flake8 test Very old, and very new, versions of flake8 treat E722 as an error: flake8 runtests: commands[0] | flake8 --ignore=H105,H405,E127 ceph_disk tests ceph_disk/main.py:1575:9: E722 do not use bare except' ceph_disk/main.py:1582:9: E722 do not use bare except' ceph_disk/main.py:3252:5: E722 do not use bare except' ceph_disk/main.py:3288:21: E722 do not use bare except' ceph_disk/main.py:3296:17: E722 do not use bare except' ceph_disk/main.py:4358:5: E722 do not use bare except' tests/test_main.py:26:1: E722 do not use bare except' ERROR: InvocationError: '/opt/j/ws/mkck/src/ceph-disk/.tox/flake8/bin/flake8 --ignore=H105,H405,E127 ceph_disk tests' Fixes: http://tracker.ceph.com/issues/22207 References: https://gitlab.com/pycqa/flake8/issues/361 Signed-off-by: Nathan Cutler (cherry picked from commit 3600cd7bfdca76485d2998c5da1a0dd25816d1ce) commit ecec65906af60c16fc99f669c1591bb821305a64 Merge: 613634c1da aac7a85a3a Author: Kefu Chai Date: Wed Nov 22 13:36:58 2017 +0800 Merge pull request #19080 from smithfarm/wip-22212-luminous luminous: tests: ceph-disk: silence deprecation warnings Reviewed-by: Kefu Chai commit aac7a85a3ae9cce38987ce3f51be4c6cb63c8dd2 Author: Kefu Chai Date: Tue Nov 21 21:47:30 2017 +0800 qa/workunits: silence py warnings for ceph-disk tests ceph-disk now prints "depreacted" warning message when it starts. but the tests parses its stdout and stderr for a json string. so we need to silence the warnings for the tests. Fixes: http://tracker.ceph.com/issues/22154 Signed-off-by: Kefu Chai (cherry picked from commit d44334f31704487ec3574738e75145872d9932cf) commit 613634c1da4cea975d6e1da7d258600c454c9400 Merge: 5a469c0376 94839deb2f Author: Andrew Schoen Date: Mon Nov 20 09:24:41 2017 -0600 Merge pull request #18989 from ceph/wip-luminous-rm22154 luminous: ceph-disk create deprecation warnings Reviewed-by: Andrew Schoen commit 5a469c0376e299e5c60d7d9b3c16640515b11f26 Merge: 3532f233f6 37d2726751 Author: Kefu Chai Date: Mon Nov 20 22:55:06 2017 +0800 Merge pull request #19025 from tchaikov/wip-pr-19024-luminous luminous: qa/tasks: prolong revive_osd() timeout to 6 min Reviewed-by: Sage Weil commit 3532f233f691ac22c32c4533a677b95d0e4c6273 Merge: 5e519aebe7 0f7332e925 Author: Sage Weil Date: Mon Nov 20 07:23:30 2017 -0600 Merge pull request #18945 from liewegas/wip-22128 mon/OSDMonitor: add option to fix up ruleset-* to crush-* for ec profiles Reviewed-by: Joao Eduardo Luis commit 37d2726751c05a0b5a3c1b739bce49995cc72445 Author: Kefu Chai Date: Mon Nov 20 13:40:56 2017 +0800 qa/tasks: prolong revive_osd() timeout to 6 min see also #17902 Fixes: http://tracker.ceph.com/issues/21474 Signed-off-by: Kefu Chai (cherry picked from commit 749bbda07522114b99b17a4a01cfcf019520285d) commit 5e519aebe718bd16d52e974beba8e3bf97b04ae8 Merge: 9abf091336 96c981ce6b Author: Sage Weil Date: Sat Nov 18 20:34:50 2017 -0600 Merge pull request #18983 from pdvian/wip-22150-luminous luminous: os/bluestore: fix SharedBlob unregistration commit 0f7332e9259b320777770f0312233a31103bfeda Author: Sage Weil Date: Wed Nov 15 08:55:33 2017 -0600 mon/OSDMonitor: add option to fix up ruleset-* to crush-* for ec profiles The jewel->luminous upgrade test will fail if we finish the upgrade while a workload setting old-style ec profiles is running. Add option to automatically fix them up. Warn to the cluster log when this happens. For now, enable this option to ease upgrades and whitelist the warning. Only include this option in luminous so that we implicitly sunset this compatibility kludge immediately. Fixes: http://tracker.ceph.com/issues/22128 Signed-off-by: Sage Weil commit 9abf0913367a1ad8cf076c79bf3adace8c526853 Merge: c0e032c153 975229c367 Author: Sage Weil Date: Sat Nov 18 09:44:46 2017 -0600 Merge pull request #18751 from shinobu-x/wip-21701-luminous luminous: ceph-kvstore-tool does not call bluestore's umount when exit commit c0e032c153a36c00109970395fe42cc2013fdb9a Merge: 81b768ab03 4d4aee14a4 Author: Sage Weil Date: Sat Nov 18 09:44:19 2017 -0600 Merge pull request #18750 from shinobu-x/wip-21702-luminous luminous: BlueStore::umount will crash when the BlueStore is opened by start_kv_only() commit 81b768ab036e66d3f4d36ddc31febc4bbcb143fb Merge: f02d7109c5 0fe463b5e1 Author: Sage Weil Date: Sat Nov 18 09:41:37 2017 -0600 Merge pull request #18860 from liewegas/wip-filestore-rocksdb-compression-luminous luminous: os/filestore: disable rocksdb compression Reviewed-by: Douglas Fuller commit 94839deb2f32e8059206ad48ad711dd2baf2fef5 Author: Alfredo Deza Date: Fri Nov 17 13:10:29 2017 -0500 doc/ceph-volume create a migration section to help users coming from ceph-disk Signed-off-by: Alfredo Deza (cherry picked from commit e433efe51481ec63bc9753a0f446fe2b12b5bb6a) commit b8bf0d047868054135592188c7ebe186181310c5 Author: Alfredo Deza Date: Fri Nov 17 13:01:14 2017 -0500 ceph-disk add deprecation warnings in favor of ceph-volume Signed-off-by: Alfredo Deza Fixes: http://tracker.ceph.com/issues/22154 (cherry picked from commit c9da92e1ec8d60dc40d86630dab95e4fec4d0d30) commit f02d7109c5f00cd2165082c4336b3623352be7c5 Merge: 9515da2777 ebbe033936 Author: Alfredo Deza Date: Fri Nov 17 12:19:36 2017 -0500 Merge pull request #18973 from ceph/backport-18924 luminous: ceph-volume: allow using a device or partition for `lvm --data` Reviewed-by: Alfredo Deza commit ebbe033936c424a0f6ba2d102fc271d0b0e9578d Author: Alfredo Deza Date: Fri Nov 17 08:41:24 2017 -0500 ceph-volume tests.devices.lvm prepare isn't bluestore specific anymore Signed-off-by: Alfredo Deza (cherry picked from commit 2f40dbac386be631ac97e2450117c218d9bf02f9) commit 96c981ce6b00520a94385228bcf70a90c5ae8ff9 Author: Sage Weil Date: Tue Nov 7 22:05:10 2017 -0600 os/bluestore: fix SharedBlob unregistration We use the SharedBlobSet remove() in three cases: - from SharedBlob::put(), we try to remove ourselves from the set, but have to deal with a racing lookup, so the removal is conditional on nref still being 0. - from split_cache(), we move the SharedBlob to another collection - from make_blob_unshared(), we remove the entry when we clear the sbid. The problem is that the condtiional remove() (for the first case) was being used for all three cases, and in the second two cases nref is always != 0, so it doesn't actually happen. This can lead to a crash during cache shutdown. Fix by making two variants: remove() that is unconditional, and try_remove() that is conditional. Set the sb->coll pointer after because remove() asserts the parent matches where we are unregistering. Fixes: http://tracker.ceph.com/issues/22039 Signed-off-by: Sage Weil (cherry picked from commit 78a465b90b1c5187cfec9f25eb58b3bf617ca39b) commit 9515da277737ad81d48f5c032f627f3cb919f746 Merge: 5c9b93dadd d19b740359 Author: Josh Durgin Date: Thu Nov 16 13:40:32 2017 -0800 Merge pull request #18957 from tchaikov/wip-pr-18808-luminous luminous: qa: do not wait for down/out osd for pg convergence Reviewed-by: Josh Durgin commit ad4668b65cb3e32f1b3afc5ec767004f166856e6 Author: Alfredo Deza Date: Thu Nov 16 08:50:34 2017 -0500 ceph-volume tests.functional create a common playbook directory for setups Signed-off-by: Alfredo Deza (cherry picked from commit def89a411f2f4bdb23c4bbd3b71c1310a564f07d) commit d25a04114c98aa1449809248e8b87ecd1124af3d Author: Alfredo Deza Date: Tue Nov 14 11:01:48 2017 -0500 doc/ceph-volume lvm prepare can accept devices for --data Signed-off-by: Alfredo Deza (cherry picked from commit 9bfab348b2ff59c0b939a19c22d4a06bc5236b96) Conflicts: doc/ceph-volume/lvm/prepare.rst commit 7f7d62a1848e297153f88a8e476585596114e1d5 Author: Alfredo Deza Date: Tue Nov 14 08:42:27 2017 -0500 ceph-volume tests.functional add setup playbook for xenial filestore Signed-off-by: Alfredo Deza (cherry picked from commit 210c1e1d4770dde47f0008cadaaec35deb729272) commit 210e1b3bf507654167349411c5cc85641b1d8680 Author: Alfredo Deza Date: Tue Nov 14 08:42:02 2017 -0500 ceph-volume tests.functional add data device and journal to xenial filestore Signed-off-by: Alfredo Deza (cherry picked from commit d0754e25177b9bf3bce695b7991e99f49312d91c) commit 211895ac0906747bc448b2326ae5c62f3e6ee4a1 Author: Alfredo Deza Date: Tue Nov 14 08:40:37 2017 -0500 ceph-volume tests.functional add setup playbook for xenial bluestore Signed-off-by: Alfredo Deza (cherry picked from commit d0f5623963f58fc4d5727a79d8377b2191b0f9dc) commit 398a1365a12de95bc9810b89a9aeab9040bb07b8 Author: Alfredo Deza Date: Tue Nov 14 08:40:18 2017 -0500 ceph-volume tests.functional add raw data device to xenial bluestore Signed-off-by: Alfredo Deza (cherry picked from commit fbcfb522981cd4039cfe617609c5f160f0b05c52) commit 0d082d0dbb43cc418bf0786b1f708968050ae114 Author: Alfredo Deza Date: Tue Nov 14 08:38:23 2017 -0500 ceph-volume tests.functional tox.ini addition of setup playbook for partitions Signed-off-by: Alfredo Deza (cherry picked from commit d074b534643dde37d732b34c36c5089e41848d6d) commit a167556eaacb444c41299cd28f1efd4327da4747 Author: Alfredo Deza Date: Tue Nov 14 08:37:17 2017 -0500 ceph-volume tests.functional add raw data device to centos7 bluestore Signed-off-by: Alfredo Deza (cherry picked from commit 2d416a849ad7b33dde1386da09c81249e6c2299d) commit 12e180a84f1456dfe7ab940255c6c47ee8ebddf0 Author: Alfredo Deza Date: Tue Nov 14 08:36:36 2017 -0500 ceph-volume tests.functional add setup playbook for centos7 bluestore Signed-off-by: Alfredo Deza (cherry picked from commit 161fd373d2f11f64466927e5d615d039822e2ab5) commit f06ea77590336d439235bd9acc6028b854e8c181 Author: Alfredo Deza Date: Tue Nov 14 08:35:59 2017 -0500 ceph-volume tests.functional add setup playbook for centos7 filestore Signed-off-by: Alfredo Deza (cherry picked from commit b3bb147298f8816651f231f77bc91a3a1b50d96f) commit f10bfbeefe06e59629a4e9d904e8035bf060920b Author: Alfredo Deza Date: Tue Nov 14 08:35:42 2017 -0500 ceph-volume tests.functional add raw device to centos7 filestore Signed-off-by: Alfredo Deza (cherry picked from commit c5b8d72570b24dfd0d2f8c1cc8f13812c6c11a69) commit 968006a271dc82f3dcada4e5de93ecb0e23238c6 Author: Alfredo Deza Date: Mon Nov 13 14:29:32 2017 -0500 ceph-volume lvm.common update --data flag to reflect device acceptance Signed-off-by: Alfredo Deza (cherry picked from commit 4df7ea8d83cfa02c8864d043b5d5fb2f5409dc96) commit c8f64af027689a1fb1bb916e6d479519a12b764e Author: Alfredo Deza Date: Mon Nov 13 14:28:32 2017 -0500 ceph-volume lvm.prepare add example of raw device or partition in help menu Signed-off-by: Alfredo Deza (cherry picked from commit 6fb8d566aac0e0fa081e6500852855ad177f784f) commit f930934159edd947024eef1a33e15a30467cb332 Author: Alfredo Deza Date: Mon Nov 13 14:26:44 2017 -0500 ceph-volume lvm.prepare allow partition or raw device as input for --data in filestore Signed-off-by: Alfredo Deza (cherry picked from commit 89236ee3f81e50b9b059c10f6d6e7ce048c06474) commit d19b7403595bef1c3ba73f4a5fb83f5ef2dd3f81 Author: Kefu Chai Date: Wed Nov 8 14:13:54 2017 +0800 qa: do not wait for down/out osd for pg convergence that osd is not invovlved in the PG state changes. Signed-off-by: Kefu Chai (cherry picked from commit 7f549af459b40c5bb51a24a3a443ed7a7116915e) commit 5c9b93dadd8f05b1a146e3a19f7480ae4019815f Merge: cc41c393b3 2f380b32da Author: Yuri Weinstein Date: Tue Nov 14 08:38:35 2017 -0800 Merge pull request #18908 from smithfarm/wip-luminous-fixup qa/suites/fs/basic_functional/clusters: more osds Reviewed-by: Patrick Donnelly Reviewed-by: Abhishek Lekshmanan commit cc41c393b3a76a5f1a18cbda45acc01e2d6fdfc2 Merge: d69f42e1af 806e49aadb Author: Alfredo Deza Date: Tue Nov 14 11:06:00 2017 -0500 Merge pull request #18910 from ceph/backport-18882 luminous: ceph-volume: add functional tests for simple, rearrange lvm tests Reviewed-by: Alfredo Deza commit 806e49aadb0d832a4bf46fcd681d889d93158232 Author: Alfredo Deza Date: Fri Nov 10 13:18:07 2017 -0500 ceph-volume tests.functional update lvm/tox.ini paths after moving it Signed-off-by: Alfredo Deza (cherry picked from commit 7d787849556788961155534039886aedfcdb2a88) commit 724ad01afc0af79fc7f808aae819ff9dc0971b33 Author: Alfredo Deza Date: Fri Nov 10 13:14:55 2017 -0500 ceph-volume tests.functional move top level tox.ini to lvm/tox.ini Signed-off-by: Alfredo Deza (cherry picked from commit 9b14b96d3bf6c8326cfc91e657161ffab3b8c2b4) commit 754c237597cc91af4bfe7199f891a9c2a6938ad4 Author: Alfredo Deza Date: Fri Nov 10 13:12:51 2017 -0500 ceph-volume tests.functional move xenial to lvm/xenial Signed-off-by: Alfredo Deza (cherry picked from commit 00d576d266b63080129e58482bd6fba62534ee97) commit 762c5f1cc10266d5efa62628c0686b3f8d696bff Author: Alfredo Deza Date: Fri Nov 10 13:11:58 2017 -0500 ceph-volume tests.functional lvm move xenial/bluestore to xenial/bluestore/create Signed-off-by: Alfredo Deza (cherry picked from commit d5ec7d1c58141f4593ed9d1e4501c9f3be6d9755) commit 5b1bde338fbb1e689e3e1305e91f5450ef175edd Author: Alfredo Deza Date: Fri Nov 10 13:09:48 2017 -0500 ceph-volume tests.functional lvm move xenial/create to xenial/filestore/create Signed-off-by: Alfredo Deza (cherry picked from commit c504b8781b3a31475280a032cd75c9e1eaa6d8c9) commit 5a00604f27acad8cb368495f841ce76cce383414 Author: Alfredo Deza Date: Fri Nov 10 13:02:05 2017 -0500 ceph-volume tests.functional move centos7 to lvm/centos7 Signed-off-by: Alfredo Deza (cherry picked from commit dc1da3fd0aeafe59a82343fb16fe1fcea66ae3eb) commit 57d64a86d0758d174c1ca0b432a43194cbe6162c Author: Alfredo Deza Date: Fri Nov 10 12:58:27 2017 -0500 ceph-volume tests.functional lvm move bluestore to bluestore/create Signed-off-by: Alfredo Deza (cherry picked from commit aec1a80a3e1e61720f494980711808bfee783a03) commit 94a9f3f43d9c3a2ac08be4025e65c47c3767d2c1 Author: Alfredo Deza Date: Fri Nov 10 12:54:01 2017 -0500 ceph-volume tests.functional lvm move create to filestore/create Signed-off-by: Alfredo Deza (cherry picked from commit d6905f3f620d4c733835e21a275d4127cdcb769b) commit bc134c20a5409bd6f966d205dd7ba850ff28fc91 Author: Alfredo Deza Date: Fri Nov 10 12:42:19 2017 -0500 ceph-volume tests.functional simple xenial bluestore scenario Signed-off-by: Alfredo Deza (cherry picked from commit 8f0fdc975951a2a8af31bbaa3abd2b5d807a0add) commit 004efbfbe4d1eb56190e5e7497a9613776494fae Author: Alfredo Deza Date: Fri Nov 10 12:40:18 2017 -0500 ceph-volume tests.functional simple xenial filestore scenario Signed-off-by: Alfredo Deza (cherry picked from commit 7452e1b6f9b82c2ef62bae19c632e81039638ba5) commit b51744f26f3c6db4d2927a2ac1733f0b80e44010 Author: Alfredo Deza Date: Fri Nov 10 12:30:05 2017 -0500 ceph-volume tests.functional simple tox.ini update for xenial scnearios Signed-off-by: Alfredo Deza (cherry picked from commit 76ed6a8f2ecab7c58def0453ffed9d8a997e0d5b) commit 19983b9ba5ed45c8e6fac967fb702fd586132c35 Author: Alfredo Deza Date: Fri Nov 10 09:12:09 2017 -0500 ceph-volume tests.functional simple tox.ini addition for bluestore Signed-off-by: Alfredo Deza (cherry picked from commit 02154e445b87a8fdcabbd699a2532833764140b5) commit 040e7913b028c54aeb9acd854610f6a4f112465d Author: Alfredo Deza Date: Fri Nov 10 09:11:39 2017 -0500 ceph-volume tests.functional add initial files for simple bluestore activate Signed-off-by: Alfredo Deza (cherry picked from commit 3320f28e799e1d77d1ea69be9ced603bb69cc5fa) commit 23f6c3e779c743bc1302d1ad542201434ff9431f Author: Alfredo Deza Date: Thu Nov 9 16:55:30 2017 -0500 ceph-volume tests.functional add initial files for simple filestore activate Signed-off-by: Alfredo Deza (cherry picked from commit 9d18bae52ad77be0e64ef105c33a019446b6f1cf) commit 8f425b678f6c947a8c83f121be4fa46151fd6664 Author: Alfredo Deza Date: Thu Nov 9 16:07:53 2017 -0500 ceph-volume tests.functional create a specific test.yml for simple activate Signed-off-by: Alfredo Deza (cherry picked from commit 37bde0d3b37a87c456438b0edc0cb9e2dcd39120) commit 6f1fff00f5210840fe143aac7506ce3773600f05 Author: Alfredo Deza Date: Thu Nov 9 15:40:51 2017 -0500 ceph-volume tests.functional create a separate tox.ini environ for `simple` Signed-off-by: Alfredo Deza (cherry picked from commit 89757ad12e6938fe2bf997e0f21a5ffac082e30b) commit d69f42e1afaf8759de1544b3d381671a01cfb55a Merge: a7c8c8101d 90976ef6b9 Author: Andrew Schoen Date: Mon Nov 13 10:55:24 2017 -0600 Merge pull request #18907 from ceph/luminous-wip-bz1498200 luminous: ceph-volume: support GPT and other deployed OSDs Reviewed-by: Andrew Schoen commit 2f380b32da1b6294b6b0f60622b09b6517f9afdd Author: Sage Weil Date: Mon Oct 9 13:44:51 2017 -0500 qa/suites/fs/basic_functional/clusters: more osds Signed-off-by: Sage Weil (cherry picked from commit 998122c324047c96c16d60d7fbb3a21f4dd63c56) commit 90976ef6b9815b68df620aac46f2c04117766f99 Author: Alfredo Deza Date: Thu Nov 9 10:50:43 2017 -0500 doc/ceph-volume include the top level systemd.rst in the toctree Signed-off-by: Alfredo Deza (cherry picked from commit e2f281c115c15e820588b478214c3600ff97470f) commit 9a8a00d3a17f03d2b1b3a92662f7288f7d18fe7a Author: Alfredo Deza Date: Thu Nov 9 06:49:49 2017 -0500 ceph-volume simple.activate add --file in help docstring as an example Signed-off-by: Alfredo Deza (cherry picked from commit ff60592897b63581ebce372b3f6ffa0f10d2fe5a) commit 34194b8d788d91b4f7c969be5f0099a279bc61bf Author: Alfredo Deza Date: Thu Nov 9 06:46:59 2017 -0500 doc/ceph-volume activate has a --file option for json files Signed-off-by: Alfredo Deza (cherry picked from commit 29913951b71d68b3f3f9b9fa076f309c0252da36) commit 582262b57bcc959d1698636000d7a804cab9c611 Author: Alfredo Deza Date: Wed Nov 8 17:23:41 2017 -0500 ceph-volume simple.activate enable the osd when activating Signed-off-by: Alfredo Deza (cherry picked from commit 366fb8c43af54e9c92f4e95d9a0582c1fc378dbc) commit a32e72cdb0efb55d185488fb02f44055f9d3dffe Author: Alfredo Deza Date: Wed Nov 8 15:25:05 2017 -0500 ceph-volume systemd.systemctl always force symlink when masking ceph-disk Signed-off-by: Alfredo Deza (cherry picked from commit cb470622fd4f13fafe9cb2f38fb25fd1fbd8d89c) commit 3af40c35f3cf42be722f9d905cb2d0ba2baf1b1b Author: Alfredo Deza Date: Wed Nov 8 15:21:10 2017 -0500 ceph-volume simple.activate use the contents of the JSON file, fallback to CLI args Signed-off-by: Alfredo Deza (cherry picked from commit c954c913395557bcd89f25b0496c4428e6109494) commit 161e9657d985aa4ceffad2f56e18edd78ca2a81f Author: Alfredo Deza Date: Wed Nov 8 14:50:16 2017 -0500 ceph-volume simple.activate only check for missing id and uuid if file is not passed in Signed-off-by: Alfredo Deza (cherry picked from commit 826c2709dd5ab0fbf6eb419a735d15e718a7712c) commit 585072afc814c6ac87fd64f15c561c51c1e55070 Author: Alfredo Deza Date: Wed Nov 8 14:28:12 2017 -0500 ceph-volume simple.activate allow to pass a file to activate Signed-off-by: Alfredo Deza (cherry picked from commit bd3498f3eabf07637cb936e6c036d1d8d75415b0) commit 01266c193588066425b44fea4c1729692f999692 Author: Alfredo Deza Date: Wed Nov 8 12:49:31 2017 -0500 ceph-volume tests.util add tests for OSDPath validator Signed-off-by: Alfredo Deza (cherry picked from commit d7fc26ab4e2cebe485647510119ea300d47db184) commit 56fdc1c973b41d855563e459d23682990e8940bf Author: Alfredo Deza Date: Wed Nov 8 11:56:36 2017 -0500 ceph-volume systemd.systemctl masking ceph-disk needs to be done by directly symlinking to dev/null Signed-off-by: Alfredo Deza (cherry picked from commit cda47c247b929401a7dce3f1ef70371ea80eeceb) commit 0a0c289de605b6b74b762b2a0a4ed0f3a04de8ad Author: Alfredo Deza Date: Wed Nov 8 09:47:12 2017 -0500 ceph-volume systemd.systemctl masking needs to use systemctl mask, not disable Signed-off-by: Alfredo Deza (cherry picked from commit bad2f62b34243addc854abf691854b886aecdc19) commit 32913844b36285aae40f9786e5cb035bfca22f2d Author: Alfredo Deza Date: Wed Nov 8 09:46:20 2017 -0500 ceph-volume simple.scan if mounted device cannot be detected, raise an error Signed-off-by: Alfredo Deza (cherry picked from commit 6b2fc4c0ba63e4bd51774ea09cc6542738128621) commit fe73229fb123fdb45f4ef743d8584f6748318868 Author: Alfredo Deza Date: Wed Nov 8 09:17:52 2017 -0500 ceph-volume simple.activate remove str formatting from logging Signed-off-by: Alfredo Deza (cherry picked from commit 230faf04a185c0b95c8ad497ddd5501dc70c1409) commit 50c19ef335305215a636595506eda1b5d16f3687 Author: Alfredo Deza Date: Wed Nov 8 09:11:41 2017 -0500 ceph-volume util.arg_validators always convert the path to an absolute path Signed-off-by: Alfredo Deza (cherry picked from commit b5acf4d720b5395ae4ccbf007c94c99610e40089) commit d27c35c972abba2b230e1cff1fa35437500dcdf6 Author: Alfredo Deza Date: Wed Nov 8 09:10:47 2017 -0500 ceph-volume util.arg_validators require superuser privs to check paths Signed-off-by: Alfredo Deza (cherry picked from commit 376dad74159532f901d0044f95330fc02f93bd25) commit 0832ddc52496cd81aa7c2e1e85252b50d2f33044 Author: Alfredo Deza Date: Wed Nov 8 09:09:06 2017 -0500 ceph-volume simple.scan skip directories to scan Signed-off-by: Alfredo Deza (cherry picked from commit 6bc1e9ee0b8d7bc475f218ae3dd8c6c7fe5a983a) commit d6071cb75ba1a702c551d073f032336560654e97 Author: Alfredo Deza Date: Tue Nov 7 15:08:13 2017 -0500 ceph-volume tests.devices.simple add activate CLI unit tests Signed-off-by: Alfredo Deza (cherry picked from commit cde0d517563b537319d06c758b93e421cdabe59d) commit 2bff2e7fdbba37280791d3bc9a363e47aa26ab53 Author: Alfredo Deza Date: Tue Nov 7 15:07:44 2017 -0500 ceph-volume tests.devices add help menu unit test Signed-off-by: Alfredo Deza (cherry picked from commit dcc7c72196a327f9501045f025761b228b5379df) commit ec8624005082913faf89a9b001d4abeb7c27a589 Author: Alfredo Deza Date: Tue Nov 7 13:42:45 2017 -0500 doc/ceph-volume systemd describe the interaction with 'simple' activation Signed-off-by: Alfredo Deza (cherry picked from commit 7071eeaeb5d77c6ab6e7a24f1b630cb0cf8a8e7c) commit aacd22681d1e7f5a604025b2019b195041151b3f Author: Alfredo Deza Date: Tue Nov 7 13:42:11 2017 -0500 doc/ceph-volume index initial description of simple Signed-off-by: Alfredo Deza (cherry picked from commit f9ee51f9feadb6d0f91b76a2b2ba8a24f1523a77) commit e9527e65d5da503107e21dc5b695f287c8529049 Author: Alfredo Deza Date: Tue Nov 7 13:41:12 2017 -0500 doc/ceph-volume lvm add back a system.rst with just the lvm portion of it Signed-off-by: Alfredo Deza (cherry picked from commit f872cd29b93e02b77172f7b382c4746af9ba845d) commit ef027a737989dca83c0e5810935bad2280146356 Author: Alfredo Deza Date: Tue Nov 7 13:40:44 2017 -0500 doc/ceph-volume make systemd.rst generic enough to be related to other sub-commands Signed-off-by: Alfredo Deza (cherry picked from commit cf4e32a0dc3ab4f55fe7857b51afbc63a638ec66) commit a83b746eaeece5896770b40949804316c29f6bc1 Author: Alfredo Deza Date: Tue Nov 7 13:39:52 2017 -0500 doc/ceph-volume move lvm/systemd.rst to top level, making it generic Signed-off-by: Alfredo Deza (cherry picked from commit 0672dd608d4601a936d703ea5ba6edc14a161d8e) commit bc492490e15d3ea91654cd4dabfb3115f835b9ac Author: Alfredo Deza Date: Tue Nov 7 11:52:17 2017 -0500 doc/ceph-volume simple.activate initial take on documenting activation Signed-off-by: Alfredo Deza (cherry picked from commit 80c796bd1c43125f5678710b494b03c486b3fce6) commit 6301c83097729749fc712846954a3c0f89647f07 Author: Alfredo Deza Date: Tue Nov 7 08:35:57 2017 -0500 doc/ceph-volume lvm update systemd anchor to reflect lvm parent Signed-off-by: Alfredo Deza (cherry picked from commit 53420a4756a5e4fa6b0f2aa74d6ad1e2919f1954) commit 5fe4122bcfdc9c8e4f69079e45c907148eccdb7d Author: Alfredo Deza Date: Tue Nov 7 08:35:43 2017 -0500 doc/ceph-volume lvm.activate update systemd anchor to reflect lvm parent Signed-off-by: Alfredo Deza (cherry picked from commit 7fc0e7ebcae06f28f6ed08c0db14075ada34e8a7) commit c98886a99914748d73d0f695808a74850bc9e66e Author: Alfredo Deza Date: Tue Nov 7 07:56:35 2017 -0500 doc/ceph-volume lvm.systemd update anchor to reflect lvm parent Signed-off-by: Alfredo Deza (cherry picked from commit 01a8b7e6a45f6fd85bf53d3cab15acc81d0bae1b) commit 68f89b52d608c3946be9ad92e733fe9472bd35f0 Author: Alfredo Deza Date: Mon Nov 6 15:53:39 2017 -0500 doc/ceph-volume document the scan functionality Signed-off-by: Alfredo Deza (cherry picked from commit b30a3eaf17a20a1273724b2cfcaf0a7b36eb08f6) commit 19d0ca22908fcd3fbe83d39c1b708c5818b34f47 Author: Alfredo Deza Date: Mon Nov 6 14:17:20 2017 -0500 doc/ceph-volume update the index to include the simple sub command and files Signed-off-by: Alfredo Deza (cherry picked from commit d7f85c64e4b409ea95e30ccb5a1388278843c478) commit 61c90bacc562e1e2d65b767dca97b08973832131 Author: Alfredo Deza Date: Mon Nov 6 11:15:28 2017 -0500 ceph-volume tests.simple add tests for trigger parsing from systemd Signed-off-by: Alfredo Deza (cherry picked from commit 0e2620234e82abf85274b3c08b39cdf76430ce48) commit 80ae5b1fb024827cceb0594a60e3361fd4a1e7a3 Author: Alfredo Deza Date: Mon Nov 6 10:36:14 2017 -0500 ceph-volume tests.simple create etc_path scan tests Signed-off-by: Alfredo Deza (cherry picked from commit 90ebc4e1ffa7b3273787d708bb2c05180530b5d1) commit b9ad514b45e0233a17834e19a232bca4467a8140 Author: Alfredo Deza Date: Mon Nov 6 10:06:02 2017 -0500 ceph-volume tests.simple add checks for get_contents in scan Signed-off-by: Alfredo Deza (cherry picked from commit 647bfe0883d2bb05b4ff7f99894075df3e73d2d5) commit c9308af91a322f92ab7d1f45a423b2a94f64c429 Author: Alfredo Deza Date: Mon Nov 6 09:43:58 2017 -0500 ceph-volume tests.util add binary verification tests Signed-off-by: Alfredo Deza (cherry picked from commit 827aa57557561355dd4068e26801b5f1ed62f295) commit c94f52dffe0ea4b48c9fb316e8c53fd06a81b870 Author: Alfredo Deza Date: Mon Nov 6 09:42:53 2017 -0500 ceph-volume tests create a fixture for temporary files Signed-off-by: Alfredo Deza (cherry picked from commit 23da3445718c01f71e2a5748fb4fc1c5fdc85ca5) commit bbb25df806609083efd377befb44d6d3bef3338b Author: Alfredo Deza Date: Fri Nov 3 14:36:38 2017 -0400 ceph-volume systemd.systemctl helpers to mask ceph-disk Signed-off-by: Alfredo Deza (cherry picked from commit 3da23fa4caea8c9dc91da3eee5b5acc1860bf242) commit c971321430fd1a4de7c6fa2687ecf35a5b3ec71c Author: Alfredo Deza Date: Fri Nov 3 14:36:13 2017 -0400 ceph-volume devices include simple at the module import level Signed-off-by: Alfredo Deza (cherry picked from commit 027a09b3ea6bac416ff632a69a53393badd51708) commit 1615d947db50ac37e4ef554acc6debe8fd870962 Author: Alfredo Deza Date: Fri Nov 3 14:35:45 2017 -0400 ceph-volume main add simple to the sub-commands Signed-off-by: Alfredo Deza (cherry picked from commit 94579c77ffe4d5ba2d457ad3981a2b9ed232a90f) commit abb77a8d9824eef3bd9c75d17f8df101440c25d1 Author: Alfredo Deza Date: Fri Nov 3 14:35:03 2017 -0400 ceph-volume util add an OSDPath argument validator Signed-off-by: Alfredo Deza (cherry picked from commit a826fdf4a2ed9ff704fb823d05d3cb706ba89e38) commit ff2a1833da385d6c0c9087e98cb2b93925b06cab Author: Alfredo Deza Date: Fri Nov 3 14:34:39 2017 -0400 ceph-volume simple.system add a helper to check if a file is a binary or not Signed-off-by: Alfredo Deza (cherry picked from commit 7bf8676045b4f966b60cdadf4d5af1c89fca69ae) commit 84d2a15ce9f1a045ed67f79fa7d1dc99dc1e4b8f Author: Alfredo Deza Date: Fri Nov 3 14:34:06 2017 -0400 ceph-volume util.system add a context manager for temporary mounting/unmounting Signed-off-by: Alfredo Deza (cherry picked from commit 6394cdd41943e3cda6867ecc59ab835cb2151707) commit 3d8ff08c2d46017749e7ecaa4e4d49dda4d55d74 Author: Alfredo Deza Date: Fri Nov 3 14:32:42 2017 -0400 ceph-volume simple.scan initial take on directory/device scanning Signed-off-by: Alfredo Deza (cherry picked from commit 0e54b9be621cc0969a3bbc972d1870080ccdae77) commit d701569a6486d7ccccf14b5eadf702ad07db4821 Author: Alfredo Deza Date: Fri Nov 3 14:27:23 2017 -0400 ceph-volume simple.trigger enable systemd triggering for simple setups Signed-off-by: Alfredo Deza (cherry picked from commit cbc0c0f775e96a5b0ab5d5bc141acebe4cc48373) commit 76fe659a19a8a450b4c61f6d2b56e03dc46746ae Author: Alfredo Deza Date: Fri Nov 3 14:26:13 2017 -0400 ceph-volume simple.activate initial take on activation Signed-off-by: Alfredo Deza (cherry picked from commit 5a97374605408d18e9be29bcc051ae7b913d1675) commit 206c7551435983b4cbdaece61b737e4a1f265dbd Author: Alfredo Deza Date: Fri Nov 3 14:23:47 2017 -0400 ceph-volume simple add __init__ for the module Signed-off-by: Alfredo Deza (cherry picked from commit 285e5bdd28a42d98418ca90ad0cdec6bd87849db) commit 8a5edb9a2acc3f0b5477f09edae4f97f5ec7f5b5 Author: Alfredo Deza Date: Fri Nov 3 14:23:29 2017 -0400 ceph-volume simple create the initial, main module Signed-off-by: Alfredo Deza (cherry picked from commit f050502425a1993ae8edfdb5785b24129cffeeba) commit 0fe463b5e1fe62e03b498c32aa708be8c10eb0b8 Author: Sage Weil Date: Thu Nov 9 20:10:59 2017 -0600 os/filestore: disable rocksdb compression Experience working with customer escalations suggests that disabling compression improves performance, and the storage overhead is generally not a concern for the metadata and omap data we are storing. Signed-off-by: Sage Weil (cherry picked from commit b878ead071b328e5fe7309a2368383e67679e9f7) # Conflicts: # src/common/options.cc commit a7c8c8101d4b78b4d6e437620b2c1a38cd752c3f Merge: 1071fdcf73 f56de67980 Author: Abhishek L Date: Thu Nov 9 18:08:27 2017 +0100 Merge pull request #18801 from linuxbox2/luminous-rgw-readdir2 rgw_file: implement variant offset readdir processing -- pullup 2 of 2 Reviewed-By: Abhishek Lekshmanan commit 1071fdcf73faa387d0df18489ab7b0359a0c0afb Merge: 4db48ca836 8cacd2a568 Author: Kefu Chai Date: Thu Nov 9 22:36:27 2017 +0800 Merge pull request #18734 from shinobu-x/wip-21648-luminous luminous: mgr[zabbix] float division by zero Reviewed-by: Wido den Hollander commit 4db48ca8369d1dae021131c93a1c8484512aa99b Merge: 9cd31820f2 d858570755 Author: Kefu Chai Date: Thu Nov 9 22:35:44 2017 +0800 Merge pull request #18851 from tchaikov/wip-18759-pr-luminous mon/LogMonitor: "log last" should return up to n entries Reviewed-by: John Spray commit 9cd31820f22d746e680f4ce3b0c780157d24194f Merge: 60ff142c8e 8a7f983f33 Author: Kefu Chai Date: Thu Nov 9 22:27:47 2017 +0800 Merge pull request #18741 from shinobu-x/wip-22019-luminous luminous: tests: "ceph osd create" is not idempotent Reviewed-by: Kefu Chai commit d858570755d88e9a3708952827a99cc24f09f9ab Author: Kefu Chai Date: Mon Nov 6 16:01:23 2017 +0800 mon/LogMonitor: "log last" should return up to n entries limit the # of returned entries to "num", and backoff the start iterator by one if it reaches the rend(). Signed-off-by: Kefu Chai (cherry picked from commit de14103f5201da69b360d9b01e249896d59376b3) commit f56de6798077399f472c8d2df25fb1c3ccaf8090 Author: Matt Benjamin Date: Sun Oct 15 21:48:19 2017 -0400 rgw_file: implement variant offset readdir processing Introduce new rgw_readdir2(...), which in which continues from an arbitrary dirent name, which presumably has been seen in a prior partial enumeration. Add single-file unit test for READDIR cases, librgw_file_marker.cc. Signed-off-by: Matt Benjamin (cherry picked from commit a2c34c597371809bd8fd40ee1fc6b8e6a5145609) Signed-off-by: Matt Benjamin commit 60ff142c8e8ed8fb294f7d13a5461f070aa1ff4f Merge: f1776bc788 7efa91c7ce Author: Abhishek L Date: Thu Nov 9 10:10:53 2017 +0100 Merge pull request #18800 from linuxbox2/luminous-rgw_file-fsid-mount Luminous rgw file fsid mount -- pullup 1 of 2 Reviewed-By: Abhishek Lekshmanan commit f1776bc7887d2245e632d5a89189294a823ff3b2 Merge: 2d3c14e5b8 f7dd3be6b7 Author: Abhishek L Date: Thu Nov 9 10:04:08 2017 +0100 Merge pull request #18767 from smithfarm/wip-22020-luminous luminous: multisite: race between sync of bucket and bucket instance metadata Reviewed-By: Casey Bodley commit 2d3c14e5b88c773548af61312438a69dcb454726 Merge: 2bad0ac00c 36e214c67b Author: Abhishek L Date: Thu Nov 9 09:56:16 2017 +0100 Merge pull request #18794 from theanalyst/wip-18709 luminous: rgw: Stale bucket index entry remains after object deletion Reviewed-By: Casey Bodley commit 2bad0ac00c898d50c9c036400b4fba09918fb331 Merge: 9787509155 f81e6dc8ab Author: Kefu Chai Date: Wed Nov 8 18:19:27 2017 +0800 Merge pull request #18726 from shinobu-x/wip-luminous-22035 luminous: Spurious ceph-mgr failovers during mon elections Reviewed-by: John Spray commit 9787509155df02bd9df21f389968f77d3de2d988 Merge: 0c3e9e6aab 7e08cdf539 Author: Kefu Chai Date: Wed Nov 8 18:18:39 2017 +0800 Merge pull request #18738 from shinobu-x/wip-21547-luminous luminous: ceph-mgr gets process called "exe" after respawn Reviewed-by: John Spray commit 0c3e9e6aab5cfbd3d8e998b148e19bab30f3fa0e Merge: 628e76fe2a ed4a692254 Author: Kefu Chai Date: Wed Nov 8 18:17:54 2017 +0800 Merge pull request #18727 from shinobu-x/wip-luminous-22034 luminous: key mismatch for mgr after upgrade from jewel to luminous(dev) Reviewed-by: John Spray commit 628e76fe2a939aae4996f43e9e9c9f7d1269e0bb Merge: ef5264afc9 be5363a771 Author: Kefu Chai Date: Wed Nov 8 14:44:41 2017 +0800 Merge pull request #18723 from shinobu-x/wip-luminous-22023 luminous: osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL commit ef5264afc992d78c1b34c8a96a43216191fa5406 Merge: a0a2d27c1e 31c56c6e6b Author: Kefu Chai Date: Wed Nov 8 14:38:10 2017 +0800 Merge pull request #18737 from shinobu-x/wip-21549-luminous luminous: the dashboard uses absolute links for filesystems and clients Reviewed-by: John Spray commit a0a2d27c1e40280ae214696dbf5c25567c28ba38 Merge: 947cae47db 76ab4eba99 Author: Kefu Chai Date: Wed Nov 8 14:37:19 2017 +0800 Merge pull request #18736 from shinobu-x/wip-21638-luminous luminous: dashboard OSD list has servers and osds in arbitrary order Reviewed-by: John Spray commit 947cae47db9f5dc6d77a85834515670e20b5faf8 Merge: 2e11e6662a e14de28f9e Author: Kefu Chai Date: Wed Nov 8 14:36:40 2017 +0800 Merge pull request #18728 from shinobu-x/wip-luminous-22032 luminous: dashboard barfs on nulls where it expects numbers Reviewed-by: John Spray commit 2e11e6662ae167276ece7a96783f02a24fc1f89f Merge: 31afe858a9 cd18f84304 Author: Kefu Chai Date: Wed Nov 8 14:34:35 2017 +0800 Merge pull request #18699 from shinobu-x/wip-luminous-21538 luminous: upmap does not respect osd reweights Reviewed-by: Sage Weil commit 31afe858a96f1d5c67124b92f19cf4542df7e647 Merge: d9df23cf3a 388b5016d9 Author: Kefu Chai Date: Wed Nov 8 14:30:42 2017 +0800 Merge pull request #18719 from tchaikov/wip-pr-18373-luminous luminous: osd,mon: add max-pg-per-osd limit Reviewed-by: Sage Weil commit 388b5016d981c9dd98b6505b2d75b0220cfee772 Author: Kefu Chai Date: Wed Oct 18 16:07:50 2017 +0800 doc: fix the desc of "osd max pg per osd hard ratio" Signed-off-by: Kefu Chai (cherry picked from commit d940f6220346ad21dfc4f351c57389e8430c0257) commit 04852a400483b6459985be80a7773ef153c04b1e Author: Kefu Chai Date: Tue Oct 17 12:17:09 2017 +0800 mon: change "mon_pg_warn_min_per_osd" to uint64_t * this silences a warning from -Wsign-compare. * also switch all its user to the new-style option Signed-off-by: Kefu Chai (cherry picked from commit c3cd0c13a1585388df0f17ab53c170e39c711615) commit d9df23cf3aa002315e15db46d195be7d5657f9d0 Merge: 567c29f964 53096755c0 Author: Sage Weil Date: Tue Nov 7 22:27:15 2017 -0600 Merge pull request #18702 from liewegas/wip-pr-18670-luminous luminous: qa/tasks/thrashosds: set min_in default to 4 commit 567c29f96461901bd86d4c2d4f84f7b84342837f Merge: 77c2b0de60 8cbb2eb937 Author: Abhishek L Date: Tue Nov 7 22:28:58 2017 +0100 Merge pull request #18596 from smithfarm/wip-21955-luminous luminous: tests: add EC data pool to testing Reviewed-By: Patrick Donelly commit 7efa91c7ced61f0fde81adfc0540a36fafafb10d Author: Gui Hecheng Date: Sat Jul 22 13:12:06 2017 +0800 rgw_file: introduce rgw_mount2 with a bucket name parameter This explicitly allow mount a specified bucket in librgw. Originally, mounting a bucket is implemented like a sub-directory mount in nfs-ganesha FSAL_RGW with lookup_path. With this change, we allow root_fh to points exactly to the root of a mounted fs instance, which is a bucket or "/" rather than always let root_fh points to "/". Signed-off-by: Gui Hecheng (cherry picked from commit 860716e17e840df11a7e1d8b6ca0c1ee694b038b) commit b53c3948a5e48623bb437b2d36718ce3a799aa54 Author: Gui Hecheng Date: Fri May 26 17:45:29 2017 +0800 rgw_file: new fsid for fs instance We use an in-memory fs_inst_counter for fsid currently, but it inevitably cause nfs automount problem across a nfs-ganesha crash. So here we relate the fsid to the s3 uid with a hash. With this, fsid remains consistent across all fs instances and across crashes. We should be able to upgrade from old fsid to new fsid with update_fh(). Signed-off-by: Gui Hecheng (cherry picked from commit 178b547e83a2c0eeb024bbe4ca2ff8d8e7d50572) commit 77c2b0de6042707b42d6e717e90a525fda8ead76 Merge: 14e0530f50 555746a09e Author: Abhishek L Date: Tue Nov 7 22:07:45 2017 +0100 Merge pull request #18628 from batrick/i21953 luminous: mds: sanitize mdsmap of removed pools Reviewed-By: Sage Weil commit 14e0530f50cd5fba46da914cdd09489f13adb2ec Merge: bc9f508477 c2237c7c6d Author: Abhishek L Date: Tue Nov 7 18:33:21 2017 +0100 Merge pull request #18626 from dillaman/wip-21968 luminous: cls/journal: fixed possible infinite loop in expire_tags Reviewed-By: Mykola Golub commit bc9f5084778607c95dcbc93ea60ce296e8e8ee88 Merge: 6bc121ddee 1aeec0c032 Author: Abhishek L Date: Tue Nov 7 18:32:20 2017 +0100 Merge pull request #18688 from liewegas/wip-balancer-luminous luminous: mgr/balancer: backport crush-compat work from master Reviewed-By: John Spray commit 36e214c67b6982cc7a85e08dedc83ea61e56d6a4 Author: J. Eric Ivancich Date: Fri Nov 3 09:15:13 2017 -0400 rgw: fix BZ 1500904, Stale bucket index entry remains after object deletion We have a race condition: 1. RGW client #1: requests an object be deleted. 2. RGW client #1: sends a prepare op to bucket index OSD #1. 3. OSD #1: prepares the op, adding pending ops to the bucket dir entry 4. RGW client #2: sends a list bucket to OSD #1 5. RGW client #2: sees that there are pending operations on bucket dir entry, and calls check_disk_state 6. RGW client #2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (#1) 7. RGW client #1: sends a delete object to object OSD (#2) 8. OSD #2: deletes the object 9. RGW client #2: sends a complete op to bucket index OSD (#1) 10. OSD #1: completes the op 11. OSD #1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby **RECREATING** it Solution implemented: At step #5 the object's dir entry exists. If we get to beginning of step #11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it. Signed-off-by: J. Eric Ivancich (cherry picked from commit b33f529e79b74314a2030231e1308ee225717743) commit 6bc121ddeec4b12c91ae5ac982ad113bda14d2bf Merge: ee27efad8b f26f3dc842 Author: Abhishek L Date: Tue Nov 7 18:21:41 2017 +0100 Merge pull request #18569 from kmroz/wip-21939-luminous luminous: list bucket which enable versioning get wrong result when user marker Reviewed-By: Casey Bodley commit ee27efad8ba4d751828dd73012de12ab16c2f78d Merge: 601249c6aa a340f1acab Author: Abhishek L Date: Tue Nov 7 18:21:03 2017 +0100 Merge pull request #18591 from cbodley/wip-21938 luminous: rgw: fix error handling in ListBucketIndexesCR (approved in irc & standups) Reviewed-By: Abhishek Lekshmanan commit 601249c6aa154d0f9e53332480ac45b5730fb503 Merge: d2226d5adb 15a52ba832 Author: Abhishek L Date: Tue Nov 7 18:17:57 2017 +0100 Merge pull request #18539 from linuxbox2/luminous-baixuyeyu-refcnt luminous: baixuyeyu refcnt Reviewed-By: Casey Bodley commit d2226d5adbb4752e0180dfa3406e188bd3b37d6e Merge: 925c68e9e9 902d467a0e Author: Abhishek L Date: Tue Nov 7 18:15:38 2017 +0100 Merge pull request #18516 from ceph/wip-bp-pr18193-luminous qa/rgw: ignore errors from 'pool application enable' Reviewed-By: Nathan Cutler Reviewed-By: Casey Bodley commit 925c68e9e9538a97f98bef1e5a884d31811aa229 Merge: 2f35da4e60 add8c3db19 Author: Abhishek L Date: Tue Nov 7 18:14:21 2017 +0100 Merge pull request #18599 from linuxbox2/lum-rgwfile-21940 rgw_file: set s->obj_size from bytes_written Reviewed-By: Casey Bodley commit 2f35da4e60c2b5fd2f45348587a5b175868d04d7 Merge: b2961db15d 26c523c3be Author: Abhishek L Date: Tue Nov 7 13:07:09 2017 +0100 Merge pull request #18566 from kmroz/wip-21914-luminous luminous: [rbd-mirror] peer cluster connections should filter out command line optionals Reviewed-By: Jason Dillaman commit 8e3227ad51fa4909a39eb0d05798b48200a03fc5 Author: Kefu Chai Date: Tue Oct 17 12:10:55 2017 +0800 osd,mon: change "mon_max_pg_per_osd" to uint64_t as it should never be a negative number. if this option is disabled, it's 0. Signed-off-by: Kefu Chai (cherry picked from commit 0d68197376b6cf48c6679d77ec9f28f279f3da68) commit ae7823187186310bac117fac437d991398cffd5e Author: Kefu Chai Date: Tue Sep 26 15:54:14 2017 +0800 osd: add max-pg-per-osd limit osd will refused to create new pgs, until its pg number is lower than the max-pg-per-osd upper bound setting. Signed-off-by: Kefu Chai (cherry picked from commit 4c7df944c7f28232873ba681eedce72cdb062ea5) commit 1db44c5bef752b3a6ce59ea21548b54b77b587d2 Author: Kefu Chai Date: Sun Oct 15 20:46:28 2017 +0800 qa/tasks/ceph: add "create_rbd_pool" option True by default, but we can opt not to create the "rbd" pool. Signed-off-by: Kefu Chai (cherry picked from commit ac1e13ce63a6f32ae9ab0db8196c8cc8100d94ed) commit ed34b00566f9b1b65c09989af44ad24cdb27fbdc Author: Kefu Chai Date: Thu Oct 12 20:07:46 2017 +0800 osd: print osd as "osd.$osdid" in log message easier to read this way Signed-off-by: Kefu Chai (cherry picked from commit c6b71ebf6b7b7c54560f331d5f8e4e24f5a60633) commit df611786d2882e346075f5a32aef46bea3c36d47 Author: Kefu Chai Date: Tue Sep 26 15:50:08 2017 +0800 mon/OSDMonitor: bypass checks if pg_temp.forced originally, monitor ignores the pg_temp messages sent from replica osds, but to re-trigger peering if replica osd's pg number drops down below the setting, we need to bypass the checks. Signed-off-by: Kefu Chai (cherry picked from commit da6f7a810055a0dc9e364aca8e6efbfbc32881b3) commit aefc9315ed6ebbf209d4496b0fb33c72dc107961 Author: Kefu Chai Date: Tue Sep 26 15:49:11 2017 +0800 messages/MOSDPGTemp: add `forced` field the monitor is supposed to bypass some checks if `forced` is true. Signed-off-by: Kefu Chai (cherry picked from commit 6335bf791003e9e33b2f738ab3da06f3c626537a) commit b2961db15dd7528019df7990f8f7690ae3a95aa9 Merge: a5899a57e4 4cac43679a Author: Kefu Chai Date: Tue Nov 7 17:48:34 2017 +0800 Merge pull request #18775 from tchaikov/wip-upgrade-luminous-x qa: add 1-ceph-install to upgrade:luminous-x Reviewed-by: Yuri Weinstein Reviewed-by: Abhishek Lekshmanan commit 4cac43679a7688cb4765728e1a75e812d0df22de Author: Yuri Weinstein Date: Tue Aug 29 09:50:51 2017 -0700 qa: add 1-ceph-install to upgrade:luminous-x Signed-off-by: Yuri Weinstein (cherry picked from commit aff342627cf77e2faae6904e626d9fb2c897f155) commit 1aeec0c0326678bcdba7ee0a6a1204f460e3ddf7 Author: Sage Weil Date: Mon Nov 6 21:17:22 2017 -0600 qa/suites/rados/thrash/d-*/*balancer*: debug osd in mgr Signed-off-by: Sage Weil commit f7dd3be6b7804035b47d2e6fb94463f0bc772bbe Author: Casey Bodley Date: Tue Oct 31 16:56:01 2017 -0400 rgw: remove placement_rule from cls_user_bucket_entry placement_rule is no longer needed in cls_user_bucket_entry, because the only time that it's needed, we can read it from the bucket instance in RGWRados::update_containers_stats() Signed-off-by: Casey Bodley (cherry picked from commit 8e62e3526643da67f5af7daa687120feed469785) commit 17cc13b87b5378bf03e2b0b52c274772ebaf08ef Author: Casey Bodley Date: Tue Oct 31 16:29:31 2017 -0400 rgw: remove placement_rule from rgw_link_bucket() with the fallback in RGWRados::update_containers_stats(), we no longer need to pass placement_rule into the cls_user_bucket_entry this removes the dependency between bucket entrypoint metadata and its bucket instance metadata during multisite sync Fixes: http://tracker.ceph.com/issues/21990 Signed-off-by: Casey Bodley (cherry picked from commit dd5d278a4f9b080234cbb77e448208803ec0cf93) commit 338248a8b376074c0f43b626e9f69fc59ee26487 Author: Casey Bodley Date: Tue Oct 31 16:26:54 2017 -0400 rgw: take placement_rule from bucket info in update_containers_stats in case the cls_user_bucket_entry doesn't contain a placement_rule, take it from the bucket instance info Signed-off-by: Casey Bodley (cherry picked from commit 531195d80d5bac774d58b210307eb19b57d26345) commit 975229c3674ce99704aef399ecfb770cc8bcf1da Author: Chang Liu Date: Tue Oct 3 00:12:43 2017 +0800 tool: ceph-kvstore-tool doesn't umount BlueStore properly Fixes: http://tracker.ceph.com/issues/21625 Signed-off-by: Chang Liu (cherry picked from commit 81e4560781c4e5a5a033bfd4a4fe4094fc27d964) commit 4d4aee14a46ed36cdb40c0ff67193c8650b106da Author: Chang Liu Date: Tue Oct 3 00:01:43 2017 +0800 os/bluestore: make BlueStore opened by start_kv_only umountable ceph-kvstore-tool use start_kv_only to debug the kvstore. we will get a crash when we try to umount bluestore in kvstore-tool. Fixes: http://tracker.ceph.com/issues/21624 Signed-off-by: Chang Liu (cherry picked from commit 2754c5acad442e50d97c8daf648cfcadd38222a6) commit 3f7eadb79f4bb0a2fe25b229c24b652e7dcbaf00 Author: Sage Weil Date: Sat Nov 4 09:32:51 2017 -0500 qa/suites/rados/thrash: combine balancer with require-luminous We can only do the balancer stuff if we've done the mkfs-time require luminous because otherwise the mgr commands aren't proxied via the mon. Only do balancer for thrash tests; not for basic, monthrash, verify. Signed-off-by: Sage Weil commit 8a7f983f3322b62cb18421c088e652341125eb1f Author: Kefu Chai Date: Wed Nov 1 10:45:09 2017 +0800 qa: stop testing deprecated "ceph osd create" "ceph osd create" is not idempotent, and is considered deprecated. Fixes: http://tracker.ceph.com/issues/21993 Signed-off-by: Kefu Chai (cherry picked from commit 371d3f40911917704f4992c48236b4c60905bf4e) commit 7e08cdf53992570d27b47d0028c698b78908ba83 Author: John Spray Date: Fri Sep 15 12:23:39 2017 -0400 mgr: set explicit thread name This gets used as our process name in some situations when respawning. This is the same as what commit 4f177bb6b did for the MDS. Fixes: http://tracker.ceph.com/issues/21404 Signed-off-by: John Spray (cherry picked from commit 44dce0a5664639b2ae0949cc434f1be1176d872d) commit 31c56c6e6b50b44fef56852e58ab97520ee8614a Author: Nick Erdmann Date: Tue Aug 22 16:55:11 2017 +0000 pybind/mgr/dashboard: updated favicon (old one was 404) Signed-off-by: Nick Erdmann (cherry picked from commit 7a54520f5c3d4db27fe7e2a12760dcd838ba89aa) commit d822c15af9b838dc2eea08b5f13f1dba785d776e Author: Nick Erdmann Date: Tue Aug 22 08:08:45 2017 +0000 pybind/mgr/dashboard: clean up Signed-off-by: Nick Erdmann (cherry picked from commit f512ac3d7712850f274cd01bb589aa9fbc8c8bef) commit 1f47cc0b53f7e20e156f74b28502c7705ddb4700 Author: Nick Erdmann Date: Mon Aug 21 17:21:10 2017 +0000 pybind/mgr/dashboard: add url_prefix This adds a configuration variable url_prefix to the dashboard that that is prepended to all URLs so you can access the dashboard at http://$IP:$PORT/$PREFIX/. This is necessary if you wish to use a reverse http proxy that forwards to the dashboard under a sub-path. Fixes: http://tracker.ceph.com/issues/20568 Signed-off-by: Nick Erdmann (cherry picked from commit 6d285fff0df598d66f46d38bf0a6a2cc9c0dc62f) commit 76ab4eba99785edf01c8ff9464d997b2ec691b31 Author: John Spray Date: Wed Sep 27 15:13:48 2017 +0100 mgr/dashboard: sort servers and OSDs in OSD list Fixes: http://tracker.ceph.com/issues/21572 Signed-off-by: John Spray (cherry picked from commit f409099fa25b73fe580ee4662aff51636f118ed2) commit 8cacd2a568fca023a51f3be6e20f90b4ec08c82b Author: John Spray Date: Sat Sep 23 15:22:34 2017 +0100 mgr/zabbix: log exceptions with backtrace Logging just the exception object gave a sometimes-enigmatic single line. Let's use the logger exception method so that we get a backtrace. Signed-off-by: John Spray (cherry picked from commit 316dcc8f0b42b85503632d5472c45c9828882f2c) commit 06706bea9339e26d1a0ab395e9d42d391ee8c03c Author: John Spray Date: Sat Sep 23 15:18:18 2017 +0100 mgr/zabbix: fix div by zero Fixes: http://tracker.ceph.com/issues/21518 Signed-off-by: John Spray (cherry picked from commit 9c02738b4ea0303f5b3cbfc4748d6791007be834) commit a5899a57e4f077f4061fc2aa33a61377de7ac025 Merge: 52f66ef5ba 35de92b259 Author: Sage Weil Date: Sat Nov 4 10:52:27 2017 -0500 Merge pull request #18673 from dzafman/wip-21833 osd: build_past_intervals_parallel: Ignore new partially created PGs commit e14de28f9e524575c633dc98c6ebb9a234c8d1c5 Author: John Spray Date: Wed Sep 27 14:33:58 2017 +0100 mgr/dashboard: handle null in format_number Fixes: http://tracker.ceph.com/issues/21570 Signed-off-by: John Spray (cherry picked from commit f43859d4dca696ba4cac224e0c7e87e4b2aad40f) commit ed4a692254e71ce7b30566dfd6d16e8a070b3461 Author: John Spray Date: Thu Oct 19 09:28:18 2017 -0400 mon: don't blow away bootstrap-mgr on upgrades Fixes: http://tracker.ceph.com/issues/20950 Signed-off-by: John Spray (cherry picked from commit 57229ea2a4369518c7a16b7a09b045b7896f5a70) commit f81e6dc8abe3535a54cbf683992c95c4eaf4e186 Author: John Spray Date: Thu Oct 12 11:57:50 2017 +0100 mon: handle monitor lag when killing mgrs Fixes: http://tracker.ceph.com/issues/20629 Signed-off-by: John Spray (cherry picked from commit 3c3776b30a2da3f5d2ece15c57713c0ce191c778) commit be5363a771172d3260049b0012519b485e4d728e Author: Yao Zongyou Date: Tue Oct 31 13:23:49 2017 +0800 osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL These two counters should be reported to mgr because the dashboard plugin needing them to display osd detailed information. Fixes: http://tracker.ceph.com/issues/21981 Signed-off-by: Yao Zongyou (cherry picked from commit 40bb3c5f05b6f2a917a7e400f88677f41a9629a3) commit 77766609d808a974aafdff359dadc46af5bd26f4 Author: Sage Weil Date: Thu Nov 2 16:18:03 2017 -0500 pybind/mgr/mgr_module: fix calc_pg_upmaps This was fixed in the object cleanup. Signed-off-by: Sage Weil (cherry picked from commit ed442164d52c803916ca43a5ed76047089d3da66) commit 56c27e6be813e6a8b972733468eb099596ff7381 Author: Sage Weil Date: Thu Nov 2 16:11:26 2017 -0500 mgr/balancer: enable module by default It will still be "off". Signed-off-by: Sage Weil (cherry picked from commit 26710f0a9b176289237a52e5fa5894342ad163dc) commit 53096755c02d953867d0fff3e08bc3a62d872ec6 Author: Sage Weil Date: Wed Nov 1 08:31:31 2017 -0500 qa/tasks/thrashosds: set min_in default to 4 We have EC tests with k=2,m=2, so we need a min of 4. Fixes: http://tracker.ceph.com/issues/21997 Signed-off-by: Sage Weil (cherry picked from commit d21809b14ea58dc1f44df844e407ebab5a315062) commit 52f66ef5ba291fccc265fd7a34a4eae201667153 Merge: 1b00bdfeef e58656005d Author: Abhishek L Date: Fri Nov 3 10:26:44 2017 +0100 Merge pull request #18446 from theanalyst/wip-21854-luminous luminous: rgw_file: explicit NFSv3 open() emulation Reviewed-By: Abhishek Lekshmanan Reviewed-By: Matt Benjamin commit cd18f84304124e1341516194fb73c8bde48db445 Author: Theofilos Mouratidis Date: Mon Sep 25 16:17:47 2017 +0200 osd: upmap should respect osd reweights Modify OSDMap::calc_pg_upmaps to take the osd reweight into account when computing the size of each OSD. Signed-off-by: Theofilos Mouratidis Fixes: http://tracker.ceph.com/issues/21538 (cherry picked from commit b8561cefa705d31954d47723125376a3c087f4f4) commit 03a11694feee0d78dcc1f296a77200635a651707 Author: Theofilos Mouratidis Date: Mon Sep 25 16:15:45 2017 +0200 test/cli/osdmaptool: test upmap with an out osd Test upmap with one osd marked out. Signed-off-by: Theofilos Mouratidis (cherry picked from commit ff9e024a59d87eda6cbf3be9464ee7ae6f1140d7) commit 2c7972d3a4928cec039e8637048c26e94c6278d2 Author: Theofilos Mouratidis Date: Mon Sep 25 16:14:09 2017 +0200 osdmaptool: add --mark-out option Add a --mark-down option which sets the reweight for a given osd id to 0.0. Signed-off-by: Theofilos Mouratidis (cherry picked from commit bd75a78187143f750f90fb937984eaa6333159d3) commit 1b00bdfeefd7b724f14634383ae4baaf7a42c996 Merge: c6052f2706 92d0545781 Author: Abhishek L Date: Thu Nov 2 21:23:50 2017 +0100 Merge pull request #18439 from theanalyst/wip-21655-luminous luminous: expose --sync-stats via admin api Reviewed-By: Abhishek Lekshmanan Reviewed-By: Casey Bodley commit 282e21bd07c3dfb5d18f338abd3ced4f4eeaa247 Author: Sage Weil Date: Fri Oct 27 12:53:52 2017 -0500 mgr/balancer: simplify pool_info tracking Signed-off-by: Sage Weil (cherry picked from commit a59a1df85382d559362c51bb45284b61caaca1d4) commit e2346f539f477a4b7a4685828876eb89c84ad00d Author: Sage Weil Date: Wed Oct 25 22:47:02 2017 -0500 mgr/balancer: less verbose on 'eval' by default; add 'eval-verbose' The verbose output is helpful for debugging and understanding what is being measured, but most of the time all that you care about is the score. Signed-off-by: Sage Weil (cherry picked from commit 2c300bc8d7189b464f59c93deb5eb56915df62b8) commit 64d75fa26d3ceb6ccdbee3b31b3705c4db2fc59f Author: Sage Weil Date: Sat Oct 21 10:46:27 2017 -0500 mgr/balancer: fix pg vs object terminology Signed-off-by: Sage Weil (cherry picked from commit 6513e8b0fc2ebcbbbda937ec6d533ea084ef278f) commit 6082f53391a46076bda1b5d56685d928acd2cd4b Author: Sage Weil Date: Tue Oct 24 17:12:54 2017 -0500 mgr/balancer: restrict to time of day Signed-off-by: Sage Weil (cherry picked from commit 3f0b50b50aeb34ed39767289b8475ec5b877cc27) commit 91b84cb24466f84ce3e189e08f8cba1e2900a4cc Author: Sage Weil Date: Thu Oct 5 17:31:06 2017 -0500 mgr/module: adjust osd_weight min step to .005 That should be ~1 PG or less on average; no real sense it taking a step smaller than that! Signed-off-by: Sage Weil (cherry picked from commit 297e2d65abcebe45c6be69a768cf3ea98e1b354d) commit acbf539d326403b6249ed0e2b0ca5ffc4a52152d Author: Sage Weil Date: Thu Oct 5 17:26:16 2017 -0500 mgr/balancer: if score regresses, take a few more steps To avoid getting stuck in a local minima, take a few additional steps if our score drops and see if it gets better. Signed-off-by: Sage Weil (cherry picked from commit 70b503495a0ceb7d1aa4f5fce4403a8eff9a03b5) commit b9cc962adfd95304cbd56833bb84b827e5eed4d4 Author: Sage Weil Date: Thu Oct 5 17:25:19 2017 -0500 mgr/balancer: allow 5% misplaced Signed-off-by: Sage Weil (cherry picked from commit 1ad3101818d1855c8eea8386d7682fab2fe9c8b8) commit 452962767b4f899c8c2589d01a6b8acc3cac9a69 Author: Sage Weil Date: Thu Oct 5 17:25:06 2017 -0500 mgr/balancer: more aggressive steps We are smart enough to back off if we misplace too much. Signed-off-by: Sage Weil (cherry picked from commit 0dbcf737eaed02b5aa94bc6f27231b4b87f09ce5) commit 901ae5920c590a2a83f30897b05017e001572e5f Author: Sage Weil Date: Sun Oct 1 16:00:06 2017 -0500 qa/suites/rados/thrash/d-balancer: enable balancer in various modes Signed-off-by: Sage Weil (cherry picked from commit 2c9c18d1ec3d33d385adf2a30dc1d3b8e987f9a5) commit a1ebe97886f7a578bc56a109f10a3d6149954087 Author: Sage Weil Date: Wed Sep 27 17:05:05 2017 -0400 mgr/balancer: crush-compat: phase out osd_weights Phase out the osd_weight values as we optimize the crush compat weight set. Allow a small regression in overall score if it means we are taking a step to remove the osd_weight value. Fix the 'osd reweightn' invocation. Signed-off-by: Sage Weil (cherry picked from commit 861c37fd99a76d5f1665a610224778fcbb41e9e1) commit 72716f058cd132d836c3251b4e6d4b056bb4c0a8 Author: Sage Weil Date: Wed Sep 27 15:12:54 2017 -0400 mgr/balancer: crush_compat: cope with 'out' osds Signed-off-by: Sage Weil (cherry picked from commit c2bba5820a0bf2225dfc9ba39cc67c78083aba11) commit c982848f32ed85639346931e86d331fb52458355 Author: Sage Weil Date: Wed Sep 27 15:12:36 2017 -0400 mgr/balancer: stop if we get a perfect score ..and fix the 'worse' threshold (we're just working around floating point imprecision; 1.01 was too coarse). Signed-off-by: Sage Weil (cherry picked from commit f1f3014ffcc67200c20e6fb2b2e6e8f1141a0a1d) commit 37ed6f218ba4fafd5360b8f62feb321a1feec56f Author: Sage Weil Date: Wed Sep 27 11:40:36 2017 -0400 mgr/balancer: more dead code Signed-off-by: Sage Weil (cherry picked from commit 37bd64405b8a578329b9a903c9a53322ea29caaf) commit b3dd7abc79fa3dff7768898caf814bcdfb8f8690 Author: Sage Weil Date: Wed Sep 27 11:27:42 2017 -0400 mgr/balancer: crush-compat: throttle changes based on max_misplaced Take smaller steps if we overshoot max_misplaced. Signed-off-by: Sage Weil (cherry picked from commit df2e008d3388b657d62ea4547e6b1b7a0251aa04) commit 25bfd12cf252dbba81f482fe9abe53959f992b4b Author: Sage Weil Date: Wed Sep 27 11:19:30 2017 -0400 mgr/balancer: remove dead code Signed-off-by: Sage Weil (cherry picked from commit 020be37fd06a75ec86e05af966be318c2199da9f) commit 1aff9dfadc148fb1cf9b60400c35bd11011772b2 Author: Sage Weil Date: Wed Sep 27 11:19:23 2017 -0400 mgr/balancer: include pg up mapping in MappingState Signed-off-by: Sage Weil (cherry picked from commit f6f2d253c379a6235e810808e7ce3b83cc4e212c) commit 9121e73f639bd47310ee29dce69a4c105582a3ab Author: Sage Weil Date: Tue Sep 26 18:36:06 2017 -0400 mgr/balancer: normalize weight-set weights to sum to target weight Signed-off-by: Sage Weil (cherry picked from commit 99fcd161bd222a4f0d2ddde0f1a4caecd1a78882) commit 619ae53354aa5682aeb7d77d3795a323188dad1a Author: Sage Weil Date: Tue Sep 26 18:35:42 2017 -0400 mgr/balancer: note root id in Eval Signed-off-by: Sage Weil (cherry picked from commit 26a7ec0c8a4c215791369d8650cd17d41317aefe) commit afcce93591c7e5c53596cec03165033f049a0c63 Author: Sage Weil Date: Tue Sep 26 18:00:08 2017 -0400 mgr/balancer: make crush-compat mode work! - it does multiple iterations, like the upmap optimizer. - it decreases the step size if it isn't improving, in the hope that it is overshooting the minimum - debug output is cleaned up a bit (the info level should be genuinely useful) Signed-off-by: Sage Weil (cherry picked from commit d9a31595ba15de5fda104a0154778e3200fc46a0) commit c6052f2706ad6b7c92f9d6fa381527e8922b30c6 Merge: 4e1eec777c 56359c367d Author: Sage Weil Date: Thu Nov 2 14:54:50 2017 -0500 Merge pull request #18590 from liewegas/wip-prime-fixes-luminous luminous: ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing commit 4e1eec777cf194cdf9bd4db1e7c4672d5b2adac7 Merge: 79736e781f 338af16881 Author: Sage Weil Date: Thu Nov 2 14:54:18 2017 -0500 Merge pull request #18620 from tchaikov/wip-21527-luminous luminous: mon/mgr: sync "mgr_command_descs","osd_metadata" and "mgr_metadata" prefixes to new mons Reviewed-by: Nathan Cutler commit 79736e781fbd36083c639d3f4c22f8f3ae426cce Merge: 815d8ab573 9d8e5d763b Author: Sage Weil Date: Thu Nov 2 14:53:16 2017 -0500 Merge pull request #18621 from tchaikov/wip-21534-luminous luminous: mon: update get_store_prefixes implementations Reviewed-by: Nathan Cutler Reviewed-by: John Spray commit 815d8ab573215277a5283af1ecabaac84eaf6200 Merge: c23865f07e 6b6eceee02 Author: Sage Weil Date: Thu Nov 2 14:52:32 2017 -0500 Merge pull request #18623 from liewegas/wip-pr-18585-luminous luminous: ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing Reviewed-by: Kefu Chai commit c23865f07e9147f8ac8a8d3837f735558af30dba Merge: 24bd1de6af 796e336470 Author: Sage Weil Date: Thu Nov 2 14:51:58 2017 -0500 Merge pull request #18625 from tchaikov/wip-ceph-disk-unlock-dmcrypted-partitions-luminous luminous: ceph-disk: unlocks dmcrypted partitions when activating them Reviewed-by: Nathan Cutler commit 24bd1de6afaa0d3dab763ae946b03ffb7b333430 Merge: 240edcfb18 3c3609b7e3 Author: Sage Weil Date: Thu Nov 2 14:51:32 2017 -0500 Merge pull request #18650 from b-ranto/wip-allow-getattr-luminous luminous: selinux: Allow getattr on lnk sysfs files Reviewed-by: Ken Dreyer commit 240edcfb18475bef84bd5d6345dd365411398773 Merge: 98edc3b9f2 deb78542ed Author: Sage Weil Date: Thu Nov 2 14:51:11 2017 -0500 Merge pull request #18675 from jcsp/wip-luminous-mgr luminous: bulk backport of ceph-mgr improvements commit 98edc3b9f2012db277c5796055223f6e69b67e66 Merge: 96102df540 0c0d8e6f40 Author: Alfredo Deza Date: Thu Nov 2 13:39:05 2017 -0400 Merge pull request #18687 from ceph/backport-18656 luminous: "ceph-volume: adds functional testing for bluestore" Reviewed-by: Alfredo Deza commit 96102df5405fc470696df84996853e254c29fd31 Merge: 172e6c1e0c 2e6b29084a Author: Kefu Chai Date: Fri Nov 3 01:02:26 2017 +0800 Merge pull request #18293 from smithfarm/wip-21795-luminous luminous: Ubuntu amd64 client can not discover the ubuntu arm64 ceph cluster Reviewed-by: Kefu Chai commit 0c0d8e6f402ae78ae8110a717188f8b366a67696 Author: Andrew Schoen Date: Tue Oct 31 15:25:23 2017 -0500 ceph-volume tests: adds objecstore as a testing factor Signed-off-by: Andrew Schoen (cherry picked from commit f8b114be61d033c853a2e8295b966e30c6a5f721) commit 9d970edddc1ab453a683bff1dfd19cab46e8cf9a Author: Andrew Schoen Date: Tue Oct 31 15:10:28 2017 -0500 ceph-volume: enable the centos7-bluestore and xenial-bluestore tests This also updates the ansible and testinfra version to match the current ceph-ansible master branch. Signed-off-by: Andrew Schoen (cherry picked from commit b49a19a9773599de295197979b498d06ca500cf8) commit f9554c4e3ef0eea1f4f09828f98c58838390d4fa Author: Andrew Schoen Date: Tue Oct 31 15:09:52 2017 -0500 ceph-volume: adds the xenial-bluestore testing scenario Signed-off-by: Andrew Schoen (cherry picked from commit 38372890c9c40e6f536026f61c5ef6e6d23d7111) commit b7fef119d0730a3cb8f19a461f773e66ce97af0f Author: Andrew Schoen Date: Tue Oct 31 15:05:46 2017 -0500 ceph-volume: adds the centos7-bluestore testing scenario Signed-off-by: Andrew Schoen (cherry picked from commit 45e4b0c9d7d4fea1d7bd3481d46edf3b42c40d26) commit deb78542ed45ac402b4492d5b9ae4a9cdf182724 Author: John Spray Date: Thu Nov 2 07:42:56 2017 -0400 qa: fix mgr _load_module helper I inadvertently broke this with the latest change to the module ls output. Signed-off-by: John Spray (cherry picked from commit 4fb3025682d812f86c50fa36532290fc0f8857ae) commit d4fcb97839a501094db2e1704c9799a107ad100e Author: John Spray Date: Wed Nov 1 19:10:19 2017 +0000 mgr: fix up make_unique usage for backport This was getting the definition some other way in master, but in luminous we need to include the backport14 header. Signed-off-by: John Spray commit f41cbb72f39359d7efc9feb0825cbad61e51c07c Author: John Spray Date: Fri Oct 20 13:54:29 2017 +0100 mon: fix up mgr_inactive_grace for backport This was converted to options.cc only, but we cannot backport another commit that removes this legacy health reporting code, so this commit updates the legacy health reporting code for the options.cc style config opt. Signed-off-by: John Spray commit a80a7a4d7f9ac73b4d3b86f9787dd9de9865530a Author: John Spray Date: Wed Nov 1 06:35:14 2017 -0400 mon: include disabled modules in `mgr module ls` Otherwise, when someone wants to see what's possible to do with `mgr module enable` they have to trawl through the whole mgr map dump. Signed-off-by: John Spray (cherry picked from commit 5861c4c022e825fe28347ba7e61ae86a6f260f24) commit fad998cb11628db970099cc192277c1824be884a Author: John Spray Date: Wed Nov 1 06:34:36 2017 -0400 doc: describe using `mgr module ...` commands ...including the new "mgr services" command. Signed-off-by: John Spray (cherry picked from commit d220e1adc337daeac418563dee125f161e85afdf) commit a46c5733881403f65872ca36d300ab6a4ca9d2cc Author: John Spray Date: Wed Nov 1 06:28:30 2017 -0400 doc: describe how to implement standby modules Signed-off-by: John Spray (cherry picked from commit ceb2a91190441b19171e5b5726748e43ee4d1e92) commit c704b344a4a739e5ba18535a0fd4f05308c89019 Author: John Spray Date: Mon Oct 23 05:37:49 2017 -0400 qa: fix mgr caps This was still using Kraken era settings Signed-off-by: John Spray (cherry picked from commit 9988ebed9530718b523a23d729d499dedab5eb6d) commit f58cb7035cf8992b8772f3358f2d67787ab585c3 Author: John Spray Date: Thu Oct 19 07:50:19 2017 -0400 qa: expand mgr testing Some extra coverage of the dashboard, including its standby redirect mode and the publishing of URIs. Also invoking the command_spam mode of the selftest module. Signed-off-by: John Spray (cherry picked from commit 05e648be6a51d3efa110ad9826bbdd0adcc4dd4d) commit a067468b3717dbab20f84996e76e2336d5fb5f30 Author: John Spray Date: Tue Oct 17 18:39:17 2017 -0400 mgr/selftest: extend test and add background spam mode Signed-off-by: John Spray (cherry picked from commit a382c3f1ca98b9fb7300e2d410bb2a1bb10b35ae) commit 1430e448ff93f73b8e8fba61f591ce4ec8c02380 Author: John Spray Date: Tue Oct 17 18:16:22 2017 -0400 mgr: drop GIL around set_uri, set_health_checks These didn't need to keep the GIL to go and do their pure C++ parts, and by keeping it they could deadlock while trying to take ActiveMgrModules::lock. Signed-off-by: John Spray (cherry picked from commit 27ee148e040ebaf512f8e11f814b3a7c8cf21f8b) commit 2a94381b53241f9ecb98c4cc605ef6547b910ca2 Author: John Spray Date: Tue Oct 17 18:14:43 2017 -0400 mgr: fix ~MonCommandCompletion This was doing a Py_DECREF outside of the Gil. Fixes: http://tracker.ceph.com/issues/21593 Signed-off-by: John Spray (cherry picked from commit 58dfa97ba88882fb3540d15e31bcac48a1aef5ef) commit 045ed0e023ae2d1a567b3426ef2b48980a851b7e Author: John Spray Date: Mon Oct 16 10:51:34 2017 -0400 mgr: update for SafeThreadState A bunch of the previous commits were done before this class existed, so updating in one go instead of trying to edit history in fine detail. Signed-off-by: John Spray (cherry picked from commit 29193a47e6cf8297d9b1ceecc7695f2c85434999) commit ef67307966575b618614225d6206019781d71647 Author: John Spray Date: Fri Oct 13 11:31:22 2017 -0400 mgr: refactor PyOSDMap etc implementation Implement real python classes from the C side, rather than exposing only module methods. Signed-off-by: John Spray (cherry picked from commit 7e61f79f5d56b568103a067d9a1eb87af997ad61) commit d85f91cf3e2f747cc23218a52d9271dd6e27febb Author: Sage Weil Date: Tue Sep 26 18:35:29 2017 -0400 mgr/PyOSDMap: add CRUSH get_item_weight Signed-off-by: Sage Weil (cherry picked from commit eacc9021459b31e42232bb958536d594d03b07b3) commit b1307e074b5fb708420686bf814bd4a7aa76f6d8 Author: John Spray Date: Mon Oct 16 06:33:48 2017 -0400 mgr: fix py_module_registry shutdown Was calling way too early, which did a Py_Finalize before the modules had been joined. Signed-off-by: John Spray (cherry picked from commit 0d5b1d70e616d7d1c2d6360375770f5c4754649d) commit 642a26dfcd9bef06fceb119008f5c3078bbf10e7 Author: John Spray Date: Thu Oct 12 13:14:02 2017 -0400 mgr: fix thread naming Was passing a reference to a local stringstream into Thread::create, not realising that it was taking a char* reference instead of a copy. Result was garbage (or usually, all threads having the name of the last one created) Signed-off-by: John Spray (cherry picked from commit bb4e71ed2ebdee1ac5e4b3eee390060e19fea0d8) commit f9223ad56d8e01c1f6e8690a6a186d02eed8d96d Author: John Spray Date: Fri Oct 6 11:02:44 2017 -0400 mgr: cut down duplication between active+standby ...by using PyModuleRunner class from ActivePyModule too. Signed-off-by: John Spray (cherry picked from commit df8797320bed7ad9f121477e35d7e3862efd89bd) commit 728e1e4e136aab5b4d185c738de8df7bcdce18ee Author: John Spray Date: Wed Oct 4 13:13:25 2017 -0400 mgr: fix os._exit overrides These would throw an exception when passed a status code. Signed-off-by: John Spray (cherry picked from commit e2442c1e20bf4ff12d58af500b34a18cc60d2de1) commit ca51a1b9d2cf0a3bf0fd7b69b833fa6696e61888 Author: John Spray Date: Thu Aug 24 14:07:37 2017 -0400 mon/MgrMonitor: reset services map on drop_active Signed-off-by: John Spray (cherry picked from commit 7b629ae46599d79ca1929cfc6637b367c6bb9029) commit 8d4512b5636eb41b2cbb50a42d071580cff728c6 Author: John Spray Date: Tue Aug 22 14:47:10 2017 -0400 mgr/dashboard: implement standby mode Signed-off-by: John Spray (cherry picked from commit 4f7007d1b0226af3f0cc33627ebf5051975657ac) commit 3953c0b3b7d0012507ac047085a3f863dc4929d3 Author: John Spray Date: Tue Aug 22 11:41:26 2017 -0400 pybind/mgr: add MgrStandbyModule Signed-off-by: John Spray (cherry picked from commit 3048e85cd712b7da77cf6ac55dd6a689d00e47e5) commit e37df9459d6d9abdbe310153bd53d1c1c4384a6b Author: John Spray Date: Tue Aug 22 14:42:11 2017 -0400 mgr: standby modules come up and run now ...they still don't have access to any config though. Signed-off-by: John Spray (cherry picked from commit c1471c7501948004096581ee415ab4a1fa2d9379) commit 977583eba266df834a7e27516cacb750bf7121ec Author: John Spray Date: Wed Aug 16 10:23:59 2017 -0400 mgr: enable running modules in standby mode Modules can implement a second, separate class that has access to very little state about the system and can't implement commands. They have just enough information to redirect or forward incoming requests/traffic to the active instance of the module on the active mgr. This enables module authors to create modules that end users can access via any (running) mgr node at any time, rather than having to first work out which mgr node is active. Signed-off-by: John Spray (cherry picked from commit 25566d1edca638bd15b3ba3326ee7e4d3e573cbb) commit 6a35a96ea838760945073b756ea7ae13e9a68ccf Author: John Spray Date: Tue Aug 15 06:53:18 2017 -0400 mgr: clean up python source file naming Signed-off-by: John Spray (cherry picked from commit 70d45a6b93c92baf8d6a3b15765110a5384c5e60) commit d37bf83f419fd5b34f0cbc8ee5313425e5177bc8 Author: John Spray Date: Mon Aug 14 06:31:18 2017 -0400 mgr: refactor python module management Separate out the *loading* of modules from the *running* of modules. This is a precursor to enabling modules to run in standby mode. Signed-off-by: John Spray (cherry picked from commit 9718896c8b844db2f3c07df1d344636da4605e61) commit b563555edb51b576fcbf94a3b92598a7300aaf09 Author: John Spray Date: Thu Jul 27 13:49:27 2017 -0400 pybind/mgr: use set_uri hook from dashboard+restful modules No more guessing the URL! Signed-off-by: John Spray (cherry picked from commit 089e105dd7ec762572ac06794caa7f5543075001) commit 6dd4d0504de9806fcf7c398a8d66b956b45986d9 Author: John Spray Date: Thu Jul 27 11:50:23 2017 -0400 mgr: enable python modules to advertise their service URI Fixes: http://tracker.ceph.com/issues/17460 Signed-off-by: John Spray (cherry picked from commit a0183a63fa791954d14c57632e184858cefe893d) commit 778322d0913d4d9c70609a3c40d809f62a366020 Author: John Spray Date: Thu Jul 27 11:49:45 2017 -0400 mon/MgrMonitor: store services in map and expose with command Signed-off-by: John Spray (cherry picked from commit c3c3e4e90ba6b09e29879b500f211d607ebabb53) commit a6b87a8e7c859607d4554833c5f9ad4cb2fea187 Author: John Spray Date: Thu Jul 27 11:46:40 2017 -0400 messages: `services` in MMgrBeacon Signed-off-by: John Spray (cherry picked from commit 236841b3b62af92ce0c4852045327fcfbc5c1651) commit b44cf5f70d0286c9dcdcda8dbb9f014a3386813a Author: John Spray Date: Thu Jul 27 11:45:53 2017 -0400 mon/MgrMap: store list of services Signed-off-by: John Spray (cherry picked from commit 3f703bd91f07b2fe43a16df0083d7b7c23803fd5) commit 5b677f3f02500b87757b9965d20151e345b8ba3f Author: John Spray Date: Thu Jul 27 06:31:01 2017 -0400 mgr: carry PyModules ref in MonCommandCompletion Signed-off-by: John Spray (cherry picked from commit e938bf9b9d27e192765c805e5f532c9dd4808b21) commit b69a656ac76fe38431bcd66c626883e324be1ba4 Author: John Spray Date: Wed Jul 26 12:31:13 2017 -0400 pybind: update MgrModule for ceph_state->ceph_module & tidy up the places where ceph_state was getting used outside of MgrModule. Signed-off-by: John Spray (cherry picked from commit 62cb512e4740f1f78f516b4f2179c1123fae1b36) commit 8b5f302706e91c5c6a88e918a9d4baafd726c211 Author: John Spray Date: Wed Jul 26 07:44:00 2017 -0400 mgr: refactor python interface Expose a python class instead of a module, so that we have a place to carry our reference to our MgrPyModule* and to PyModules*, rather than passing a handle for the former and using a global pointer for the latter. Signed-off-by: John Spray (cherry picked from commit 563878ba217491dd0a6fbd588cd56d09e3456c14) commit 35b4518c4a0c477f0190bfc006434093fe720c05 Author: John Spray Date: Thu Aug 3 06:22:35 2017 -0400 mgr/dashboard: remove blue highlight on scrubbing pg states This was kind of unnecessary, highlighting a completely normal and healthy situation in a different colour. The blue was also really hard to read against a grey background. Signed-off-by: John Spray (cherry picked from commit 99fa1fdf4e1be57792f50907147781d12009b32b) commit 764e7011d0e69e08aa5f56a5c8c56b8875820b7e Author: John Spray Date: Thu Jul 27 11:42:16 2017 -0400 mgr/dashboard: clean up fs standby list when empty Signed-off-by: John Spray (cherry picked from commit 5e64787c0ae0ac2a365c89bf89dfea425adc17d4) commit 1d1dce056664a40ea4dc9ebd1dc3826f0e449555 Author: John Spray Date: Wed Aug 30 13:56:39 2017 +0100 mgr: remove old-style config opt usage Signed-off-by: John Spray (cherry picked from commit ec09a7abc515f802451bf7ef3d22ce8ee6c6c7b3) commit a0131144036a9ea64c6f8e3289562bf7db18c746 Author: John Spray Date: Wed Aug 30 12:12:40 2017 +0100 mon: remove old-style mgr config opt usage Signed-off-by: John Spray (cherry picked from commit 6af4120d63324150ba19022c41fe4fa8a38cacbb) commit 526425c63b6a8f2d3f79fdd75c360d7c2ecfeec6 Author: John Spray Date: Wed Aug 30 11:48:25 2017 +0100 common: populate manager config option metadata Signed-off-by: John Spray (cherry picked from commit eba4c3f2762ae40ba746091e32364c2d68e780d9) commit 1eee973b90418979c33ca42b0720045f2de670c3 Author: Kefu Chai Date: Thu Jul 13 14:49:48 2017 +0800 common,mds,mgr,mon,osd: store event only if it's added otherwise * we will try to cancel it even it's never been added * we will keep a dangling pointer around. which is, well, scaring. * static analyzer will yell at us: Memory - illegal accesses (USE_AFTER_FREE) Signed-off-by: Kefu Chai (cherry picked from commit 2449b3a5c365987746ada095fde30e3dc63ee0c7) commit 66aa481be983fdd60beddc7ce621dad3edbf5490 Author: John Spray Date: Tue Oct 3 08:16:10 2017 -0400 mgr: safety checks on pyThreadState usage Previously relied on the caller of Gil() to pass new_thread=true if they would be calling from a different thread. Enforce this with an assertion, by wrapping PyThreadState in a SafeThreadState class that remembers which POSIX thread it's meant to be used in. Signed-off-by: John Spray (cherry picked from commit 625e1b5cfb9b8a5843dfe75e97826f70a57d6ebe) commit 385a6a0e8e6a6dd315ab358dcea7f3c069d9ad4d Author: John Spray Date: Tue Aug 22 11:38:25 2017 -0400 mgr: move Gil implementation into .cc The inclusion of Python.h in the .h was awkward for other files including Gil.h. Signed-off-by: John Spray (cherry picked from commit 23c3a075ee1a27e1b57fcb452a4d6ce53080264e) commit 241d655e127c71e9d9012d9205005c97d61def8a Author: John Spray Date: Wed Jul 26 07:21:40 2017 -0400 mgr: reduce Gil verbosity at level 20 Even at 20, it's pretty heavy to be logging every lock acquire/release. Signed-off-by: John Spray (cherry picked from commit 987612a97529be7e67b89977c4a0cf47906a5ecb) commit 30db4f5a54eac28e546c43a2ce099aa89d179495 Author: Jan Fajerski Date: Wed Oct 11 12:28:19 2017 +0200 pybind/mgr/prometheus: no ports in osd_metadata Ports might change on a OSD restart and this would create a new metadata metric for this osd. Signed-off-by: Jan Fajerski (cherry picked from commit 48fec7db4b214fe8ef6a04f8cb53fb8a2fb9c2ca) commit 80fc65cafaad3bb5499a15edbd97ecf483b4ad82 Author: Jan Fajerski Date: Wed Oct 11 10:59:33 2017 +0200 pybind/mgr/prometheus: add osd_in/out metric; make osd_weight a metric Signed-off-by: Jan Fajerski (cherry picked from commit e4c44c1d702ce242f2cb9a58ca7ce1c31fe0a498) commit db09ea1c75c80dc189042c59e09ebc1c1f8d1249 Author: Jan Fajerski Date: Wed Oct 11 20:07:19 2017 +0200 pybind/mgr_module: move PRIO_* and PERFCOUNTER_* to MgrModule class Signed-off-by: Jan Fajerski (cherry picked from commit f69484debade5f4fa2bd3a0d1badc9291cc9d7b7) commit bbe8fbe9ca98c1b0c8b5b43c0296fd9cc4d09c77 Author: John Spray Date: Mon Oct 9 12:10:22 2017 +0100 qa/mgr: fix influx/prometheus test names This was a typo: they were swapped around. Signed-off-by: John Spray (cherry picked from commit d96a59e74b6984b77c9f3b15f702e3bf45053590) commit be207ede6028e515a0a632e12330848c93d04a34 Author: John Spray Date: Thu Sep 28 10:50:53 2017 -0400 doc: flesh out prometheus docs Explain ceph_disk_occupation, importance of instance labels and honor_labels, provide example prometheus configuration yaml. Signed-off-by: John Spray (cherry picked from commit 5227afed5f33fa9487e1bfa3fd8ce0d82eb4a20f) commit 8a9e1cb00110ed73f492f1f6cf23b3595756efc4 Author: John Spray Date: Thu Sep 28 10:10:14 2017 -0400 mgr/prometheus: add ceph_disk_occupation series This is the magic series that enables consumers to easily get the drive stats that go with their OSD stats. Fixes: http://tracker.ceph.com/issues/21594 Signed-off-by: John Spray (cherry picked from commit 284be75524f7125dc1409b9c05fe47b37484964e) commit 51d74e41223397abcde006a4ce53e693a2125852 Author: Benjeman Meekhof Date: Wed Oct 4 10:05:17 2017 -0400 mgr/influx: Correct name of daemon stat measurement to 'ceph_daemon_stats' Signed-off-by: Benjeman Meekhof (cherry picked from commit f9014a1c75c6a3adf414b48a707fd444e65b3024) commit 2934dda9133a3a8876ab2701b11d4379440a5e41 Author: Benjeman Meekhof Date: Tue Oct 3 16:30:43 2017 -0400 mgr/influx: modify module database check to not require admin privileges - existing check tried to list all DB and fails even if DB exists if user is not admin level - still tries to create database if not found and user has privs Signed-off-by: Benjeman Meekhof (cherry picked from commit 06d7d37c7b9a8c3f4435eff04b6f4934be5e676f) commit 8c816b8e0fc1fd3fe227690287bcc6e6fce7c54d Author: Jan Fajerski Date: Tue Oct 10 08:40:31 2017 +0200 pybind/mgr/prometheus: fix metric type undef -> untyped Signed-off-by: Jan Fajerski (cherry picked from commit 6306392492d103200b21ea91bce10a315d7c4e16) commit 2ebab2f19cdd44f4c568eb96388fc4296f0f1814 Author: John Spray Date: Mon Sep 25 11:14:57 2017 -0400 mgr: respect perf counter prio_adjust in MgrClient This awkwardly involves re-ordering some definitions in perf_counters.h in order to refer to the prio names defined in PerfCountersBuilder. Signed-off-by: John Spray (cherry picked from commit 88163749b572ffd2bfe0850136fad5dbed2a9180) commit f8e9c37286d37d43bb768531f9cfbf70f2cb06b6 Author: John Spray Date: Mon Sep 18 09:06:13 2017 -0400 test: update perfcounters test for priority in output Signed-off-by: John Spray (cherry picked from commit 0f531f7871a68db96b2fb66ffdf6fae6935e6107) commit f073fc4663db17ea44e9c36831d21df1597d193a Author: John Spray Date: Wed Sep 13 17:16:54 2017 -0400 qa: add mgr module selftest task The module self test commands give us a chance to catch any other ceph changes that change something that a module was relying on reading. Signed-off-by: John Spray (cherry picked from commit 99352ceced9d0fe92ddad6b97b1393b41de75d50) commit 4c22f0f2669d343a8e7e83f0bc2a2dacbe194f34 Author: John Spray Date: Wed Sep 13 10:46:56 2017 -0400 mgr/prometheus: remove explicit counter list These have had their priorities bumped up to USEFUL, so they'll appear in the default get_all_counters output. Signed-off-by: John Spray (cherry picked from commit ad5a31efbea8081f03dd73669e891d03857ef9cc) commit fec2b3abf9d9abdc12bfeedaf2b22e8b0cc8c9f4 Author: John Spray Date: Wed Sep 13 10:45:21 2017 -0400 mon: elevate priority of many perf counters We can be quite liberal here, because mons are small in number. However, we don't want to expose KV database counters at this database from OSDs, so use the prio_adjust mechanism for that. Signed-off-by: John Spray (cherry picked from commit ac8320f23dd4c00eb80da0d9837c29744e38bd57) commit 8b9a18d3a0c40cb0ef10e32018ea1b23ff53f51e Author: John Spray Date: Wed Sep 13 07:07:50 2017 -0400 osd: upgrade a bunch of perf counters to PRIO_USEFUL These are broadly the OSD-wide IO stats, which happen to also be the ones that were named in the prometheus plugin until I changed it to be priority-based. Signed-off-by: John Spray (cherry picked from commit a1cc4ba2993de62b60fd1e58a9704877a6da5fe4) commit 278188c76ebdbaaa36b3d185b1378abf7665f3fc Author: John Spray Date: Wed Sep 13 07:06:24 2017 -0400 common: PerfCountersBuilder helper for priorities Let the caller set a priority as the defaul, to enable them to create a bunch at a given priority. This is just a convenience. Signed-off-by: John Spray (cherry picked from commit 66f61eeda6a2465b5fc0e40a4f1300913db065dc) commit 3dff5c0f39289e02e882bf86bf3e23e94ee33aa8 Author: John Spray Date: Tue Sep 12 10:27:12 2017 -0400 mgr/prometheus: add a self-test command Signed-off-by: John Spray (cherry picked from commit 76e1ba52b1b95d417cdd04b8fe985acee648f0e9) commit a6bc96dfe93f7cbcefa8030a3b6830117516931e Author: John Spray Date: Tue Sep 12 08:05:28 2017 -0400 mgr/influx: remove file-based config ...and also trim down the configuration to what's really needed. In general users don't need to pick and choose metrics. We could add it back if there was a strong motivation. Signed-off-by: John Spray (cherry picked from commit 6776d4645afc49a4bfb4b62673c91384239037f4) commit bda26fe0e707ec9e609be613805340dadd17e8f3 Author: John Spray Date: Tue Sep 12 06:51:21 2017 -0400 mgr/influx: enable self-test without dependencies The idea of self-test commands is that they're self contained and just exercise the module's calls to the Ceph-side. Signed-off-by: John Spray (cherry picked from commit 125294ab9d6e99aa4c960fea147a4e86624b869e) commit 9abd779d3321d1cda9c8677600e1e35af46109e7 Author: John Spray Date: Tue Sep 12 06:18:15 2017 -0400 mgr/influx: revise perf counter handling - Use new get_all_perf_counters path - Consequently get counters for all daemons, not just OSD - Tag stats with ceph_daemon rather than osd_id, as some stats appear from more than one daemon type - Remove summing of perf counters, external TSDB and/or queries can do this. - Remove mgr_id tag: this would change depending on which mgr was active, which is certainly not desirable. Signed-off-by: John Spray (cherry picked from commit 59b48e7660f4b757804974835027cd08a59843c2) commit 32f5f0fe515e6890e8aefad6ad8cdc2574ae8a34 Author: John Spray Date: Thu Aug 3 13:00:56 2017 -0400 mgr: omit module list in beacon logging This is useful in itself, but awkward when dealing with logs generally, because it means that when you grep on the name of a module, you get mostly beacon messages rather than the log messages from the module. Signed-off-by: John Spray (cherry picked from commit 8d1277fa5c578ce0ea23a70cc58c6cf99921ee25) commit 10ab4f8b6821e2d6593bc09161c23a9163b93611 Author: John Spray Date: Tue Sep 12 05:42:23 2017 -0400 mgr: define perf counter constants in mgr_module So that modules can consume perf counter data intelligently without having to hunt around in C land for these constants and redefine them. Signed-off-by: John Spray (cherry picked from commit 39ab28ed47e869e1466cb3a316a2cb11bdedd23a) commit 290d15ed912285803a40d02308b921b415c580a2 Author: John Spray Date: Mon Sep 11 09:12:25 2017 -0400 ceph.in: use PRIO_INTERESTING as daemonperf threshold Using PRIO_USEFUL as the threshold for what goes into time series databases. I'm claiming that we have more "useful" counters than fit on the screen, so daemonperf's "a screen's worth" threshold should be at the "interesting" level. Signed-off-by: John Spray (cherry picked from commit 30a74ce343caec2a433cb532ba697fe7013ed05c) commit 5ee9e15858d5aa2a71a89c13dea9a5db72579726 Author: John Spray Date: Mon Sep 11 09:12:01 2017 -0400 mon: set some priorities on perf counters Signed-off-by: John Spray (cherry picked from commit 29a71c35c39fbe1d4887e3f5ebb93232daab3487) commit 3bd478794e2626251eda91850974188da208b591 Author: John Spray Date: Mon Sep 4 05:39:11 2017 -0400 mgr/prometheus: tag stats by daemon name Using osd=0 or similar tags was problematic because daemons of different types have some same-named counters (e.g. MDS and OSD both have objecter perf counters). Signed-off-by: John Spray (cherry picked from commit eb524c272c89f8f99f22969b78caa016db7c671e) commit af92c011b20eaef073730be2f8d7112a0f7e99f8 Author: John Spray Date: Fri Sep 1 12:02:37 2017 -0400 mgr/prometheus: use new get_all_perf_counters interface Signed-off-by: John Spray (cherry picked from commit 11137aa269271ad15dcf19a8d51ce6f4acb7a98e) commit 3ce4a950975188051b6a3e6cb366c2fb23f0e88f Author: John Spray Date: Fri Sep 1 12:01:35 2017 -0400 common: used fixed size int for perf counter prio ...to avoid any ambiguity in allowed range and make clear how to encode it down the wire. Signed-off-by: John Spray (cherry picked from commit ba08fc1008d17aa7a5f285ea2705705ce1a0bda0) commit d4a0c778747970c231a55baea32f59a1009af4dd Author: John Spray Date: Fri Sep 1 12:00:59 2017 -0400 mgr: transmit perf counter prio to the mgr Signed-off-by: John Spray (cherry picked from commit f304f84cfbc22c1a54d152cc38227077bc564a7e) commit cdcac6e92ff57f1a4e5f5cac676049c6584452c4 Author: John Spray Date: Fri Sep 1 10:46:56 2017 -0400 common: always include priority in perf counter dump JSON output with inconsistent sets of members is annoying to use on the receiving side. Signed-off-by: John Spray (cherry picked from commit e631f1a72735ec618e2f3012ad7b9c5830d6c0eb) commit 802cd49bae2cce6c55eddd68feb617df893e1046 Author: John Spray Date: Tue Aug 29 11:55:28 2017 -0400 mgr: add get_all_perf_counters to MgrModule interface This is for use by modules that dump counters in bulk, e.g. to a TSDB. Signed-off-by: John Spray (cherry picked from commit 9a42d4255d9d968d6162b53b71db292d9d3de2e4) commit 85a93dcdddc4cd82c8935bfe123cb20f13f8928a Author: Jan Fajerski Date: Fri Aug 11 13:09:24 2017 +0200 pybind/mgr/prometheus: export cluster-wide pg stats, not per osd Signed-off-by: Jan Fajerski (cherry picked from commit 13b1236b96d4563e0985cad40d3009b60cc475e7) commit 1d89c0809f64009d646b929fec675e1aaa6c2b1a Author: Jan Fajerski Date: Fri Aug 11 12:51:47 2017 +0200 pybind/mgr/prometheus: add more osd metadata Signed-off-by: Jan Fajerski (cherry picked from commit e7704fa9cc35549dba526212c2830df589670416) commit 6a65408c1364669916643329494825aae394b200 Author: Jan Fajerski Date: Fri Aug 11 12:05:09 2017 +0200 pybind/mgr/prometheus: don't get perf counters that are not in schema Signed-off-by: Jan Fajerski (cherry picked from commit d4ba07d04477ccae3a89dcdcafbb7e76149dfd1c) commit 62092680ad966eabcdc6b557e9858fe0562147ee Author: Jan Fajerski Date: Fri Aug 11 12:04:28 2017 +0200 pybind/mgr/prometheus: add mon and osd perf counters to export Signed-off-by: Jan Fajerski (cherry picked from commit fa25d31263a26074225e2a00cb82448066b54069) commit 752c888533aaa9c7ae5781fafe4a6aa3302059fc Author: Jan Fajerski Date: Thu Aug 10 19:46:07 2017 +0200 pybind/mgr/prometheus: add index page, export metrics under metrics/ Signed-off-by: Jan Fajerski (cherry picked from commit d99a506ed37c2d0991d68ecd34ac5fb213a3eea4) commit b279ae25d95f6c4b8290042fc9f33c4a6f7afac2 Author: Jan Fajerski Date: Thu Aug 10 18:19:42 2017 +0200 pybind/mgr/prometheus: export selected perf_counters Signed-off-by: Jan Fajerski (cherry picked from commit f6e2e36ba72caf6347f3bb6a985925d0e35077a2) commit 4c092524929db048f6863acf1a9f12281bdc1646 Author: Jan Fajerski Date: Thu Aug 10 18:18:36 2017 +0200 pybind/mgr/prometheus: export osd and pool metadata Signed-off-by: Jan Fajerski (cherry picked from commit 2bea3814699c27baa8f633b56a8800d697685898) commit e3dafc3dd682a10e2310efdac4615d834933e7d6 Author: Jan Fajerski Date: Thu Aug 10 18:15:56 2017 +0200 pybind/mgr/prometheus: actually emit reported pg counts Signed-off-by: Jan Fajerski (cherry picked from commit c288624eed862559b2c86c5dfc85c837716739ab) commit 1ef5c88ef08ab9950ee8d7d4acda032b5a54f2f5 Author: Jan Fajerski Date: Thu Aug 10 18:09:17 2017 +0200 pybind/mgr/prometheus: no need to wait for notify event If stats or perf counters are not available they won't be emitted. Signed-off-by: Jan Fajerski (cherry picked from commit ead0973d7dd12fe985390891c80f1bc15f7b9aec) commit d0ef1cd2570644451205e5306feca0da94462d1f Author: Jan Fajerski Date: Thu Aug 10 18:07:14 2017 +0200 pybind/mgr/prometheus: no need to convert perf_schema to ordered_dict Signed-off-by: Jan Fajerski (cherry picked from commit 5e4b4b5ea2a217731691c1c391c252b08452798a) commit 7f191ff9dff33cac551ae5f64027d566b17b6d98 Author: Jan Fajerski Date: Wed Aug 9 17:22:49 2017 +0200 pybind/mgr/prometheus: add device_class label to osd metrics Signed-off-by: Jan Fajerski (cherry picked from commit 76d1918724320b7d6b1120b57b3002bb24099001) commit b24263387e46ac537a8ac189f9e0e80699518e0b Author: Jan Fajerski Date: Wed Aug 9 16:19:38 2017 +0200 pybind/mgr/prometheus: add cluster wide metrics; no perf counters for now Signed-off-by: Jan Fajerski (cherry picked from commit 49b3ff83cd231066d2a8f1809fadbdeb2c0c1f88) commit 5f0ef3a73a55a3f24997e415a848308afa0263ba Author: Jan Fajerski Date: Fri Aug 4 10:23:11 2017 +0200 pybind/mgr/prometheus: prefix metrics with 'ceph'; replace :: with _ Both follow prometheus best practices. While : is a legal metric character, "Exposed metrics should not contain colons, these are for users to use when aggregating." Signed-off-by: Jan Fajerski (cherry picked from commit 177afcc7886aa3898d092ebd1e101697bc6539fd) commit a303218c85498ccf472d2f1b35fd391376fb0faf Author: mhdo2 Date: Mon Aug 21 12:13:01 2017 -0400 doc/mgr: add influx plugin docs Signed-off-by: My Do (cherry picked from commit e345fe3c5780976a4e33488b3a75cd24bb2c96c5) commit de89603e54f550ffc4ff994026f32c5b7d5f6529 Author: mhdo2 Date: Tue Jul 18 18:33:55 2017 -0400 mgr/influx: added influx plugin Signed-off-by: My Do (cherry picked from commit 68ae26c014d0471cc3f2f979dc8d822b2e50740f) commit 2fea47d9710ec1e388db044065bc42b8046dae22 Author: John Spray Date: Sat Sep 23 11:55:55 2017 -0400 mgr: store declared_types in MgrSession Because we don't (yet) properly prevent multiple sessions from daemons reporting the same name (e.g. rgws), storing it in the DaemonPerfCounters meant that one daemon's report was referring to another daemon's set of reported types. This should always have been a property of the session. The behaviour will still be ugly when multiple daemons are using the same name (stomping on each other's stats/statsu) but it shouldn't crash. Fixes: http://tracker.ceph.com/issues/21197 Signed-off-by: John Spray (cherry picked from commit dc415f1ae09a308bd448614934a4c168eb9cf07b) commit 951d67fc9fee23f3f8f7d7df8fbfb8c29e60f82f Author: John Spray Date: Mon Sep 18 10:12:00 2017 +0100 mgr: make pgmap_ready atomic to avoid taking lock Signed-off-by: John Spray (cherry picked from commit d20915741d985e080a723cd6563bc6f4a657276f) commit 946d1541bd2b14d6f8abbaa201c3521dd117f01b Author: John Spray Date: Mon Aug 28 07:29:36 2017 -0400 mgr/DaemonServer: handle MMgrReports in parallel The DaemonStateIndex locking is sufficient to make all the report processing safe: holding DaemonServer::lock through all ms_dispatch was unnecessarily serializing dispatch. Signed-off-by: John Spray (cherry picked from commit 64af9d3da0fceff9ad0ff668f60d272c46912f34) commit d70fae092db920e2db77c7c8b044cfb4d9687992 Author: John Spray Date: Thu Aug 24 12:53:24 2017 -0400 mgr: clean up DaemonStateIndex locking Various things here were dangerously operating outside locks. Additionally switch to a RWLock because this lock will be relatively read-hot when it's taken every time a MMgrReport is handled, to look up the DaemonState for the sender. Fixes: http://tracker.ceph.com/issues/21158 Signed-off-by: John Spray (cherry picked from commit 806f10847cefe5c7a78fc319b1b130d372197dd3) commit b86cc9a2e49a4d1b53e83f47876a42b42819efcb Author: John Spray Date: Thu Aug 31 12:13:23 2017 -0400 mgr: runtime adjustment of perf counter threshold ceph-mgr has missed out on the `config set` command that the other daemons got recently: add it here and hook it all up to the stats period and threshold settings. Signed-off-by: John Spray (cherry picked from commit 057b73d641decb9403aba50caae9d139f3a34dd4) commit c278580d207f0b99daed48afdc689ae77fe73e9b Author: John Spray Date: Mon Jul 31 09:24:09 2017 -0400 mgr: apply a threshold to perf counter prios ...so that we can control the level of load we're putting on ceph-mgr with perf counters. Don't collect anything below PRIO_USEFUL by default. Signed-off-by: John Spray (cherry picked from commit bdc775fdd8acdad5c58ff3065a21396f80ce5db4) commit 11720b96ec8f9ef683710dffa3da4cda2ecf096d Author: Sage Weil Date: Tue Aug 8 16:36:23 2017 -0400 pybind/mgr/balancer: make auto mode work (with upmap at least) Signed-off-by: Sage Weil (cherry picked from commit ef1a3be05671ad31907cf8c4beb64a766359bc66) commit b6a000f0efef3cd6a143b82ae33cacf660cd8e4c Author: Spandan Kumar Sahu Date: Mon Aug 7 04:01:57 2017 +0530 src/pybind/mgr/balancer/module.py: improve scoring method * score lies in [0, 1), 0 being perfect distribution * use shifted and scaled cdf of normal distribution to prioritize highly over-weighted device. * consider only over-weighted devices to calculate score Signed-off-by: Spandan Kumar Sahu (cherry picked from commit c09308c49ca087fb8c5e7d4261b0234190f863d9) commit 6090ae6c224904516736c5f1a4ae5bcb6d7e6caa Author: Sage Weil Date: Fri Aug 4 17:59:20 2017 -0400 pybind/mgr/balancer: make 'crush-compat' sort of work Signed-off-by: Sage Weil (cherry picked from commit 7a00e02acd1b2ff21dac829de30f80fd69eae602) commit add3cd36db7c02d52c2fa429c034c114cac526d8 Author: Sage Weil Date: Thu Aug 3 16:23:08 2017 -0400 pybind/mgr/balancer: rough framework Signed-off-by: Sage Weil (cherry picked from commit d5e5c68c374e7d5514f89aac2d3df6008d103a76) commit 0c73e433ab6583fca6eea7678c23b469d643ae04 Author: Sage Weil Date: Thu Jul 27 23:33:06 2017 -0400 mgr/PyOSDMap: OSDMap.map_pool_pgs_up, CRUSHMap.get_item_name Signed-off-by: Sage Weil (cherry picked from commit a928bf62316c32f37dd1791192fd9a2ddaef0d33) commit 423947563c8e88f89bef9d71b3116c3302bd40c9 Author: Sage Weil Date: Sun Jul 23 00:10:56 2017 -0400 mgr/PyOSDMap: get_crush, find_takes, get_take_weight_osd_map These let us identify distinct CRUSH hierarchies that rules distribute data over, and create relative weight maps for the OSDs they map to. Signed-off-by: Sage Weil (cherry picked from commit 3b8a276c437cfd599c55a935d141375afda676ff) commit 073f23734c8058e3efcb440df89adab818510695 Author: Sage Weil Date: Thu Jul 27 10:07:31 2017 -0400 crush/CrushWrapper: rule_has_take Signed-off-by: Sage Weil (cherry picked from commit ef140de639078b40c05971fb219f7b8c12d83228) commit df426b5c24e12b1156bccdd8948cbed5977c348a Author: Sage Weil Date: Sat Jul 22 23:50:27 2017 -0400 crush/CrushWrapper: refactor get_rule_weight_osd_map to work with roots too Allow us to specify a root node in the hierarchy instead of a rule. This way we can use it in conjunction with find_takes(). Signed-off-by: Sage Weil (cherry picked from commit 69454e0570274ff7f252e7f081965dcc9bb04459) commit 89cac2d5176300838c23a28814257d0f395e39c9 Author: Sage Weil Date: Sat Jul 22 23:17:18 2017 -0400 pybind/mgr/balancer: do upmap by pool, in random order Signed-off-by: Sage Weil (cherry picked from commit 028a66d43244c15a77e71f3d3e4f41773837ab02) commit bfdc955ab6ab424b6a46dc1b0022fb3dc13ce157 Author: Sage Weil Date: Tue Jul 11 16:27:08 2017 -0400 pybind/mgr/balancer: add balancer module - wake up every minute - back off when unknown, inactive, degraded - throttle against misplaced ratio - apply some optimization step - initially implement 'upmap' only Signed-off-by: Sage Weil (cherry picked from commit 0d9685c50f79fbb53dbc8bd98c95900ef6e902b8) commit c57a55e8d426e6f87d4649a799d9ddee665618e0 Author: Sage Weil Date: Tue Jul 11 16:26:16 2017 -0400 pybind/mgr/mgr_module: add default arg to get_config Signed-off-by: Sage Weil (cherry picked from commit 39c42ddb9339c1950a3a474e8083db8b24e775a6) commit 4d686ee875d5bc0125426d348e044988aaca9f91 Author: Sage Weil Date: Mon Jul 10 23:23:19 2017 -0400 mgr: add trivial OSDMap wrapper class Signed-off-by: Sage Weil (cherry picked from commit 2ef005196ba2eb49c34c32def624938c7a8beb03) commit 1b721632aa8271d74b319714db0a9fedacb2629b Author: Sage Weil Date: Thu Jul 27 10:06:45 2017 -0400 mgr/PyModules: add 'pg_dump' get Signed-off-by: Sage Weil (cherry picked from commit bfb9286f4212947183c46543d609b664ea13b489) commit 68d411cf7b934c724f1f8ebeaa7148daebd79599 Author: Sage Weil Date: Tue Jul 11 16:25:42 2017 -0400 mgr/PyModules: add 'pg_status' dump This is summary info, same as what's in 'ceph status'. Signed-off-by: Sage Weil (cherry picked from commit 85b5b80906d00e098d4b1af1354c60a357022dd2) commit 35de92b259f50b52e51ab1daf57041553170406d Author: David Zafman Date: Wed Nov 1 09:53:34 2017 -0700 osd: build_past_intervals_parallel: Ignore new partially created PGs Fixes: http://tracker.ceph.com/issues/21833 Signed-off-by: David Zafman commit 172e6c1e0c5bb86db7cc0017acaa674aa893e3a0 Merge: 2988a39abe 966683fde6 Author: Casey Bodley Date: Wed Nov 1 16:28:25 2017 -0400 Merge pull request #18674 from ceph/wip-rgw-s3-branch qa/tests: use ceph-luminous branch for s3tests Reviewed-by: Casey Bodley commit 966683fde600837eb9a0f82ca6fd00f52e721809 Author: Vasu Kulkarni Date: Wed Nov 1 10:32:07 2017 -0700 qa: use ceph-luminous branch for s3tests Signed-off-by: Vasu Kulkarni commit 2988a39abedc6a092b8b3c2f0b33a3559a55461b Merge: 4dc0a0e38a 8f87fa2d3a Author: John Spray Date: Wed Nov 1 15:24:55 2017 +0100 Merge pull request #18412 from kmroz/wip-21659-luminous luminous: mgr: fix crashable DaemonStateIndex::get calls Reviewed-by: John Spray commit 4dc0a0e38a3f1a988c180d47970102df4b326b9e Merge: 273e035dbe 72c8583107 Author: John Spray Date: Wed Nov 1 12:33:45 2017 +0100 Merge pull request #18113 from jcsp/wip-prometheus-port-backport luminous: ceph-mgr: can not change prometheus port for mgr Reviewed-by: Kefu Chai Reviewed-by: John Spray commit 273e035dbe158f162c2ee559744c456c942f1737 Merge: a7f519913d f463cd84ae Author: Yuri Weinstein Date: Tue Oct 31 14:45:27 2017 -0700 Merge pull request #18655 from ceph/wip-yuriw-bp-PR18634-luminous qa: add "restful" to ceph_mgr_modules in ceph-ansible suite commit f463cd84ae1fb38817078967a4a40ba4b600f0e5 Author: Kefu Chai Date: Tue Oct 31 11:21:58 2017 +0800 qa: add "restful" to ceph_mgr_modules in ceph-ansible suite backport of https://github.com/ceph/ceph/pull/18634 Signed-off-by: Kefu Chai (cherry picked from commit caf9ee5c60d999951979d0b67afda8d56e1cd91d) Signed-off-by: Yuri Weinstein commit a7f519913df3aba83aa3ead9eee6b94aa8f90ffd Merge: f2749114c5 b84803d609 Author: Karol Mroz Date: Tue Oct 31 10:26:42 2017 -0700 Merge pull request #17889 from smithfarm/wip-21372-luminous luminous: core: Improve OSD startup time by only scanning for omap corruption once Reviewed-by: David Zafman commit f2749114c596d4016eb14e4777cb93a11b14b705 Merge: 528c3b6d32 1236e0f2e9 Author: Karol Mroz Date: Tue Oct 31 10:18:07 2017 -0700 Merge pull request #18004 from linuxbox2/luminous-rgwf-wr luminous: rgw_file: fix write error when the write offset overlaps. Reviewed-by: Casey Bodley commit 528c3b6d32f9c3319fd429de37bbca578538ed7c Merge: ec2d294684 d800747fa1 Author: Karol Mroz Date: Tue Oct 31 09:03:48 2017 -0700 Merge pull request #18438 from theanalyst/wip-21696-luminous luminous: fix a bug about inconsistent unit of comparison Reviewed-by: Casey Bodley commit 3c3609b7e3ce7339a845f9191a709d12025fbbb2 Author: Boris Ranto Date: Thu Sep 21 17:24:07 2017 +0200 selinux: Allow getattr on lnk sysfs files This showed up during downstream testing for luminous. We are doing getattr on the sysfs lnk files and the current policy does not allow this. Fixes: http://tracker.ceph.com/issues/21523 Signed-off-by: Boris Ranto (cherry picked from commit 394c26adb97cd150233fe8760355f486d03624a4) commit ec2d2946840fac65d9588f225adaa6f51c32f6b0 Merge: 90017e7d93 def3d55eb7 Author: Kefu Chai Date: Tue Oct 31 13:04:00 2017 +0800 Merge pull request #18410 from kmroz/wip-21732-luminous luminous: qa/suites/rest/basic/tasks/rest_test: whitelisting Reviewed-by: xie xingguo Reviewed-by: Kefu Chai commit 90017e7d933b52794d32fef69d97d5ba529d3936 Merge: 4f024c6831 c603faaf17 Author: Yuri Weinstein Date: Mon Oct 30 20:23:39 2017 -0700 Merge pull request #18629 from ceph/wip-yuriw-21978-luminous qa/suites/upgrade/jewel-x: Changed typo ('hammer' to 'jewel') commit c603faaf171fd3074ffe5e1342e8b96cd261e097 Author: Yuri Weinstein Date: Mon Oct 30 14:52:52 2017 -0700 Changed typo ('hammer' to 'jewel') Fixes http://tracker.ceph.com/issues/21978 Signed-off-by: Yuri Weinstein commit 555746a09e8bb7bcf99da91f9d00dfab9791d80c Author: Patrick Donnelly Date: Fri Oct 27 13:20:07 2017 -0700 MDSMonitor: wait for readable OSDMap before sanitizing Fixes: http://tracker.ceph.com/issues/21945 Signed-off-by: Patrick Donnelly (cherry picked from commit ca52f3bd93e6c743aa05171108527d877807b426) commit be13cdc0d4e54ce23c6ecf658100ddd860b1b96f Author: Patrick Donnelly Date: Tue Oct 3 12:25:12 2017 -0700 mds: clean up non-existent data pools in MDSMap Older versions of Ceph weren't strict about preventing pool deletion when the MDSMap referred to to-be-deleted pool. If we are dealing with a cluster upgrade, we should try to gracefully handle that by cleaning out data pools that have been removed. Reproduced this by allowing CephFS pools to be deleted: diff --git a/src/mon/OSDMonitor.cc b/src/mon/OSDMonitor.cc index 85c47c13da6..694b240cb9f 100644 --- a/src/mon/OSDMonitor.cc +++ b/src/mon/OSDMonitor.cc @@ -10962,7 +10962,7 @@ int OSDMonitor::_check_remove_pool(int64_t pool_id, const pg_pool_t& pool, FSMap const &pending_fsmap = mon->mdsmon()->get_pending(); if (pending_fsmap.pool_in_use(pool_id)) { *ss << "pool '" << poolstr << "' is in use by CephFS"; - return -EBUSY; + //return -EBUSY; } if (pool.tier_of >= 0) { pdonnell@icewind ~/ceph/build$ bin/ceph osd pool create derp 4 4 pool 'derp' created pdonnell@icewind ~/ceph/build$ bin/ceph fs add_data_pool cephfs_a derp added data pool 3 to fsmap pdonnell@icewind ~/ceph/build$ bin/ceph osd pool rm derp derp --yes-i-really-really-mean-it pool 'derp' is in use by CephFSpool 'derp' removed pdonnell@icewind ~/ceph/build$ bin/ceph fs ls ... 2017-10-03 12:50:48.409561 7f9e2e05b700 -1 /home/pdonnell/ceph/src/osd/OSDMap.h: In function 'const string& OSDMap::get_pool_name(int64_t) const' thread 7f9e2e05b700 time 2017-10-03 12:50:48.407897 /home/pdonnell/ceph/src/osd/OSDMap.h: 1184: FAILED assert(i != pool_name.end()) ceph version 12.1.2-2624-g37884a41964 (37884a419640b446fffc1fa4d6074c97339fdd96) mimic (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xf5) [0x564ebb5420f5] 2: (()+0x41dade) [0x564ebb3cbade] 3: (MDSMonitor::preprocess_command(boost::intrusive_ptr)+0x1fb9) [0x564ebb4cd119] Note when testing this fix, use something like this after removing the data pool: pdonnell@icewind ~/ceph/build$ bin/ceph fs set cephfs_a max_mds 2 Setting max_mds will cause a new FSMap to be created where MDSMap::sanitize is called; this is simulating the initial load+sanitize of a Hammer legacy MDSMap by the mons. Fixes: http://tracker.ceph.com/issues/21568 Signed-off-by: Patrick Donnelly (cherry picked from commit 7adf0fb819cc98702cd97214192770472eab5d27) commit 23fa3b726368f036b31e53a0bec8ad4bc654d993 Author: Patrick Donnelly Date: Tue Oct 3 12:23:03 2017 -0700 mds: reduce variable scope Signed-off-by: Patrick Donnelly (cherry picked from commit 37884a419640b446fffc1fa4d6074c97339fdd96) commit 4f024c68319a79bef9d937d7f2363a5b69575773 Merge: 6166148078 5a0016131b Author: Alfredo Deza Date: Mon Oct 30 14:05:00 2017 -0400 Merge pull request #18627 from ceph/backport-18513 ceph-volume lvm zap backport Reviewed-by: Alfredo Deza commit 5a0016131b1ec900255057876c98fc70e00542f0 Author: Andrew Schoen Date: Fri Oct 27 11:29:50 2017 -0500 ceph-volume: set journal_uuid and journal_device when using a partition This correctly sets the tags when a partition is used for a filestore journal. Signed-off-by: Andrew Schoen (cherry picked from commit 1a8561d38dd7363dc920ae82ec1343b2b75a1ca2) commit 05522e75f8d98658b5440d3c2f70e93ce65c65ce Author: Andrew Schoen Date: Tue Oct 24 10:13:40 2017 -0500 docs for ceph-volume lvm zap Signed-off-by: Andrew Schoen (cherry picked from commit 3e93a31deda96ed6fb23fa19ce3e273de05f7d88) commit cb1e81447f5c0c4eb8c0fd9fb11d0ceae3e26199 Author: Andrew Schoen Date: Mon Oct 23 09:51:43 2017 -0500 ceph-volume: add tests for ceph-volume lvm zap Signed-off-by: Andrew Schoen (cherry picked from commit 3d5e391693fc64747a4774287c968f842294eaa6) commit 33927c10e78842bb4c4ce4c0e4051ccceb8bb9e8 Author: Andrew Schoen Date: Mon Oct 23 09:44:07 2017 -0500 ceph-volume: print success message if zap succeeds Signed-off-by: Andrew Schoen (cherry picked from commit 6d70762dee0cd9047c291cf9869f666a375e956b) commit d8381ab509d9f60f6fe62c5692a9374b3187d5c9 Author: Andrew Schoen Date: Fri Oct 20 09:58:48 2017 -0500 ceph-volume: terminal.write only takes one argument Signed-off-by: Andrew Schoen (cherry picked from commit a5454eba241abd8cc1b1a660b12a1aec7c3c16e0) commit 6f2bd88278f2c7dc4cfd156969f98d7cc8c3e763 Author: Andrew Schoen Date: Thu Oct 19 16:29:39 2017 -0500 ceph-volume: the metadata field for the lv path is lv_path not path Signed-off-by: Andrew Schoen (cherry picked from commit e3a9113e7025f1e3a3130ec1e2d565f37bf3d2dc) commit 1dfe7e9854b00850b86168049821447d33298fff Author: Andrew Schoen Date: Thu Oct 19 14:59:26 2017 -0500 ceph-volume: update help text for ceph-volume lvm zap Signed-off-by: Andrew Schoen (cherry picked from commit cbc38c4e4a7dcc24b31bcfa6af73eb8cf04f56ad) commit c5206435a9bae9d93e6cd9a3c2b98127871ad94e Author: Andrew Schoen Date: Thu Oct 19 14:55:44 2017 -0500 ceph-volume: remove lvm metadata when zapping an lv Signed-off-by: Andrew Schoen (cherry picked from commit 5f57fc87205c2d35da9a1f28c72e233ffb3fe4d9) commit eb53ef67b4e3bf5afb6e414fd42caab67c4cfdf2 Author: Andrew Schoen Date: Thu Oct 19 14:55:02 2017 -0500 ceph-volume: adds Volume.clear_tags Will remove all tags for the current lv Signed-off-by: Andrew Schoen (cherry picked from commit 3c9401f1618f7fcbea827fe7279e8dfde617d957) commit 4ee13e0587c5e152fdac18004bc4336f8b90e4af Author: Andrew Schoen Date: Thu Oct 19 12:01:57 2017 -0500 ceph-volume: zap should leave lvs and partitions intact This will remove filesystems and wipe data from any lv or partition given to 'ceph-volume lvm zap' but still leave it intact for further use. Signed-off-by: Andrew Schoen (cherry picked from commit caba9fb80b358222192f736bd1b0ab31dca25cec) commit 8c3ab3459abfebac20bb4ce3ddecf1a7a58cf401 Author: Andrew Schoen Date: Thu Oct 19 11:44:11 2017 -0500 ceph-volume: adds utilites to wipe the fs and clear data when zapping These should eventually move to a disk api. Signed-off-by: Andrew Schoen (cherry picked from commit 2f64d4a0abd41afbcb9eba6a237642502d9a9135) commit 8a0e7fad57e41e10d51e0e167b671668ee1aa286 Author: Andrew Schoen Date: Thu Oct 19 10:35:51 2017 -0500 ceph-volume: adds tests for api.lvm.remove_lv Signed-off-by: Andrew Schoen (cherry picked from commit 937b57438147681c2c6e32c6db38d8bea68d4731) commit c6dd47ab6d1ec6784ec3cef8f65139baeaf4fa3a Author: Andrew Schoen Date: Thu Oct 19 10:35:10 2017 -0500 ceph-volume: api.lvm.remove_lv should return True when successful Signed-off-by: Andrew Schoen (cherry picked from commit 7f055aa6deb3904cf0334e214c13a26098b08aa8) commit 71ae0b8cabc0a696e6762d7f377f9f4b28d2a963 Author: Andrew Schoen Date: Wed Oct 18 10:19:25 2017 -0500 ceph-volume: zap logical volumes Signed-off-by: Andrew Schoen (cherry picked from commit 1d083a2191315ee94301c2f5f102a0906dd05fa8) commit 7110922c42ee30fa25dc9dacbfc8d0f9ae2f2a79 Author: Andrew Schoen Date: Wed Oct 18 10:19:00 2017 -0500 ceph-volume: adds a remove_lv command to the lvm api Signed-off-by: Andrew Schoen (cherry picked from commit cf98b6971ced0caa29ff4f7ac346df652177fa29) commit 731610f18b9e4109ce72a4467af209144e4a127d Author: Andrew Schoen Date: Tue Oct 17 14:15:18 2017 -0500 ceph-volume: stubs out the ceph-volume lvm zap command Signed-off-by: Andrew Schoen (cherry picked from commit 2e64b797ef6ae91623ffba8ae28d3f8ccc7d7b93) commit c2237c7c6dbcd5420d8ed87c6f2648c3f4e6edae Author: Jason Dillaman Date: Fri Oct 27 09:58:20 2017 -0400 qa/suites/rbd: run cls tests for all dependencies Signed-off-by: Jason Dillaman (cherry picked from commit 27613a63d1f1ee6fa7327cd1c63b875c0e5247f8) commit 6b75efab390b3019e404f007090e9031328e1301 Author: Jason Dillaman Date: Fri Oct 27 09:57:43 2017 -0400 cls/journal: fixed possible infinite loop in expire_tags Fixes: http://tracker.ceph.com/issues/21956 Signed-off-by: Jason Dillaman (cherry picked from commit 9e66dca49591e50b9cab5df311f1dc217eb58fcc) commit 796e33647022de1fd831b31045242fb8e9d0e4d2 Author: Kefu Chai Date: Wed Aug 30 18:46:49 2017 +0800 ceph-disk: unlock all partitions when activate should unlock all dmcrypted partitions when activating a device. Signed-off-by: Kefu Chai (cherry picked from commit ba2aa0cee9e077d8439ba31228b41beb2d827a04) commit 3e9aad1a116e02bb126010f310ca2267177b5162 Author: Felix Winterhalter Date: Mon Jul 17 02:04:39 2017 +0200 ceph-disk activate unlocks bluestore data partition Signed-off-by: Felix Winterhalter (cherry picked from commit 1287caf2dbb5ef6e5f243fe1d23633946aef26f9) commit 6b6eceee027e571d8d3da4af144391f8da85b53e Author: Yao Zongyou Date: Sat Oct 28 18:23:30 2017 +0800 ceph-bluestore-tool: the link target should not ending with new line Signed-off-by: Yao Zongyou (cherry picked from commit 39c68d128cf29f8e0a617009d16c5edf471ead47) commit cc84812507607d0a91233a513088094533a2d6a7 Author: Yao Zongyou Date: Sat Oct 28 18:22:27 2017 +0800 ceph-bluestore-tool: the right action is prime-osd-dir not prime-osd-dev Signed-off-by: Yao Zongyou (cherry picked from commit d418a04e9fed6bf91cc9acc5ac777eadf664c974) commit 9d8e5d763b173247f95f0769fd52372a8a7a5371 Author: John Spray Date: Mon Sep 25 10:44:28 2017 +0100 mon: implement MDSMonitor::get_store_prefixes Fixes: http://tracker.ceph.com/issues/21534 Signed-off-by: John Spray (cherry picked from commit a3c317b406c69b9d6a2d7df94806f4c308e6ee6a) commit 338af1688147babb4e92523972175fe09ebb34b6 Author: huanwen ren Date: Mon Sep 25 15:01:02 2017 +0800 mon/mgr: sync mgr_command_descs table and mgr_metadata table sync mgr_command_descs table and mgr_metadata table to new mons Fixes: http://tracker.ceph.com/issues/21527 Signed-off-by: huanwen ren (cherry picked from commit 13f6aa3aaa6de0aeccec67d7c6f3effe43dcae49) Conflict: in master PaxosService::get_store_prefixes(..) is marked const while in luminous, the cleanup commit which added the `const` is not backported yet, so drop the `const` in the backported commit. commit de07d2954afb012e51d2eaca12c1763e29a19eca Author: John Spray Date: Mon Sep 25 10:37:18 2017 +0100 mon/OSDMonitor: tidy prefix definitions We should define them in one place to make it easy when updating get_store_prefixes. Fixes: http://tracker.ceph.com/issues/21534 Signed-off-by: John Spray (cherry picked from commit 889ac5166ad8fd30678325ddc2da59f45db53f06) commit 981e552d72c151b6bf2c9bb64b969f9446ef4789 Author: huanwen ren Date: Mon Sep 25 14:55:55 2017 +0800 mon/osd_metadata: sync osd_metadata table sync osd_metadata table to new mons when add new mons Signed-off-by: huanwen ren (cherry picked from commit 755ec735107c75156ae4935f7255bbfe5dc384d9) commit 3f0ad5f453a259cc53f50b1a9515e8a720da4a1d Author: huanwen ren Date: Mon Sep 25 14:55:55 2017 +0800 mon/osd_metadata: sync osd_metadata table sync osd_metadata table to new mons when add new mons Signed-off-by: huanwen ren (cherry picked from commit 6c20433bfb049ac6c69f6f7a979006e8b9ea0e29) commit 61661480780e555fc501aec7c32163596e1e18d3 Merge: c0ec364ad6 410434b3d2 Author: Patrick Donnelly Date: Sun Oct 29 11:06:28 2017 -0700 Merge PR #18385 into luminous * refs/pull/18385/head: mds: fix race in PurgeQueue::wait_for_recovery() mds: open purge queue when transitioning out of standby replay mds: always re-probe mds log when standby replay done Reviewed-by: John Spray commit c0ec364ad6f54da7efbbb6b84ad34ea9782e3df2 Merge: 5ee57e1b42 695d7ad511 Author: Patrick Donnelly Date: Sun Oct 29 11:06:24 2017 -0700 Merge PR #18316 into luminous * refs/pull/18316/head: mds: prevent trim count from underflowing Reviewed-by: Zheng Yan commit 5ee57e1b42d93fc0dc77b1221c362753684b280e Merge: 84f4bf40da b99acfd601 Author: Patrick Donnelly Date: Sun Oct 29 11:06:20 2017 -0700 Merge PR #18300 into luminous * refs/pull/18300/head: mds: keep CInode::STATE_QUEUEDEXPORTPIN state when exporting inode Reviewed-by: Zheng Yan Reviewed-by: Amit Kumar commit 84f4bf40dade5ef55eba139d63187959b98e84e5 Merge: 2638d5039e 460268b5e9 Author: Patrick Donnelly Date: Sun Oct 29 11:06:16 2017 -0700 Merge PR #17729 into luminous * refs/pull/17729/head: ceph.in: validate service glob Reviewed-by: Patrick Donnelly Reviewed-by: Amit Kumar commit 2638d5039e615035563338cd428b44fb23d879ff Merge: f6f88c2a9b 2973b6d418 Author: Patrick Donnelly Date: Sun Oct 29 11:00:18 2017 -0700 Merge PR #18299 into luminous * refs/pull/18299/head: mds: update client metadata for already open session Reviewed-by: Zheng Yan Reviewed-by: Amit Kumar commit f6f88c2a9b40cf84658319a07a2bc26858763093 Merge: 388901f091 f353a1e805 Author: Patrick Donnelly Date: Sun Oct 29 11:00:14 2017 -0700 Merge PR #18298 into luminous * refs/pull/18298/head: osdc/ObjectCacher: limit memory usage of BufferHead Reviewed-by: Zheng Yan commit 388901f0911b2d45f5b5ca2d653da5c605d99e2d Merge: 45cfeae6eb c5ccbf1108 Author: Patrick Donnelly Date: Sun Oct 29 11:00:10 2017 -0700 Merge PR #18085 into luminous * refs/pull/18085/head: ceph_volume_client: fix setting caps for IDs Reviewed-by: Ramana Raja commit 45cfeae6ebd5a0910c03ff323ecdd5740e569b41 Merge: ecd5b0066f 1021fe235c Author: Patrick Donnelly Date: Sun Oct 29 11:00:06 2017 -0700 Merge PR #18030 into luminous * refs/pull/18030/head: qa: relax cap expected value check mds: improve cap min/max ratio descriptions mds: fix whitespace mds: cap client recall to min caps per client mds: fix conf types mds: fix whitespace doc/cephfs: add client min cache and max cache ratio describe mds: adding tunable features for caps_per_client Reviewed-by: Zheng Yan commit ecd5b0066fc091f34a166989b2e68e165c390dd0 Merge: 04acd559db d5e583490d Author: Patrick Donnelly Date: Sun Oct 29 10:55:32 2017 -0700 Merge PR #17921 into luminous * refs/pull/17921/head: ceph_volume_client: perform snapshot operations in Reviewed-by: Amit Kumar Reviewed-by: Ramana Raja commit add8c3db195acfda75a1628196e839b0147e7c3f Author: Matt Benjamin Date: Thu Oct 26 18:28:10 2017 -0400 rgw_file: set s->obj_size from bytes_written Required to store correct final accounted_size of objects in RGWWriteWriteRequest::write_finish. Fixes: http://tracker.ceph.com/issues/21940 Signed-off-by: Matt Benjamin (cherry picked from commit b1f528d35aeccabb4e5818aec6feb8e53e562500) commit 8cbb2eb937cae4e136ef77bc93e6d5346e21c8da Author: Sage Weil Date: Mon Oct 9 08:15:21 2017 -0500 qa/cephfs: test ec data pool Signed-off-by: Sage Weil (cherry picked from commit d0732fc96fbc6849dd51b391d85f765c74cfb593) commit 04acd559db55c6e8040a2ba65a850e72a04ad2ab Merge: d294493c42 8a87d43db2 Author: Andrew Schoen Date: Fri Oct 27 11:34:15 2017 -0500 Merge pull request #18593 from ceph/luminous-wip-bz1499840 luminous ceph-volume lvm bluestore support Reviewed-by: Andrew Schoen commit d294493c42ea21e85a1fd36eb8623229a2ff6ea6 Merge: 58c1b8c4f2 34cd96d342 Author: Abhishek L Date: Fri Oct 27 17:30:35 2017 +0200 Merge pull request #18334 from kmroz/wip-21816-luminous luminous: rgw: fix bilog entries on multipart complete Reviewed-By: Casey Bodley commit 58c1b8c4f28b73d9b7db1cf3fe79ae5062832c88 Merge: 8742c619b0 d7f6b93af4 Author: Abhishek L Date: Fri Oct 27 17:30:07 2017 +0200 Merge pull request #17861 from smithfarm/wip-21441-luminous luminous: rbd: [cli] mirror getter commands will fail if mirroring has never been enabled commit 8742c619b03cc5f048dc08f76d67ea64c109b4b3 Merge: 7c03693d0b f8bfd9458d Author: Abhishek L Date: Fri Oct 27 17:29:55 2017 +0200 Merge pull request #18416 from kmroz/wip-21855-luminous luminous: librbd: object map batch update might cause OSD suicide timeout Reviewed-By: Jason Dillaman commit 7c03693d0bd1a7108bd2d55eda7b7bd1e4e7f904 Merge: fc486a26cb 8b3e9917a7 Author: Abhishek L Date: Fri Oct 27 17:29:43 2017 +0200 Merge pull request #18337 from kmroz/wip-21640-luminous luminous: rbd-mirror: forced promotion can result in incorrect status Reviewed-By: Jason Dillaman commit fc486a26cb12466df6269ec4c7717332be2ed92c Merge: c944c688e5 4ebd4b3928 Author: Abhishek L Date: Fri Oct 27 17:29:22 2017 +0200 Merge pull request #18336 from kmroz/wip-21639-luminous luminous: librbd: snapshots should be created/removed against data pool Reviewed-By: Jason Dillaman commit c944c688e5a4a7935233e7fb3824b97d9afe44d9 Merge: bc7e648b18 34af07f97a Author: Abhishek L Date: Fri Oct 27 17:29:03 2017 +0200 Merge pull request #17860 from smithfarm/wip-21299-luminous luminous: rbd: [rbd-mirror] asok hook names not updated when image is renamed Reviewed-By: Jason Dillaman commit 8a87d43db25a364841d969636bd74b7590c3d563 Author: Alfredo Deza Date: Thu Oct 26 15:09:18 2017 -0400 ceph-volume lvm.activate remove links on activate before priming Signed-off-by: Alfredo Deza (cherry picked from commit 634b5caaed7c9188e426727e83a0768bdbc51f0d) commit 869d13037dbb2b8882c4a4a91ec20c9814bcc438 Author: Alfredo Deza Date: Thu Oct 26 07:48:24 2017 -0400 ceph-volume lvm.activate ceph-bluestore-tool needs --cluster too Signed-off-by: Alfredo Deza (cherry picked from commit fb36087bc3c7e7bb8358cbf8ad84409fa23fac68) commit 3baab9f4e286c3b54c179b0f471435c2dbc859b2 Author: Alfredo Deza Date: Wed Oct 25 18:57:28 2017 -0400 ceph-volume lvm.activate consume cluster_name from lvm metadata Signed-off-by: Alfredo Deza (cherry picked from commit 2b7ff497fe9e0e167ab01bf0d8ce5e26f0da836f) commit 8f41a983356ee9d182f8247f2da8db15848afae0 Author: Alfredo Deza Date: Wed Oct 25 18:57:03 2017 -0400 ceph-volume lvm.prepare persist cluster_name on lvm metadata Signed-off-by: Alfredo Deza (cherry picked from commit 6d38ad987e08791be379f6c006c1eb5bf9464415) commit 1ae30d3130af845ebf563a74a1e2642f5236f1ed Author: Alfredo Deza Date: Wed Oct 25 15:30:43 2017 -0400 ceph-volume util.prepare bluestore db and wal paths are to devices, not osd dir paths Signed-off-by: Alfredo Deza (cherry picked from commit fd616acd6ae0f184c1757e7de38fc746d526908c) commit fdfbfb8140fc1be3123d1254f4463613dc7c7cb0 Author: Alfredo Deza Date: Wed Oct 25 15:30:06 2017 -0400 ceph-volume lvm.prepare bluestore wal and db links get created with mkfs Signed-off-by: Alfredo Deza (cherry picked from commit 80698d2309a221d2318b11be9b8f8a95b51416ea) commit 6ac58f3cd5e292153c82d881a30db1f6c4bfd936 Author: Alfredo Deza Date: Wed Oct 25 10:28:43 2017 -0400 ceph-volume lvm.prepare use wal and db for mkfs Signed-off-by: Alfredo Deza (cherry picked from commit 4970ab5c804111ce4b2b57817f1e29d58b18a893) commit c9ecd14ca220e45960650715703cae3e4376e3c0 Author: Alfredo Deza Date: Tue Oct 24 16:28:33 2017 -0400 ceph-volume lvm.prepare ensure tags are set for ceph.type correctly for block and data Signed-off-by: Alfredo Deza (cherry picked from commit 58dbefd3be7bad51e6c56c50fbfb450e205b3036) commit 76fb5fec94a9bbfed1163a23faf87e19d5617bd4 Author: Alfredo Deza Date: Tue Oct 24 16:26:03 2017 -0400 ceph-volume lvm.activate check if lv device is mounted at path Signed-off-by: Alfredo Deza (cherry picked from commit 1ead135bef801280e7cec21283c95c7ba440d84b) commit 343362027368c4170bd785f1633afeb5ed90f3a2 Author: Alfredo Deza Date: Tue Oct 24 15:55:21 2017 -0400 ceph-volume lvm.prepare default to using bluestore Signed-off-by: Alfredo Deza (cherry picked from commit 39dd0d1d10aba08c3e8299eb1702260274a756a7) commit 0a27fe88266458eead5a011c27cc323f5b7173a5 Author: Alfredo Deza Date: Tue Oct 24 15:54:57 2017 -0400 ceph-volume lvm.create default to using bluestore Signed-off-by: Alfredo Deza (cherry picked from commit d3145da6e643dedb2fbed72e7d4d0cddf397c5e3) commit bd2e0aa78ad1a500935c3134f4d1e6c3413a26ae Author: Alfredo Deza Date: Tue Oct 24 15:54:11 2017 -0400 ceph-volume lvm.common remove boolean defaults for CLI flags Signed-off-by: Alfredo Deza (cherry picked from commit a7d5965ab532a621f402bad55d45dfed61d90cc5) commit c3d474af562ec5b8658bed6362046b0208a2c0eb Author: Alfredo Deza Date: Tue Oct 24 13:29:18 2017 -0400 ceph-volume lvm.prepare update to use create_osd_path Signed-off-by: Alfredo Deza (cherry picked from commit bafb72144666f891773bed95e9a0f757673af2e0) commit bc7e648b182569865d1c2a2aea6da1e52a3d3cb1 Merge: 690be9c711 b015ce1161 Author: Abhishek L Date: Fri Oct 27 16:57:48 2017 +0200 Merge pull request #18138 from kmroz/wip-21684-luminous luminous: rgw: stop/join TokenCache revoke thread only if started. Reviewed-By: Casey Bodley commit 690be9c711b34be1ec4254ba0cedf7d6755f59f6 Merge: 5727e4a33e 06c31a6cee Author: Abhishek L Date: Fri Oct 27 16:55:22 2017 +0200 Merge pull request #17994 from ukernel/luminous-21337 luminous: mds: make sure snap inode's last matches its parent dentry's last Reviewed-By: Patrick Donelly commit 5727e4a33e0b82e0f9591251572b61458ec2cfae Merge: d973d6d7ae 624b3ac505 Author: Abhishek L Date: Fri Oct 27 16:52:43 2017 +0200 Merge pull request #18431 from theanalyst/wip-21857-luminous luminous: rgw: We cant't get torrents if objects are encrypted using SSE-C Reviewed-By: Casey Bodley commit 02013a714ec5fa6c2ff1073e432b8f6b9ff41305 Author: Alfredo Deza Date: Mon Oct 23 09:44:27 2017 -0400 ceph-volume lvm.activate only prime-osd-dir when directory is empty Signed-off-by: Alfredo Deza (cherry picked from commit 9bb6cfd867c2d8e972093d023622f8c2e5b440d4) commit 28593dbf1caeeec8f7c69dbd06e2ff9759afd072 Author: Alfredo Deza Date: Mon Oct 23 09:20:26 2017 -0400 ceph-volume lvm.activate safeguard against auto detect objectstore flag not being set Signed-off-by: Alfredo Deza (cherry picked from commit c9b5f352a8f9e2743e5dca3a0479c8099a59fd12) commit 8df51c8af5d95d67effece720ec2ff82517155b9 Author: Alfredo Deza Date: Fri Oct 20 15:18:20 2017 -0400 ceph-volume util.system use strings for comparison in py3 Signed-off-by: Alfredo Deza (cherry picked from commit 1b671a8602b036cb8d53bc40eb70198c13143051) commit 1f09ee754fec40c4b17ea33829e6449d8749690a Author: Alfredo Deza Date: Fri Oct 20 15:10:08 2017 -0400 f ceph-volume tests update changes from the help menu in lvm Signed-off-by: Alfredo Deza (cherry picked from commit 3ddcd3c2f19cc848a11a2882f150462ad2f26c65) commit ec7c42bd59dc05cb8bad082d64ffdc28f13d3dc8 Author: Alfredo Deza Date: Fri Oct 20 14:12:02 2017 -0400 ceph-volume lvm.common --journal-size doesn't accept a volume group Signed-off-by: Alfredo Deza (cherry picked from commit a8282e9a580bcfa74abdd33032e2ce2851b5cf29) commit 243098815d1d49570c193767d008cbc4fa0f174b Author: Alfredo Deza Date: Fri Oct 20 14:04:31 2017 -0400 ceph-volume lvm.prepare filestore does not require a volume group Signed-off-by: Alfredo Deza (cherry picked from commit 3653f105927d455b8df143797af0cfe72f8d6834) commit da5f764a083613e5d74d1401da65c2ba3aff5fb4 Author: Alfredo Deza Date: Fri Oct 20 14:04:05 2017 -0400 ceph-volume lvm.common volume groups are no longer required Signed-off-by: Alfredo Deza (cherry picked from commit 526c0aa81ac8eed95c2b4664a00ded2118372fff) commit eaa9150d4a6bacf62fcf485e8d36430743ac06cf Author: Alfredo Deza Date: Fri Oct 20 12:07:06 2017 -0400 doc/ceph-volume activate workflow mention tmpfs in osd dirs Signed-off-by: Alfredo Deza (cherry picked from commit 8788fe3721034f390240c01493bc1d27d65a73a5) commit d65af56eaaab728d8c81ddb69b967cb13f177766 Author: Alfredo Deza Date: Fri Oct 20 12:02:10 2017 -0400 doc/ceph-volume update activation to include bluestore support Signed-off-by: Alfredo Deza (cherry picked from commit 288a3124793d11d71b540555fd03c2f61a79dd16) commit 6545b1c14e84af5f25fd4bbd405b56f7b5797113 Author: Alfredo Deza Date: Fri Oct 20 11:58:47 2017 -0400 doc/ceph-volume update create to indicate bluestore support Signed-off-by: Alfredo Deza (cherry picked from commit 61ce8b4b844fcb64fc95373c5515de902a52fc8c) commit d973d6d7aefae1c34b664f98955baed4c841579b Merge: cf621f8b4e 5b5f0b5a33 Author: Abhishek L Date: Fri Oct 27 16:49:03 2017 +0200 Merge pull request #17858 from smithfarm/wip-21448-luminous luminous: rgw: string_view instance points to expired memory in PrefixableSignatureHelper Reviewed-By: Casey Bodley commit cf621f8b4e8138284843f6d979aca61651a854bd Merge: e0c3a05959 0b3a974be7 Author: Abhishek L Date: Fri Oct 27 16:48:54 2017 +0200 Merge pull request #18442 from theanalyst/wip-21637-luminous luminous: encryption: PutObj response does not include sse-kms headers Reviewed-By: Casey Bodley Reviewed-By: Matt Benjamin commit e0c3a05959a70b4cf4213320ba7c990578bf8c19 Merge: dd7aa70301 14a1dcb1ff Author: Abhishek L Date: Fri Oct 27 16:48:26 2017 +0200 Merge pull request #18437 from theanalyst/wip-21698-luminous luminous: radosgw-admin usage show loops indefinitly Reviewed-By: Casey Bodley commit dd7aa7030144d42958f92af5c7f6804a402f3b6d Merge: 3a8f046716 c3a719da27 Author: Abhishek L Date: Fri Oct 27 16:48:12 2017 +0200 Merge pull request #18434 from theanalyst/wip-21817-luminous luminous: rgw: zone compression type is not validated Reviewed-By: Casey Bodley Reviewed-By: Matt Benjamin commit 3a8f046716456eb0f51db79f152f3dc67d84c319 Merge: 0d9b3a01d0 4c18ac1240 Author: Abhishek L Date: Fri Oct 27 16:47:48 2017 +0200 Merge pull request #18440 from theanalyst/wip-21652-luminous luminous: policy checks missing from Get/SetRequestPayment operations Reviewed-By: Casey Bodley commit 0d9b3a01d0972a2fc5ae9550ba80c38acd9edbeb Merge: aa71baeabe a3e2ae4925 Author: Abhishek L Date: Fri Oct 27 16:47:33 2017 +0200 Merge pull request #18441 from theanalyst/wip-21651-luminous luminous: rgw: avoid logging keystone revocation failures when no keystone is configured Reviewed-By: Casey Bodley commit aa71baeabef5aa7eb6903593b3dfcc8878227c99 Merge: 1067293a7e 0b84dd07c4 Author: Abhishek L Date: Fri Oct 27 16:47:13 2017 +0200 Merge pull request #18443 from theanalyst/wip-21634-luminous luminous: s3:GetBucketLocation bucket policy fails with 403 Reviewed-By: Casey Bodley commit 1067293a7e21fc835447c9e5e1d8416902593ba1 Merge: f2f8aef89c abefb578c3 Author: Abhishek L Date: Fri Oct 27 16:46:59 2017 +0200 Merge pull request #18444 from theanalyst/wip-21635-luminous luminous: s3:GetBucketCORS/s3:PutBucketCORS policy fails with 403 Reviewed-By: Casey Bodley commit f2f8aef89c3144ac5d859ea0fe8a7db57cfa8ad6 Merge: 385d84b8eb 0e65ea466f Author: Abhishek L Date: Fri Oct 27 16:46:44 2017 +0200 Merge pull request #18430 from theanalyst/wip-21695-luminous luminous: failed CompleteMultipartUpload request does not release lock Reviewed-By: Casey Bodley Reviewed-By: Matt Benjamin commit 385d84b8ebb3a4192ce4817aaf6c99b27af0190d Merge: 4a74a41c47 f9850d2148 Author: Abhishek L Date: Fri Oct 27 16:46:13 2017 +0200 Merge pull request #18445 from theanalyst/wip-21633-luminous luminous: s3:GetBucketWebsite/PutBucketWebsite fails with 403 Reviewed-By: Casey Bodley commit 4a74a41c473fcda61f156d8bf720bf0e4cf0aec0 Merge: 14cb83cd07 8b420c4627 Author: Abhishek L Date: Fri Oct 27 16:45:54 2017 +0200 Merge pull request #18436 from theanalyst/wip-21789-luminous luminous: user creation can overwrite existing user even if different uid is given Reviewed-By: Casey Bodley commit 14cb83cd077258e061cfa09733cf0b9a82f7507a Merge: 0339b8b599 fe45086a54 Author: Abhishek L Date: Fri Oct 27 16:45:36 2017 +0200 Merge pull request #18435 from theanalyst/wip-21790-luminous luminous: RGW: Multipart upload may double the quota Reviewed-By: Casey Bodley commit 0339b8b599f313e6a7b176f0f8b94b8101ab50c3 Merge: 68cc868d1f 248bd94dab Author: Abhishek L Date: Fri Oct 27 16:45:19 2017 +0200 Merge pull request #18432 from theanalyst/wip-21856-luminous disable dynamic resharding in multisite enviorment Reviewed-By: Casey Bodley commit 68cc868d1fdca55308fa1aba0bb708d07170e668 Merge: f337bef88a 17df0ba8f0 Author: Abhishek L Date: Fri Oct 27 16:44:40 2017 +0200 Merge pull request #18429 from theanalyst/wip-21792-luminous luminous: encryption: reject requests that don't provide all expected headers Reviewed-By: Casey Bodley commit 29470eac4ca939c824b56f4fbc03c8b7814f9d02 Author: Alfredo Deza Date: Fri Oct 20 11:51:55 2017 -0400 doc/ceph-volume update prepare with bluestore workflow Signed-off-by: Alfredo Deza (cherry picked from commit 9f1a5627399e2589fe706b158a32a9fb8642ac23) commit c90e473ff35d15d309eb92dbdc6b9df2e8bd5ecc Author: Alfredo Deza Date: Fri Oct 20 10:15:09 2017 -0400 doc/ceph-volume update filestore to indicate it is not the only/default objectstore Signed-off-by: Alfredo Deza (cherry picked from commit c93603e5764e021305cfeb1c297ee16318126fe1) commit 3890c47dddd0d028b0c003cb46a9bec670c445f1 Author: Alfredo Deza Date: Fri Oct 20 10:00:17 2017 -0400 doc/ceph-volume elaborate on env vars for interval and tries in activation Signed-off-by: Alfredo Deza (cherry picked from commit 1dc24d1a7065b4e23b1bfa0e7c6d263b5ebc6906) commit 88240acf19176ae5c139836b7d46848e15ef5454 Author: Alfredo Deza Date: Fri Oct 20 09:33:55 2017 -0400 ceph-volume lvm.common update cli flags help for bluestore support Signed-off-by: Alfredo Deza (cherry picked from commit c9693fb3f8386016aba5d7c8f53f5f71f948c88d) commit a945e7776309d39955918cec7e0c5f1895abc39b Author: Alfredo Deza Date: Fri Oct 20 09:23:35 2017 -0400 ceph-volume util.disk st_mode is needed for stat() Signed-off-by: Alfredo Deza (cherry picked from commit e3b5663f7968e44ec0cd6db2bfc43acdb233e314) commit e833916afe22e9c4765344229b5b89cfc9af1b14 Author: Alfredo Deza Date: Fri Oct 20 09:23:02 2017 -0400 ceph-volume decorators always log to the file with exceptions Signed-off-by: Alfredo Deza (cherry picked from commit cb3533a72571251d9f1b6171f783246462726857) commit 53e9cf6fcd7549b4bfa78d7568d56b30b38e81fd Author: Alfredo Deza Date: Fri Oct 20 08:49:32 2017 -0400 ceph-volume tests verify behavior of get_lv_from_argument Signed-off-by: Alfredo Deza (cherry picked from commit adabbd462bcb19a6da65211e24dfb2b9b5043ef4) commit 40cfec281cc2584b2e2987f356109c6c160d4597 Author: Alfredo Deza Date: Fri Oct 20 08:48:01 2017 -0400 ceph-volume api.lvm return result of get_lv when using arguments If the argument is an absolute path it is fine to just return whatever get_lv finds since it is a "safe" call, it will return a None if nothing is found Signed-off-by: Alfredo Deza (cherry picked from commit f209c9d4d50bd7abe0f7f6b9fb29359cb7196718) commit 7139077aab892c3213815d1dcf71d93df80a32c8 Author: Alfredo Deza Date: Fri Oct 20 08:12:01 2017 -0400 ceph-volume lvm.listing support bluestore, not only filestore Signed-off-by: Alfredo Deza (cherry picked from commit 6a5b89a475c2a36022ca09dbb7f6b94cc3b44881) commit 654e06f1202596d5cecef9d03134861eed053882 Author: Alfredo Deza Date: Fri Oct 20 08:10:51 2017 -0400 ceph-volume api.lvm allow full paths to detect lv from arguments Signed-off-by: Alfredo Deza (cherry picked from commit 8d257101b1556ec5ce805622e9b7b6b74f6b000d) commit 454d48eb29f4429370b831645a29e0450b5b46b1 Author: Alfredo Deza Date: Fri Oct 20 08:10:26 2017 -0400 ceph-volume tests add support for bluestore list verification Signed-off-by: Alfredo Deza (cherry picked from commit 4674da9c07f871bf1f25bb782c136fe082095a14) commit 8d72737f90b0cd1db860678ebb66596e233fd646 Author: Alfredo Deza Date: Thu Oct 19 13:20:55 2017 -0400 ceph-volume tests activate with fsid works for bluestore and filestore Signed-off-by: Alfredo Deza (cherry picked from commit 5b9f0853f780db3b0af6b4b7c1ad1d59184069c9) commit 846aa2492f6db342d9fb4c477c2e13e61e414aaf Author: Alfredo Deza Date: Thu Oct 19 13:14:05 2017 -0400 ceph-volume tests arg validator can accept absolute paths Signed-off-by: Alfredo Deza (cherry picked from commit 15586715f506505a57ec5122416541479df74bc8) commit 2c9ee51a0a7ed5edc900316c19ba27facb9163eb Author: Alfredo Deza Date: Thu Oct 19 12:03:17 2017 -0400 ceph-volume tests update changes from the help menu in lvm Signed-off-by: Alfredo Deza (cherry picked from commit 39bdc2e3f621a32bcbb3dfe64f156bbfd7e4f5ee) commit 5724bb04bb74e17b9f7e96b3a3e434cac720c4ed Author: Alfredo Deza Date: Thu Oct 19 11:46:57 2017 -0400 ceph-volume tests update to get_lv from get_journal_lv Signed-off-by: Alfredo Deza (cherry picked from commit 9be1012072f36e21a3f3cef42a42f7f1eb83c64e) commit 43e25933701605ac8cd7cb3d6fbd4cbf6705565e Author: Alfredo Deza Date: Thu Oct 19 11:30:29 2017 -0400 ceph-volume tests.util new path/device mount interface Signed-off-by: Alfredo Deza (cherry picked from commit d5984bad97347e58318793fc05da84832d323113) commit 9b36a8bf8d471047dc94cdfcac1f84a9802b9fc5 Author: Alfredo Deza Date: Thu Oct 19 09:49:25 2017 -0400 ceph-volume tests update api.lvm tests The create_lv signature changed to require full size description and tags need to be an actual dictionary (vs. keyword args) Signed-off-by: Alfredo Deza (cherry picked from commit e8a18637531cde59d73a97d106958489c1d12117) commit ea57e70e6f826ac17eae0226b9d916e8bff4d021 Author: Alfredo Deza Date: Wed Oct 18 16:06:55 2017 -0400 ceph-volume lvm.activate ceph-bluestore-tool should populate the osd dir Signed-off-by: Alfredo Deza (cherry picked from commit f3bb616bcf8a4737a3701ebf812de831785d6341) commit fe3b61ac0da67d9c80ead24c6ccf79e1507fa132 Author: Alfredo Deza Date: Wed Oct 18 15:06:01 2017 -0400 ceph-volume lvm.activate auto detect objectstore type Signed-off-by: Alfredo Deza (cherry picked from commit 472aa0b1997a0b1301b699b7ce30dce728aaea0c) commit 5da9775184e4bc9bf1b83d18eae602c9a510a89f Author: Alfredo Deza Date: Wed Oct 18 15:00:19 2017 -0400 ceph-volume lvm.trigger enable objectstore auto detection for activate Signed-off-by: Alfredo Deza (cherry picked from commit f579ef07026564f1a94b2923a2e4640ade2da503) commit 307c99e4382895e6b42c193e9b98b9206fd57f9a Author: Alfredo Deza Date: Wed Oct 18 12:00:34 2017 -0400 ceph-volume lvm.activate add bluestore support for activation Signed-off-by: Alfredo Deza (cherry picked from commit 5696fed09b65044142b34f89492a6d91d3243e14) commit 70632b0fac2fa7e411418c90a50b99391379f436 Author: Alfredo Deza Date: Wed Oct 18 11:40:38 2017 -0400 ceph-volume util.system create mappings of mounts This makes it easier for consumers to detect a path, or a device that might be mounted or might have more than one mount, including tmpfs and devtmpfs support, which was just not possible before. Signed-off-by: Alfredo Deza (cherry picked from commit f61007cc650f6964ef2ff3ea98a1debf02713198) commit 839dc1f554a12fb109ca1f992784dd4f4933898a Author: Alfredo Deza Date: Mon Oct 16 06:51:05 2017 -0400 ceph-volume util.prepare separate filestore vs. bluestore prepare utils Signed-off-by: Alfredo Deza (cherry picked from commit 85cdc783008ff0e0566c34aaab7cd5f12db1cbc7) commit deae88983d80cedcdc842d192f8d659b7533d3dc Author: Alfredo Deza Date: Mon Oct 16 06:50:27 2017 -0400 ceph-volume process allow obfuscating a key or index when logging commands Signed-off-by: Alfredo Deza (cherry picked from commit bbcca70d962855d991f0aa8e86a19f1855345668) commit 18336242b85065d8a2b0a92f22b2e91a81b997ba Author: Alfredo Deza Date: Fri Oct 13 09:22:53 2017 -0400 ceph-volume util.prepare add other ln helpers for bluestore Signed-off-by: Alfredo Deza (cherry picked from commit 555b664882db2cd247087f907bdf4d426fb652f8) commit 37a1564a41ba8ae6a89eb0903f0fdf6b463d90f0 Author: Alfredo Deza Date: Thu Oct 12 16:08:52 2017 -0400 ceph-volume util.arg_validators allow to pass a group as --data for lvm Signed-off-by: Alfredo Deza (cherry picked from commit 211c38f591945dee53196068533784c385bae102) commit afd1588fe06a80f2277082fd7bd15f43d8e601b7 Author: Alfredo Deza Date: Thu Oct 12 16:08:16 2017 -0400 ceph-volume util.disk add utilities for is_device and is_partition Signed-off-by: Alfredo Deza (cherry picked from commit 6b23fc72240ced158cf5335f6e815dbfd804dfab) commit 5e8f798d2b8f80b21f3ce4701d22e47992dd0938 Author: Alfredo Deza Date: Thu Oct 12 16:05:22 2017 -0400 ceph-volume api.lvm udpate create_lv for bluestore Require full size notation for LVM (e.g. 50G or 100M) and do not "translate" tags by pre-fixing them with 'ceph'. That is already how the rest of the API works, no need to make that translation in one place Signed-off-by: Alfredo Deza (cherry picked from commit ac0d1555d3d9caeef9d4fc3e46bc11b7e437413b) commit f337bef88a31a5d23420665406a06f761e8d4e6e Merge: 3db3ca9938 d2be1e0ccb Author: Abhishek L Date: Fri Oct 27 16:43:43 2017 +0200 Merge pull request #17857 from smithfarm/wip-21446-luminous luminous: rgw:multisite: Get bucket location which is located in another zonegroup, will return 301 Moved Permanently Reviewed-By: Casey Bodley commit 3db3ca99380521c79596b06b695eefbb81d57908 Merge: ced01a5b85 c4d6539374 Author: Abhishek L Date: Fri Oct 27 16:43:00 2017 +0200 Merge pull request #17856 from smithfarm/wip-21444-luminous luminous: rgw: setxattrs call leads to different mtimes for bucket index and object Reviewed-By: Casey Bodley commit 3bbd840fe0d34d42c04fac523aee5fc023d8bd3b Author: Alfredo Deza Date: Thu Oct 12 15:42:22 2017 -0400 ceph-volume lvm.prepare update filestore workflow to use new helpers Signed-off-by: Alfredo Deza (cherry picked from commit 1e8a79b8f4b9e6360e8e3ace580e46caeedca8c7) commit ced01a5b85c9ec5657d8f09a447b6308ea597754 Merge: 2c9a9af424 fe0a41b4a9 Author: Abhishek L Date: Fri Oct 27 16:41:15 2017 +0200 Merge pull request #18417 from kmroz/wip-21782-luminous luminous: cls/journal: possible infinite loop within tag_list class method Reviewed-By: Jason Dillaman commit 2c9a9af424c618c50334c0efa7f5a5f807aeb4fd Merge: a1d286252a 7d2b7e8e31 Author: Abhishek L Date: Fri Oct 27 16:38:04 2017 +0200 Merge pull request #18287 from smithfarm/wip-retract-assertions-luminous luminous: rgw: Remove assertions in IAM Policy Reviewed-By: Adam Emerson Reviewed-By: Abhishek Lekshmanan commit fb95f3f9fd6122f9b8c3c2215463fc8a1ef62ba2 Author: Alfredo Deza Date: Thu Oct 12 14:19:07 2017 -0400 ceph-volume lvm.prepare initial take on bluestore support Signed-off-by: Alfredo Deza (cherry picked from commit e4fc3464af472a8dbdf049917eed73519ff82c3b) commit a1d286252a75c3632a7ecc63a366e5262f9e16cd Merge: b151619a45 fd207447d1 Author: Abhishek L Date: Fri Oct 27 16:29:02 2017 +0200 Merge pull request #17859 from smithfarm/wip-21451-luminous luminous: rgw: lc process only schdule the first item of lc objects Reviewed-By: Daniel Gryniewicz commit 26ef33efc37cdd626c6a7577812bb87947417e01 Author: Alfredo Deza Date: Thu Oct 12 14:16:25 2017 -0400 ceph-volume lvm.common update help flags for bluestore usage Signed-off-by: Alfredo Deza (cherry picked from commit 17028f3a3c1f9394fa82dc6f2bbd596d96acb037) commit 005359b92a910b72712edbcaa7b2e27a54d3a093 Author: Alfredo Deza Date: Wed Oct 11 14:30:41 2017 -0400 ceph-volume api.lvm allow to create a vg Signed-off-by: Alfredo Deza (cherry picked from commit 16d7feb2b5d5eb5fa0bf23a95b60623ae2e1676f) commit 795e69673f452227afc12d76ed5aa0d6a8af8665 Author: Alfredo Deza Date: Tue Oct 10 12:30:06 2017 -0400 ceph-volume lvm add block CLI flags for bluestore Signed-off-by: Alfredo Deza (cherry picked from commit 63d2afbc605ac1531bc9f00a1891017bfd42fbce) commit cbfb082c180e93cd147e164afd1d32ef9234924e Author: Alfredo Deza Date: Tue Oct 10 12:15:14 2017 -0400 ceph-volume lvm make bluestore the default option Signed-off-by: Alfredo Deza (cherry picked from commit dc3ced30747312dedd0e895a75ec15b1925ef1ea) commit a340f1acab04587f5b2c323abc1a14b8dd7de915 Author: Casey Bodley Date: Mon Oct 9 14:46:47 2017 -0400 rgw: RGWDataSyncControlCR retries on all errors similar to RGWMetaSyncShardControlCR, we don't want to exit and stop the data sync processor thread on failures. we want to keep retrying with backoff Signed-off-by: Casey Bodley (cherry picked from commit 065e67bbd312240f5128c31f5525098c62f3afce) Conflicts: src/rgw/rgw_data_sync.cc (sync tracing) commit fdec63a28e24e056938139591e5799a0e3693c20 Author: Casey Bodley Date: Mon Oct 9 14:09:40 2017 -0400 rgw: fix error handling in ListBucketIndexesCR the call to set_state() returns 0, when we want operate() to return the error code instead. use set_cr_error() to do this Fixes: http://tracker.ceph.com/issues/21735 Signed-off-by: Casey Bodley (cherry picked from commit ed6340a47f32472bb1f650407dbe739c44bf4bd6) commit e711cf319fa12ec7b03edda769685353e24cbfcb Author: Casey Bodley Date: Mon Oct 9 14:08:21 2017 -0400 rgw: ListBucketIndexesCR spawns entries_index after listing metadata if the metadata listing fails, we won't have to clean up entries_index Signed-off-by: Casey Bodley (cherry picked from commit 7f127f54406ef0e08a1265b9bc20433fe3f21523) commit 56359c367d8267281f714dcecefdb50c598718a6 Author: Sage Weil Date: Thu Oct 26 13:51:40 2017 -0500 ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing If the symlink points to the right location, do nothing. If it doesn't, replace it. If it's not a symlink, bail with EEXIST. Signed-off-by: Sage Weil (cherry picked from commit de8dc42d42218bc1a1779e1bcc5831c567853c8d) commit b151619a4523b148026e613860a05add6aff3831 Merge: 84fc30873a 352373ff4a Author: Abhishek L Date: Fri Oct 27 15:11:00 2017 +0200 Merge pull request #18433 from theanalyst/wip-21545-luminous luminous: rgw file write error Reviewed-By: Matt Benjamin commit 84fc30873aa6ed3c1bf0c91307c362d27e9bc72a Merge: 2bc24caa00 bb8586b259 Author: Andrew Schoen Date: Fri Oct 27 08:03:16 2017 -0500 Merge pull request #18587 from ceph/luminous-wip-volume-api-change luminous ceph-volume lvm api refactor/move Reviewed-by: Andrew Schoen commit 2bc24caa003ea3c4898202a887a5864078c0f3fd Merge: a5548a8247 5f96edfb41 Author: Sage Weil Date: Fri Oct 27 06:39:58 2017 -0500 Merge pull request #18568 from liewegas/wip-fix-rop-leak-luminous-2 luminous: osd/PG: fix recovery op leak due to recovery preemption Reviewed-by: David Zafman commit bb8586b259726eb0ae72db7c6a238b2d2432fc9b Author: Alfredo Deza Date: Tue Oct 3 16:13:33 2017 -0400 ceph-volume create an api module to hold common api modules Signed-off-by: Alfredo Deza (cherry picked from commit f72368641d7f38b6395afb7ed70a0a9d0794a03a) commit 3d32fb30193437b57535c79657ee726704f4b037 Author: Alfredo Deza Date: Wed Oct 4 06:44:54 2017 -0400 ceph-volume tests move lvm api tests into its new test module Signed-off-by: Alfredo Deza (cherry picked from commit eec0c8a5c27adc9b3fc5e999cbee8165bdb45736) commit e6145af90c3f1c8c520bb7671194e6d8a13bb5d2 Author: Alfredo Deza Date: Wed Oct 4 06:43:36 2017 -0400 ceph-volume tests refactor lvm api import for test_listing Signed-off-by: Alfredo Deza (cherry picked from commit 66fd41fd4b35e275196bfb94794f3fda9fd77f76) commit c63d3ba89489cc42aaa364c99b3d3b4c1bba226c Author: Alfredo Deza Date: Wed Oct 4 06:43:18 2017 -0400 ceph-volume tests refactor lvm api import for api tests Signed-off-by: Alfredo Deza (cherry picked from commit 2a9a36bd24bddb35a7618127a0ec57f661a71566) commit 6efd25344230c0abb51cd82f99310a4b94f2fcd3 Author: Alfredo Deza Date: Wed Oct 4 06:42:47 2017 -0400 ceph-volume tests refactor lvm api import for activate tests Signed-off-by: Alfredo Deza (cherry picked from commit d81aa8c112312f2fb49d94ce3512b3356b0b8814) commit 47cd005b672828aedf11aeb67a6099477783f7f0 Author: Alfredo Deza Date: Wed Oct 4 06:42:01 2017 -0400 ceph-volume tests refactor lvm api import in conftest Signed-off-by: Alfredo Deza (cherry picked from commit a00aff7066fe6a679498ac8a02ec22ef29feced3) commit 4a6dfa9b6949bdcdba22a7003c451390105d287e Author: Alfredo Deza Date: Tue Oct 3 16:17:08 2017 -0400 ceph-volume devices.lvm refactor importing api to come from api/lvm.py Signed-off-by: Alfredo Deza (cherry picked from commit f4bee6cb42a017abce269ed3926deff04038a47e) commit 9a20f184676973222c931cecff26ca313e636e1d Author: Alfredo Deza Date: Tue Oct 3 16:14:16 2017 -0400 ceph-volume move lvm/api.py to api/lvm.py so disk can consume it Signed-off-by: Alfredo Deza (cherry picked from commit bb72480d2bf44ff04ea93c98f4a4e59032a28896) commit f26f3dc842c04ac70e309266aaedbae9d45eeae8 Author: yuliyang Date: Sun Sep 24 08:41:04 2017 +0800 rgw:fix list objects with marker when bucket is enable versioning fix: http://tracker.ceph.com/issues/21500 Signed-off-by: yuliyang (cherry picked from commit bc16b162cf757b9c6ceae30912eb647ef9304f75) commit a5548a8247cd9b072a2e0357cf6d0358550e7565 Merge: 724609a8a5 1751a094d7 Author: Sage Weil Date: Thu Oct 26 16:52:28 2017 -0500 Merge pull request #18498 from liewegas/wip-recovery-fixes-luminous luminous: osd: fix recovery priority and pg state on recovery->backfill transition Reviewed-by: David Zafman Date: Thu Oct 26 16:50:20 2017 -0500 Merge pull request #18567 from ceph/backport-wip-bz1491250 luminous: ceph-volume lvm list Reviewed-by: Andrew Schoen commit 5f96edfb41c4b028732cf12da5ab44322e251b2a Author: Sage Weil Date: Thu Oct 26 16:23:41 2017 -0500 osd/PG: fix recovery op leak due to recovery preemption This was fixed in master in a different patch, but are not yet ready to backport the bits there that came before this. For now, fix it specifically for luminous. We can either sort out the conflicts later or revert this and backport the master parts conflict-free. Signed-off-by: Sage Weil commit 50c4cc06101fd1c8a9323cd3a04c92c8a9ab2580 Merge: a175c0321e 33aa167331 Author: Sage Weil Date: Thu Oct 26 16:14:43 2017 -0500 Merge pull request #17805 from tchaikov/wip-luminous-pr-17508 luminous: mon: more aggressively convert crush rulesets -> distinct rules Reviewed-by: Sage Weil commit a175c0321e745ac095707d5a2d3cd25232272bfc Merge: 72b057187a b382db7cfc Author: Sage Weil Date: Thu Oct 26 16:14:07 2017 -0500 Merge pull request #18364 from kmroz/wip-21544-luminous luminous: mon: osd feature checks with 0 up osds Reviewed-by: Sage Weil commit 72b057187aed9b4e5f3a4ecf82fafa6e9cd2d3fd Merge: f6362e3110 b60d8cf0cb Author: Sage Weil Date: Thu Oct 26 16:13:31 2017 -0500 Merge pull request #18413 from kmroz/wip-21693-luminous luminous: osd: additional protection for out-of-bounds EC reads Reviewed-by: Josh Durgin commit f6362e3110e4e482ff5a676035b862b1b0b666df Merge: f36670e190 4c6b1b65f7 Author: Sage Weil Date: Thu Oct 26 16:11:42 2017 -0500 Merge pull request #18529 from smithfarm/wip-21918-luminous luminous: Disable messenger logging (debug ms = 0/0) for clients unless overridden. Reviewed-by: Sage Weil commit f36670e190ea96589c373d87d30ee8f4b3c6b20b Merge: 9a28a5c9d7 cec24f99ab Author: Sage Weil Date: Thu Oct 26 16:11:21 2017 -0500 Merge pull request #18540 from liewegas/wip-pr-17894-luminous luminous: ceph.spec.in,debian/rules: change aio-max-nr to 1048576 Reviewed-by: Kefu Chai Reviewed-by: Boris Ranto commit 565b5cd1d68437096e283ccb942d8f108c26d3d4 Author: Alfredo Deza Date: Wed Sep 13 08:23:40 2017 -0400 ceph-volume lvm add listing module to main Signed-off-by: Alfredo Deza (cherry picked from commit ff72fd0ed52ef2d166007c7769c0140424982dc7) commit 58095350953140a1fa4b968c2e0a5774a960e739 Author: Alfredo Deza Date: Tue Oct 3 11:41:10 2017 -0400 doc/ceph-volume add lvm/list to the toctree Signed-off-by: Alfredo Deza (cherry picked from commit d11c80c917f44b33077715b19564aeea30c78e42) commit 3893947d3e04f049a85245750b841adbbcefeac8 Author: Alfredo Deza Date: Tue Oct 3 11:36:06 2017 -0400 doc/ceph-volume create the reference for the list sub-command Signed-off-by: Alfredo Deza (cherry picked from commit 9a52939ef8fc4abe720d2525c59997f4f9760902) commit 0992c3d7c136a58ff38f521fc0978ff036843d27 Author: Alfredo Deza Date: Tue Oct 3 11:35:39 2017 -0400 doc/ceph-volume add create and list to the index Signed-off-by: Alfredo Deza (cherry picked from commit ed5fa519aca1335039bfc32bc060af5181265517) commit 148537cc333d88b75640506c15a1bc221c9e09d6 Author: Alfredo Deza Date: Tue Oct 3 08:59:37 2017 -0400 ceph-volume tests create a unit test module for listing features Signed-off-by: Alfredo Deza (cherry picked from commit 2c2cf547b07f9510146b86d0746190b6d3fead3d) commit 285bc06569c8ed06890d23e9537a9d1106825b7f Author: Alfredo Deza Date: Tue Oct 3 08:59:01 2017 -0400 ceph-volume tests create a small factory fixture Signed-off-by: Alfredo Deza (cherry picked from commit 5a0b17de81770ef3b851a1ff98232d93c78f49f9) commit 1e4e241d9c77796874a003799a0546fc185c5241 Author: Alfredo Deza Date: Mon Oct 2 14:40:41 2017 -0400 ceph-volume tests ensure Volume.as_dict gets populated correctly Signed-off-by: Alfredo Deza (cherry picked from commit c5b933a96d067fb8c7f753a67123e5662398f964) commit a0ab94e8d70ae9fc5d741c47a8f0dcfe033f9a5e Author: Alfredo Deza Date: Mon Oct 2 14:31:57 2017 -0400 ceph-volume lvm.listing initial take on listing command Signed-off-by: Alfredo Deza (cherry picked from commit 2d80190a039798a637383513f292e6aab62cd3f8) commit 454c655f2eb0b1e79ca8e1e1d87ff9c0fe4886da Author: Alfredo Deza Date: Mon Oct 2 12:35:13 2017 -0400 ceph-volume lvm.api be able to return a dictionary repr of a Volume object Signed-off-by: Alfredo Deza (cherry picked from commit 0cae22750f396d1c1664875edb2742c50a41bffd) commit 1390fb76d76b76428e6a9210005dbf01fd3b3a1a Author: Alfredo Deza Date: Mon Oct 2 12:12:57 2017 -0400 ceph-volume process allows to fully mute terminal output on commands Signed-off-by: Alfredo Deza (cherry picked from commit 233f64a4fd90a19c50d57b849ffaa6a25c835f34) commit 00e9ec1fa25ac3950f43ec3cf84f8321b04507eb Author: Alfredo Deza Date: Thu Sep 14 07:35:47 2017 -0400 ceph-volume lvm.api use a helper to get an lv from an arg Signed-off-by: Alfredo Deza (cherry picked from commit 3c5bbbc5822f02a038f093647c5de58bef2fe8f5) commit 26c523c3beeb5da7e855be64a053ac9dca7d10e2 Author: Jason Dillaman Date: Mon Oct 23 14:53:30 2017 -0400 rbd-mirror: strip environment/CLI overrides for remote cluster Fixes: http://tracker.ceph.com/issues/21894 Signed-off-by: Jason Dillaman (cherry picked from commit 70dc22e03a115b130b9c214030d5996276a1634b) commit 9a28a5c9d7ae618f5b79d3727ba155b5066a2653 Merge: 42172a4431 18a99f5f6b Author: Josh Durgin Date: Wed Oct 25 11:45:38 2017 -0700 Merge pull request #18456 from liewegas/wip-21882-luminous luminous: messages/MOSDMap: do compat reencode of crush map, too Reviewed-by: Josh Durgin commit 18a99f5f6b4976f87dcd0d4fe7e34fddd90de22b Author: Sage Weil Date: Wed Oct 25 07:00:14 2017 -0500 qa/suites/upgrade/jewel-x: make sure min compat client is hammer for straw2 Signed-off-by: Sage Weil commit cec24f99ab5e368d2dae7d53628fee09515bd873 Author: chenliuzhong Date: Tue Oct 24 10:54:33 2017 +0800 ceph.spec.in,debian/rules: change aio-max-nr to 1048576 when osd is more than 14 in one host,it report error that aio is not enough. As the default aio-max-nr is 65536, one OSD needs 4096 aios and other programs may use aios. This patch change aio-max-nr to 1048576 when install ceph-osd rpm package and debian package Signed-off-by: chenliuzhong (cherry picked from commit 36326dc7104fc2f20f19d51b6f618a029ba072d7) commit 15a52ba8329831ddf350664e6e6612523f98dcf4 Author: baixueyu Date: Wed Oct 18 11:16:20 2017 +0800 You can find the problem do like this: upload obj to bucket1 s3cmd put obj s3://bucket1 cp obj from bucket1 to bucket2 s3cmd cp s3://bucket1/obj s3://bucket2 del obj from bucket1 and bucket2 s3cmd del s3://bucket1/obj s3cmd del s3://bucket2/obj you can see the pool 'default.rgw.buckets.data', the data of obj can not deleted. ceph df | grep default.rgw.buckets.data Fixes: http://tracker.ceph.com/issues/21819 Signed-off-by: Xueyu Bai baixueyu@inspur.com (cherry picked from commit 2e11545077c485d5c155ab932b6274a8ac863e61) commit 4c6b1b65f7337809682bf907be94fe946909959a Author: Jason Dillaman Date: Thu Oct 19 22:13:36 2017 -0400 common/common_init: disable ms subsystem log gathering for clients The log gathering causes large performance degradation to clients with high message throughputs. This is hopefully a short-term workaround until per-message logging can be replaced with an efficient data recording system for post-incident analysis use-cases. Fixes: http://tracker.ceph.com/issues/21860 Signed-off-by: Jason Dillaman (cherry picked from commit a3a40413f7908b08c40dec4020034cca4a0c4798) commit 42172a443183ffe6b36e85770e53fe678db293bf Merge: f2afb7a0d5 9f49698fab Author: Sage Weil Date: Tue Oct 24 21:36:18 2017 -0500 Merge pull request #18503 from liewegas/wip-21878-luminous luminous: os/bluestore/BlueFS: fix race with log flush during async log compaction Reviewed-by: Varada Kari commit f2afb7a0d590ff55ae11f5bf3b13264a56380ce6 Merge: d0be2106f9 3ad21109ed Author: Sage Weil Date: Tue Oct 24 21:35:29 2017 -0500 Merge pull request #18501 from liewegas/wip-21766-luminous luminous: os/bluestore: handle compressed extents in blob unsharing checks commit d0be2106f96377f059ec08800e32e492ec6f82a3 Merge: d52e59bad7 e6eeb796b4 Author: Sage Weil Date: Tue Oct 24 21:34:52 2017 -0500 Merge pull request #18459 from liewegas/wip-pr-18457-luminous luminous: qa/suites/rados/rest/mgr-restful: whitelist more health commit 902d467a0ec23baebf28a2419a4c0c78e0b1b662 Author: Casey Bodley Date: Mon Oct 9 10:11:14 2017 -0400 qa/rgw: ignore errors from 'pool application enable' Fixes: http://tracker.ceph.com/issues/21715 Signed-off-by: Casey Bodley (cherry picked from commit 160240e0c1c216c950b7210a561b27436198de1f) Signed-off-by: Yuri Weinstein commit 1751a094d76ac84fbcecfc9816904cbb496bd765 Author: Sage Weil Date: Mon Oct 23 17:11:59 2017 -0500 osd/PrimaryLogPG: clear DEGRADED at recovery completion even if more backfill We may have log recovery *and* backfill to do, but cease to be degraded as soon as the log recovery portion is done. If that's the case, clear the DEGRADED bit so that the PG state is not misleading. Signed-off-by: Sage Weil (cherry picked from commit 3e91fda79d3be771f46394312a6b72ab75660012) commit d52e59bad700e8add6cd4cf895817aa8fd7a601a Merge: 724a51cd44 0048e6a58c Author: Kefu Chai Date: Tue Oct 24 11:51:19 2017 +0800 Merge pull request #18491 from liewegas/wip-21573-luminous luminous: buffer: fix ABI breakage by removing list _mempool member Reviewed-by: Kefu Chai commit 9f49698fab5de9ade79d3aba957afe82daa4fecc Author: Sage Weil Date: Fri Oct 20 08:51:17 2017 -0500 os/bluestore/BlueFS: fix race with log flush during async log compaction During async log compaction we rely on _flush-and_sync_log to update the log_writer to jump_to. However, if racing threads are also trying to flush the log and manage to flush our new log events for us, then our flush will turn into a no-op, and we won't update jump_to correctly at all. This results in a corrupted log size a bit later one. Fix by ensuring that there are no in-progress flushes before we add our log entries. Also, add asserts to _flush_and_sync_log to make sure we never bail out early if jump_to is set (which would indicate this or another similar bug is still present). Fixes: http://tracker.ceph.com/issues/21878 Signed-off-by: Sage Weil (cherry picked from commit 4324c8bc7e66633035c15995e3f82ef91d3a5e8c) commit 3ad21109ed2df068bb113e8551505b850fbfcb30 Author: Sage Weil Date: Wed Oct 11 16:48:41 2017 -0500 os/bluestore: handle compressed extents in blob unsharing checks If the blob is compressed, we aren't mapping to a range within the allocated extents, but rather referencing the entire blob. Fixes: http://tracker.ceph.com/issues/21766 Signed-off-by: Sage Weil (cherry picked from commit eb26cfbfbb2b9b9fe84dfe9c999e2d67fd2661c0) commit 8f176f31d091cb53e1baa8c317df9e64bef4d95e Author: Sage Weil Date: Sun Oct 22 22:46:00 2017 -0500 osd/PG: on recovery done, requeue for backfill We were keeping our existing recovery reservation slot (with a high priority) and going straight to waiting for backfill reservations on the peers. This is a problem because the reserver thinks we're doing high priority work when we're actually doing lower-priority backfill. Fix by closing out our recovery reservation and going to the WaitLocalBackfillReserved state, where we'll re-request backfill at the appropriate priority. Signed-off-by: Sage Weil (cherry picked from commit 09c50e2fe4fc3e8951a483e4b2693f0ffc008dcf) commit cc707155ea475836f97791c08c31994fb40e4374 Author: Sage Weil Date: Sun Oct 22 22:43:18 2017 -0500 osd/PG: move local_reserver recovery cancel to Recovering state transition This is easier to follow than canceling the reservation in the next state. Signed-off-by: Sage Weil (cherry picked from commit dde007749bb960122f399b4cb8608a8c1ea4eb9c) commit def29079dc623926fed26a4946d6b9a074dfa2ac Author: Sage Weil Date: Fri Oct 20 22:55:29 2017 -0500 qa/suites/upgrade/jewel-x/parallel: run some jewel after completed upgrade Signed-off-by: Sage Weil commit 8697eca5ea24c4724f637e4d4f75d3e0d2b95278 Author: Sage Weil Date: Fri Oct 20 22:52:45 2017 -0500 qa/suites/upgrade/jewel-x/: set up compat weight-set after cluster upgrade ...but before we run some client workloads. This will hopefully ensure that older clients will behave. Signed-off-by: Sage Weil commit 89ee508ac51839c3d1f71c5e21443e8f2679c59e Author: Sage Weil Date: Sat Oct 21 13:03:17 2017 -0500 mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2' This is a common and recommended step that users should take once they have clients that support straw2 (hammer!). Let's make it easy. This may result in a small amount of data movement, but usually it is very little. Signed-off-by: Sage Weil (cherry picked from commit 928770513cbe3a244095e36dcb41c83c0724cfe9) commit 94a4ca6c8e4d26465cbefd1492ee05f21b08ff1e Author: Sage Weil Date: Sat Oct 21 12:59:16 2017 -0500 crush/CrushWrapper: add bucket_set_alg() Signed-off-by: Sage Weil (cherry picked from commit edc411ee4cb7b88a970ccfc9d29b3a3ba15d066c) commit 0048e6a58c7cdf3b3d98df575bc47db8397cd5a9 Author: Sage Weil Date: Thu Oct 19 16:19:35 2017 -0500 buffer: remove list _mempool member This broke the C++ ABI by changing the list structure size. Also, it's not necessary as we can infer the mempool by looking at the other list contents. We don't (currently) have a need to map an empty list to a particular mempool and have that state stick. Fixes: http://tracker.ceph.com/issues/21573 Signed-off-by: Sage Weil (cherry picked from commit 9b92d87d37e3892b0096cd728b46154aed1e2d86) commit 3cd334b6264998490e6fb8321fb7242ff82b36e6 Author: Sage Weil Date: Thu Oct 19 16:09:10 2017 -0500 buffer: allow mempool to be passed into raw* ctors and create methods This is more convenient, and also faster than initializing it in buffer_anon and the immediately moving it elsewhere. Drop the optionality of the alignment argument. No users yet. Signed-off-by: Sage Weil (cherry picked from commit 481277b8c92368b8cac8fdafe1848720ec1035c2) commit 724a51cd442d35c950414b86608c48936b01b78b Author: Sage Weil Date: Mon Oct 23 12:05:39 2017 -0500 /etc/sysconfig/ceph: remove jemalloc option This breaks when used with rocksdb, which is now the default. See http://tracker.ceph.com/issues/20557 Signed-off-by: Sage Weil (cherry picked from commit 19efccef248c2d66fe3d7fdac8f56d455f2c7b27) commit 95b6bc9ac41c4985ad97a515c4105351a0e6b94d Author: Sage Weil Date: Mon Oct 23 11:16:26 2017 -0500 etc/default/ceph: remove jemalloc option This breaks when used with rocksdb, which is now the default. See http://tracker.ceph.com/issues/20557 Signed-off-by: Sage Weil (cherry picked from commit fba2055b4e54128db1e9a83f27bb3526d65d4db7) commit e6eeb796b4d24fcf6797519be06aca9771e73028 Author: Sage Weil Date: Fri Oct 20 23:14:36 2017 -0500 qa/suites/rados/rest/mgr-restful: whitelist more health The test is fiddling with OSDs. Signed-off-by: Sage Weil (cherry picked from commit ddf3e9162da542af0c5f025957b8304e7359c924) commit 33aa1673316b77b71e473d1ab71cf61587e425a6 Author: Sage Weil Date: Tue Sep 5 22:46:48 2017 -0400 mon/OSDMonitor: improve crush map validation - move into OSDMap method - ensure that rules exist for each pool - ensure pool type matches rule type - ensure rule mask min/max size cover the pool size Signed-off-by: Sage Weil (cherry picked from commit 8f8f0b25058ff7319ec9d023f0eceb5832d0ecff) commit d825de7681ba1cff257ba4108d6c22c81e3a6aab Author: John Spray Date: Fri Aug 25 11:06:21 2017 +0100 mon: more forceful renumbering of legacy ruleset IDs Previously, the rules were only modified in the trivial case, so we continued to potentially have CRUSH maps with the legacy ruleset functionality in use. In order to ultimately remove rulesets entirely, we need to do this more aggressively, renumbering all the rules and then updating any pools as needed. Signed-off-by: John Spray (cherry picked from commit 71d4b2bed54371657693cfb999ade44449be0efd) Conflicts: src/mon/OSDMonitor.cc: the check for multiple rules was removed in master, but not in luminous. once we renumber the legacy ruleset IDs, it's not need to check for and to warn the user at seeing the case where 1-to-n mapping from ruleset to rule IDs. commit f52d1ad23d97e1ab8c70f9b17b75daa0773698b7 Author: John Spray Date: Fri Aug 25 10:42:58 2017 +0100 osd: s/crush_ruleset_in_use/crush_rule_in_use/ Signed-off-by: John Spray (cherry picked from commit 675483ecca06686fd4c626bfea33748092aef0c6) commit 17d73e5a2e592719df8099450c9cd5f46d6fd582 Merge: d4158133f7 e25a583323 Author: Sage Weil Date: Fri Oct 20 23:10:49 2017 -0500 Merge pull request #17773 from xiexingguo/wip-luminous-object-legacy-flag luminous: osd/PrimaryLogPG: _delete_oid - fix incorrect 'legacy' flag Reviewed-by: Kefu Chai commit d4158133f7586472d01d9c6e669db92ddcd2e73d Merge: 84fbb0ff70 8b4fe14ba0 Author: Sage Weil Date: Fri Oct 20 23:10:06 2017 -0500 Merge pull request #18362 from kmroz/wip-21543-luminous luminous: qa/tasks: prolong revive_osd() timeout to 6 min commit 84fbb0ff705df5bc9f164dd5a02bd5a82e6478d7 Merge: 87ab8d5c80 993d098505 Author: Sage Weil Date: Fri Oct 20 23:09:52 2017 -0500 Merge pull request #18388 from tchaikov/wip-pr-18346-luminous os/filestore: print out the error if do_read_entry() fails Reviewed-by: xie xingguo commit 87ab8d5c80374a73ad95f9ed9dd8826c54e64e9f Merge: 00daa8462a fd42f2a9cc Author: Sage Weil Date: Fri Oct 20 23:09:17 2017 -0500 Merge pull request #18398 from smithfarm/wip-21783-luminous luminous: cli/crushtools/build.t sometimes fails in jenkins' make check run Reviewed-by: Kefu Chai commit 00daa8462a5a337ecc57bd8664bfcc748befeead Merge: 5a4850c8c6 112069c70d Author: Sage Weil Date: Fri Oct 20 23:08:58 2017 -0500 Merge pull request #18401 from liewegas/wip-bluestore-label-luminous luminous: os/bluestore: ceph-bluestore-tool repair updates and prime-osd-dir support commit dbddf21bd3aac284122fd5fc84abdd4c89b12b3e Author: Sage Weil Date: Fri Oct 20 22:32:33 2017 -0500 messages/MOSDMap: do compat reencode of crush map, too If we are reencoding an incremental, and it embeds a crush map, we need to reencode that in a compatible way too. This is especially true now because we have the compat crush weight-sets. Otherwise, a client may learn the crush map through an incremental but not understand choose_args, and not see/understand the alternate weight set. It will send requests to the wrong OSDs where they will just get dropped. Fixes: http://tracker.ceph.com/issues/21882 Signed-off-by: Sage Weil (cherry picked from commit 9eaafc66278d59447b29ce6cc378d0cd3ad14ad4) commit c5ccbf1108532547f54f084b8f06c7f5ddf8a4e8 Author: Ramana Raja Date: Sun Sep 24 20:23:12 2017 +0530 ceph_volume_client: fix setting caps for IDs ... that have empty OSD and MDS caps. Don't add a ',' at the start of OSD and MDS caps. Fixes: http://tracker.ceph.com/issues/21501 Signed-off-by: Ramana Raja (cherry picked from commit baf3b88800c63ef7467abbc7b54e40c0da669f38) commit b382db7cfce84adc897cd68e4cd7a4eb1b40a9f1 Author: Brad Hubbard Date: Wed Sep 20 13:15:30 2017 +1000 mon/OSDMonitor: mon osd feature checks with 0 up osds get_up_osd_features() returns 0 if no osds are up which causes feature checks to fail. Fixes: http://tracker.ceph.com/issues/21471 Signed-off-by: Brad Hubbard (cherry picked from commit 26ba0ba044846d18f9bdfdbee6e2a9c46882c742) Conflicts: src/mon/MonCommands.h - removed mimic reference src/mon/OSDMonitor.cc - removed mimic-only command path commit c845db3f8c2a0c2494fbdc9d6cc63c8e0c3b0563 Author: Sage Weil Date: Tue Sep 19 18:25:56 2017 -0400 osd/OSDMap: ignore xinfo if features == 0 Some old bug (e.g., http://tracker.ceph.com/issues/20751) could result in an UP+EXISTS osd having features==0. If that happens, we shouldn't crash the mon, which (reasonably) does if (osdmap.get_epoch()) { if (osdmap.get_num_up_osds() > 0) { assert(osdmap.get_up_osd_features() & CEPH_FEATURE_MON_STATEFUL_SUB); check_subs(); } } Signed-off-by: Sage Weil (cherry picked from commit 8399833f3d221364a8ededb97cd4e1c5a258d924) commit e58656005d681ee0e35a25b324321cfeea8c7b3a Author: Matt Benjamin Date: Tue Oct 17 17:45:37 2017 -0400 rgw_file: explicit NFSv3 open() emulation Expect RGW_OPEN_FLAG_V3 in rgw_write(), and if present, attempt a stateless open inline. Required by Ganesha v2.5 and later Signed-off-by: Matt Benjamin (cherry picked from commit 6ed23b4a0cea7e41b4743e27319737af693947ec) commit f9850d2148b396032634f7ffa18ae458bd208195 Author: Adam C. Emerson Date: Thu Sep 28 13:54:32 2017 -0400 rgw: Check bucket Website operations in policy Add code to check s3:GetBucketWebsite and s3:PutBucketWebsite operations against bucket policy. Fixes: http://tracker.ceph.com/issues/21597 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493896 Signed-off-by: Adam C. Emerson (cherry picked from commit ceed535957ac186e241fcff26b103cf7efa959b1) commit abefb578c345a0ddd7100cae987e582b530d3729 Author: Adam C. Emerson Date: Wed Sep 27 15:42:27 2017 -0400 rgw: Check bucket CORS operations in policy Add code to check s3:GetCORS and s3:PutCORS operations against bucket policy. Fixes: http://tracker.ceph.com/issues/21578 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494140 Signed-off-by: Adam C. Emerson (cherry picked from commit 27eb13fe568cc802feaf69131a21db076bcb6746) commit 0b84dd07c41a6d69d99672218c72a4f69c21b040 Author: Adam C. Emerson Date: Wed Sep 27 16:08:56 2017 -0400 rgw: Check bucket GetBucketLocation in policy Add code to check s3:GetBucketLocation against bucket policy. Fixes: http://tracker.ceph.com/issues/21582 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493934 Signed-off-by: Adam C. Emerson (cherry picked from commit 79188d679edeb6e2f7ca852fdc4224368412cb72) commit 0b3a974be7bb9cd1d25ee9c2b861f38ea721c66b Author: Casey Bodley Date: Wed Sep 27 14:26:25 2017 -0400 rgw: include SSE-KMS headers in encrypted upload response Fixes: http://tracker.ceph.com/issues/21576 Signed-off-by: Casey Bodley (cherry picked from commit a5b3db7acb4150d9d27ba42f72658e418dd444c0) commit a3e2ae4925e7e25c213ef37072d576d1f1ffdc13 Author: Abhishek Lekshmanan Date: Fri Sep 22 16:11:55 2017 +0200 rgw: defer constructing keystone engine unless url is configured currently we create a keystone revocation thread even when keystone url is empty, lets defer the construction of keystone unless the urls are configured Signed-off-by: Abhishek Lekshmanan (cherry picked from commit 104c2f59011e6b515e3845cce529ee44334e76c0) commit 4c18ac124048f72a10d92aecfe1eadabab0a80b4 Author: Adam C. Emerson Date: Wed Sep 27 14:35:59 2017 -0400 rgw: Check bucket versioning operations in policy Add code to check s3:GetBucketVersioning and s3:PutBucketVersioning operations against bucket policy. Fixes: http://tracker.ceph.com/issues/21389 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278 Signed-off-by: Adam C. Emerson (cherry picked from commit 16de0fc1c5ede961ebd481f13753214a971c206c) commit be1ded107ac4e1a1b1515d40553aed3581e0afac Author: Adam C. Emerson Date: Thu Sep 14 18:07:15 2017 -0400 rgw: Check payment operations in policy Add code to check s3:GetBucketRequestPayment and s3:PutBucketRequestPayment operations against bucket policy. Fixes: http://tracker.ceph.com/issues/21389 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278 Signed-off-by: Adam C. Emerson (cherry picked from commit f9d1ae1d153319e870c3ccaf7afdc92786cdaa3b) commit 92d0545781ca2bff39f7bd8e9d63bb8c2d8cb997 Author: Nathan Johnson Date: Thu Sep 7 23:27:40 2017 -0500 rgw: admin api - add ability to sync user stats from admin api Fixes: http://tracker.ceph.com/issues/21301 Signed-off-by: Nathan Johnson (cherry picked from commit 828412d573a0b31387fc9f6a879f9a6083535730) commit d800747fa18a1872f1d748119bfcc7eff34ae6cf Author: gaosibei Date: Tue Sep 26 10:19:22 2017 +0800 RGW: fix a bug about inconsistent unit of comparison Fixes：http://tracker.ceph.com/issues/21590 Signed-off-by: gaosibei (cherry picked from commit cb39f065d6eba87cf1fc2e99334322bf63092df3) commit 14a1dcb1ff70b2a468cb47e7f239b98e45c63a88 Author: Mark Kogan Date: Mon Sep 25 09:53:00 2017 +0300 rgw: update the usage read iterator in truncated scenario Fixes: http://tracker.ceph.com/issues/21196 Signed-off-by: Mark Kogan (cherry picked from commit 7306514a813661b77bfcbfc6f534dbabbdd3aa78) commit 8b420c4627faca27f61f472597c6f67cca2efa2f Author: Casey Bodley Date: Thu Oct 5 16:39:30 2017 -0400 rgw: RGWUser::init no longer overwrites user_id if an admin op specifies a user_id and does not find a user with that id, but does find a user based on a later field (email, access key, etc), RGWUser::user_id will be overwritten with the existing user's id when this happens on 'radosgw-admin user create', RGWUser::execute_add() will modify that existing user, instead of trying to create a new user with the given user_id (and failing due to the conflicting email, access key, etc) by preserving the original user_id (when specified), this uid conflict is detected in RGWUser::check_op() and a "user id mismatch" error is returned Fixes: http://tracker.ceph.com/issues/21685 Signed-off-by: Casey Bodley (cherry picked from commit 4e65c9e9af86e832cb96d57d487771aa28251e7c) commit fe45086a54d76cd9b913663d4537b36a8a276ed4 Author: gaosibei Date: Tue Sep 26 10:44:09 2017 +0800 RGW: Multipart upload may double the quota Fixes: http://tracker.ceph.com/issues/21586 Signed-off-by: Sibei Gao (cherry picked from commit 97f95e457fb7f9e36031f41e0f2dd3955daedd87) commit c3a719da278a830cc1b8101e3d8ef25316da33d7 Author: Casey Bodley Date: Thu Oct 12 13:26:29 2017 -0400 rgw: 'zone placement' commands validate compression type Fixes: http://tracker.ceph.com/issues/21775 Signed-off-by: Casey Bodley (cherry picked from commit 33f8ee8b6fa5605edbbfe12d8e097123e509ad9f) commit 352373ff4a333c9b3869c9f168ee14530067907a Author: Yao Zongyou Date: Sun Sep 24 22:11:22 2017 +0800 rgw_file: fix write error when the write offset overlaps. Signed-off-by: Yao Zongyou (cherry picked from commit 872d73f945364002f0fa31762e6976db5b4b3c19) commit 248bd94dab4359dfea34bfa0cbdd16d9a9b76a02 Author: Orit Wasserman Date: Mon Oct 9 13:25:36 2017 +0300 rgw: disable dynamic resharding in multisite environment Fixes:http://tracker.ceph.com/issues/21725 Signed-off-by: Orit Wasserman (cherry picked from commit f1ec7a49bde054a19a0a8fd80f37866e1e40379f) commit 624b3ac5053dbef008d2e9238c6958d07206f6c3 Author: Zhang Shaowen Date: Tue Sep 26 10:00:59 2017 +0800 rgw: Torrents are not supported for objects encrypted using SSE-C Fixes: http://tracker.ceph.com/issues/21720 Signed-off-by: Zhang Shaowen (cherry picked from commit 9a8ae664da54eb74e74a12cc16a52b0df7df70a0) commit 0e65ea466f05a2bc8825e51b32901d74082c1b65 Author: Matt Benjamin Date: Tue Oct 3 17:48:29 2017 -0400 rgw: release cls lock if taken in RGWCompleteMultipart Follows Casey's proposal to conditionally release the lock in ::complete(), in order to avoid duplicated code in various early return cases. Fixes: http://tracker.ceph.com/issues/21596 Signed-off-by: Matt Benjamin (cherry picked from commit 704f793f08a02760d23eb5778b738bb07be0e7cf) commit 17df0ba8f00abbf4b30e0370090251b0c7f6b643 Author: Enming Zhang Date: Sat Aug 26 04:46:35 2017 +0800 rgw: encryption add exception handling for from_base64 on bad input If calling from_base64 with charactors that is not base64 encoded, rgw will crash. Signed-off-by: Enming Zhang (cherry picked from commit f0b88e51be86d84793b794f6fe87fdda62004a4d) commit 46f09e3803ba3c4b463bfa6a2eeb40a0bd7cc0ef Author: Enming Zhang Date: Fri Aug 25 19:48:53 2017 +0800 rgw: encryption fix the issue when not provide encryption mode Now, in RGW, if someone want to upload an object using server-side encryption with providing customer key or kms key id, but not specify the encryption mode in the "x-amz-server-side-encryption-customer-algorithm" or "x-amz-server-side-encryption", the object will be uploaded successfully without encryption. This is not a correct way to deal with it. It is better to return error. Fixes: http://tracker.ceph.com/issues/21581 Signed-off-by: Enming Zhang (cherry picked from commit f8c4be8a8943096217d89f7c5abd129fa2414b36) commit fefbc9fec47cc6858f74b7ea372d39efeb95c61f Author: Enming Zhang Date: Fri Aug 25 19:47:21 2017 +0800 rgw: encryption SSE-KMS add the details of error msg in response Signed-off-by: Enming Zhang (cherry picked from commit 3f16785daf4b3609f2bff7c107a35f908103e32a) commit 2e493e1f7a1f20d2f33e66816cc72d6b5f193b61 Author: Enming Zhang Date: Fri Aug 25 19:37:52 2017 +0800 rgw: encryption SSE-C add the details of error msg in response Signed-off-by: Enming Zhang (cherry picked from commit 9502549ac2133e969a5f268601b92fd4063c1bd9) commit fe0a41b4a921ac3510b344a6edf6e930a2fa2465 Author: Jason Dillaman Date: Thu Oct 12 08:58:03 2017 -0400 cls/journal: fixed possible infinite loop which could kill the OSD Fixes: http://tracker.ceph.com/issues/21771 Signed-off-by: Jason Dillaman (cherry picked from commit 9e7475f8fb08b0ef30cad2c0d680cecf331b44ef) commit e0d57c544d55f3693f858464f7995b4b30cfa88d Author: Jason Dillaman Date: Thu Oct 12 08:56:57 2017 -0400 test: ceph_test_cls_journal was dropped when converting to cmake Signed-off-by: Jason Dillaman (cherry picked from commit 5d9cfebabac33b645c6199bda63ff4619e6f538f) commit f8bfd9458d44d96d4ef56a28195a9e36fc0190c6 Author: Jason Dillaman Date: Sun Oct 15 17:26:25 2017 -0400 librbd: batch large object map updates into multiple chunks Fixes: http://tracker.ceph.com/issues/21797 Signed-off-by: Jason Dillaman (cherry picked from commit 04c5d12acc045731fbf1d0ddce276b5743d9fd89) commit 6866abc01be5fd92acc918472a604a3af9894b44 Author: Jason Dillaman Date: Fri Oct 13 18:55:47 2017 -0400 test/librbd: initial test cases for trim state machine Signed-off-by: Jason Dillaman (cherry picked from commit 4674b578d8233da3b3c145606ed9c0e4e88a540a) commit 3b87dea6584010ae05e540825867c9961708ff36 Author: Jason Dillaman Date: Fri Oct 13 15:49:00 2017 -0400 librbd: tweaks to support testing of trim state machine Signed-off-by: Jason Dillaman (cherry picked from commit 3220480d4a22c81cb74f25e528f2bcf3986342c1) commit ce628329641ac6322d5c59055e8db9eecafbfef4 Author: Jason Dillaman Date: Fri Oct 13 14:44:40 2017 -0400 librbd: combine trim state machine object map batch update states The PRE/POST states were previously divided into two halves for handling the copy-up batch and the direct removal batch. This can be simplified by just using a single PRE/POST for the entire deletion region. Signed-off-by: Jason Dillaman (cherry picked from commit 72ce4576fa2b562799a5bc78bd423cfabe097d67) Conflicts: src/librbd/operation/TrimRequest.h commit 80ff76db5495ae41baf6d4292b6b10c2f789c155 Author: Jason Dillaman Date: Fri Oct 13 11:06:38 2017 -0400 cls/rbd: object map update now utilizes constant-time bit vector operations Signed-off-by: Jason Dillaman (cherry picked from commit fc99603ea2b5b268181d38507b8b6fb55ae0a6d0) commit 5dc46378370da8eede9958a88d6764737d55b09b Author: Jason Dillaman Date: Fri Oct 13 11:05:48 2017 -0400 common/bit_vector: provide constant time iteration of underlying bufferlist Signed-off-by: Jason Dillaman (cherry picked from commit daa29f7d2b50bc4b44a108a0f91d45ce90fc16e5) commit ae4ebaca231e4ae94efc8d445fee792f76e656e1 Author: Jason Dillaman Date: Fri Oct 13 11:03:25 2017 -0400 common/buffer: expose hidden const deference operator Signed-off-by: Jason Dillaman (cherry picked from commit f35947e3c6661dfe9f951896bc54bc8da158b490) commit b60d8cf0cbe56500af98b3d3857245e94d904b19 Author: Jason Dillaman Date: Mon Oct 2 17:28:04 2017 -0400 osd: additional protection for out-of-bounds EC reads Fixes: http://tracker.ceph.com/issues/21629 Signed-off-by: Jason Dillaman (cherry picked from commit 881f2a02aecef0cd6bf8ec3e5045b76c8441e832) commit 8f87fa2d3af44840a3044cd736a2f8da0b602b07 Author: John Spray Date: Sat Sep 23 13:21:47 2017 -0400 mgr: fix crashable DaemonStateIndex::get calls This function was recently fixed to return null on missing entries: handle that properly. Fixes: http://tracker.ceph.com/issues/17737 Signed-off-by: John Spray (cherry picked from commit ade4827d86bb2bc79466a2da040475651c2aae0d) commit def3d55eb71e0c62b7fd523865c2ba196342aa1a Author: huangjun Date: Fri Oct 6 11:58:50 2017 +0800 qa/suites/rest/basic/tasks/rest_test: whiltelist OSD_DOWN Fixes: http://tracker.ceph.com/issues/21425 Signed-off-by: huangjun (cherry picked from commit 838e12cfc84b386fd4eae09e526b078286f0771e) commit 20c9e641f7096bf3e208b4d6d28494603eeba891 Author: huangjun Date: Tue Sep 19 16:13:34 2017 +0800 qa/suites/rest/basic/tasks/rest_test: more whitelisting Fixes: http://tracker.ceph.com/issues/21425 Signed-off-by: huangjun (cherry picked from commit 4d30d02e522039461762d51a3a826a337214ca7e) commit 5a4850c8c66b75d02fc2597e1ca744c6b0303836 Merge: 17c6f35703 03e2a68e43 Author: Sage Weil Date: Thu Oct 19 16:43:39 2017 -0500 Merge pull request #17692 from smithfarm/wip-21176-luminous build/ops: python-numpy-devel build dependency for SUSE Reviewed-by: Abhishek Lekshmanan Reviewed-by: Amit Kumar commit 17c6f357030315040f8af95e751921c475b32289 Merge: 97b30154f8 5aa445897c Author: Sage Weil Date: Thu Oct 19 16:42:45 2017 -0500 Merge pull request #17730 from xiexingguo/wip-pr-17371 luminous: mon, osd: per pool space-full flag support Reviewed-by: Sage Weil commit 97b30154f884f4ea987e3330799bf412b08ba3e4 Merge: 0a8f1540ba fb7e6a581d Author: Sage Weil Date: Thu Oct 19 16:42:07 2017 -0500 Merge pull request #17779 from smithfarm/wip-rh-74-luminous tests: CentOS 7.4 is now the latest Reviewed-by: Ken Dreyer commit 0a8f1540ba08bb79e9ec8f3358490a31be734b2b Merge: 5ca7af785a d7cde5f316 Author: Sage Weil Date: Thu Oct 19 16:41:34 2017 -0500 Merge pull request #17863 from smithfarm/wip-21343-luminous luminous: mon: DNS SRV default service name not used anymore commit 5ca7af785a6695121f841909a6118a5103795d99 Merge: 37d7cfc9f1 49e030da55 Author: Sage Weil Date: Thu Oct 19 16:40:45 2017 -0500 Merge pull request #17864 from smithfarm/wip-21438-luminous luminous: core: Daemons(OSD, Mon...) exit abnormally at injectargs command Reviewed-by: Kefu Chai commit 37d7cfc9f136c9604a1e53604a8e0d939eee60db Merge: 223b0956b9 077b3601d6 Author: Sage Weil Date: Thu Oct 19 16:40:18 2017 -0500 Merge pull request #17866 from smithfarm/wip-21320-luminous luminous: mgr: Quieten scary RuntimeError from restful module on startup commit 223b0956b93504881a5f06d2589a0b138abf37a9 Merge: 4f1a7a82a1 82651da2d5 Author: Sage Weil Date: Thu Oct 19 16:39:53 2017 -0500 Merge pull request #17951 from smithfarm/wip-21548-luminous luminous: tests: ceph_manager: bad AssertionError: failed to recover before timeout expired commit 4f1a7a82a1134d6688537653ac07d924275e1f3a Merge: 37b13acabe a670921f4f Author: Sage Weil Date: Thu Oct 19 16:39:32 2017 -0500 Merge pull request #18342 from xiexingguo/wip-pr-18318 luminous: bluestore: using bluestore_compression_min_blob_size instead of bluestore_compression_max_blob_size to config comp_min_blob_size Reviewed-by: Sage Weil commit 112069c70df5c7719af9291a8de2b4fcf48d049a Author: Sage Weil Date: Mon Oct 16 17:46:03 2017 -0500 os/bluestore: fsync on _write_bdev_label Signed-off-by: Sage Weil (cherry picked from commit 7aca8af6625156eeaa936629f83ede33d3bbe866) commit 1e9aa60e3f6acca6649f5d5d51421cc2bd51e55a Author: Sage Weil Date: Mon Oct 16 17:45:54 2017 -0500 ceph-bluestore-tool: implement 'set-label-key' and 'rm-label-key' Signed-off-by: Sage Weil (cherry picked from commit 4b468c989f9e8609e7f640cfa4385c454ae6dc17) commit 170499a1e570bc110be08ca2a38063229fee75dc Author: Sage Weil Date: Mon Oct 16 17:42:29 2017 -0500 os/bluestore: make _write_bdev_label static and public Signed-off-by: Sage Weil (cherry picked from commit 0b0550c967ae0d6ae49641c6859f0b63267e2cd9) commit 895a277568f902f1c00b9df475ea56b00b115841 Author: Sage Weil Date: Thu Oct 12 11:40:11 2017 -0500 doc/man/8/ceph-bluestore-tool: add man page Signed-off-by: Sage Weil (cherry picked from commit 7b91e50dbd289e65f10cb98c5eea9f3171c2d1f1) commit 24b3b7960e5c5d118aacb63753c794466e801f84 Author: Sage Weil Date: Fri Oct 13 09:20:53 2017 -0500 common/options: document 'key', 'keyfile', 'keyring' Signed-off-by: Sage Weil (cherry picked from commit cdee8b26b9d93189dcd2875038361fb59ed21ba2) commit 508889d4a7aa93f3bbe13ad8bfc181f3fc9ed2a9 Author: Sage Weil Date: Thu Oct 12 13:54:06 2017 -0500 ceph-bluestore-tool: fix show-label to use dev name as key Signed-off-by: Sage Weil (cherry picked from commit 4785345ace7b675a0e1c34c457aad1103aab5a1d) commit 6e17557f288aa7686b8b68c1df31a9a8c26912ac Author: Sage Weil Date: Thu Oct 12 08:48:59 2017 -0500 ceph-bluestore-tool: drop "action ..." print Signed-off-by: Sage Weil (cherry picked from commit 5e486c2bd3414f4ee6519574d4d4584618144867) commit a128aad320e99c396ff17bcaadb48866667b87df Author: Sage Weil Date: Thu Oct 12 08:48:46 2017 -0500 ceph-bluestore-tool: only infer devs if devs.empty() Signed-off-by: Sage Weil (cherry picked from commit 2d217367af0df5e18394b1c6179d92db3c0b4448) commit 24d86603faf4a826684a5e124ed9b447de0b50e8 Author: Sage Weil Date: Wed Oct 11 17:36:40 2017 -0500 ceph-bluestore-tool: implement prime-osd-dir This populates an osd dir based on the metadata in a bluestore device label. Signed-off-by: Sage Weil (cherry picked from commit 72ee7e29fb77f581a537b1ff2b274c9f3a7de53f) commit 0f43e6907cfd27f2d6791b896480f2c537115212 Author: Sage Weil Date: Wed Oct 11 16:33:54 2017 -0500 os/bluestore: keep meta items for extra bdev paths Signed-off-by: Sage Weil (cherry picked from commit 8d07fa5ad0772b4d979dc10695d230074d1f1e69) commit bb9b70c33a3e013ef78d931fd865ab44c255d772 Author: Sage Weil Date: Wed Oct 11 17:33:00 2017 -0500 vstart.sh: specify block files This is mostly just to test ceph-bluestore-tool prime-osd-dir. Signed-off-by: Sage Weil (cherry picked from commit d4e0c46d89c2ebe6ae5f9af1c909a09daaea9687) commit ce83461a60cea6026a7abdd51b2a7a35302392c6 Author: Sage Weil Date: Wed Oct 11 16:32:16 2017 -0500 osd: store osd cephx key in "osd_key" meta slot Note that this *only* works with --mkkey (deprecated except for vstart.sh) and --key . If you write out a full keyring before running --mkfs it doesn't get populated. Users should pass --key instead of --keyring in that case. Signed-off-by: Sage Weil (cherry picked from commit 65b707358e5442eab0fd6ba4d05d9f0a7adc969f) commit 60d43cb51d7b05cc778cda6b5fcd368db618e3db Author: Sage Weil Date: Wed Oct 11 16:17:09 2017 -0500 ceph-osd: mkkey before mkfs Signed-off-by: Sage Weil (cherry picked from commit 75400ead3061ba1e2820878783fe7b1ddcd9a5fb) commit faad6afa3441b5ce9455a4de0535baef78b8e7de Author: Sage Weil Date: Wed Oct 11 15:53:56 2017 -0500 os/bluestore: store meta values in main device label Signed-off-by: Sage Weil (cherry picked from commit 33b8e8e86f6bdbee120729561c6cf433d286e2f8) commit 1027e57d9af374f1fa1dee2a585dfe539dc4f5eb Author: Sage Weil Date: Sun Sep 10 15:37:10 2017 -0400 os/bluestore: fsck: fix error prefixes __func__ is now _fsck instead of fsck Signed-off-by: Sage Weil (cherry picked from commit 7d381af7aed14e08823eca4d0b12de14456ef6a1) commit 68642c43384d5a30155cd67129267d38d6cc3a06 Author: Sage Weil Date: Thu Sep 7 12:29:32 2017 -0400 os/bluestore: allow repair invocation Signed-off-by: Sage Weil (cherry picked from commit 659fcd279ae82c90172fd6ca07a032d75fe50c96) commit fd42f2a9ccc5016660b9c205f6aa0104ffdd0b10 Author: Kefu Chai Date: Wed Oct 11 17:45:19 2017 +0800 crushtool: print error message to stderr not dout(1) in hope to fix the mysterious test failure where cli/crushtool/build.t prints nothing when error message is expected. Fixes: http://tracker.ceph.com/issues/21758 Signed-off-by: Kefu Chai (cherry picked from commit dc78d40af77ff8d5e4f3d39497e6e3c98c309306) commit 6902627cb3cb41547be5134aba5c10a39b78163a Author: Sage Weil Date: Sat Jul 22 23:51:47 2017 -0400 crush/CrushWrapper: fix output arg for find_{takes,roots}() Signed-off-by: Sage Weil (cherry picked from commit 60b9cfafc3128cc0cb1f89137221fcc46fcd3802) commit 993d09850580ca7a697c1a221a8e258f4cd64237 Author: Kefu Chai Date: Tue Oct 17 17:08:59 2017 +0800 os/filestore: print out the error if do_read_entry() fails Signed-off-by: Kefu Chai (cherry picked from commit 42e85fe35d2fe8f3e99bd110021fd5157cf589d7) commit 410434b3d23c54971f3e2c4df179b139979a33c0 Author: Yan, Zheng Date: Thu Oct 19 11:32:38 2017 +0800 mds: fix race in PurgeQueue::wait_for_recovery() After PurgeQueue::wait_for_recovery() locks the mutex, purge queue may have already been recovered. Signed-off-by: "Yan, Zheng" commit ee96c34a4d1fc48dec6a1afb4fe98634805c02ce Author: Yan, Zheng Date: Wed Sep 27 19:59:47 2017 +0800 mds: open purge queue when transitioning out of standby replay MDS opens the purge queue when it starts standby replay. This is wrong because purge queue may change during standby replay. Fixes: http://tracker.ceph.com/issues/19593 Signed-off-by: "Yan, Zheng" (cherry picked from commit e515e84f69eeab90ea5c5831f7d3e684e48fb62e) commit 681cddfa420a0f5f4335564843cdfdb1a4f2371d Author: Yan, Zheng Date: Wed Sep 27 18:49:55 2017 +0800 mds: always re-probe mds log when standby replay done Signed-off-by: "Yan, Zheng" (cherry picked from commit 82c1f5fcfd151aa1e8c39c7f198abec9bf807b65) commit 37b13acabe3a0cbdcd88c06f49548146d2c69c14 Merge: 90b6067f6f f30e9a5e6b Author: vasukulkarni Date: Tue Oct 17 15:47:31 2017 -0700 Merge pull request #18363 from ceph/wip-yuriw-21822-luminous tests - ceph-ansible vars additions commit f30e9a5e6bd47129f8a35c1f9067216cfe8a6a70 Author: Yuri Weinstein Date: Tue Oct 17 13:14:36 2017 -0700 tests - ceph-ansible vars additions added symlinks for distros Fixes http://tracker.ceph.com/issues/21822 Signed-off-by: Yuri Weinstein commit 90b6067f6f5cbfaf48afb08bd17cb9e5f46a5af9 Merge: 1718fd969c ab53002186 Author: Sage Weil Date: Tue Oct 17 16:31:25 2017 -0500 Merge pull request #17975 from dillaman/wip-systemd-rbd-mirror-luminous luminous: systemd: rbd-mirror does not start on reboot Reviewed-by: Sébastien Han Reviewed-by: Mykola Golub commit 1718fd969c32cc09cfc208a18455291ab2886e93 Merge: 345286b775 9d12fa5e6d Author: Sage Weil Date: Tue Oct 17 16:30:46 2017 -0500 Merge pull request #18234 from jdurgin/wip-filestore-rocksdb-luminous filestore: set default readahead and compaction threads for rocksdb Reviewed-by: Sage Weil Reviewed-by: Amit Kumar commit 345286b775fd269eb9f13cfcb8d0aab533dbfbc6 Merge: 9223ace029 ab644b89e0 Author: Sage Weil Date: Tue Oct 17 16:29:38 2017 -0500 Merge pull request #18236 from tchaikov/wip-21609-luminous luminous: mon,osd: fix "pg ls {forced_backfill, backfilling}" Reviewed-by: Sage Weil commit 9223ace02909d6551f6126181d11bf00bab1cfe3 Merge: fdefc8094e 099801e616 Author: Sage Weil Date: Tue Oct 17 16:29:08 2017 -0500 Merge pull request #18257 from smithfarm/wip-21699-luminous luminous: mgr status module uses base 10 units Reviewed-by: John Spray commit 8b4fe14ba0e9f8e1b09ae4112dbebd370136bd3f Author: Kefu Chai Date: Fri Sep 22 10:58:40 2017 +0800 qa/tasks: prolong revive_osd() timeout to 6 min bluestore_fsck_on_mount and bluestore_fsck_on_mount_deep are enabled by default. and bluestore is used as the default store backend. it takes longer to perform the deep fsck with verbose log. so prolong the revive_osd()'s timeout from 150 sec to 360 sec. Fixes: http://tracker.ceph.com/issues/21474 Signed-off-by: Kefu Chai (cherry picked from commit 42be200c564184feff1056b7771ce3c1b2cc5527) commit a670921f4fb2e15f5ea44e9dda238c2eef96897c Author: linbing Date: Mon Oct 16 09:57:23 2017 +0800 bluestore: using bluestore_compression_min_blob_size instead of bluestore_compression_max_blob_size to config comp_min_blob_size Signed-off-by: linbing (cherry picked from commit da8dc4d4c60f5fd66f6eb2ba7257c52948f24681) commit fdefc8094ed639c859db5b4eba858cfe05329e5c Merge: bac56cccf9 3a94d69264 Author: Kefu Chai Date: Tue Oct 17 15:19:01 2017 +0800 Merge pull request #18339 from tchaikov/wip-gen-state-diagram-luminous doc: build with multiple-line state transition function signature Reviewed-By: Nathan Cutler commit bac56cccf9e3fb44a0ed85c151a8b1a0fb5fa4b0 Merge: 61aca44033 37ddfc61c3 Author: Kefu Chai Date: Tue Oct 17 12:44:04 2017 +0800 Merge pull request #17835 from rzarzynski/wip-rgw-per-storage-policy-stats-luminous luminous: rgw: add support for Swift's per storage policy statistics Reviewed-by: Matt Benjamin Reviewed-by: Casey Bodley commit 3a94d6926403ac1f55a9d5723f6b07486f35060b Author: Alfredo Deza Date: Fri Oct 6 08:57:01 2017 -0400 doc parse two lines for getting the context So that signatures can get parsed when they are split like: PG::RecoveryState::RepWaitBackfillReserved::react( const RemoteReservationCanceled &evt) Signed-off-by: Alfredo Deza (cherry picked from commit 9e2a52ccec4a5c833dd861ae528d52efbc0f9e5f) commit ddd88ff365c5399d88890abe1608f12c4a960488 Author: Alfredo Deza Date: Fri Oct 6 08:03:36 2017 -0400 doc raise exceptions with a base class Although valid in very old Python, it is no longer possible to raise "bare" exceptions without a class. Signed-off-by: Alfredo Deza (cherry picked from commit 347b7b1f193f97a3577a971e074a23366b0c8617) commit 8b3e9917a70452cc4182aae469aac331314df4e6 Author: Jason Dillaman Date: Tue Sep 26 15:46:28 2017 -0400 rbd-mirror: ensure forced-failover cannot result in sync state If the local image already exists and the local image isn't registered in the remote journal, it should be assumed that it's already in the replaying state so it can verify the ancestry to detect split-brains. Fixes: http://tracker.ceph.com/issues/21559 Signed-off-by: Jason Dillaman (cherry picked from commit bc96c53d947625d12f7b30a45da68a96c09faf70) commit d7da395e5196eae13b31c018bf503b9e0c169bda Author: Jason Dillaman Date: Tue Sep 26 14:21:42 2017 -0400 rbd-mirror: forced-promotion should interrupt replay delay to shut down Fixes: http://tracker.ceph.com/issues/21559 Signed-off-by: Jason Dillaman (cherry picked from commit e3b58312572ceb7abc675128dfb231c9b888655e) commit 4ebd4b39282f715a30a849c01510a281faec2ca6 Author: Jason Dillaman Date: Fri Sep 29 15:11:38 2017 -0400 librbd: snapshots should be created/removed against data pool Fixes: http://tracker.ceph.com/issues/21567 Signed-off-by: Jason Dillaman (cherry picked from commit 5a3baf1bd852d6c5d0de10a33839658789edc4eb) commit 34cd96d3425dcea4053ce0514632194dd7b2eaab Author: Casey Bodley Date: Thu Oct 12 11:25:13 2017 -0400 cls/rgw: increment header version to avoid overwriting bilog entries Fixes: http://tracker.ceph.com/issues/21772 Signed-off-by: Casey Bodley (cherry picked from commit 081ba106a910583851b9acdcaf5971e37a439329) commit 972916bf69192a2c504bf855e2af0480815ecc91 Author: Casey Bodley Date: Wed Oct 11 19:27:55 2017 -0400 test/rgw: add test_multipart_object_sync Signed-off-by: Casey Bodley (cherry picked from commit 1c14d4daf9201d58b1d0038bdd733a16275e037a) commit 695d7ad51179d4f1f5e4168dec3758e3ffa4fe4d Author: Patrick Donnelly Date: Sat Oct 14 13:47:53 2017 -0700 mds: prevent trim count from underflowing Fixes: http://tracker.ceph.com/issues/21807 Signed-off-by: Patrick Donnelly (cherry picked from commit 452bdd8b35643ed82d8614efcd4ca22688392eb6) commit 61aca440334219d5d04df632812dbdf164de3fbc Merge: 462b93ecc5 bb92cc93ac Author: Matt Benjamin Date: Sun Oct 15 14:30:57 2017 -0400 Merge pull request #17834 from rzarzynski/wip-rgw-21148-luminous luminous: rgw: add support for Swift's reversed account listings commit 462b93ecc5e8caf7b0c35dd83dbbe7e33cc015d9 Merge: c4ac0d64a9 119995a8f0 Author: Matt Benjamin Date: Sun Oct 15 12:48:02 2017 -0400 Merge pull request #17811 from smithfarm/wip-21456-luminous luminous: rgw: wrong error message is returned when putting container with a name that is too long commit c4ac0d64a95a6f9ec974446a6971f9d60ab67b9f Merge: 5695829ffd 957e221f1f Author: Marcus Watts Date: Fri Oct 13 20:42:16 2017 -0400 Merge pull request #17832 from rzarzynski/wip-rgw-18977-luminous luminous: rgw: list_objects() honors end_marker regardless of namespace. commit b99acfd6018178ad8cd7db187d6892780f82a276 Author: Yan, Zheng Date: Thu Oct 12 12:01:35 2017 +0800 mds: keep CInode::STATE_QUEUEDEXPORTPIN state when exporting inode Fixes: http://tracker.ceph.com/issues/21768 Signed-off-by: "Yan, Zheng" (cherry picked from commit 1d160098a93092904026b62326fd9890de48044a) commit 2973b6d4184cb5c36f8823010ab83a2b8ebb12d0 Author: Yan, Zheng Date: Tue Oct 10 17:47:33 2017 +0800 mds: update client metadata for already open session session opened by Server::prepare_force_open_sessions() has no client metadata. Fixes: http://tracker.ceph.com/issues/21746 Signed-off-by: "Yan, Zheng" (cherry picked from commit 9d5c92a3367369789836d8669aa421074c69e4f3) commit f353a1e805f4be3e31d8c3346f0f41fa2619c770 Author: Yan, Zheng Date: Mon Oct 9 18:00:38 2017 +0800 osdc/ObjectCacher: limit memory usage of BufferHead when doing small size random writes, size of data structures that used for tracking dirty data can be larger than the dirty data size. Fixes: http://tracker.ceph.com/issues/21726 Signed-off-by: "Yan, Zheng" (cherry picked from commit 27db0255992354e15b4285891a33fede6849cf62) commit 5695829ffd72a03f5611393c6037caf73ff1e813 Merge: d3804d216d 6d032e6ce3 Author: Josh Durgin Date: Fri Oct 13 15:29:45 2017 -0700 Merge pull request #18275 from ceph/wip-yuriw-21776-luminous tests - Removed `distors` and added `supported` to run on all support… Reviewed-by: Vasu Kulkarni Reviewed-by: Josh Durgin commit 6d032e6ce33fe153958a445066a2a7ba00ffd260 Author: Yuri Weinstein Date: Thu Oct 12 13:36:20 2017 -0700 tests - Removed `distors` and added `supported` to run on all supported OSs Added `openstack` fragment to run on vps Forced/hard-coded `machine_type=vps` Added `print`s Added only centos and ubuntu latest to support systemd (in distros dir) Added `ceph osd set-require-min-compat-client luminous` Fixes http://tracker.ceph.com/issues/21776 Signed-off-by: Yuri Weinstein commit d3804d216d21d7e2b12e69eae1c1ef293229a6ca Merge: c9b493d2db 28e7d5645e Author: Sage Weil Date: Fri Oct 13 14:34:56 2017 -0500 Merge pull request #18050 from xiexingguo/wip-pr-17610 luminous: os/bluestore: set bitmap freelist resolution to min_alloc_size Reviewed-by: Sage Weil commit c9b493d2db39765c6ecfb31a3c9e8a38465e6ba0 Merge: 09d597d34a b435af1137 Author: Sage Weil Date: Fri Oct 13 14:34:30 2017 -0500 Merge pull request #18247 from liewegas/wip-seesaw-warning-luminous qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN commit 2e6b29084ae505098fe6889d69133a8527c773be Author: Kefu Chai Date: Fri Sep 1 17:32:22 2017 +0800 arch/arm: set ceph_arch_aarch64_crc32 only if the build host supports crc32cx HWCAP_CRC32 is defined by the linux kernel source. so it's defined as long as the linux kernel source is new enough. but the compiler on the building host is not necessarily able to build the `crc32cx` instruction. if we happen to have an incapable compiler on a machine with recent linux kernel source, the dummy "ceph_crc32c_aarch64()" will be selected by `ceph_choose_crc32()`. and it always return 0. See-also: http://tracker.ceph.com/issues/19705 Signed-off-by: Kefu Chai (cherry picked from commit 8a077bd2a93d3e12e20013e76e57a35240f5f481) commit 7d2b7e8e3113daf40fd56d8258294d45b854ca55 Author: Adam C. Emerson Date: Tue Oct 10 16:21:48 2017 -0400 rgw: Remove assertions in IAM Policy A couple of them could be triggered by user input. Signed-off-by: Adam C. Emerson (cherry picked from commit b3118cabb8060a8cc6a01c4e8264cb18e7b1745a) commit 09d597d34a1d567dc5350570f044eaa3d74a024f Merge: 7fdd9a6dac 37ea1976b3 Author: Sage Weil Date: Thu Oct 12 14:57:38 2017 -0500 Merge pull request #18253 from ceph/wip-yuriw-21660_2-luminous_1 tests - Added yaml fragmet to cover testing... commit 7fdd9a6dac4e047ae710dfaad38d5dbe3caa2634 Merge: 8ad4617f3d 2b7bd94f13 Author: Sage Weil Date: Thu Oct 12 14:57:23 2017 -0500 Merge pull request #18254 from ceph/wip-yuriw-21660_3-luminous_1 tests - Added yaml fragmet to cover testing... commit ab644b89e0aad7b0c1a26ed3b9a2f3ea685c7a15 Author: Kefu Chai Date: Wed Oct 11 11:45:33 2017 +0800 qa: s/backfill/backfilling/ it's renamed "backfilling" in 4015343f . Signed-off-by: Kefu Chai (cherry picked from commit e21114274f6a2742748a5a9b965d415241c80f3c) commit 099801e616c3c9e8501f153501b74d5220383f33 Author: Yanhu Cao Date: Wed Oct 11 09:31:14 2017 +0800 mgr/status: fix ceph fs status returns error Fixes: http://tracker.ceph.com/issues/21752 Signed-off-by: Yanhu Cao (cherry picked from commit 4a90daa700142ed1f58f5f2a2cc85f3049bccd64) commit 0618f19128d0eb04004e65e6eab477b7b1b7fb6a Author: John Spray Date: Wed Aug 30 20:32:11 2017 +0100 mgr/status: format byte quantities in base 2 multiples Fixes: http://tracker.ceph.com/issues/21189 Signed-off-by: John Spray (cherry picked from commit 2a5ae210928f8c9f1e3b72863c277ae5655bb771) commit 2b7bd94f139251b4373a9662e5add86456566486 Author: Yuri Weinstein Date: Wed Oct 11 13:12:45 2017 -0700 tests - Added yaml fragmet to cover testing... ... for http://tracker.ceph.com/issues/21660#note-17 for luminous-x (see similar #18200, #18226) Signed-off-by: Yuri Weinstein (cherry picked from commit e7fdda31803cd37604a9296a876320bfcb849d34) Signed-off-by: Yuri Weinstein commit 37ea1976b30e4e03ac535ee353c3a5277c53dbd3 Author: Yuri Weinstein Date: Tue Oct 10 13:48:33 2017 -0700 tests - Added yaml fragmet to cover testing... ... for http://tracker.ceph.com/issues/21660#note-17 for jewel-x (see similar https://github.com/ceph/ceph/pull/18200) Signed-off-by: Yuri Weinstein (cherry picked from commit b552e636ddd8ce2cfb605b4c124440d8dd1e8e47) Signed-off-by: Yuri Weinstein commit 8ad4617f3dea71984d0d2a6119e9e51ef76779fd Merge: edf28e6912 59a5a10904 Author: Sage Weil Date: Wed Oct 11 12:20:04 2017 -0500 Merge pull request #18025 from liewegas/wip-recovery-preemption-luminous luminous: osd: allow recovery preemption Reviewed-by: Greg Farnum commit edf28e6912c183de4d955f125ef18d8a52594f59 Merge: 91cb216c01 3354937698 Author: Sage Weil Date: Wed Oct 11 12:19:45 2017 -0500 Merge pull request #18154 from ktdreyer/luminous-gitignore-debian-patches luminous: .gitignore: allow debian .patch files commit 91cb216c01eca89ff7e338328d64216998effd16 Merge: 27d3f8c748 b3f9439cfd Author: Sage Weil Date: Wed Oct 11 12:19:08 2017 -0500 Merge pull request #18189 from tchaikov/wip-retry-oserror-luminous luminous: ceph-disk: retry on OSError commit 27d3f8c748de33a35c5006513d198bbcbee19b45 Merge: a33dd01164 f60a942023 Author: Sage Weil Date: Wed Oct 11 12:18:27 2017 -0500 Merge pull request #18227 from liewegas/wip-bluestore-mempool-luminous luminous: os/bluestore: several mempool accounting fixes Reviewed-by: xie xingguo commit b435af1137ede418740d2dcf39114d8476775fd8 Author: Sage Weil Date: Wed Oct 11 08:24:50 2017 -0500 qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN Mgr can get marked down when mon weirdness is happening. Signed-off-by: Sage Weil (cherry picked from commit 0a886bc9f3e9805de04c8c60bd3d78b937737b96) commit 51c6f0def379ba013a82204398a021663f1b5d56 Author: Kefu Chai Date: Tue Oct 10 20:32:08 2017 +0800 mon/PGMap: let pg_string_state() return boost::optional<> better to be specific, so we don't run into the same problem even we are using the MSB of uint64_t for a pg state bit in future. we could, however use uint64_t(-1) to indicate the pg_string_state()'s failure to parse the state string, because pg_string_state() only translate a single state a time. but it's always better to be explicit than implicit. Fixes: http://tracker.ceph.com/issues/21609 Signed-off-by: Kefu Chai (cherry picked from commit 260f87a9f1c4e32ff3a596ea341ad07a901f44a9) Conflicts: src/mon/PGMap.cc src/osd/osd_types.cc src/osd/osd_types.h: trivial resolution commit 9d12fa5e6dad5492491f4718ba7c593077d984a6 Author: Josh Durgin Date: Tue Oct 10 19:54:31 2017 -0400 common/options: enable multiple rocksdb compaction threads for filestore One of the major benefits of rocksdb over leveldb is multithreaded compaction. The default of 1 thread does not provide much benefit, and is insufficient for heavy rgw workloads. For high-write and delete omap workloads I've seen up to 8 compaction threads be used. There's little overhead to having a higher max than are needed, so set the default to 8. Signed-off-by: Josh Durgin (cherry picked from commit 023fa810aa6b3af305e9027e3f717e54d1bb2712) Conflicts: src/common/options.cc (trivial) commit 6f2f8cdc6a6fa64928fd2d5459b4974e281d857e Author: Mark Nelson Date: Thu Sep 21 17:21:16 2017 -0500 common/options.cc: Set Filestore rocksdb compaction readahead option. fixes: http://tracker.ceph.com/issues/21505 Signed-off-by: Mark Nelson (cherry picked from commit 66567e573836c570040c8d8148c34b0cf9dc9ce2) commit f60a942023088cbba53a816e6ef846994921cab3 Author: Sage Weil Date: Thu Sep 28 08:02:50 2017 -0400 os/bluestore: move several buffer{ptr,list}s into cache_other mempool Signed-off-by: Sage Weil (cherry picked from commit 80c60fcde22cf1269ada45d0914543b8f7d49b3e) commit 37dd32f449bf2b0aaa3dbcba5cc06da076adbb1e Author: Sage Weil Date: Thu Sep 28 08:14:37 2017 -0400 os/bluestore: put new attrs in correct mempool too Signed-off-by: Sage Weil (cherry picked from commit e7762ea1477e861ebc738aa0588a01c0da883d62) commit dea2c1e6704c40de11c041e9982e9a9f7d9d4cb3 Author: Sage Weil Date: Fri Sep 22 08:09:37 2017 -0400 os/bluestore: put attrs in mempool Signed-off-by: Sage Weil (cherry picked from commit ef6524436fe095f94fce421ce26ed4c64e1424fe) commit c2ed1f9b1801e43cf1d345da4d4876de5334cc40 Author: Sage Weil Date: Fri Sep 22 08:08:52 2017 -0400 buffer: add ptr::[try_]reassign_to_mempool Signed-off-by: Sage Weil (cherry picked from commit 937379d57a0f8097ae713adf714ba0414d36dffe) commit a33dd01164bb18b24a0909c9419e15fcd2e79446 Merge: 9e48df69de c0a1168ec2 Author: Josh Durgin Date: Tue Oct 10 13:17:46 2017 -0700 Merge pull request #18200 from ceph/wip-yuriw-21660-luminous tests - Added yaml fragmet to cover testing Reviewed-by: Josh Durgin Reviewed-by: Jason Dillaman commit c0a1168ec2bacf6d78041f3ae49bfe6fff669913 Author: Yuri Weinstein Date: Mon Oct 9 12:51:59 2017 -0700 tests - Added yaml fragmet to cover testing on standanone client.4 for http://tracker.ceph.com/issues/21660#note-17 Signed-off-by: Yuri Weinstein commit b3f9439cfd9141ddde9874c06fd43d346c10c5da Author: Kefu Chai Date: Sat Oct 7 22:15:11 2017 +0800 ceph-disk: retry on OSError we are likely to 1) create partition, for instance, sdc1 2) partprobe sdc 3) udevadm settle 4) check the device by its path: /dev/sdc1 but there is chance that the uevent sent from kernel fails to reach udev before we call "udevadm", hence "/dev/sdc1" does not exist even after "udevadm settle" returns. so we retry in case of OSError here. Signed-off-by: Kefu Chai (cherry picked from commit 4f82dfb9e761a09484e6ba3bd027da535162783e) commit 83b3377c6b04d78b20b7c23d145e015d63e471a4 Author: Kefu Chai Date: Sat Oct 7 21:13:47 2017 +0800 ceph-disk: factor out the retry logic into a decorator Signed-off-by: Kefu Chai (cherry picked from commit 02a8620de7f200736900aafc944b2c1ac47d9910) commit 9e48df69de14d8415eed008b824204313774e2ba Merge: a81b0da8e4 41af4ca8f3 Author: Alfredo Deza Date: Sun Oct 8 12:05:25 2017 -0400 Merge pull request #18167 from tchaikov/wip-luminous-sphinx luminous: admin: bump sphinx to 1.6 Reviewed-by: Alfredo Deza commit 41af4ca8f3cd3db8dc076c07e1a082a053b91b26 Author: Kefu Chai Date: Mon Sep 11 21:54:33 2017 +0800 doc: switch to upstream sphinx-ditaa it works with setuptools and is now compatible with py3 the py3 branch is created to track the upstream's master branch Signed-off-by: Kefu Chai (cherry picked from commit fda079d5ed4c8a80c677f714a680233a568d512e) commit e49cae8b03340834e0b44491a4a903c0d61e1f74 Author: Alfredo Deza Date: Tue Aug 15 09:12:27 2017 -0400 admin: bump sphinx to 1.6 Signed-off-by: Alfredo Deza (cherry picked from commit 88a1114cb3f90e52ba718dc832617fa004d63b3c) commit a81b0da8e4aca275cc3195c04f3e0d485e4a2f31 Merge: 1e521b5fb4 e4164c43f5 Author: Alfredo Deza Date: Fri Oct 6 17:42:30 2017 -0400 Merge pull request #17914 from liewegas/wip-21498-luminous ceph-disk: fix '--runtime' omission for ceph-osd service Reviewed-by: Alfredo Deza commit 1e521b5fb47de74720d6aab39750c53aa2b2d8e7 Merge: d550610512 ee5b9ea46c Author: Sage Weil Date: Fri Oct 6 15:35:06 2017 -0500 Merge pull request #18135 from liewegas/wip-vta-luminous luminous: cmake: disable VTA on options.cc commit 33549376987b18a9bdcf5f56e62268c5791124bc Author: Ken Dreyer Date: Thu Sep 7 11:07:59 2017 -0600 .gitignore: allow debian .patch files The Ubuntu packaging layout with git-buildpackage assumes a "debian/patches/" directory with several .patch files in it. When upstream's .gitignore tells Git to ignore .patch files, we have to edit that line out downstream. When we forget to do that downstream, it can lead to missing patches and broken downstream builds. Allow patches in the /debian/patches directory so it's easier to maintain an Ubuntu package based on upstream's Git repo. Signed-off-by: Ken Dreyer (cherry picked from commit c734b0c0296152721b658af7b699a64b3a49d251) commit d5506105120479c1831ab559b555e2d4ea4c357c Merge: d46675a1ec 0dc73f1210 Author: Sage Weil Date: Fri Oct 6 07:38:53 2017 -0500 Merge pull request #18140 from liewegas/wip-21660-luminous luminous: src/messages/MOSDMap: reencode OSDMap for older clients commit d46675a1ecb10f7176265cb1fc5ca971209aecb1 Merge: 39ad203313 8b2bd38785 Author: Sage Weil Date: Thu Oct 5 22:12:22 2017 -0500 Merge pull request #18037 from ajarr/wip-21602-luminous luminous: ceph_volume_client: add get, put, and delete object interfaces Reviewed-by: Patrick Donnelly commit 39ad2033136e32d76e89e86d3f00bd1b928242ce Merge: 473f9d1317 80f6508fc2 Author: Sage Weil Date: Thu Oct 5 21:03:42 2017 -0500 Merge pull request #18103 from liewegas/wip-21259-luminous luminous: osd: dump bluestore debug on shutdown if debug option is set commit 473f9d13173aa751d54b9a322060111d627f71f1 Merge: c50df6511f 49294965f9 Author: Sage Weil Date: Thu Oct 5 21:03:28 2017 -0500 Merge pull request #18112 from liewegas/wip-localpool-min-size-luminous luminous: mgr/localpool: fix min_size, 3x default, crush rule Reviewed-by: Alexander Marangone commit c50df6511fda7d03c3cfa828acd5e56f2ef45d0f Merge: 4ff03e5192 16c6dd5129 Author: Sage Weil Date: Thu Oct 5 21:02:53 2017 -0500 Merge pull request #18126 from smithfarm/wip-sigpipe-porting-luminous luminous: msg: reimplement sigpipe blocking Reviewed-by: Greg Farnum commit 4ff03e51928c0d7da9a90d50ec504b30eb9e4b32 Merge: 766c6f3647 7986d4566a Author: Sage Weil Date: Thu Oct 5 21:02:09 2017 -0500 Merge pull request #18127 from liewegas/wip-21470-luminous luminous: os/bluestore: fix another aio stall/deadlock commit 766c6f3647b57b482f76ef254c95a651741d9ad5 Merge: f972702229 3850f8e0b7 Author: Sage Weil Date: Thu Oct 5 21:01:53 2017 -0500 Merge pull request #18128 from liewegas/wip-bluefs-allocation-luminous luminous: os/bluestore: make bluefs behave better near enospc commit f97270222935d1c43a41102c3ada586a7deca190 Merge: e4c51f62ff f99fe83a37 Author: Sage Weil Date: Thu Oct 5 21:01:27 2017 -0500 Merge pull request #18131 from liewegas/wip-ifname-luminous luminous: common/pick_address: add {public,cluster}_network_interface option commit e4c51f62ffc0f2fcc9214f6b7895210aa79c288b Merge: 66af04cd9b f0a6f616c6 Author: Sage Weil Date: Thu Oct 5 21:01:06 2017 -0500 Merge pull request #18132 from gregsfortytwo/wip-20416-bitwise-assert-luminous osd: make the PG's SORTBITWISE assert a more generous shutdown Reviewed-by: Sage Weil commit 0dc73f12105ec271a43fe176c2f2a22849e8475c Author: Sage Weil Date: Thu Oct 5 15:26:16 2017 -0500 src/messages/MOSDMap: reencode OSDMap for older clients We explicitly select which missing bits trigger a reencode. We already had jewel and earlier covered, but kraken includes all of the previously mentioned bits but not SERVER_LUMINOUS. This prevents kraken clients from decoding luminous maps. Fixes: http://tracker.ceph.com/issues/21660 Signed-off-by: Sage Weil (cherry picked from commit d2664aa34dc4c13da64d1ad187019737080069d7) commit 3850f8e0b791f2d05906647866b14a390ab2f5da Author: Sage Weil Date: Tue Oct 3 11:18:17 2017 -0500 os/bluestore: add bluestore_bluefs_min_free We need at least ~1GB free so we can write out new SSTs (which are 256MB each). Signed-off-by: Sage Weil (cherry picked from commit 65be614f2bfd65a1f13205a3075c87acc72c4836) (avoided 1_G syntax that luminous doesn't support) commit b015ce116159ec57d20bcd29306e450a816610f9 Author: Karol Mroz Date: Mon Oct 2 14:01:47 2017 -0700 rgw: stop/join TokenCache revoke thread only if started. Thread::join triggers an assert otherwise. Fixes http://tracker.ceph.com/issues/21666 Signed-off-by: Karol Mroz (cherry picked from commit 26f2da083c7dd21b89c1c1e6c498b14e034364a6) commit ee5b9ea46c8b1e831c60f9cc0b10819b14d5efb8 Author: Kefu Chai Date: Thu Aug 31 18:15:28 2017 +0800 cmake: disable VTA on options.cc to silence following warning and to avoid compiling this file twice: ceph/src/common/options.cc: In function ‘std::vector