diff options
Diffstat (limited to 'doc/releases/dumpling.rst')
| -rw-r--r-- | doc/releases/dumpling.rst | 947 |
1 files changed, 947 insertions, 0 deletions
diff --git a/doc/releases/dumpling.rst b/doc/releases/dumpling.rst new file mode 100644 index 000000000..b9946546e --- /dev/null +++ b/doc/releases/dumpling.rst @@ -0,0 +1,947 @@ +======== +Dumpling +======== + +Dumpling is the 4th stable release of Ceph. It is named after the +dumpling squid (Euprymna tasmanica). + +v0.67.12 "Dumpling" (draft) +=========================== + +This stable update for Dumpling fixes a few longstanding issues with +backfill in the OSD that can lead to stalled IOs. There is also a fix +for memory utilization for reads in librbd when caching is enabled, +and then several other small fixes across the rest of the system. + +Dumpling users who have encountered IO stalls during backfill and who +do not expect to upgrade to Firefly soon should upgrade. Everyone +else should upgrade to Firefly already. This is likely to be the last stable +release for the 0.67.x Dumpling series. + + +Notable Changes +--------------- + +* buffer: fix buffer rebuild alignment corner case (#6614 #6003 Loic Dachary, Samuel Just) +* ceph-disk: reprobe partitions after zap (#9665 #9721 Loic Dachary) +* ceph-disk: use partx instead of partprobe when appropriate (Loic Dachary) +* common: add $cctid meta variable (#6228 Adam Crume) +* crush: fix get_full_location_ordered (Sage Weil) +* crush: pick ruleset id that matches rule_id (#9675 Xiaoxi Chen) +* libcephfs: fix tid wrap bug (#9869 Greg Farnum) +* libcephfs: get osd location on -1 should return EINVAL (Sage Weil) +* librados: fix race condition with C API and op timeouts (#9582 Sage Weil) +* librbd: constrain max number of in-flight read requests (#9854 Jason Dillaman) +* librbd: enforce cache size on read requests (Jason Dillaman) +* librbd: fix invalid close in image open failure path (#10030 Jason Dillaman) +* librbd: fix read hang on sparse files (Jason Dillaman) +* librbd: gracefully handle deleted/renamed pools (#10270 #10122 Jason Dillaman) +* librbd: protect list_children from invalid child pool ioctxs (#10123 Jason Dillaman) +* mds: fix ctime updates from clients without dirty caps (#9514 Greg Farnum) +* mds: fix rare NULL dereference in cap update path (Greg Farnum) +* mds: fix assertion caused by system clock backwards (#11053 Yan, Zheng) +* mds: store backtrace on straydir (Yan, Zheng) +* osd: fix journal committed_thru update after replay (#6756 Samuel Just) +* osd: fix memory leak, busy loop on snap trim (#9113 Samuel Just) +* osd: fix misc peering, recovery bugs (#10168 Samuel Just) +* osd: fix purged_snap field on backfill start (#9487 Sage Weil, Samuel Just) +* osd: handle no-op write with snapshot corner case (#10262 Sage Weil, Loic Dachary) +* osd: respect RWORDERED rados flag (Sage Weil) +* osd: several backfill fixes and refactors (Samuel Just, David Zafman) +* rgw: send http status reason explicitly in fastcgi (Yehuda Sadeh) + +v0.67.11 "Dumpling" +=================== + +This stable update for Dumpling fixes several important bugs that +affect a small set of users. + +We recommend that all Dumpling users upgrade at their convenience. If +none of these issues are affecting your deployment there is no +urgency. + + +Notable Changes +--------------- + +* common: fix sending dup cluster log items (#9080 Sage Weil) +* doc: several doc updates (Alfredo Deza) +* libcephfs-java: fix build against older JNI headesr (Greg Farnum) +* librados: fix crash in op timeout path (#9362 Matthias Kiefer, Sage Weil) +* librbd: fix crash using clone of flattened image (#8845 Josh Durgin) +* librbd: fix error path cleanup when failing to open image (#8912 Josh Durgin) +* mon: fix crash when adjusting pg_num before any OSDs are added (#9052 Sage Weil) +* mon: reduce log noise from paxos (Aanchal Agrawal, Sage Weil) +* osd: allow scrub and snap trim thread pool IO priority to be adjusted (Sage Weil) +* osd: fix mount/remount sync race (#9144 Sage Weil) + + +v0.67.10 "Dumpling" +=================== + +This stable update release for Dumpling includes primarily fixes for +RGW, including several issues with bucket listings and a potential +data corruption problem when multiple multi-part uploads race. There is also +some throttling capability added in the OSD for scrub that can mitigate the +performance impact on production clusters. + +We recommend that all Dumpling users upgrade at their convenience. + +Notable Changes +--------------- + +* ceph-disk: partprobe befoere settle, fixing dm-crypt (#6966, Eric Eastman) +* librbd: add invalidate cache interface (Josh Durgin) +* librbd: close image if remove_child fails (Ilya Dryomov) +* librbd: fix potential null pointer dereference (Danny Al-Gaaf) +* librbd: improve writeback checks, performance (Haomai Wang) +* librbd: skip zeroes when copying image (#6257, Josh Durgin) +* mon: fix rule(set) check on 'ceph pool set ... crush_ruleset ...' (#8599, John Spray) +* mon: shut down if mon is removed from cluster (#6789, Joao Eduardo Luis) +* osd: fix filestore perf reports to mon (Sage Weil) +* osd: force any new or updated xattr into leveldb if E2BIG from XFS (#7779, Sage Weil) +* osd: lock snapdir object during write to fix race with backfill (Samuel Just) +* osd: option sleep during scrub (Sage Weil) +* osd: set io priority on scrub and snap trim threads (Sage Weil) +* osd: 'status' admin socket command (Sage Weil) +* rbd: tolerate missing NULL terminator on block_name_prefix (#7577, Dan Mick) +* rgw: calculate user manifest (#8169, Yehuda Sadeh) +* rgw: fix abort on chunk read error, avoid using extra memory (#8289, Yehuda Sadeh) +* rgw: fix buffer overflow on bucket instance id (#8608, Yehuda Sadeh) +* rgw: fix crash in swift CORS preflight request (#8586, Yehuda Sadeh) +* rgw: fix implicit removal of old objects on object creation (#8972, Patrycja Szablowska, Yehuda Sadeh) +* rgw: fix MaxKeys in bucket listing (Yehuda Sadeh) +* rgw: fix race with multiple updates to a single multipart object (#8269, Yehuda Sadeh) +* rgw: improve bucket listing with delimiter (Yehuda Sadeh) +* rgw: include NextMarker in bucket listing (#8858, Yehuda Sadeh) +* rgw: return error early on non-existent bucket (#7064, Yehuda Sadeh) +* rgw: set truncation flag correctly in bucket listing (Yehuda Sadeh) +* sysvinit: continue starting daemons after pre-mount error (#8554, Sage Weil) + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.10.txt>`. + + +v0.67.9 "Dumpling" +================== + +This Dumpling point release fixes several minor bugs. The most +prevalent in the field is one that occasionally prevents OSDs from +starting on recently created clusters. + +We recommend that all Dumpling users upgrade at their convenience. + +Notable Changes +--------------- + +* ceph-fuse, libcephfs: client admin socket command to kick and inspect MDS sessions (#8021, Zheng Yan) +* monclient: fix failure detection during mon handshake (#8278, Sage Weil) +* mon: set tid on no-op PGStatsAck messages (#8280, Sage Weil) +* msgr: fix a rare bug with connection negotiation between OSDs (Guang Yang) +* osd: allow snap trim throttling with simple delay (#6278, Sage Weil) +* osd: check for splitting when processing recover/backfill reservations (#6565, Samuel Just) +* osd: fix backfill position tracking (#8162, Samuel Just) +* osd: fix bug in backfill stats (Samuel Just) +* osd: fix bug preventing OSD startup for infant clusters (#8162, Greg Farnum) +* osd: fix rare PG resurrection race causing an incomplete PG (#7740, Samuel Just) +* osd: only complete replicas count toward min_size (#7805, Samuel Just) +* rgw: allow setting ACLs with empty owner (#6892, Yehuda Sadeh) +* rgw: send user manifest header field (#8170, Yehuda Sadeh) + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.9.txt>`. + + +v0.67.8 "Dumpling" +================== + +This Dumpling point release fixes several non-critical issues since +v0.67.7. The most notable bug fixes are an auth fix in librbd +(observed as an occasional crash from KVM), an improvement in the +network failure detection with the monitor, and several hard to hit +OSD crashes or hangs. + +We recommend that all users upgrade at their convenience. + +Upgrading +--------- + +* The 'rbd ls' function now returns success and returns an empty when a pool + does not store any rbd images. Previously it would return an ENOENT error. + +* Ceph will now issue a health warning if the 'mon osd down out + interval' config option is set to zero. This warning can be + disabled by adding 'mon warn on osd down out interval zero = false' + to ceph.conf. + +Notable Changes +--------------- + +* all: improve keepalive detection of failed monitor connections (#7888, Sage Weil) +* ceph-fuse, libcephfs: pin inodes during readahead, fixing rare crash (#7867, Sage Weil) +* librbd: make cache writeback a bit less aggressive (Sage Weil) +* librbd: make symlink for qemu to detect librbd in RPM (#7293, Josh Durgin) +* mon: allow 'hashpspool' pool flag to be set and unset (Loic Dachary) +* mon: commit paxos state only after entire quorum acks, fixing rare race where prior round state is readable (#7736, Sage Weil) +* mon: make elections and timeouts a bit more robust (#7212, Sage Weil) +* mon: prevent extreme pool split operations (Greg Farnum) +* mon: wait for quorum for get_version requests to close rare pool creation race (#7997, Sage Weil) +* mon: warn on 'mon osd down out interval = 0' (#7784, Joao Luis) +* msgr: fix byte-order for auth challenge, fixing auth errors on big-endian clients (#7977, Dan Mick) +* msgr: fix occasional crash in authentication code (usually triggered by librbd) (#6840, Josh Durgin) +* msgr: fix rebind() race (#6992, Xihui He) +* osd: avoid timeouts during slow PG deletion (#6528, Samuel Just) +* osd: fix bug in pool listing during recovery (#6633, Samuel Just) +* osd: fix queue limits, fixing recovery stalls (#7706, Samuel Just) +* osd: fix rare peering crashes (#6722, #6910, Samuel Just) +* osd: fix rare recovery hang (#6681, Samuel Just) +* osd: improve error handling on journal errors (#7738, Sage Weil) +* osd: reduce load on the monitor from OSDMap subscriptions (Greg Farnum) +* osd: rery GetLog on peer osd startup, fixing some rare peering stalls (#6909, Samuel Just) +* osd: reset journal state on remount to fix occasional crash on OSD startup (#8019, Sage Weil) +* osd: share maps with peers more aggressively (Greg Farnum) +* rbd: make it harder to delete an rbd image that is currently in use (#7076, Ilya Drymov) +* rgw: deny writes to secondary zone by non-system users (#6678, Yehuda Sadeh) +* rgw: do'nt log system requests in usage log (#6889, Yehuda Sadeh) +* rgw: fix bucket recreation (#6951, Yehuda Sadeh) +* rgw: fix Swift range response (#7099, Julien Calvet, Yehuda Sadeh) +* rgw: fix URL escaping (#8202, Yehuda Sadeh) +* rgw: fix whitespace trimming in http headers (#7543, Yehuda Sadeh) +* rgw: make multi-object deletion idempotent (#7346, Yehuda Sadeh) + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.8.txt>`. + +v0.67.7 "Dumpling" +================== + +This Dumpling point release fixes a few critical issues in v0.67.6. + +All v0.67.6 users are urgently encouraged to upgrade. We also +recommend that all v0.67.5 (or older) users upgrade. + +Upgrading +--------- + +* Once you have upgraded a radosgw instance or OSD to v0.67.7, you should not + downgrade to a previous version. + +Notable Changes +--------------- + +* ceph-disk: additional unit tests +* librbd: revert caching behavior change in v0.67.6 +* osd: fix problem reading xattrs due to incomplete backport in v0.67.6 +* radosgw-admin: fix reading object policy + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.7.txt>`. + + +v0.67.6 "Dumpling" +================== + +.. note: This release contains a librbd bug that is fixed in v0.67.7. Please upgrade to v0.67.7 and do not use v0.67.6. + +This Dumpling point release contains a number of important fixed for +the OSD, monitor, and radosgw. Most significantly, a change that +forces large object attributes to spill over into leveldb has been +backported that can prevent objects and the cluster from being damaged +by large attributes (which can be induced via the radosgw). There is +also a set of fixes that improves data safety and RADOS semantics when +the cluster becomes full and then non-full. + +We recommend that all 0.67.x Dumpling users skip this release and upgrade to v0.67.7. + +Upgrading +--------- + +* The OSD has long contained a feature that allows large xattrs to + spill over into the leveldb backing store in situations where not + all local file systems are able to store them reliably. This option + is now enabled unconditionally in order to avoid rare cases where + storing large xattrs renders the object unreadable. This is known to + be triggered by very large multipart objects, but could be caused by + other workloads as well. Although there is some small risk that + performance for certain workloads will degrade, it is more important + that data be retrievable. Note that newer versions of Ceph (e.g., + firefly) do some additional work to avoid the potential performance + regression in this case, but that is current considered too complex + for backport to the Dumpling stable series. + +* It is very dangerous to downgrade from v0.67.6 to a prior version of + Dumpling. If the old version does not have 'filestore xattr use + omap = true' it may not be able to read all xattrs for an object and + can cause undefined behavior. + +Notable changes +--------------- + +* ceph-disk: misc bug fixes, particularly on RHEL (Loic Dachary, Alfredo Deza, various) +* ceph-fuse, libcephfs: fix crash from read over certain sparseness patterns (Sage Weil) +* ceph-fuse, libcephfs: fix integer overflow for sync reads racing with appends (Sage Weil) +* ceph.spec: fix udev rule when building RPM under RHEL (Derek Yarnell) +* common: fix crash from bad format from admin socket (Loic Dachary) +* librados: add optional timeouts (Josh Durgin) +* librados: do not leak budget when resending localized or redirected ops (Josh Durgin) +* librados, osd: fix and improve full cluster handling (Josh Durgin) +* librbd: fix use-after-free when updating perfcounters during image close (Josh Durgin) +* librbd: remove limit on objects in cache (Josh Durgin) +* mon: avoid on-disk full OSDMap corruption from pg_temp removal (Sage Weil) +* mon: avoid stray pg_temp entries from pool deletion race (Joao Eduardo Luis) +* mon: do not generate spurious MDSMaps from laggy daemons (Joao Eduardo Luis) +* mon: fix error code from 'osd rm|down|out|in ...' commands (Loic Dachary) +* mon: include all health items in summary output (John Spray) +* osd: fix occasional race/crash during startup (Sage Weil) +* osd: ignore stray OSDMap messages during init (Sage Weil) +* osd: unconditionally let xattrs overflow into leveldb (David Zafman) +* rados: fix a few error checks for the CLI (Josh Durgin) +* rgw: convert legacy bucket info objects on demand (Yehuda Sadeh) +* rgw: fix bug causing system users to lose privileges (Yehuda Sadeh) +* rgw: fix CORS bugs related to headers and case sensitivity (Robin H. Johnson) +* rgw: fix multipart object listing (Yehuda Sadeh) +* rgw: fix racing object creations (Yehuda Sadeh) +* rgw: fix racing object put and delete (Yehuda Sadeh) +* rgw: fix S3 auth when using response-* query string params (Sylvain Munaut) +* rgw: use correct secret key for POST authentication (Robin H. Johnson) + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.6.txt>`. + + +v0.67.5 "Dumpling" +================== + +This release includes a few critical bug fixes for the radosgw, +including a fix for hanging operations on large objects. There are also +several bug fixes for radosgw multi-site replications, and a few +backported features. Also, notably, the 'osd perf' command (which dumps +recent performance information about active OSDs) has been backported. + +We recommend that all 0.67.x Dumpling users upgrade. + +Notable changes +--------------- + +* ceph-fuse: fix crash in caching code +* mds: fix looping in populate_mydir() +* mds: fix standby-replay race +* mon: accept 'osd pool set ...' as string +* mon: backport: 'osd perf' command to dump recent OSD performance stats +* osd: add feature compat check for upcoming object sharding +* osd: fix osd bench block size argument +* rbd.py: increase parent name size limit +* rgw: backport: allow wildcard in supported keystone roles +* rgw: backport: improve swift COPY behavior +* rgw: backport: log and open admin socket by default +* rgw: backport: validate S3 tokens against keystone +* rgw: fix bucket removal +* rgw: fix client error code for chunked PUT failure +* rgw: fix hang on large object GET +* rgw: fix rare use-after-free +* rgw: various DR bug fixes +* sysvinit, upstart: prevent starting daemons using both init systems + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.5.txt>`. + + +v0.67.4 "Dumpling" +================== + +This point release fixes an important performance issue with radosgw, +keystone authentication token caching, and CORS. All users +(especially those of rgw) are encouraged to upgrade. + +Notable changes +--------------- + +* crush: fix invalidation of cached names +* crushtool: do not crash on non-unique bucket ids +* mds: be more careful when decoding LogEvents +* mds: fix heap check debugging commands +* mon: avoid rebuilding old full osdmaps +* mon: fix 'ceph crush move ...' +* mon: fix 'ceph osd crush reweight ...' +* mon: fix writeout of full osdmaps during trim +* mon: limit size of transactions +* mon: prevent both unmanaged and pool snaps +* osd: disable xattr size limit (prevents upload of large rgw objects) +* osd: fix recovery op throttling +* osd: fix throttling of log messages for very slow requests +* rgw: drain pending requests before completing write +* rgw: fix CORS +* rgw: fix inefficient list::size() usage +* rgw: fix keystone token expiration +* rgw: fix minor memory leaks +* rgw: fix null termination of buffer + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.4.txt>`. + + +v0.67.3 "Dumpling" +================== + +This point release fixes a few important performance regressions with +the OSD (both with CPU and disk utilization), as well as several other +important but less common problems. We recommend that all production users +upgrade. + +Notable Changes +--------------- + +* ceph-disk: partprobe after creation journal partition +* ceph-disk: specify fs type when mounting +* ceph-post-file: new utility to help share logs and other files with ceph developers +* libcephfs: fix truncate vs readahead race (crash) +* mds: fix flock/fcntl lock deadlock +* mds: fix rejoin loop when encountering pre-dumpling backpointers +* mon: allow name and addr discovery during election stage +* mon: always refresh after Paxos store_state (fixes recovery corner case) +* mon: fix off-by-4x bug with osd byte counts +* osd: add and disable 'pg log keys debug' by default +* osd: add option to disable throttling +* osd: avoid leveldb iterators for pg log append and trim +* osd: fix readdir_r invocations +* osd: use fdatasync instead of sync +* radosgw: fix sysvinit script return status +* rbd: relicense as LGPL2 +* rgw: flush pending data on multipart upload +* rgw: recheck object name during S3 POST +* rgw: reorder init/startup +* rpm: fix debuginfo package build + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.3.txt>`. + + +v0.67.2 "Dumpling" +================== + +This is an important point release for Dumpling. Most notably, it +fixes a problem when upgrading directly from v0.56.x Bobtail to +v0.67.x Dumpling (without stopping at v0.61.x Cuttlefish along the +way). It also fixes a problem with the CLI parsing of the CEPH_ARGS +environment variable, high CPU utilization by the ceph-osd daemons, +and cleans up the radosgw shutdown sequence. + +Notable Changes +--------------- + +* objecter: resend linger requests when cluster goes from full to non-full +* ceph: parse CEPH_ARGS environment variable +* librados: fix small memory leak +* osd: remove old log objects on upgrade (fixes bobtail -> dumpling jump) +* osd: disable PGLog::check() via config option (fixes CPU burn) +* rgw: drain requests on shutdown +* rgw: misc memory leaks on shutdown + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.2.txt>`. + + +v0.67.1 "Dumpling" +================== + +This is a minor point release for Dumpling that fixes problems with +OpenStack and librbd hangs when caching is disabled. + +Notable changes +--------------- + +* librados, librbd: fix constructor for python bindings with certain + usages (in particular, that used by OpenStack) +* librados, librbd: fix aio_flush wakeup when cache is disabled +* librados: fix locking for aio completion refcounting +* fixes 'ceph --admin-daemon ...' command error code on error +* fixes 'ceph daemon ... config set ...' command for boolean config + options. + +For more detailed information, see :download:`the complete changelog <../changelog/v0.67.1.txt>`. + +v0.67 "Dumpling" +================ + +This is the fourth major release of Ceph, code-named "Dumpling." The +headline features for this release include: + +* Multi-site support for radosgw. This includes the ability to set up + separate "regions" in the same or different Ceph clusters that share + a single S3/Swift bucket/container namespace. + +* RESTful API endpoint for Ceph cluster administration. + ceph-rest-api, a wrapper around ceph_rest_api.py, can be used to + start up a test single-threaded HTTP server that provides access to + cluster information and administration in very similar ways to the + ceph commandline tool. ceph_rest_api.py can be used as a WSGI + application for deployment in a more-capable web server. See + ceph-rest-api.8 for more. + +* Object namespaces in librados. + + +.. _dumpling-upgrade: + +Upgrade Sequencing +------------------ + +It is possible to do a rolling upgrade from Cuttlefish to Dumpling. + +#. Upgrade ceph-common on all nodes that will use the command line + 'ceph' utility. +#. Upgrade all monitors (upgrade ceph package, restart ceph-mon + daemons). This can happen one daemon or host at a time. Note that + because cuttlefish and dumpling monitors can't talk to each other, + all monitors should be upgraded in relatively short succession to + minimize the risk that an a untimely failure will reduce + availability. +#. Upgrade all osds (upgrade ceph package, restart ceph-osd daemons). + This can happen one daemon or host at a time. +#. Upgrade radosgw (upgrade radosgw package, restart radosgw daemons). + + +Upgrading from v0.66 +-------------------- + +* There is monitor internal protocol change, which means that v0.67 + ceph-mon daemons cannot talk to v0.66 or older daemons. We + recommend upgrading all monitors at once (or in relatively quick + succession) to minimize the possibility of downtime. + +* The output of 'ceph status --format=json' or 'ceph -s --format=json' + has changed to return status information in a more structured and + usable format. + +* The 'ceph pg dump_stuck [threshold]' command used to require a + --threshold or -t prefix to the threshold argument, but now does + not. + +* Many more ceph commands now output formatted information; select + with '--format=<format>', where <format> can be 'json', 'json-pretty', + 'xml', or 'xml-pretty'. + +* The 'ceph pg <pgid> ...' commands (like 'ceph pg <pgid> query') are + deprecated in favor of 'ceph tell <pgid> ...'. This makes the + distinction between 'ceph pg <command> <pgid>' and 'ceph pg <pgid> + <command>' less awkward by making it clearer that the 'tell' + commands are talking to the OSD serving the placement group, not the + monitor. + +* The 'ceph --admin-daemon <path> <command ...>' used to accept the + command and arguments as either a single string or as separate + arguments. It will now only accept the command spread across + multiple arguments. This means that any script which does something + like:: + + ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok 'config set debug_ms 1' + + needs to remove the quotes. Also, note that the above can now be + shortened to:: + + ceph daemon osd.0 config set debug_ms 1 + +* The radosgw caps were inconsistently documented to be either 'mon = + allow r' or 'mon = allow rw'. The 'mon = allow rw' is required for + radosgw to create its own pools. All documentation has been updated + accordingly. + +* The radosgw copy object operation may return extra progress info + during the operation. At this point it will only happen when doing + cross zone copy operations. The S3 response will now return extra + <Progress> field under the <CopyResult> container. The Swift + response will now send the progress as a json array. + +* In v0.66 and v0.65 the HASHPSPOOL pool flag was enabled by default + for new pools, but has been disabled again until Linux kernel client + support reaches more distributions and users. + +* ceph-osd now requires a max file descriptor limit (e.g., ``ulimit -n + ...``) of at least + filestore_wbthrottle_(xfs|btrfs)_inodes_hard_limit (5000 by default) + in order to accommodate the new write back throttle system. On + Ubuntu, upstart now sets the fd limit to 32k. On other platforms, + the sysvinit script will set it to 32k by default (still + overridable via max_open_files). If this field has been customized + in ceph.conf it should likely be adjusted upwards. + +Upgrading from v0.61 "Cuttlefish" +--------------------------------- + +In addition to the above notes about upgrading from v0.66: + +* There has been a huge revamp of the 'ceph' command-line interface + implementation. The ``ceph-common`` client library needs to be + upgrade before ``ceph-mon`` is restarted in order to avoid problems + using the CLI (the old ``ceph`` client utility cannot talk to the + new ``ceph-mon``). + +* The CLI is now very careful about sending the 'status' one-liner + output to stderr and command output to stdout. Scripts relying on + output should take care. + +* The 'ceph osd tell ...' and 'ceph mon tell ...' commands are no + longer supported. Any callers should use:: + + ceph tell osd.<id or *> ... + ceph tell mon.<id or name or *> ... + + The 'ceph mds tell ...' command is still there, but will soon also + transition to 'ceph tell mds.<id or name or \*> ...' + +* The 'ceph osd crush add ...' command used to take one of two forms:: + + ceph osd crush add 123 osd.123 <weight> <location ...> + ceph osd crush add osd.123 <weight> <location ...> + + This is because the id and crush name are redundant. Now only the + simple form is supported, where the osd name/id can either be a bare + id (integer) or name (osd.<id>):: + + ceph osd crush add osd.123 <weight> <location ...> + ceph osd crush add 123 <weight> <location ...> + +* There is now a maximum RADOS object size, configurable via 'osd max + object size', defaulting to 100 GB. Note that this has no effect on + RBD, CephFS, or radosgw, which all stripe over objects. If you are + using librados and storing objects larger than that, you will need + to adjust 'osd max object size', and should consider using smaller + objects instead. + +* The 'osd min down {reporters|reports}' config options have been + renamed to 'mon osd min down {reporters|reports}', and the + documentation has been updated to reflect that these options apply + to the monitors (who process failure reports) and not OSDs. If you + have adjusted these settings, please update your ``ceph.conf`` + accordingly. + + +Notable changes since v0.66 +--------------------------- + +* mon: sync improvements (performance and robustness) +* mon: many bug fixes (paxos and services) +* mon: fixed bugs in recovery and io rate reporting (negative/large values) +* mon: collect metadata on osd performance +* mon: generate health warnings from slow or stuck requests +* mon: expanded --format=<json|xml|...> support for monitor commands +* mon: scrub function for verifying data integrity +* mon, osd: fix old osdmap trimming logic +* mon: enable leveldb caching by default +* mon: more efficient storage of PG metadata +* ceph-rest-api: RESTful endpoint for administer cluster (mirrors CLI) +* rgw: multi-region support +* rgw: infrastructure to support georeplication of bucket and user metadata +* rgw: infrastructure to support georeplication of bucket data +* rgw: COPY object support between regions +* rbd: /etc/ceph/rbdmap file for mapping rbd images on startup +* osd: many bug fixes +* osd: limit number of incremental osdmaps sent to peers (could cause osds to be wrongly marked down) +* osd: more efficient small object recovery +* osd, librados: support for object namespaces +* osd: automatically enable xattrs on leveldb as necessary +* mds: fix bug in LOOKUPINO (used by nfs reexport) +* mds: fix O_TRUNC locking +* msgr: fixed race condition in inter-osd network communication +* msgr: fixed various memory leaks related to network sessions +* ceph-disk: fixes for unusual device names, partition detection +* hypertable: fixes for hypertable CephBroker bindings +* use SSE4.2 crc32c instruction if present + + +Notable changes since v0.61 "Cuttlefish" +---------------------------------------- + +* add 'config get' admin socket command +* ceph-conf: --show-config-value now reflects daemon defaults +* ceph-disk: add '[un]suppress-active DEV' command +* ceph-disk: avoid mounting over an existing osd in /var/lib/ceph/osd/* +* ceph-disk: fixes for unusual device names, partition detection +* ceph-disk: improved handling of odd device names +* ceph-disk: many fixes for RHEL/CentOS, Fedora, wheezy +* ceph-disk: simpler, more robust locking +* ceph-fuse, libcephfs: fix a few caps revocation bugs +* ceph-fuse, libcephfs: fix read zeroing at EOF +* ceph-fuse, libcephfs: fix request refcounting bug (hang on shutdown) +* ceph-fuse, libcephfs: fix truncatation bug on >4MB files (Yan, Zheng) +* ceph-fuse, libcephfs: fix for cap release/hang +* ceph-fuse: add ioctl support +* ceph-fuse: fixed long-standing O_NOATIME vs O_LAZY bug +* ceph-rest-api: RESTful endpoint for administer cluster (mirrors CLI) +* ceph, librados: fix resending of commands on mon reconnect +* daemons: create /var/run/ceph as needed +* debian wheezy: fix udev rules +* debian, specfile: packaging cleanups +* debian: fix upstart behavior with upgrades +* debian: rgw: stop daemon on uninstall +* debian: stop daemons on uninstall; fix dependencies +* hypertable: fixes for hypertable CephBroker bindings +* librados python binding cleanups +* librados python: fix xattrs > 4KB (Josh Durgin) +* librados: configurable max object size (default 100 GB) +* librados: new calls to administer the cluster +* librbd: ability to read from local replicas +* librbd: locking tests (Josh Durgin) +* librbd: make default options/features for newly created images (e.g., via qemu-img) configurable +* librbd: parallelize delete, rollback, flatten, copy, resize +* many many fixes from static code analysis (Danny Al-Gaaf) +* mds: fix O_TRUNC locking +* mds: fix bug in LOOKUPINO (used by nfs reexport) +* mds: fix rare hang after client restart +* mds: fix several bugs (Yan, Zheng) +* mds: many backpointer improvements (Yan, Zheng) +* mds: many fixes for mds clustering +* mds: misc stability fixes (Yan, Zheng, Greg Farnum) +* mds: new robust open-by-ino support (Yan, Zheng) +* mds: support robust lookup by ino number (good for NFS) (Yan, Zheng) +* mon, ceph: huge revamp of CLI and internal admin API. (Dan Mick) +* mon, osd: fix old osdmap trimming logic +* mon, osd: many memory leaks fixed +* mon: better trim/compaction behavior +* mon: collect metadata on osd performance +* mon: enable leveldb caching by default +* mon: expanded --format=<json|xml|...> support for monitor commands +* mon: fix election timeout +* mon: fix leveldb compression, trimming +* mon: fix start fork behavior +* mon: fix units in 'ceph df' output +* mon: fix validation of mds ids from CLI commands +* mon: fixed bugs in recovery and io rate reporting (negative/large values) +* mon: generate health warnings from slow or stuck requests +* mon: many bug fixes (paxos and services, sync) +* mon: many stability fixes (Joao Luis) +* mon: more efficient storage of PG metadata +* mon: new --extract-monmap to aid disaster recovery +* mon: new capability syntax +* mon: scrub function for verifying data integrity +* mon: simplify PaxosService vs Paxos interaction, fix readable/writeable checks +* mon: sync improvements (performance and robustness) +* mon: tuning, performance improvements +* msgr: fix various memory leaks +* msgr: fixed race condition in inter-osd network communication +* msgr: fixed various memory leaks related to network sessions +* osd, librados: support for object namespaces +* osd, mon: optionally dump leveldb transactions to a log +* osd: automatically enable xattrs on leveldb as necessary +* osd: avoid osd flapping from asymmetric network failure +* osd: break blacklisted client watches (David Zafman) +* osd: close narrow journal race +* osd: do not use fadvise(DONTNEED) on XFS (data corruption on power cycle) +* osd: fix for an op ordering bug +* osd: fix handling for split after upgrade from bobtail +* osd: fix incorrect mark-down of osds +* osd: fix internal heartbeart timeouts when scrubbing very large objects +* osd: fix memory/network inefficiency during deep scrub +* osd: fixed problem with front-side heartbeats and mixed clusters (David Zafman) +* osd: limit number of incremental osdmaps sent to peers (could cause osds to be wrongly marked down) +* osd: many bug fixes +* osd: monitor both front and back interfaces +* osd: more efficient small object recovery +* osd: new writeback throttling (for less bursty write performance) (Sam Just) +* osd: pg log (re)writes are now vastly more efficient (faster peering) (Sam Just) +* osd: ping/heartbeat on public and private interfaces +* osd: prioritize recovery for degraded PGs +* osd: re-use partially deleted PG contents when present (Sam Just) +* osd: recovery and peering performance improvements +* osd: resurrect partially deleted PGs +* osd: verify both front and back network are working before rejoining cluster +* rados: clonedata command for cli +* radosgw-admin: create keys for new users by default +* rbd: /etc/ceph/rbdmap file for mapping rbd images on startup +* rgw: COPY object support between regions +* rgw: fix CORS bugs +* rgw: fix locking issue, user operation mask, +* rgw: fix radosgw-admin buckets list (Yehuda Sadeh) +* rgw: fix usage log scanning for large, untrimmed logs +* rgw: handle deep uri resources +* rgw: infrastructure to support georeplication of bucket and user metadata +* rgw: infrastructure to support georeplication of bucket data +* rgw: multi-region support +* sysvinit: fix enumeration of local daemons +* sysvinit: fix osd crush weight calculation when using -a +* sysvinit: handle symlinks in /var/lib/ceph/osd/* +* use SSE4.2 crc32c instruction if present + + +v0.66 +===== + +Upgrading +--------- + +* There is now a configurable maximum rados object size, defaulting to 100 GB. If you + are using librados and storing objects larger than that, you will need to adjust + 'osd max object size', and should consider using smaller objects instead. + +Notable changes +--------------- + +* osd: pg log (re)writes are now vastly more efficient (faster peering) (Sam Just) +* osd: fixed problem with front-side heartbeats and mixed clusters (David Zafman) +* mon: tuning, performance improvements +* mon: simplify PaxosService vs Paxos interaction, fix readable/writeable checks +* rgw: fix radosgw-admin buckets list (Yehuda Sadeh) +* mds: support robust lookup by ino number (good for NFS) (Yan, Zheng) +* mds: fix several bugs (Yan, Zheng) +* ceph-fuse, libcephfs: fix truncatation bug on >4MB files (Yan, Zheng) +* ceph/librados: fix resending of commands on mon reconnect +* librados python: fix xattrs > 4KB (Josh Durgin) +* librados: configurable max object size (default 100 GB) +* msgr: fix various memory leaks +* ceph-fuse: fixed long-standing O_NOATIME vs O_LAZY bug +* ceph-fuse, libcephfs: fix request refcounting bug (hang on shutdown) +* ceph-fuse, libcephfs: fix read zeroing at EOF +* ceph-conf: --show-config-value now reflects daemon defaults +* ceph-disk: simpler, more robust locking +* ceph-disk: avoid mounting over an existing osd in /var/lib/ceph/osd/* +* sysvinit: handle symlinks in /var/lib/ceph/osd/* + + +v0.65 +===== + +Upgrading +--------- + +* Huge revamp of the 'ceph' command-line interface implementation. + The ``ceph-common`` client library needs to be upgrade before + ``ceph-mon`` is restarted in order to avoid problems using the CLI + (the old ``ceph`` client utility cannot talk to the new + ``ceph-mon``). + +* The CLI is now very careful about sending the 'status' one-liner + output to stderr and command output to stdout. Scripts relying on + output should take care. + +* The 'ceph osd tell ...' and 'ceph mon tell ...' commands are no + longer supported. Any callers should use:: + + ceph tell osd.<id or *> ... + ceph tell mon.<id or name or *> ... + + The 'ceph mds tell ...' command is still there, but will soon also + transition to 'ceph tell mds.<id or name or \*> ...' + +* The 'ceph osd crush add ...' command used to take one of two forms:: + + ceph osd crush add 123 osd.123 <weight> <location ...> + ceph osd crush add osd.123 <weight> <location ...> + + This is because the id and crush name are redundant. Now only the + simple form is supported, where the osd name/id can either be a bare + id (integer) or name (osd.<id>):: + + ceph osd crush add osd.123 <weight> <location ...> + ceph osd crush add 123 <weight> <location ...> + +* There is now a maximum RADOS object size, configurable via 'osd max + object size', defaulting to 100 GB. Note that this has no effect on + RBD, CephFS, or radosgw, which all stripe over objects. + + +Notable changes +--------------- + +* mon, ceph: huge revamp of CLI and internal admin API. (Dan Mick) +* mon: new capability syntax +* osd: do not use fadvise(DONTNEED) on XFS (data corruption on power cycle) +* osd: recovery and peering performance improvements +* osd: new writeback throttling (for less bursty write performance) (Sam Just) +* osd: ping/heartbeat on public and private interfaces +* osd: avoid osd flapping from asymmetric network failure +* osd: re-use partially deleted PG contents when present (Sam Just) +* osd: break blacklisted client watches (David Zafman) +* mon: many stability fixes (Joao Luis) +* mon, osd: many memory leaks fixed +* mds: misc stability fixes (Yan, Zheng, Greg Farnum) +* mds: many backpointer improvements (Yan, Zheng) +* mds: new robust open-by-ino support (Yan, Zheng) +* ceph-fuse, libcephfs: fix a few caps revocation bugs +* librados: new calls to administer the cluster +* librbd: locking tests (Josh Durgin) +* ceph-disk: improved handling of odd device names +* ceph-disk: many fixes for RHEL/CentOS, Fedora, wheezy +* many many fixes from static code analysis (Danny Al-Gaaf) +* daemons: create /var/run/ceph as needed + + +v0.64 +===== + +Upgrading +--------- + +* New pools now have the HASHPSPOOL flag set by default to provide + better distribution over OSDs. Support for this feature was + introduced in v0.59 and Linux kernel version v3.9. If you wish to + access the cluster from an older kernel, set the 'osd pool default + flag hashpspool = false' option in your ceph.conf prior to creating + the cluster or creating new pools. Note that the presence of any + pool in the cluster with the flag enabled will make the OSD require + support from all clients. + +Notable changes +--------------- + +* osd: monitor both front and back interfaces +* osd: verify both front and back network are working before rejoining cluster +* osd: fix memory/network inefficiency during deep scrub +* osd: fix incorrect mark-down of osds +* mon: fix start fork behavior +* mon: fix election timeout +* mon: better trim/compaction behavior +* mon: fix units in 'ceph df' output +* mon, osd: misc memory leaks +* librbd: make default options/features for newly created images (e.g., via qemu-img) configurable +* mds: many fixes for mds clustering +* mds: fix rare hang after client restart +* ceph-fuse: add ioctl support +* ceph-fuse/libcephfs: fix for cap release/hang +* rgw: handle deep uri resources +* rgw: fix CORS bugs +* ceph-disk: add '[un]suppress-active DEV' command +* debian: rgw: stop daemon on uninstall +* debian: fix upstart behavior with upgrades + + +v0.63 +===== + +Upgrading +--------- + +* The 'osd min down {reporters|reports}' config options have been + renamed to 'mon osd min down {reporters|reports}', and the + documentation has been updated to reflect that these options apply + to the monitors (who process failure reports) and not OSDs. If you + have adjusted these settings, please update your ``ceph.conf`` + accordingly. + +Notable Changes +--------------- + +* librbd: parallelize delete, rollback, flatten, copy, resize +* librbd: ability to read from local replicas +* osd: resurrect partially deleted PGs +* osd: prioritize recovery for degraded PGs +* osd: fix internal heartbeart timeouts when scrubbing very large objects +* osd: close narrow journal race +* rgw: fix usage log scanning for large, untrimmed logs +* rgw: fix locking issue, user operation mask, +* initscript: fix osd crush weight calculation when using -a +* initscript: fix enumeration of local daemons +* mon: several fixes to paxos, sync +* mon: new --extract-monmap to aid disaster recovery +* mon: fix leveldb compression, trimming +* add 'config get' admin socket command +* rados: clonedata command for cli +* debian: stop daemons on uninstall; fix dependencies +* debian wheezy: fix udev rules +* many many small fixes from coverity scan + + +v0.62 +===== + +Notable Changes +--------------- + +* mon: fix validation of mds ids from CLI commands +* osd: fix for an op ordering bug +* osd, mon: optionally dump leveldb transactions to a log +* osd: fix handling for split after upgrade from bobtail +* debian, specfile: packaging cleanups +* radosgw-admin: create keys for new users by default +* librados python binding cleanups +* misc code cleanups |