1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
|
ceph (10.2.5-1) unstable; urgency=medium
## Upgrades from Debian Jessie
Online upgrades from Ceph versions prior to Hammer (0.94.x) are not
supported by upstream. As Debian Jessie has Ceph Firefly (0.80.x) an
online upgrade from Jessie to Stretch is not possible. You have to first
shutdown all Ceph daemons on all nodes, upgrade everything to the new
version and start all daemons again.
Ceph daemons are not automatically restarted on upgrade to minimize
disruption. You have to manually restart them after the upgrade.
-- Gaudenz Steinlin <gaudenz@debian.org> Sun, 08 Jan 2017 14:57:35 +0100
ceph (9.2.0-1) experimental; urgency=medium
## systemd Enablement
For all distributions that support systemd (Debian Jessie 8.x,
Ubuntu >= 16.04), Ceph daemons are now managed using upstream provided
systemd files instead of the legacy sysvinit scripts or distro provided
systemd files. For example:
systemctl start ceph.target # start all daemons
systemctl status ceph-osd@12 # check status of osd.12
To upgrade existing deployments that use the older systemd service
configurations (Ubuntu >= 15.04, Debian >= Jessie), you need to switch
to using the new ceph-mon@ service:
systemctl stop ceph-mon
systemctl disable ceph-mon
systemctl start ceph-mon@`hostname`
systemctl enable ceph-mon@`hostname`
and also enable the ceph target post upgrade:
systemctl enable ceph.target
The main notable distro that is *not* using systemd is Ubuntu 14.04
(The next Ubuntu LTS, 16.04, will use systemd instead of upstart).
## Ceph daemons no longer run as root
Ceph daemons now run as user and group 'ceph' by default. The
ceph user has a static UID assigned by Debian to ensure consistency
across servers within a Ceph deployment.
If your systems already have a ceph user, upgrading the package will cause
problems. We suggest you first remove or rename the existing 'ceph' user
and 'ceph' group before upgrading.
When upgrading, administrators have two options:
1. Add the following line to 'ceph.conf' on all hosts:
setuser match path = /var/lib/ceph/$type/$cluster-$id
This will make the Ceph daemons run as root (i.e., not drop
privileges and switch to user ceph) if the daemon's data
directory is still owned by root. Newly deployed daemons will
be created with data owned by user ceph and will run with
reduced privileges, but upgraded daemons will continue to run as
root.
2. Fix the data ownership during the upgrade. This is the
preferred option, but it is more work and can be very time
consuming. The process for each host is to:
1. Upgrade the ceph package. This creates the ceph user and group. For
example:
apt-get install ceph
NOTE: the permissions on /var/lib/ceph/mon will be set to ceph:ceph
as part of the package upgrade process on existing *systemd*
based installations; the ceph-mon systemd service will be
automatically restarted as part of the upgrade. All other
filesystem permissions on systemd based installs will
remain unmodified by the upgrade.
2. Stop the daemon(s):
systemctl stop ceph-osd@* # debian, ubuntu >= 15.04
stop ceph-all # ubuntu 14.04
3. Fix the ownership:
chown -R ceph:ceph /var/lib/ceph
4. Restart the daemon(s):
start ceph-all # ubuntu 14.04
systemctl start ceph.target # debian, ubuntu >= 15.04
Alternatively, the same process can be done with a single daemon
type, for example by stopping only monitors and chowning only
'/var/lib/ceph/osd'.
## KeyValueStore OSD on-disk format changes
The on-disk format for the experimental KeyValueStore OSD backend has
changed. You will need to remove any OSDs using that backend before you
upgrade any test clusters that use it.
## Deprecated commands
'ceph scrub', 'ceph compact' and 'ceph sync force' are now DEPRECATED.
Users should instead use 'ceph mon scrub', 'ceph mon compact' and
'ceph mon sync force'.
## Full pool behaviour
When a pool quota is reached, librados operations now block indefinitely,
the same way they do when the cluster fills up. (Previously they would
return -ENOSPC). By default, a full cluster or pool will now block. If
your librados application can handle ENOSPC or EDQUOT errors gracefully,
you can get error returns instead by using the new librados
OPERATION_FULL_TRY flag.
-- James Page <james.page@ubuntu.com> Mon, 30 Nov 2015 09:23:09 +0000
ceph (0.80.9-2) unstable; urgency=medium
## CRUSH fixes in 0.80.9
The 0.80.9 point release fixes several issues with CRUSH that trigger excessive
data migration when adjusting OSD weights. These are most obvious when a very
small weight change (e.g., a change from 0 to .01) triggers a large amount of
movement, but the same set of bugs can also lead to excessive (though less
noticeable) movement in other cases.
However, because the bug may already have affected your cluster, fixing it
may trigger movement back to the more correct location. For this reason, you
must manually opt-in to the fixed behavior.
In order to set the new tunable to correct the behavior:
ceph osd crush set-tunable straw_calc_version 1
Note that this change will have no immediate effect. However, from this
point forward, any ‘straw’ bucket in your CRUSH map that is adjusted will get
non-buggy internal weights, and that transition may trigger some rebalancing.
You can estimate how much rebalancing will eventually be necessary on your
cluster with:
ceph osd getcrushmap -o /tmp/cm
crushtool -i /tmp/cm --num-rep 3 --test --show-mappings > /tmp/a 2>&1
crushtool -i /tmp/cm --set-straw-calc-version 1 -o /tmp/cm2
crushtool -i /tmp/cm2 --reweight -o /tmp/cm2
crushtool -i /tmp/cm2 --num-rep 3 --test --show-mappings > /tmp/b 2>&1
wc -l /tmp/a # num total mappings
diff -u /tmp/a /tmp/b | grep -c ^+ # num changed mappings
Divide the total number of lines in /tmp/a with the number of lines
changed. We've found that most clusters are under 10%.
You can force all of this rebalancing to happen at once with:
ceph osd crush reweight-all
Otherwise, it will happen at some unknown point in the future when
CRUSH weights are next adjusted.
## Mapping rbd devices with rbdmap on systemd systems
If you have setup rbd mappings in /etc/ceph/rbdmap and corresponding mounts
in /etc/fstab things might break with systemd because systemd waits for the
rbd device to appear before the legacy rbdmap init file has a chance to run
and drops into emergency mode if it times out.
This can be fixed by adding the nofail option in /etc/fstab to all rbd
backed mount points. With this systemd does not wait for the device and
proceeds with the boot process. After rbdmap mapped the device, systemd
detects the new device and mounts the file system.
-- Gaudenz Steinlin <gaudenz@debian.org> Mon, 04 May 2015 22:49:48 +0200
|
