1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
|
Cache pool
==========
Purpose
-------
Use a pool of fast storage devices (probably SSDs) and use it as a
cache for an existing slower and larger pool.
Use a replicated pool as a front-end to service most I/O, and destage
cold data to a separate erasure coded pool that does not currently (and
cannot efficiently) handle the workload.
We should be able to create and add a cache pool to an existing pool
of data, and later remove it, without disrupting service or migrating
data around.
Use cases
---------
Read-write pool, writeback
~~~~~~~~~~~~~~~~~~~~~~~~~~
We have an existing data pool and put a fast cache pool "in front" of
it. Writes will go to the cache pool and immediately ack. We flush
them back to the data pool based on the defined policy.
Read-only pool, weak consistency
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We have an existing data pool and add one or more read-only cache
pools. We copy data to the cache pool(s) on read. Writes are
forwarded to the original data pool. Stale data is expired from the
cache pools based on the defined policy.
This is likely only useful for specific applications with specific
data access patterns. It may be a match for rgw, for example.
Interface
---------
Set up a read/write cache pool foo-hot for pool foo::
ceph osd tier add foo foo-hot
ceph osd tier cache-mode foo-hot writeback
Direct all traffic for foo to foo-hot::
ceph osd tier set-overlay foo foo-hot
Set the target size and enable the tiering agent for foo-hot::
ceph osd pool set foo-hot hit_set_type bloom
ceph osd pool set foo-hot hit_set_count 1
ceph osd pool set foo-hot hit_set_period 3600 # 1 hour
ceph osd pool set foo-hot target_max_bytes 1000000000000 # 1 TB
ceph osd pool set foo-hot min_read_recency_for_promote 1
ceph osd pool set foo-hot min_write_recency_for_promote 1
Drain the cache in preparation for turning it off::
ceph osd tier cache-mode foo-hot forward
rados -p foo-hot cache-flush-evict-all
When cache pool is finally empty, disable it::
ceph osd tier remove-overlay foo
ceph osd tier remove foo foo-hot
Read-only pools with lazy consistency::
ceph osd tier add foo foo-east
ceph osd tier cache-mode foo-east readonly
ceph osd tier add foo foo-west
ceph osd tier cache-mode foo-west readonly
Tiering agent
-------------
The tiering policy is defined as properties on the cache pool itself.
HitSet metadata
~~~~~~~~~~~~~~~
First, the agent requires HitSet information to be tracked on the
cache pool in order to determine which objects in the pool are being
accessed. This is enabled with::
ceph osd pool set foo-hot hit_set_type bloom
ceph osd pool set foo-hot hit_set_count 1
ceph osd pool set foo-hot hit_set_period 3600 # 1 hour
The supported HitSet types include 'bloom' (a bloom filter, the
default), 'explicit_hash', and 'explicit_object'. The latter two
explicitly enumerate accessed objects and are less memory efficient.
They are there primarily for debugging and to demonstrate pluggability
for the infrastructure. For the bloom filter type, you can additionally
define the false positive probability for the bloom filter (default is 0.05)::
ceph osd pool set foo-hot hit_set_fpp 0.15
The hit_set_count and hit_set_period define how much time each HitSet
should cover, and how many such HitSets to store. Binning accesses
over time allows Ceph to independently determine whether an object was
accessed at least once and whether it was accessed more than once over
some time period ("age" vs "temperature").
The ``min_read_recency_for_promote`` defines how many HitSets to check for the
existence of an object when handling a read operation. The checking result is
used to decide whether to promote the object asynchronously. Its value should be
between 0 and ``hit_set_count``. If it's set to 0, the object is always promoted.
If it's set to 1, the current HitSet is checked. And if this object is in the
current HitSet, it's promoted. Otherwise not. For the other values, the exact
number of archive HitSets are checked. The object is promoted if the object is
found in any of the most recent ``min_read_recency_for_promote`` HitSets.
A similar parameter can be set for the write operation, which is
``min_write_recency_for_promote``. ::
ceph osd pool set {cachepool} min_read_recency_for_promote 1
ceph osd pool set {cachepool} min_write_recency_for_promote 1
Note that the longer the ``hit_set_period`` and the higher the
``min_read_recency_for_promote``/``min_write_recency_for_promote`` the more RAM
will be consumed by the ceph-osd process. In particular, when the agent is active
to flush or evict cache objects, all hit_set_count HitSets are loaded into RAM.
Cache mode
~~~~~~~~~~
The most important policy is the cache mode:
ceph osd pool set foo-hot cache-mode writeback
The supported modes are 'none', 'writeback', 'forward', and
'readonly'. Most installations want 'writeback', which will write
into the cache tier and only later flush updates back to the base
tier. Similarly, any object that is read will be promoted into the
cache tier.
The 'forward' mode is intended for when the cache is being disabled
and needs to be drained. No new objects will be promoted or written
to the cache pool unless they are already present. A background
operation can then do something like::
rados -p foo-hot cache-try-flush-evict-all
rados -p foo-hot cache-flush-evict-all
to force all data to be flushed back to the base tier.
The 'readonly' mode is intended for read-only workloads that do not
require consistency to be enforced by the storage system. Writes will
be forwarded to the base tier, but objects that are read will get
promoted to the cache. No attempt is made by Ceph to ensure that the
contents of the cache tier(s) are consistent in the presence of object
updates.
Cache sizing
~~~~~~~~~~~~
The agent performs two basic functions: flushing (writing 'dirty'
cache objects back to the base tier) and evicting (removing cold and
clean objects from the cache).
The thresholds at which Ceph will flush or evict objects is specified
relative to a 'target size' of the pool. For example::
ceph osd pool set foo-hot cache_target_dirty_ratio .4
ceph osd pool set foo-hot cache_target_dirty_high_ratio .6
ceph osd pool set foo-hot cache_target_full_ratio .8
will begin flushing dirty objects when 40% of the pool is dirty and begin
evicting clean objects when we reach 80% of the target size.
The target size can be specified either in terms of objects or bytes::
ceph osd pool set foo-hot target_max_bytes 1000000000000 # 1 TB
ceph osd pool set foo-hot target_max_objects 1000000 # 1 million objects
Note that if both limits are specified, Ceph will begin flushing or
evicting when either threshold is triggered.
Other tunables
~~~~~~~~~~~~~~
You can specify a minimum object age before a recently updated object is
flushed to the base tier::
ceph osd pool set foo-hot cache_min_flush_age 600 # 10 minutes
You can specify the minimum age of an object before it will be evicted from
the cache tier::
ceph osd pool set foo-hot cache_min_evict_age 1800 # 30 minutes
|
