1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
|
===============================
RBD Persistent Read-only Cache
===============================
.. index:: Ceph Block Device; Persistent Read-only Cache
Shared, Read-only Parent Image Cache
====================================
`Cloned RBD images`_ usually modify only a small fraction of the parent
image. For example, in a VDI use-case, VMs are cloned from the same
base image and initially differ only by hostname and IP address. During
booting, all of these VMs read portions of the same parent
image data. If we have a local cache of the parent
image, this speeds up reads on the caching host. We also achieve
reduction of client-to-cluster network traffic.
RBD cache must be explicitly enabled in
``ceph.conf``. The ``ceph-immutable-object-cache`` daemon is responsible for
caching the parent content on the local disk, and future reads on that data
will be serviced from the local cache.
.. note:: RBD shared read-only parent image cache requires the Ceph Nautilus release or later.
.. ditaa::
+--------------------------------------------------------+
| QEMU |
+--------------------------------------------------------+
| librbd (cloned images) |
+-------------------+-+----------------------------------+
| librados | | ceph--immutable--object--cache |
+-------------------+ +----------------------------------+
| OSDs/Mons | | local cached parent image |
+-------------------+ +----------------------------------+
Enable RBD Shared Read-only Parent Image Cache
----------------------------------------------
To enable RBD shared read-only parent image cache, the following Ceph settings
need to added in the ``[client]`` `section`_ of your ``ceph.conf`` file::
rbd parent cache enabled = true
rbd plugins = parent_cache
Immutable Object Cache Daemon
=============================
Introduction and Generic Settings
---------------------------------
The ``ceph-immutable-object-cache`` daemon is responsible for caching parent
image content within its local caching directory. Using SSDs as the underlying
storage is recommended because doing so provides better performance.
The key components of the daemon are:
#. **Domain socket based IPC:** The daemon listens on a local domain socket at
startup and waits for connections from librbd clients.
#. **LRU based promotion/demotion policy:** The daemon maintains in-memory
statistics of cache hits for each cache file. It demotes the cold cache
if capacity reaches the configured threshold.
#. **File-based caching store:** The daemon maintains a simple file-based cache
store. On promotion, the RADOS objects are fetched from RADOS cluster and
stored in the local caching directory.
When each cloned RBD image is opened, ``librbd`` tries to connect to the cache
daemon through its Unix domain socket. After ``librbd`` is successfully
connected, it coordinates with the daemon upon every subsequent read. In the
case of an uncached read, the daemon promotes the RADOS object to the local
caching directory and the next read of the object is serviced from the cache.
The daemon maintains simple LRU statistics, which are used to evict cold cache
files when required (for example, when the cache is at capacity and under
pressure).
Here are some important cache configuration settings:
``immutable_object_cache_sock``
:Description: The path to the domain socket used for communication between
librbd clients and the ceph-immutable-object-cache daemon.
:Type: String
:Required: No
:Default: ``/var/run/ceph/immutable_object_cache_sock``
``immutable_object_cache_path``
:Description: The immutable object cache data directory.
:Type: String
:Required: No
:Default: ``/tmp/ceph_immutable_object_cache``
``immutable_object_cache_max_size``
:Description: The max size for immutable cache.
:Type: Size
:Required: No
:Default: ``1G``
``immutable_object_cache_watermark``
:Description: The high-water mark for the cache. The value is between (0, 1).
If the cache size reaches this threshold the daemon will start
to delete cold cache based on LRU statistics.
:Type: Float
:Required: No
:Default: ``0.9``
The ``ceph-immutable-object-cache`` daemon is available within the optional
``ceph-immutable-object-cache`` distribution package.
.. important:: ``ceph-immutable-object-cache`` daemon requires the ability to
connect RADOS clusters.
Running the Immutable Object Cache Daemon
-----------------------------------------
``ceph-immutable-object-cache`` daemon should use a unique Ceph user ID.
To `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create``
command, user name, monitor caps, and OSD caps::
ceph auth get-or-create client.ceph-immutable-object-cache.{unique id} mon 'allow r' osd 'profile rbd-read-only'
The ``ceph-immutable-object-cache`` daemon can be managed by ``systemd`` by specifying the user
ID as the daemon instance::
systemctl enable ceph-immutable-object-cache@ceph-immutable-object-cache.{unique id}
The ``ceph-immutable-object-cache`` can also be run in foreground by ``ceph-immutable-object-cache`` command::
ceph-immutable-object-cache -f --log-file={log_path}
QOS Settings
------------
The immutable object cache supports throttling, controlled by the following settings:
``immutable_object_cache_qos_schedule_tick_min``
:Description: Minimum schedule tick for immutable object cache.
:Type: Milliseconds
:Required: No
:Default: ``50``
``immutable_object_cache_qos_iops_limit``
:Description: The desired immutable object cache IO operations limit per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``
``immutable_object_cache_qos_iops_burst``
:Description: The desired burst limit of immutable object cache IO operations.
:Type: Unsigned Integer
:Required: No
:Default: ``0``
``immutable_object_cache_qos_iops_burst_seconds``
:Description: The desired burst duration in seconds of immutable object cache IO operations.
:Type: Seconds
:Required: No
:Default: ``1``
``immutable_object_cache_qos_bps_limit``
:Description: The desired immutable object cache IO bytes limit per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``
``immutable_object_cache_qos_bps_burst``
:Description: The desired burst limit of immutable object cache IO bytes.
:Type: Unsigned Integer
:Required: No
:Default: ``0``
``immutable_object_cache_qos_bps_burst_seconds``
:Description: The desired burst duration in seconds of immutable object cache IO bytes.
:Type: Seconds
:Required: No
:Default: ``1``
.. _Cloned RBD Images: ../rbd-snapshot/#layering
.. _section: ../../rados/configuration/ceph-conf/#configuration-sections
.. _create a Ceph user: ../../rados/operations/user-management#add-a-user
|
