1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
|
Serialization (encode/decode)
=============================
When a structure is sent over the network or written to disk, it is
encoded into a string of bytes. Usually (but not always -- multiple
serialization facilities coexist in Ceph) serializable structures
have ``encode`` and ``decode`` methods that write and read from
``bufferlist`` objects representing byte strings.
Terminology
-----------
It is best to think not in the domain of daemons and clients but
encoders and decoders. An encoder serializes a structure into a bufferlist
while a decoder does the opposite.
Encoders and decoders can be referred collectively as dencoders.
Dencoders (both encoders and docoders) live within daemons and clients.
For instance, when an RBD client issues an IO operation, it prepares
an instance of the ``MOSDOp`` structure and encodes it into a bufferlist
that is put on the wire.
An OSD reads these bytes and decodes them back into an ``MOSDOp`` instance.
Here encoder was used by the client while decoder by the OSD. However,
these roles can swing -- just imagine handling of the response: OSD encodes
the ``MOSDOpReply`` while RBD clients decode.
Encoder and decoder operate accordingly to a format which is defined
by a programmer by implementing the ``encode`` and ``decode`` methods.
Principles for format change
----------------------------
It is not unusual that the format of serialization changes. This
process requires careful attention from during both development
and review.
The general rule is that a decoder must understand what had been
encoded by an encoder. Most of the problems come from ensuring
that compatibility continues between old decoders and new encoders
as well as new decoders and old decoders. One should assume
that -- if not otherwise derogated -- any mix (old/new) is
possible in a cluster. There are 2 main reasons for that:
1. Upgrades. Although there are recommendations related to the order
of entity types (mons/osds/clients), it is not mandatory and
no assumption should be made about it.
2. Huge variability of client versions. It was always the case
that kernel (and thus kernel clients) upgrades are decoupled
from Ceph upgrades. Moreover, proliferation of containerization
bring the variability even to e.g. ``librbd`` -- now user space
libraries live on the container own.
With this being said, there are few rules limiting the degree
of interoperability between dencoders:
* ``n-2`` for dencoding between daemons,
* ``n-3`` hard requirement for client-involved scenarios,
* ``n-3..`` soft requirements for clinet-involved scenarios. Ideally
every client should be able to talk any version of daemons.
As the underlying reasons are the same, the rules dencoders
follow are virtually the same as for deprecations of our features
bits. See the ``Notes on deprecation`` in ``src/include/ceph_features.h``.
Frameworks
----------
Currently multiple genres of dencoding helpers co-exist.
* encoding.h (the most proliferated one),
* denc.h (performance optimized, seen mostly in ``BlueStore``),
* the `Message` hierarchy.
Although details vary, the interoperability rules stay the same.
Adding a field to a structure
-----------------------------
You can see examples of this all over the Ceph code, but here's an
example:
.. code-block:: cpp
class AcmeClass
{
int member1;
std::string member2;
void encode(bufferlist &bl)
{
ENCODE_START(1, 1, bl);
::encode(member1, bl);
::encode(member2, bl);
ENCODE_FINISH(bl);
}
void decode(bufferlist::iterator &bl)
{
DECODE_START(1, bl);
::decode(member1, bl);
::decode(member2, bl);
DECODE_FINISH(bl);
}
};
The ``ENCODE_START`` macro writes a header that specifies a *version* and
a *compat_version* (both initially 1). The message version is incremented
whenever a change is made to the encoding. The compat_version is incremented
only if the change will break existing decoders -- decoders are tolerant
of trailing bytes, so changes that add fields at the end of the structure
do not require incrementing compat_version.
The ``DECODE_START`` macro takes an argument specifying the most recent
message version that the code can handle. This is compared with the
compat_version encoded in the message, and if the message is too new then
an exception will be thrown. Because changes to compat_version are rare,
this isn't usually something to worry about when adding fields.
In practice, changes to encoding usually involve simply adding the desired fields
at the end of the ``encode`` and ``decode`` functions, and incrementing
the versions in ``ENCODE_START`` and ``DECODE_START``. For example, here's how
to add a third field to ``AcmeClass``:
.. code-block:: cpp
class AcmeClass
{
int member1;
std::string member2;
std::vector<std::string> member3;
void encode(bufferlist &bl)
{
ENCODE_START(2, 1, bl);
::encode(member1, bl);
::encode(member2, bl);
::encode(member3, bl);
ENCODE_FINISH(bl);
}
void decode(bufferlist::iterator &bl)
{
DECODE_START(2, bl);
::decode(member1, bl);
::decode(member2, bl);
if (struct_v >= 2) {
::decode(member3, bl);
}
DECODE_FINISH(bl);
}
};
Note that the compat_version did not change because the encoded message
will still be decodable by versions of the code that only understand
version 1 -- they will just ignore the trailing bytes where we encode ``member3``.
In the ``decode`` function, decoding the new field is conditional: this is
because we might still be passed older-versioned messages that do not
have the field. The ``struct_v`` variable is a local set by the ``DECODE_START``
macro.
# Into the weeeds
The append-extendability of our dencoders is a result of the forward
compatibility that the ``ENCODE_START`` and ``DECODE_FINISH`` macros bring.
They are implementing extendibility facilities. An encoder, when filling
the bufferlist, prepends three fields: version of the current format,
minimal version of a decoder compatible with it and the total size of
all encoded fields.
.. code-block:: cpp
/**
* start encoding block
*
* @param v current (code) version of the encoding
* @param compat oldest code version that can decode it
* @param bl bufferlist to encode to
*
*/
#define ENCODE_START(v, compat, bl) \
__u8 struct_v = v; \
__u8 struct_compat = compat; \
ceph_le32 struct_len; \
auto filler = (bl).append_hole(sizeof(struct_v) + \
sizeof(struct_compat) + sizeof(struct_len)); \
const auto starting_bl_len = (bl).length(); \
using ::ceph::encode; \
do {
The ``struct_len`` field allows the decoder to eat all the bytes that were
left undecoded in the user-provided ``decode`` implementation.
Analogically, decoders tracks how much input has been decoded in the
user-provided ``decode`` methods.
.. code-block:: cpp
#define DECODE_START(bl) \
unsigned struct_end = 0; \
__u32 struct_len; \
decode(struct_len, bl); \
... \
struct_end = bl.get_off() + struct_len; \
} \
do {
Decoder uses this information to discard the extra bytes it does not
understand. Advancing bufferlist is critical as dencoders tend to be nested;
just leaving it intact would work only for the very last ``deocde`` call
in a nested structure.
.. code-block:: cpp
#define DECODE_FINISH(bl) \
} while (false); \
if (struct_end) { \
... \
if (bl.get_off() < struct_end) \
bl += struct_end - bl.get_off(); \
}
This entire, cooperative mechanism allows encoder (its further revisions)
to generate more byte stream (due to e.g. adding a new field at the end)
and not worry that the residue will crash older decoder revisions.
|
