1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
|
======================
Capabilities in CephFS
======================
When a client wants to operate on an inode, it will query the MDS in various
ways, which will then grant the client a set of **capabilities**. This
grants the client permissions to operate on the inode in various ways. One
of the major differences from other network file systems (e.g NFS or SMB) is
that the capabilities granted are quite granular, and it's possible that
multiple clients can hold different capabilities on the same inodes.
Types of Capabilities
---------------------
There are several "generic" capability bits. These denote what sort of ability
the capability grants.
::
/* generic cap bits */
#define CEPH_CAP_GSHARED 1 /* (metadata) client can read (s) */
#define CEPH_CAP_GEXCL 2 /* (metadata) client can read and update (x) */
#define CEPH_CAP_GCACHE 4 /* (file) client can cache reads (c) */
#define CEPH_CAP_GRD 8 /* (file) client can read (r) */
#define CEPH_CAP_GWR 16 /* (file) client can write (w) */
#define CEPH_CAP_GBUFFER 32 /* (file) client can buffer writes (b) */
#define CEPH_CAP_GWREXTEND 64 /* (file) client can extend EOF (a) */
#define CEPH_CAP_GLAZYIO 128 /* (file) client can perform lazy io (l) */
These are then shifted by a particular number of bits. These denote a part of
the inode's data or metadata on which the capability is being granted:
::
/* per-lock shift */
#define CEPH_CAP_SAUTH 2 /* A */
#define CEPH_CAP_SLINK 4 /* L */
#define CEPH_CAP_SXATTR 6 /* X */
#define CEPH_CAP_SFILE 8 /* F */
Only certain generic cap types are ever granted for some of those "shifts",
however. In particular, only the FILE shift ever has more than the first two
bits.
::
| AUTH | LINK | XATTR | FILE
2 4 6 8
From the above, we get a number of constants, that are generated by taking
each bit value and shifting to the correct bit in the word:
::
#define CEPH_CAP_AUTH_SHARED (CEPH_CAP_GSHARED << CEPH_CAP_SAUTH)
These bits can then be or'ed together to make a bitmask denoting a set of
capabilities.
There is one exception:
::
#define CEPH_CAP_PIN 1 /* no specific capabilities beyond the pin */
The "pin" just pins the inode into memory, without granting any other caps.
Graphically:
::
+---+---+---+---+---+---+---+---+
| p | _ |As x |Ls x |Xs x |
+---+---+---+---+---+---+---+---+
|Fs x c r w b a l |
+---+---+---+---+---+---+---+---+
The second bit is currently unused.
Abilities granted by each cap
-----------------------------
While that is how capabilities are granted (and communicated), the important
bit is what they actually allow the client to do:
* **PIN**: This just pins the inode into memory. This is sufficient to allow
the client to get to the inode number, as well as other immutable things like
major or minor numbers in a device inode, or symlink contents.
* **AUTH**: This grants the ability to get to the authentication-related metadata.
In particular, the owner, group and mode. Note that doing a full permission
check may require getting at ACLs as well, which are stored in xattrs.
* **LINK**: The link count of the inode.
* **XATTR**: Ability to access or manipulate xattrs. Note that since ACLs are
stored in xattrs, it's also sometimes necessary to access them when checking
permissions.
* **FILE**: This is the big one. This allows the client to access and manipulate
file data. It also covers certain metadata relating to file data -- the
size, mtime, atime and ctime, in particular.
Shorthand
---------
Note that the client logging can also present a compact representation of the
capabilities. For example:
::
pAsLsXsFs
The 'p' represents the pin. Each capital letter corresponds to the shift
values, and the lowercase letters after each shift are for the actual
capabilities granted in each shift.
The relation between the lock states and the capabilities
---------------------------------------------------------
In MDS there are four different locks for each inode, they are simplelock,
scatterlock, filelock and locallock. Each lock has several different lock
states, and the MDS will issue capabilities to clients based on the lock
state.
In each state the MDS Locker will always try to issue all the capabilities to the
clients allowed, even some capabilities are not needed or wanted by the clients,
as pre-issuing capabilities could reduce latency in some cases.
If there is only one client, usually it will be the loner client for all the inodes.
While in multiple clients case, the MDS will try to calculate a loner client out for
each inode depending on the capabilities the clients (needed | wanted), but usually
it will fail. The loner client will always get all the capabilities.
The filelock will control files' partial metadatas' and the file contents' access
permissions. The metadatas include **mtime**, **atime**, **size**, etc.
* **Fs**: Once a client has it, all other clients are denied **Fw**.
* **Fx**: Only the loner client is allowed this capability. Once the lock state
transitions to LOCK_EXCL, the loner client is granted this along with all other
file capabilities except the **Fl**.
* **Fr**: Once a client has it, the **Fb** capability will be already revoked from
all the other clients.
If clients only request to read the file, the lock state will be transferred
to LOCK_SYNC stable state directly. All the clients can be granted **Fscrl**
capabilities from the auth MDS and **Fscr** capabilities from the replica MDSes.
If multiple clients read from and write to the same file, then the lock state
will be transferred to LOCK_MIX stable state finally and all the clients could
have the **Frwl** capabilities from the auth MDS, and the **Fr** from the replica
MDSes. The **Fcb** capabilities won't be granted to all the clients and the
clients will do sync read/write.
* **Fw**: If there is no loner client and once a client have this capability, the
**Fsxcb** capabilities won't be granted to other clients.
If multiple clients read from and write to the same file, then the lock state
will be transferred to LOCK_MIX stable state finally and all the clients could
have the **Frwl** capabilities from the auth MDS, and the **Fr** from the replica
MDSes. The **Fcb** capabilities won't be granted to all the clients and the
clients will do sync read/write.
* **Fc**: This capability means the clients could cache file read and should be
issued together with **Fr** capability and only in this use case will it make
sense.
While actually in some stable or interim transitional states they tend to keep
the **Fc** allowed even the **Fr** capability isn't granted as this can avoid
forcing clients to drop full caches, for example on a simple file size extension
or truncating use case.
* **Fb**: This capability means the clients could buffer file write and should be
issued together with **Fw** capability and only in this use case will it make
sense.
While actually in some stable or interim transitional states they tend to keep
the **Fc** allowed even the **Fw** capability isn't granted as this can avoid
forcing clients to drop dirty buffers, for example on a simple file size extension
or truncating use case.
* **Fl**: This capability means the clients could perform lazy io. LazyIO relaxes
POSIX semantics. Buffered reads/writes are allowed even when a file is opened by
multiple applications on multiple clients. Applications are responsible for managing
cache coherency themselves.
|
