1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
|
NFS
===
NFS is commonly used in one of these ways:
1. Dovecot is run in a single computer.
2. Dovecot is run in multiple computers, users are redirected more or less
randomly to different computers.
3. Dovecot is run in multiple computers, each user is assigned a specific
computer which is used whenever possible.
*The only way to reliably implement the 2nd setup is with the <director>
[Director.txt] service.*
Dovecot configuration
---------------------
Single Dovecot server setup or <Dovecot director> [Director.txt] cluster setup:
---%<-------------------------------------------------------------------------
mmap_disable = yes
#dotlock_use_excl = no # only needed with NFSv2, NFSv3+ supports O_EXCL and
it's faster
mail_fsync = always
mail_nfs_storage = no
mail_nfs_index = no
---%<-------------------------------------------------------------------------
Multi-server setup that *tries* to flush NFS caches (increases NFS operations,
and *isn't fully reliable*), try not to use this:
---%<-------------------------------------------------------------------------
mmap_disable = yes
#dotlock_use_excl = no # only needed with NFSv2, NFSv3+ supports O_EXCL and
it's faster
mail_fsync = always
# These settings slow things down and don't fully work, use director proxy
instead:
mail_nfs_storage = yes
mail_nfs_index = yes
---%<-------------------------------------------------------------------------
Common issues
-------------
Clock synchronization
---------------------
Run ntpd in the NFS server and all the NFS clients to make sure their clocks
are synchronized. If the clocks are more than one second apart from each others
and multiple computers access the same mailbox simultaneously, you may get
errors.
NFS caching problems
--------------------
NFS caching is a big problem when multiple computers are accessing the same
mailbox simultaneously. The best fix for this is to prevent it from happening.
Configure your setup so that a user always gets redirected to the same server
(unless it's down). This also means that mail deliveries must be done by the
same server, or alternatively it shouldn't update index files.
Dovecot flushes NFS caches when needed if you set 'mail_nfs_storage=yes', but
unfortunately this doesn't work 100%, so you can get random errors.
Disabling NFS attribute cache helps a lot in getting rid of caching related
errors, but this makes the performance MUCH worse and increases the load on NFS
server. This can usually be done by giving 'actimeo=0' or 'noac' mount option.
Index files
-----------
If you keep the index files stored on NFS, you'll need to set
'mmap_disable=yes'. If you're not running lockd you'll have to set
'lock_method=dotlock', but this degrades performance. Note that some NFS
installations have problems with lockd. If you're beginning to get all kinds of
locking related errors, try if the problems go away with dotlocking.
With mbox/Maildir formats (but not dbox!) it's also possible to store index
files on local disk instead of on NFS. If the user gets redirected to different
servers, the local indexes are automatically created/updated. If the user is
(nearly) always redirected to the same server this should be fine and you would
likely get higher performance than indexes stored on NFS, but if the server
changes it can be slow to recreate the index/cache files.
Single computer setup
---------------------
This doesn't really differ from keeping mails stored locally. For better
performance you should keep index files stored in a local disk.
Random redirects to multiple servers
------------------------------------
You should avoid this setup whenever possible. Besides the NFS cache problems
described above, mailbox contents can't be cached as well in the memory either.
This is more problematic with mbox than with maildir, but in both cases if a
client is redirected to a different server when reconnecting, the new server
will have to read some data via the NFS into memory, while the original server
might have had the data already cached.
If you choose to use this setup, at the very least try to make connections from
a single IP redirected into the same server. This avoids the biggest problems
with clients that use multiple connections.
Per-user redirects to multiple servers
--------------------------------------
This method performs a lot better than random redirects. It maximizes the
caching possibilities and prevents the problems caused by simultaneous mailbox
access.
New mail deliveries are often still handled by different computers. This isn't
a problem with maildir as long as you're not using <LDA.txt> (i.e.
dovecot-uidlist file or index files shouldn't get updated). It shouldn't be a
problem with mboxes either as long as you're using fcntl locking. This problem
can be fully solved by using LMTP protocol to deliver the mails to the correct
server (possibly using Dovecot's LMTP proxy).
NFS clients
===========
Here's a list of kernels that have been tried as NFS clients:
* FreeBSD has a caching bug
[http://www.freebsd.org/cgi/query-pr.cgi?pr=123755] which causes problems
when mailbox is being accessed from different computers at the same time
* Linux 2.6.16: 'utime()' is buggy, fix in here
[http://client.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-007-fix_setattr_clobber.dif].
With the fix applied, utime() seems to work perfectly. High-volume systems
may experience VFS lock sync issues and for these the complete patchset at
http://www.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-NFS_ALL.dif is
suggested and appears to work well in production.
* Linux 2.6.18: Seems to have intermittent caching issues. The same .config
with 2.6.20.1 has been tested and appears to work well.
* Linux 2.4.8: Has caching problems, don't know if they can be solved
* Solaris: If it's completely broken, see
http://dovecot.org/list/dovecot/2006-December/018145.html
* The Connectathon test suite is very useful to verify a healthy NFS setup,
see http://www.connectathon.org/nfstests.html
Misc notes
==========
* readdirplus isn't really needed by Dovecot and it can slow down some NFS
servers. Use "nordirplus" mount option to disable it.
* Dovecot doesn't care about root_squash setting, all the root-owned files are
in /var/run typically which is not in NFS
* In an environment using Debian (2.6.18) clients with Isilon NFS cluster
nodes - the following mount options were found to be the most
successful:'rsize=32768,wsize=32768,hard,fg,lock,nfsvers=3,tcp,retrans=0,nordirplus
0 0'
* To learn more about NFS caching and other issues, mostly from a programmer's
point of view, see NFS Coding HOWTO
[http://iki.fi/tss/nfs-coding-howto.html]
* Use such permissions for the unmounted mount point root directory that
Dovecot can't create files under it. Otherwise if the NFS server isn't
mounted for any reason and user access mails, a new empty user mail
directory is created, which breaks things.
(This file was created from the wiki on 2019-06-19 12:42)
|
