diff options
Diffstat (limited to '')
276 files changed, 37641 insertions, 0 deletions
diff --git a/doc/wiki/ACL.txt b/doc/wiki/ACL.txt new file mode 100644 index 0000000..486d71c --- /dev/null +++ b/doc/wiki/ACL.txt @@ -0,0 +1,258 @@ +Access Control Lists +==================== + +This page talks mainly about how ACLs work, for more general description of how +shared mailboxes work, see <SharedMailboxes.txt>. + +Dovecot v1.0 and v1.1 supports administrator-configured ACL files. v1.2+ +supports also IMAP ACL extension, which allows users to change ACLs themselves. +The ACL code was written to allow multiple ACL backends, but currently Dovecot +supports only virtual ACL files. Note that using ACLs doesn't grant mail +processes any extra filesystem permissions that they already don't have. <You +must make sure that the processes have enough permissions> +[SharedMailboxes.Permissions.txt] to be able to access the mailboxes. When +testing you could first try accessing shared/public mailboxes without ACL +plugin even enabled. + +ACLs can be enabled in dovecot.conf with: + +---%<------------------------------------------------------------------------- +mail_plugins = acl +protocol imap { + mail_plugins = $mail_plugins imap_acl +} + +plugin { + # Without global ACLs: + acl = vfile + + # With global ACL files in /etc/dovecot/dovecot-acls file (v2.2.11+): + #acl = vfile:/etc/dovecot/dovecot-acl + + # With global ACLs in /etc/dovecot/acls/ directory (obsolete): + #acl = vfile:/etc/dovecot/acls + + # If enabled, don't try to find dovecot-acl files from mailbox directories. + # This reduces unnecessary disk I/O when only global ACLs are used. +(v2.2.31+) + #acl_globals_only = yes +} +---%<------------------------------------------------------------------------- + +ACL groups support works by returning a comma-separated 'acl_groups' <extra +field> [UserDatabase.ExtraFields.txt] from userdb, which contains all the +groups the user belongs to. User's UNIX groups have no effect on ACLs (you can +"enable" them by using a special <post-login script> [PostLoginScripting.txt]). + +The default ACL for mailboxes is to give the mailbox owner all permissions and +other users none. Mailboxes in public namespaces don't have owners, so by +default no one can access them. + +Master users +------------ + +Note that master users have their own ACLs. They're not the the mailbox owners, +so by default they have no permissions to any of the mailboxes. See +Authentication/MasterUsers#ACLs for more information. + +ACL vfile backend +----------------- + +vfile backend supports per-mailbox ACLs and global ACLs. + +Per-mailbox ACLs are stored in 'dovecot-acl' named file, which exists in: + + * maildir: The Maildir's mail directory (eg. '~/Maildir', + '~/Maildir/.folder/') + * mbox: Control directory. You should explicitly specify ':CONTROL=<path>' in + mail location. + * dbox: dbox's mail directory (eg. '~/dbox/INBOX/dbox-Mails/') + +ACL Inheritance +--------------- + +Every time you create a new mailbox, it gets its ACLs from the parent mailbox. +If you're creating a root-level mailbox, it uses the namespace's default ACLs. +There is no actual inheritance, however: If you modify parent's ACLs, the +child's ACLs stay the same. There is currently no support for ACL inheritance. + + * Maildir: Namespace's default ACLs are read from "dovecot-acl" file in the + namespace's mail root directory (e.g.'/var/public/Maildir'). Note that + currently these default ACLs are used only when creating new mailboxes, they + aren't used for mailboxes without ACLs. + * v2.2.2+: If 'plugin { acl_defaults_from_inbox=yes } ', the default ACLs for + private and shared namespaces (but not public namespaces) are taken from the + INBOX. This means that giving somebody access to your INBOX will give them + access to all your other mailboxes as well, unless the specific mailboxes' + ACLs override the INBOX's. + +*NOTE*: Currently the default ACLs are merged with the mailbox-specific ACLs. +So if a default ACL gives access to "user1" and a per-mailbox ACL gives access +to "user2", the "user1" still has access to that mailbox. + +Global ACLs +----------- + +Global ACLs can be used to apply ACLs globally to all user's specific +mailboxes. They are used mainly for two purposes: + + 1. Removing some permissions from users' personal mailboxes. For example each + user might have an "Invoices" mailbox which will be read-only. + 2. Giving permissions to master user logins. See + <Authentication/MasterUsers#ACLs> [Authentication.MasterUsers.txt] for more + information. + +If a mailbox has both global ACLs and the per-mailbox ACL file, both of them +are read and the ACLs are merged. If there are any conflicts, the global ACL +file overrides per-mailbox ACL file. This is because users can modify their own +per-mailbox ACL files via IMAP ACL extension. Global ACLs can only be modified +by administrator, so users shouldn't be able to override them. + +Global ACL file (v2.2.11+) +-------------------------- + +Global ACL file path is specified as a parameter to vfile backend in 'acl' +setting ('/etc/dovecot/dovecot-acl' in the above example). The file contains +otherwise the same data as regular per-mailbox 'dovecot-acl' files, except each +line is prefixed by the mailbox name pattern. The pattern may contain "*" and +"?" wildcards. For example: + +---%<------------------------------------------------------------------------- +* user=foo lrw +Public user=bar lrwstipekxa +Public/* user=bar lrwstipekxa +---%<------------------------------------------------------------------------- + +Global ACL directory (obsolete) +------------------------------- + +Global ACL directory is specified as a parameter to vfile backend in 'acl' +setting ('/etc/dovecot/acls/' in the above example). They are looked up using +the mailbox's virtual name. For example: + + * INBOX: '/etc/dovecot/acls/INBOX' + * archives.2007: '/etc/dovecot/acls/archives.2007' + * archives/2007: '/etc/dovecot/acls/archives/2007' + +The filenames must start with namespace prefix (if it has one). For example +with namespace 'prefix=INBOX/' containing mailbox "foo" use +'/etc/dovecot/acls/INBOX/foo'. + +There is an extra problem with mailbox formats that use '/' as the separator +(e.g. mbox, dbox): For example if you have mailboxes "foo" and "foo/bar" and +you wish to give ACLs to both of them, you can't create both +'/etc/dovecot/acls/foo' and '/etc/dovecot/acls/foo/bar' files. The 'foo' has to +be either a directory or a file, it can't be both. To solve this problem, you +can instead create a .DEFAULT file for "foo": + + * foo: '/etc/dovecot/acls/foo/.DEFAULT' + * foo/bar: '/etc/dovecot/acls/foo/bar' + +ACL files +--------- + +The files themselves are in format: + +---%<------------------------------------------------------------------------- +<identifier> <ACLs> [:<named ACLs>] +---%<------------------------------------------------------------------------- + +Where *identifier* is one of: + + * group-override=*group name* + * user=*user name* + * owner + * group=*group name* + * authenticated + * anyone (or anonymous, which is alias for anyone) + +The ACLS are processed in the precedence given above, so for example if you +have given read-access to a group, you can still remove that from specific +users inside the group. + +Group-override identifier allows you to override users' ACLs. Probably the most +useful reason to do this is to temporarily disable access for some users. For +example: + +---%<------------------------------------------------------------------------- +user=timo rw +group-override=tempdisabled +---%<------------------------------------------------------------------------- + +Now if /timo/ is in /tempdisabled/ group, he has no access to the mailbox. This +wouldn't be possible with a normal group identifier, because the 'user=timo' +would override it. + +The currently supported ACLs and their corresponding named ACLs are: ++---+---------------+---------------------------------------------------------+ +| l | lookup | Mailbox is visible in mailbox list. Mailbox can be | +| | | subscribed to. | ++---+---------------+---------------------------------------------------------+ +| r | read | Mailbox can be opened for reading. | ++---+---------------+---------------------------------------------------------+ +| w | write | Message flags and keywords can be changed, except \Seen | +| | | and \Deleted | ++---+---------------+---------------------------------------------------------+ +| s | write-seen | \Seen flag can be changed | ++---+---------------+---------------------------------------------------------+ +| t | write-deleted | \Deleted flag can be changed | ++---+---------------+---------------------------------------------------------+ +| i | insert | Messages can be written or copied to the mailbox | ++---+---------------+---------------------------------------------------------+ +| p | post | Messages can be posted to the mailbox by <LDA.txt>, e.g.| +| | | from <Sieve> [Pigeonhole.Sieve.txt] scripts | ++---+---------------+---------------------------------------------------------+ +| e | expunge | Messages can be expunged | ++---+---------------+---------------------------------------------------------+ +| k | create | Mailboxes can be created (or renamed) directly under | +| | | this mailbox (but not necessarily under its children, | +| | | see ACL Inheritance section above) (renaming also | +| | | requires delete rights) | ++---+---------------+---------------------------------------------------------+ +| x | delete | Mailbox can be deleted | ++---+---------------+---------------------------------------------------------+ +| a | admin | Administration rights to the mailbox (currently: ability| +| | | to change ACLs for mailbox) | ++---+---------------+---------------------------------------------------------+ + +The ACLs are compatible with RFC 4314 (IMAP ACL extension, updated version). + +Unknown ACL letters are complained about, but unknown named ACLs are ignored. +Named ACLs are mostly intended for future extensions. + +Note that the file is rather picky about formatting; using a tab (or multiple +spaces) instead of a space character between fields may not work. If you are +having problems, make sure to check for tabs, extra spaces and other unwanted +characters. + +Examples +-------- + +Mailbox owner has all privileges, "timo" has list-read privileges: + +---%<------------------------------------------------------------------------- +owner lrwstipekxa +user=timo lr +---%<------------------------------------------------------------------------- + +Allow everyone to list and read a public mailbox (public namespace has no +owner): + +---%<------------------------------------------------------------------------- +anyone lr +---%<------------------------------------------------------------------------- + +Prevent all users from deleting their Spam folder (notice no x flag) + +---%<------------------------------------------------------------------------- +INBOX.Spam owner lrwstipeka +---%<------------------------------------------------------------------------- + +List cache +---------- + +'dovecot-acl-list' file lists all mailboxes that have "l" rights assigned. If +you manually add/edit 'dovecot-acl' files, you may need to delete the +'dovecot-acl-list' to get the mailboxes visible. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AixPluginsSupport.txt b/doc/wiki/AixPluginsSupport.txt new file mode 100644 index 0000000..444c00d --- /dev/null +++ b/doc/wiki/AixPluginsSupport.txt @@ -0,0 +1,141 @@ +Plugins Support on AIX +====================== + +/How to build Dovecot with plugins supported on AIX./ + +Tested with: + + * AIX 5.2, VAC 5.0.2 and Dovecot 1.0rc21 to 1.0.13 /(Dovecot 1.1.x does not + build with VAC 5.0.2)/ + * AIX 5.2, VAC 8.0.0 and Dovecot 1.1.1 + +The Problem +----------- + + * When you trying to use plugins on AIX you seen error messages like this: + ---%<---------------------------------------------------------------------- + imap(root): Error: + dlopen(/usr/local/lib/dovecot/imap/lib20_zlib_plugin.so) failed: + rtld: 0712-001 Symbol i_error was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_get_data was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_skip was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_seek was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_close was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_panic was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol pool_get_exp_grown_size was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + Additional errors occurred but are not reported. + ---%<---------------------------------------------------------------------- + + * .. produced by executing + ---%<---------------------------------------------------------------------- + # MAIL_PLUGINS=zlib /usr/local/libexec/dovecot/imap + ---%<---------------------------------------------------------------------- + +Compiler Script +--------------- + + * Create a compiler script to rewrite the /xlc/ command line on the fly: + *dovecot-cc* + + ---%<---------------------------------------------------------------------- + #!/bin/bash + + xlc=/usr/bin/xlc + ar=/bin/ar + sed=/bin/sed + + dest=NOBINARY + + for i in "$@"; do + case "$i" in + '-o') dest=;; + *) if [ -z "$dest" ]; then dest="$i"; break; fi;; + esac + done + + case "$dest" in + imap-login) args1="../lib-charset/libcharset.a ../lib-mail/libmail.a + -liconv";; + imap) args1="../lib-sql/libsql.a";; + pop3) args1="../lib-sql/libsql.a";; + deliver) args1="../lib-sql/libsql.a";; + esac + + for i in "$@" $args1; do + case "$i" in + */*.a) lib="${i##*/}"; obj=`$ar -t $i | $sed "s:^:${i%/*}/:"`;; + *.a) lib="$i"; obj=`$ar -t $i | $sed "s:^:./:"`;; + *) continue;; + esac + test -d .libs || mkdir .libs + > .libs/${lib%.a}.exp + args2="$args2 -bE:.libs/${lib%.a}.exp" + + (set -x ; exec $xlc -qmkshrobj -qexpfile=.libs/${lib%.a}.exp $obj) + 2>/dev/null + done + (set -x ; exec $xlc "$@" $args1 $args2) + ---%<---------------------------------------------------------------------- + +Compiling Dovecot +----------------- + + * Expand Dovecot: + ---%<---------------------------------------------------------------------- + gzip -cd doveccot-1.0.rc21.tar.gz | tar xvf - + ---%<---------------------------------------------------------------------- + + * Setup build environment: + ---%<---------------------------------------------------------------------- + export CC=$PWD/dovecot-cc + export LDFLAGS="-bexpall -brtl" + ---%<---------------------------------------------------------------------- + + * Configure and build Dovecot + ---%<---------------------------------------------------------------------- + cd dovecot-1.0.rc21 + bash configure + make + ---%<---------------------------------------------------------------------- + + * Test a plugin (e.g. zlib) + ---%<---------------------------------------------------------------------- + echo 0 logout | MAIL_PLUGIN_DIR=src/plugins/zlib/.libs/ MAIL_PLUGINS="zlib" + MAIL=maildir:/tmp src/imap/imap + ---%<---------------------------------------------------------------------- + + you should see this: + ---%<---------------------------------------------------------------------- + * PREAUTH [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND + UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS] Logged in as root + * BYE Logging out + 0 OK Logout completed. + imap(root): Info: Disconnected: Logged out + ---%<---------------------------------------------------------------------- + + * Install Dovecot + ---%<---------------------------------------------------------------------- + make install + ---%<---------------------------------------------------------------------- + +Prebuild Binaries for AIX 5.2 +----------------------------- + +You will find prebuild AIX 5.2 binaries here: +http://www.fh-trier.de/~beckerr/dovecot/ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AttachmentIndicator.txt b/doc/wiki/AttachmentIndicator.txt new file mode 100644 index 0000000..1acac6f --- /dev/null +++ b/doc/wiki/AttachmentIndicator.txt @@ -0,0 +1,31 @@ +Attachment indicator +==================== + +Since 2.2.34/2.3.1 dovecot has a feature that indicates whether email has an +attachment or not via keywords. These keywords could be used by IMAP clients. +These keywords will likely become standardized - for now the latest information +is in https://www.ietf.org/mail-archive/web/imapext/current/msg05858.html + +Configuration +------------- + +To enable this feature, you can set *mail_attachment_detection_options*. + +It supports following options + + * add-flags-on-save - Enables the feature, attachments are detected and marked + during save + * content-type=type|!type - Include or exclude given content type. Including + will only negate an exclusion (e.g. content-type=!foo/* + content-type=foo/bar). + * exclude-inlined - Do not consider any attachment with disposition inlined. + +Usage +----- + +Once enabled, mails that are saved, are marked with $Has ''Attachment or $Has +''No ''Attachment keyword. These keywords can be cleared, and are not +protected. Since v2.3.3 it is possible to post-process attachments with +'doveadm rebuild attachments' command. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.CheckPassword.txt b/doc/wiki/AuthDatabase.CheckPassword.txt new file mode 100644 index 0000000..c285868 --- /dev/null +++ b/doc/wiki/AuthDatabase.CheckPassword.txt @@ -0,0 +1,234 @@ +CheckPassword +============= + +Since v2.3.0 You can also use Lua to write your custom authentication, see +<AuthDatabase.Lua.txt> + +Checkpassword is an authentication interface originally implemented by qmail +[http://www.qmail.org/]. Checkpassword combines both the <password database> +[PasswordDatabase.txt] and <user database> [UserDatabase.txt] lookups into a +single checkpassword lookup, which makes the standard implementation unsuitable +for a standalone userdb. With Dovecot extensions it's also possible to use +checkpassword as a userdb. + +Typically you'll use <prefetch> [UserDatabase.Prefetch.txt] as the userdb, but +it's not required that you use the checkpassword script's userdb capabilities. +You can still use for example <static userdb> [UserDatabase.Static.txt] if +you're using only a single UID and GID, and your home directory fits into a +template. + +Security +-------- + +The standard checkpassword design is incompatible with Dovecot's security +model. If the system has local users and the checkpassword script setuid()s +into a local user, the user is able to ptrace into the communication and change +the authentication results. This is of course undesirable, so v2.2.7+ will just +refuse to run in such environments by default. The possibilities to solve this +are: + + 1. If possible, change the checkpassword to return 'userdb_uid' and + 'userdb_gid' extra fields instead of using 'setuid()' and 'setgid()'. This + also improves the performance. + 2. If you can't change the script, you can make Dovecot's + 'checkpassword-reply' binary setuid or setgid (e.g.'chgrp dovecot + /usr/local/libexec/dovecot/checkpassword-reply; chmod g+s + /usr/local/libexec/dovecot/checkpassword-reply') + 3. If you don't have any untrusted local users and you just don't care about + this check, you can set 'INSECURE_SETUID=1' environment e.g. with a wrapper + checkpassword script. + +Deliver +------- + +If your checkpassword script doesn't support Dovecot extensions, you can't use +it as a user database. This means that if you wish to use <LDA.txt>, you can't +use the '-d' parameter to do userdb lookups. There are two ways to solve this: + + 1. Use another userdb which does the lookup for deliver, for example <SQL> + [AuthDatabase.SQL.txt] or <static> [UserDatabase.Static.txt]. Add this + userdb after the prefetch userdb. + 2. Use a script to look up the user's home directory and run deliver without + '-d' parameter. For example: + +---%<------------------------------------------------------------------------- +#!/bin/sh + +# <<Lookup user's home directory here.>> + +# If users have different UIDs/GIDs, make sure to also change this process's +UID and GID. +# If you want to override any settings, use dovecot-lda's -o parameter +# (e.g. dovecot-lda -o mail_location=maildir:~/Maildir). + +export HOME +exec /usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +Checkpassword Interface +----------------------- + +The interface is specified in http://cr.yp.to/checkpwd/interface.html. However +here's a quick tutorial for writing a script: + + * Read '<username> NUL <password> NUL' from fd 3. + * Verify the username and password. + * If the authentication fails, exit with code 1. This makes Dovecot give + "Authentication failed" error to user. + * This error is returned both for password mismatch and also if the user + doesn't exist at all. Internally Dovecot maps this as password + mismatch. + * If you encounter an internal error, exit with code 111. This makes + Dovecot give "Temporary authentication failure" error to user. + * If the authentication succeeds, you'll need to: + * Set user's home directory to '$HOME' environment. This isn't required, + <but highly encouraged> [VirtualUsers.txt]. + * Set '$USER' environment variable. If the user name was changed (eg. if + you lowercased "Username" to "username"), you can tell about it to + Dovecot by setting '$USER' to the changed user name. + * Return the user's <UNIX UID and GID> [UserIds.txt] using 'userdb_uid' and + 'userdb_gid' environments and add them to the 'EXTRA' environment (see + below for Dovecot extensions). + * This is recommended over actually changing the UID/GID using + setuid()/setgid() as specified by the standard checkpassword + interface, because it's <incompatible with Dovecot's security model> + [AuthDatabase.CheckPassword.txt]. + * Your program received a path to 'checkpassword-reply' binary as the first + parameter. Execute it. + +Qmail-LDAP +---------- + +Note that auth_imap that comes with qmail-ldap is not compatible with this +interface. You can get a patch that adds auth_dovecot functionality to +qmail-ldap here +[http://japc.uncovering.org/dovecot/qmail-ldap-1.03-20060201-dovecot.patch]. Or +you can use auth_pop instead, but you may need to pass /aliasempty/ to let +auth_pop find the Maildir, so it is recommended to write a +/var/qmail/bin/auth_dovecot wrapper (don't forget to chmod +x it) around +auth_pop. + +---%<------------------------------------------------------------------------- +#!/bin/sh +QMAIL="/var/qmail" +if [ -e $QMAIL/control/defaultdelivery ]; then + ALIASEMPTY=`head -n 1 $QMAIL/control/defaultdelivery 2> /dev/null` +else + ALIASEMPTY=`head -n 1 $QMAIL/control/aliasempty 2> /dev/null` +fi +ALIASEMPTY=${ALIASEMPTY:-"./Maildir/"} +exec $QMAIL/bin/auth_pop "$@" $ALIASEMPTY +---%<------------------------------------------------------------------------- + +you can also use this wrapper to pass LOGLEVEL environmental variable to +auth_pop. + +Dovecot Extensions +------------------ + +If you wish to return <extra fields> [PasswordDatabase.ExtraFields.txt] for +Dovecot, set them in environment variables and then list them in EXTRA +environment variable. The <userdb extra fields> [UserDatabase.ExtraFields.txt] +can be returned by prefixing them with 'userdb_'. For example: + +---%<------------------------------------------------------------------------- +userdb_quota_rule=*:storage=10000 +userdb_mail=mbox:$HOME/mboxes +EXTRA=userdb_quota_rule userdb_mail +---%<------------------------------------------------------------------------- + +Dovecot also sets some environment variables that the script may use: + + * 'SERVICE': contains eg. imap, pop3 or smtp + * 'TCPLOCALIP' and 'TCPREMOTEIP': Client socket's IP addresses if available + * 'MASTER_USER': If master login is attempted. This means that the password + contains the master user's password and the normal username contains the + user who master wants to log in as. + * 'AUTH_*': All of the <auth variables> [Variables.txt] are available as + 'AUTH_<long name>' extra fields. For example '%{cert}' is in 'AUTH_CERT'. + (v2.0.16+) + +Checkpassword as userdb +----------------------- + +Dovecot calls the script with 'AUTHORIZED=1' environment set when performing a +userdb lookup. The script must acknowledge this by changing the environment to +'AUTHORIZED=2', otherwise the lookup fails. Other than that, the script works +the same way as a passdb checkpassword script. If user doesn't exist, use exit +code 3. + +Checkpassword with passdb lookups (v2.1.2+) +------------------------------------------- + +Normally checkpassword answers to questions "is user X's password Y?" This +doesn't work with non-plaintext auth mechanisms, or when Dovecot wants to do a +non-authenticating passdb lookup (e.g. for LMTP proxy). These passdb +credentials lookups can be implemented the same way as a userdb lookup (i.e. +change the 'AUTHORIZED' environment). + + * 'AUTHORIZED=1' is set, just like for userdb lookup + * When doing a non-plaintext authentication: + * 'CREDENTIALS_LOOKUP=1' environment is set + * The password scheme that Dovecot wants is available in 'SCHEME' + environment (e.g.'SCHEME=CRAM-MD5') + * If a password is returned, it must be returned as + 'password={SCHEME}secret'. + * When doing a passdb lookup, e.g. a proxy which doesn't really want the + password, just the passdb extra fields: + * Neither 'CREDENTIALS_LOOKUP' nor 'SCHEME' is set. + * FIXME: Unfortunately it looks like you currently can't easily + differentiate a passdb lookup from userdb lookup! + * If user doesn't exist, use exit code 3. + * If you get an error about checkpassword exiting with code 0, you didn't + execute the 'checkpassword-reply' binary as you should have (which exits + with code 2 on success) + +Example +------- + +The standard way: + +---%<------------------------------------------------------------------------- +passdb { + driver = checkpassword + args = /usr/bin/checkpassword +} +userdb { + driver = prefetch +} +# If you want to use deliver -d and your users are in SQL: +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +Using checkpassword only to verify the password: + +---%<------------------------------------------------------------------------- +passdb { + driver = checkpassword + args = /usr/bin/checkpassword +} +userdb { + driver = static + args = uid=vmail gid=vmail home=/home/%u +} +---%<------------------------------------------------------------------------- + +Performance +----------- + +The <CheckPassword.txt> backend is not suited for heavy traffic. Especially if +the script spawned has to launch an entire language interpreter. + +If your user database is only accessible with custom code an alternative might +be using the <Dict AuthDatabase over a UNIX socket> [AuthDatabase.Dict.txt]. + +Specific checkpassword implementations +-------------------------------------- + + * phpBB dovecot checkpassword authentication, written in python: + https://github.com/ser/checkpassword-phpbb + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.Dict.txt b/doc/wiki/AuthDatabase.Dict.txt new file mode 100644 index 0000000..4a4405a --- /dev/null +++ b/doc/wiki/AuthDatabase.Dict.txt @@ -0,0 +1,391 @@ +Key-value authentication database +================================= + +Key-value databases can be used as auth backends. They probably should be used +only for caching in front of e.g. SQL auth backends. Iteration is supported if +the underlying dict provider supports iteration. See <Dictionary.txt> for list +of supported databases. + +Auth configuration +------------------ + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +userdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +---%<------------------------------------------------------------------------- + +Dict configuration +------------------ + +---%<------------------------------------------------------------------------- +uri = redis:host=127.0.0.1:port=6379 + +# Dictionary URI +#uri = + +# Default password scheme +default_pass_scheme = MD5 + +# Username iteration prefix. Keys under this are assumed to contain usernames. +iterate_prefix = userdb/ + +# Should iteration be disabled for this userdb? If this userdb acts only as a +# cache there's no reason to try to iterate the (partial & duplicate) users. +#iterate_disable = no + +# The example here shows how to do multiple dict lookups and merge the replies. +# The "passdb" and "userdb" keys are JSON objects containing key/value pairs, +# for example: { "uid": 1000, "gid": 1000, "home": "/home/user" } + +key passdb { + key = passdb/%u + format = json +} +key userdb { + key = userdb/%u + format = json +} +key quota { + key = userdb/%u/quota # or e.g. quota/%{userdb:quota_class} + #format = value + # The default_value is used if the key isn't found. If default_value setting + # isn't specified at all (even as empty), the passdb/userdb lookup fails with + # "user doesn't exist". + default_value = 100M +} + +# Space separated list of keys whose values contain key/value paired objects. +# All the key/value pairs inside the object are added as passdb fields. +# This can only be used for JSON formatted values. +passdb_objects = passdb + +#passdb_fields { +#} + +# Userdb key/value object list. +userdb_objects = userdb + +userdb_fields { + # dict:<key> refers to key names + quota_rule = *:storage=%{dict:quota} + + # dict:<key>.<objkey> refers to the objkey inside (JSON) object + mail = maildir:%{dict:userdb.home}/Maildir +} +---%<------------------------------------------------------------------------- + +Example values +-------------- + +The value formats are either "value" that contains a direct value, or "json". +For example userdb lookup should return something like: + +---%<------------------------------------------------------------------------- +{ "uid": 123, "gid": 123, "home": "/home/username" } +---%<------------------------------------------------------------------------- + +dict proxying +------------- + +It may be useful to do the lookups via the "dict" or "dict-async" service. For +example: + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +dict { + cassandra-userdb = cassandra:/etc/dovecot/dovecot-dict-userdb-cql.conf.ext +} +---%<------------------------------------------------------------------------- + +'dovecot-dict-auth.conf.ext': + +---%<------------------------------------------------------------------------- +uri = proxy:dict-async:cassandra-userdb +iterate_disable = yes +# The _key and _path suffixes are not necessary, they're just here to help +# understand how to match them between different parts of the configuration. +key email_key { + key = userdb/email_path/%u +} +key displayname_key { + key = userdb/displayname_path/%u +} +userdb_fields { + # these fields will be visible as %{userdb:u_email} and +%{userdb:u_displayname} + u_email = %{dict:email_key} + u_displayname = %{dict:displayname_key} +} +---%<------------------------------------------------------------------------- + +'dovecot-dict-userdb-cql.conf.ext': + +---%<------------------------------------------------------------------------- +driver = cassandra +connect = host=127.0.0.1 dbname=email_users + +# SELECT displayname FROM user_profile WHERE id = %u +map { + # pattern must match the "key" path, except with added shared/ prefix. %u +gets caught into $username + pattern = shared/userdb/displayname_path/$username + table = user_profile + value_field = displayname + value_type = string + fields { + id = $username + } +} + +# SELECT email FROM user_profile WHERE id = %u +map { + pattern = shared/userdb/email_path/$username + table = user_profile + value_field = email + value_type = string + fields { + id = $username + } +} +---%<------------------------------------------------------------------------- + +Complete example for authenticating via the CDB dictionary +---------------------------------------------------------- + +This example uses the CDB dictionary to store the userdb and passdb. + +Auth configuration +------------------ + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +# Access to the CDB has to go through a dict process. +dict { + auth = cdb:/etc/dovecot/auth.cdb +} + +passdb { + driver = dict + args = /etc/dovecot/dovecot-cdb.conf +} + +userdb { + driver = dict + args = /etc/dovecot/dovecot-cdb.conf +} +---%<------------------------------------------------------------------------- + +Dict configuration +------------------ + +The CDB dictionary doesn't support iteration yet. + +'dovecot-cdb.conf': + +---%<------------------------------------------------------------------------- +uri = proxy::auth + +# FIXME: obsolete configuration - should use the key { .. } instead +password_key = passdb/%u +user_key = userdb/%u +# iterate_prefix = userdb/ # no yet supported +iterate_disable = yes + +default_pass_scheme = BLF-CRYPT +---%<------------------------------------------------------------------------- + +Complete example for authenticating via a UNIX socket +----------------------------------------------------- + +The Dict auth backend can be used to query a local UNIX socket for users. This +can be handy for accessing user databases which would otherwise only be +accessible via the <CheckPassword> [AuthDatabase.CheckPassword.txt] backend and +a scripting language. + +When given a <"proxy:"> [Quota.Dict.txt] URL the Dict backend speaks a simple +protocol over a UNIX socket. The protocol is defined in +'src/lib-dict/dict-client.h' (GitHub +[https://github.com/dovecot/core/blob/master/src/lib-dict/dict-client.h]). + +Auth configuration +------------------ + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +userdb { + # optional + driver = prefetch +} +userdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +---%<------------------------------------------------------------------------- + +Dict configuration +------------------ + +The last "dictionary name" ("somewhere") argument is redundant here. + +'/etc/dovecot/dovecot-dict-auth.conf.ext': + +---%<------------------------------------------------------------------------- +uri = proxy:/var/run/auth_proxy_dovecot/socket:somewhere + +# FIXME: obsolete configuration - should use the key { .. } instead +password_key = passdb/%u +user_key = userdb/%u +iterate_disable = yes +#default_pass_scheme = plain +---%<------------------------------------------------------------------------- + +Server process for answering Dict lookups +----------------------------------------- + +The server process listening on '/var/run/auth_proxy_dovecot/socket' can be +written in any language.Here's an example in Perl: + +---%<------------------------------------------------------------------------- +package AuthProxyDovecot; +use base qw( Net::Server::PreFork ); + +use strict; +use warnings; + +use JSON::XS; + +AuthProxyDovecot->run() or die "Could not initialize"; + +sub default_values +{ + return { + port => '/var/run/auth_proxy_dovecot/socket|unix', + + log_level => 2, + log_file => 'Sys::Syslog', + syslog_logsock => 'unix', + syslog_ident => 'auth_proxy_dovecot', + syslog_facility => 'daemon', + + background => 1, + setsid => 1, + pid_file => '/var/run/auth_proxy_dovecot.pid', + + user => 'root', + group => 'root', + + max_spare_servers => 2, + min_spare_servers => 1, + min_servers => 2, + max_servers => 10, + + }; +} ## end sub default_values + +################################################## + +sub process_request { + my $self = shift; + + my %L_handler = ( + passdb => sub { + my ($arg) = @_; + my $ret = { + password => '$1$JrTuEHAY$gZA1y4ElkLHtnsrWNHT/e.', + userdb_home => "/home/username/", + userdb_uid => 1000, + userdb_gid => 1000, + }; + return $ret; + }, + userdb => sub { + my ($arg) = @_; + my $ret = { + home => "/home/username/", + uid => 1000, + gid => 1000, + }; + return $ret; + }, + ); + + # protocol from src/lib-dict/dict-client.h + my $json = JSON::XS->new; + + eval { + my $ret; + # Dict protocol is multiline... go through the lines. + while (<STDIN>) { + $self->log(2, "Got request: $_"); + chomp; + my $cmd = substr($_,0,1); + next if $cmd eq 'H'; # "hello", skip this line, assume it's ok + die "Protocol error: Bad command $cmd" unless ($cmd eq 'L'); + # Process request + + my ($namespace,$type,$arg) = split ('/',substr($_,1),3); + + if ($namespace eq 'shared') { + my $f = $L_handler{$type}; + + if (defined $f && defined $arg) { + $ret = $f->($arg); + } + else { + die 'Protocol error: Bad arg'; + } + } + else { + die 'Protocol error: Bad namespace' + } + last; # Got an "L" , now respond. + } + if ($ret) { + my $json = JSON::XS->new->indent(0)->utf8->encode($ret); + $self->log(3,"O:$json"); + print "O".$json."\n"; + } + else { + $self->log(3,"NOUSER"); + print "N\n"; + } + 1; + } or do { + $self->log(2, "Error: $@"); + print "F\n"; + }; +} + +sub pre_loop_hook { + my $self = shift; + + $self->log(1, 'Starting server'); +} + +sub pre_server_close_hook { + my $self = shift; + + $self->log(1, 'Server is shut down'); +} + +1; + +__END__ +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.AuthBinds.txt b/doc/wiki/AuthDatabase.LDAP.AuthBinds.txt new file mode 100644 index 0000000..a81eef3 --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.AuthBinds.txt @@ -0,0 +1,89 @@ +Passdb LDAP with authentication binds +===================================== + +Advantages over <password lookups> [AuthDatabase.LDAP.PasswordLookups.txt]: + + * LDAP server verifies the password, so Dovecot doesn't need to know what + format the password is stored in. + * A bit more secure, as a security hole in Dovecot doesn't give attacker + access to all the users' password hashes. (And Dovecot admins in general + don't have direct access to them.) + +You can enable authentication binds by setting 'auth_bind=yes'. Next Dovecot +needs to know what DN to use in the binding. There are two ways to configure +this: lookup or template. + +DN lookup +--------- + +DN is looked up by sending a 'pass_filter' LDAP request and getting the DN from +the reply. This is very similar to doing a <password lookup> +[AuthDatabase.LDAP.PasswordLookups.txt]. The only difference is that +userPassword attribute isn't returned. Just as with password lookups, the +'pass_attrs' may contain special <extra fields> +[PasswordDatabase.ExtraFields.txt]. + +Example: + +---%<------------------------------------------------------------------------- +auth_bind = yes +pass_attrs = uid=user +pass_filter = (&(objectClass=posixAccount)(uid=%u)) +---%<------------------------------------------------------------------------- + +DN template +----------- + +The main reason to use DN template is to avoid doing the DN lookup, so that the +authentication consists only of one LDAP request. With IMAP and POP3 logins the +same optimization can be done by using <prefetch userdb> +[UserDatabase.Prefetch.txt] and returning userdb info in the DN lookup (a total +of two LDAP requests per login in both cases). If you're also using Dovecot for +SMTP AUTH, it doesn't do a userdb lookup so the prefetch optimization doesn't +help. + +If you're using DN template, 'pass_attrs' and 'pass_filter' settings are +completely ignored. That means you can't make passdb return any <extra fields> +[PasswordDatabase.ExtraFields.txt]. You should also set 'auth_username_format = +%Lu' in 'dovecot.conf' to normalize the username by lowercasing it. + +Example: + +---%<------------------------------------------------------------------------- +auth_bind = yes +auth_bind_userdn = cn=%u,ou=people,o=org +---%<------------------------------------------------------------------------- + +Connection optimization +----------------------- + +When using + + * auth binds and + * userdb ldap lookups, + +the userdb lookups should use a separate connection to the LDAP server. That +way it can send LDAP requests asynchronously to the server, which improves the +performance. This can be done by specifying different filenames in the LDAP +passdb and userdb args. The second file could be a symlink to the first one. +For example: + +---%<------------------------------------------------------------------------- +passdb { + driver = ldap + args = /etc/dovecot/dovecot-ldap.conf.ext +} +userdb { + driver = ldap + args = /etc/dovecot/dovecot-ldap-userdb.conf.ext +} +---%<------------------------------------------------------------------------- + +And create the symlink: + +---%<------------------------------------------------------------------------- +ln -s /etc/dovecot/dovecot-ldap.conf.ext +/etc/dovecot/dovecot-ldap-userdb.conf.ext +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.PasswordLookups.txt b/doc/wiki/AuthDatabase.LDAP.PasswordLookups.txt new file mode 100644 index 0000000..a9232d5 --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.PasswordLookups.txt @@ -0,0 +1,98 @@ +Passdb LDAP with password lookups +================================= + +Advantages over <authentication binds> [AuthDatabase.LDAP.AuthBinds.txt]: + + * Faster, because Dovecot can keep sending multiple LDAP requests + asynchronously to the server. With auth binds Dovecot must wait for each + request to finish before sending the next one. + * Supports non-plaintext <authentication mechanisms> + [Authentication.Mechanisms.txt] (if returning plaintext/ <properly hashed + passwords> [Authentication.PasswordSchemes.txt]). + * When using <LDA.txt> and static userdb, deliver can check if destination + user exists. With auth binds this check isn't possible. + +LDAP server permissions +----------------------- + +Normally LDAP server doesn't give anyone access to users' passwords, so you'll +need to create an administrator account that has access to the userPassword +field. With OpenLDAP this can be done by modifying '/etc/ldap/slapd.conf': + +---%<------------------------------------------------------------------------- +# there should already be something like this in the file: +access to attribute=userPassword + by dn="<dovecot's dn>" read # just add this line + by anonymous auth + by self write + by * none +---%<------------------------------------------------------------------------- + +Replace <dovecot's dn> with the DN you specified in 'dovecot-ldap.conf's' 'dn' +setting. + +Dovecot configuration +--------------------- + +The two important settings in password lookups are: + + * 'pass_filter' specifies the LDAP filter how user is found from the LDAP. You + can use all the normal <variables> [Variables.txt] like '%u' in the filter. + * 'pass_attrs' specifies a comma-separated list of attributes that are + returned from the LDAP. If you set it to empty, all the attributes are + returned. + +Usually the LDAP attribute names aren't the same as <the field names that +Dovecot uses internally> [PasswordDatabase.txt]. You must create a mapping +between them to get the wanted results. This is done by listing the fields as +'<ldap attribute>=<dovecot field>'. For example: + +---%<------------------------------------------------------------------------- +pass_attrs = uid=user, userPassword=password +---%<------------------------------------------------------------------------- + +This maps the LDAP "uid" attribute to Dovecot's "user" field and LDAP's +"userPassword" attribute to Dovecot's "password" field. These two fields should +always be returned, but it's also possible to return other special <extra +fields> [PasswordDatabase.ExtraFields.txt]. + +Password +-------- + +Most importantly the 'pass_attrs' must return a "password" field, which +contains the user's password. The next thing Dovecot needs to know is what +format the password is in. If all the passwords are in same format, you can use +'default_pass_scheme' setting in 'dovecot-ldap.conf' to specify it. Otherwise +each password needs to be prefixed with "{password-scheme}", for example +"{plain}plaintext-password". See <Authentication.PasswordSchemes.txt> for a +list of supported password schemes. + +Username +-------- + +LDAP lookups are case-insensitive. Unless you somehow normalize the username, +it's possible that a user logging in as "user", "User" and "uSer" are treated +differently. The easiest way to handle this is to tell Dovecot to change the +username to the same case as it's in the LDAP database. You can do this by +returning "user" field in the 'pass_attrs', as shown in the above example. + +If you can't normalize the username in LDAP, you can alternatively lowercase +the username in 'dovecot.conf': + +---%<------------------------------------------------------------------------- +auth_username_format = %Lu +---%<------------------------------------------------------------------------- + +Example +------- + +A typical configuration would look like: + +---%<------------------------------------------------------------------------- +auth_bind = no +pass_attrs = uid=user, userPassword=password +pass_filter = (&(objectClass=posixAccount)(uid=%u)) +default_pass_scheme = MD5 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.Userdb.txt b/doc/wiki/AuthDatabase.LDAP.Userdb.txt new file mode 100644 index 0000000..3371325 --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.Userdb.txt @@ -0,0 +1,181 @@ +Userdb LDAP +=========== + +Usually your LDAP database also contains the <userdb information> +[UserDatabase.txt]. If your home directory can be specified with a template and +you're using only a single <UID and GID> [UserIds.txt], you should use <static +userdb> [UserDatabase.Static.txt] instead to avoid an unnecessary LDAP lookup. +You can also use <prefetch userdb> [UserDatabase.Prefetch.txt] to avoid the +userdb LDAP lookup. + +Userdb lookups are always done using the default DN ('dn' setting) bind. It's +not possible to do the lookup using the user's DN (remember that e.g. <LDA.txt> +needs to do userdb lookups without knowing the user's password). + +The userdb lookups are configured in very much the same way as <LDAP password +lookups> [AuthDatabase.LDAP.PasswordLookups.txt]. Instead of 'pass_attrs' and +'pass_filter', the userdb uses 'user_attrs' and 'user_filter'. Typically +'pass_filter' and 'user_filter' are equivalent. + +If you're using a single UID and GID for all the users, you can specify them +globally with 'mail_uid' and 'mail_gid' settings instead of returning them from +LDAP. + +Example: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =home=%{ldap:homeDirectory}, \ + =uid=%{ldap:uidNumber}, \ + =gid=%{ldap:gidNumber} +user_filter = (&(objectClass=posixAccount)(uid=%u)) + +# For using doveadm -A: +iterate_attrs = =user=%{ldap:uid} +iterate_filter = (objectClass=posixAccount) +---%<------------------------------------------------------------------------- + +Attribute templates (v2.1+) +--------------------------- + +You can mix static text with the value returned from LDAP by using %{ldap:*} +variables, which expand to the named LDAP attribute's value. Some examples: + +Create a "quota_rule" field with value "*:bytes=<n>" where <n> comes from +"quotaBytes" LDAP attribute: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =quota_rule=*:bytes=%{ldap:quotaBytes} +---%<------------------------------------------------------------------------- + +Create a "mail" field with value "maildir:/var/mail/<dir>/Maildir" where <dir> +comes from "sAMAccountName" LDAP attribute: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =mail=maildir:/var/spool/vmail/%{ldap:sAMAccountName}/Maildir +---%<------------------------------------------------------------------------- + +You can add static fields that aren't looked up from LDAP. For example create a +"mail" field with value "maildir:/var/vmail/%d/%n/Maildir": + +---%<------------------------------------------------------------------------- +user_attrs = \ + =quota_rule=*:bytes=%{ldap:quotaBytes}, \ + =mail=maildir:/var/vmail/%d/%n/Maildir +---%<------------------------------------------------------------------------- + +If you don't want a field to exist at all when its LDAP attribute doesn't +exist, you can give the attribute name before the first "=" character. For +example this doesn't return "home" or "mail" fields if "mailboxPath" doesn't +exist: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =quota_rule=*:bytes=%{ldap:quotaBytes}, \ + mailboxPath=home=/home/%{ldap:mailboxPath}, \ + mailboxPath=mail=maildir:~/Maildir +---%<------------------------------------------------------------------------- + +It's also possible to give default values to nonexistent attributes in v2.1.11+ +by using e.g.'%{ldap:userDomain:example.com} ' where if userDomain attribute +doesn't exist, example.com is used instead. + +Subqueries and pointers (v2.2) +------------------------------ + +LDAP values can now have DN pointers to other entries that are queried. + +*Note*: These aren't actually very useful anymore. See the next section for how +to do multiple queries more easily using multiple userdbs. + +Example: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =user=%{ldap:uid}, \ + @mail=%{ldap:mailDN}, \ + =uid=%{ldap:uidNumber@mail}, \ + =gid=%{ldap:gidNumber@mail}, \ + =home=%{ldap:rootPath@mail}/%d/%n +---%<------------------------------------------------------------------------- + +This will do a regular lookup first. Then does another lookup with DN taken +from mailDN's value. The *@mail attributes are assigned from the second +lookup's results. + +---%<------------------------------------------------------------------------- +user_attrs = \ + =user=%{ldap:uid}, \ + =home=%{ldap_ptr:activePath}, \ + !primaryPath, !secondaryPath +---%<------------------------------------------------------------------------- + +The activePath's value can be either "primaryPath" or "secondaryPath". The +home's value will be the contents of that field. The !field tells Dovecot to +fetch the field's value but not to do anything with it otherwise. + +Multiple queries via userdbs (v2.2+) +------------------------------------ + +Example: Give the user a class attribute, which defines the default quota: + +dovecot.conf: + +---%<------------------------------------------------------------------------- +userdb { + driver = ldap + args = /etc/dovecot/dovecot-users-ldap.conf.ext + result_success = continue-ok +} +userdb { + driver = ldap + args = /etc/dovecot/dovecot-class-ldap.conf.ext + skip = notfound +} +---%<------------------------------------------------------------------------- + +/etc/dovecot/dovecot-users-ldap.conf.ext: + +---%<------------------------------------------------------------------------- +# If user has overridden quota, quota_rule is set below. Otherwise it's still +unset. +user_attrs = \ + =class=%{ldap:userClass} + quotaBytes=quota_rule=*:bytes=%{ldap:quotaBytes} +---%<------------------------------------------------------------------------- + +/etc/dovecot/dovecot-class-ldap.conf.ext: + +---%<------------------------------------------------------------------------- +# Do the lookup using the user's class: +user_filter = (&(objectClass=userClass)(class=%{userdb:class})) +# With :protected suffix the quota_rule isn't overridden if it's already set. +user_attrs = \ + classQuotaBytes=quota_rule:protected=*:bytes=%{ldap:classQuotaBytes} +---%<------------------------------------------------------------------------- + +Variables and domains +--------------------- + +User names and domains may be distinguished using the <Variables.txt> %n and +%d. They split the /previous username/ at the "@" character. The /previous +username/ is: + + * For LMTP, it will be user@hostname, where hostname depends on e.g. the + Postfix configuration. + * For IMAP, it will be whatever the password database has designated as the + username. If the (LDAP) password database has "user_attrs = =user=%n", then + the domain part of the login name will be stripped by the password database. + The UserDB will not see any domain part, i.e. %n and %u are the same thing + for the UserDB. + +The UserDB may set a new username, too, using "user_attrs = =user=...". This +will be used for + + * Logging + * %u and %d variables in other parts of the configuration (e.g. quota file + names) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.txt b/doc/wiki/AuthDatabase.LDAP.txt new file mode 100644 index 0000000..3ad2348 --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.txt @@ -0,0 +1,79 @@ +LDAP +==== + +There are two ways to do LDAP authentication: + + * <Password lookups> [AuthDatabase.LDAP.PasswordLookups.txt] + * <Authentication binds> [AuthDatabase.LDAP.AuthBinds.txt] + +Both of these have their own advantages and disadvantages. + + * <LDAP as userdb> [AuthDatabase.LDAP.Userdb.txt] and other common LDAP query + settings. + +Configuration common to LDAP passdb and userdb +---------------------------------------------- + +Connecting +---------- + +There are two alternative ways to specify what LDAP server(s) to connect to: + + * 'hosts': A space separated list of LDAP hosts to connect to. You can also + use host:port syntax to use different ports. + * 'uris': A space separated list of LDAP URIs to connect to. This isn't + supported by all LDAP libraries. The URIs are in syntax + 'protocol://host:port'. For example 'ldap://localhost' or + 'ldaps://secure.domain.org' + +If multiple LDAP servers are specified, it's decided by the LDAP library how +the server connections are handled. Typically the first working server is used, +and it's never disconnected from. So there is no load balancing or automatic +reconnecting to the "primary" server. + +SSL/TLS +------- + +You can enable TLS in two alternative ways: + + * Connect to ldaps port (636) by using "ldaps" protocol, e.g. 'uris = + ldaps://secure.domain.org' + * Connect to ldap port (389) and use STARTTLS command. Use 'tls=yes' to enable + this. + +See the tls_* settings in 'dovecot-ldap-example.conf' for how to configure TLS. +(I think they apply to ldaps too?) + +Getting Dovecot to talk to a LDAPS signed against a custom certificate of +authority +----------------------------------------------------------------------------------- + +If you need to connect to ldaps secured against a custom certificate of +authority (CA), you will need to install the custom CA on your system.On Red +Hat Enterprise Linux 6, Dovecot uses the OpenLDAP library. By default, the CA +must be installed under the directory specified in the TLS_CACERTDIR option +found under /etc/openldap/ldap.conf (default value is /etc/openldap/certs). +After copying the CA, you'll need to run "c_rehash ." inside the directory, +this will create a symlink pointing to the CA. + +You can test the CA installation with this: openssl s_client -connect +yourldap.example.org:636 -CApath /etc/openldap/certs -showcerts + +This should report "Verify return code: 0 (ok)". + +SASL binds +---------- + +It's possible to use SASL binds instead of the regular plaintext binds if your +LDAP library supports them. See the sasl_* settings in +'dovecot-ldap-example.conf'. Note that SASL binds are currently incompatible +with authentication binds. + +Active Directory +---------------- + +When connecting to AD, you may need to use port 3268. Then again, not all LDAP +fields are available in port 3268. Use whatever +works.http://technet.microsoft.com/en-us/library/cc978012.aspx + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.Lua.txt b/doc/wiki/AuthDatabase.Lua.txt new file mode 100644 index 0000000..41caf61 --- /dev/null +++ b/doc/wiki/AuthDatabase.Lua.txt @@ -0,0 +1,293 @@ +Lua based authentication +======================== + +Since v2.3.0 you can implement passdb and userdb using Lua +[https://www.lua.org/] script. + +Contents + + + 1. Lua based authentication + + 1. Known bugs + + 2. Lua interface + + 3. Auth request methods + + 4. Password database + + 5. User database + + 6. Examples + +Known bugs +---------- + + * Before 2.3.4 when returning a table with values, the table values are + mistakenly converted into a number if they seem like a number. So if you are + using values like '012345', this would get converted into '12345' + * Before 2.3.4 returning password without scheme would cause a crash. + +Lua interface +------------- + +For details about Dovecot Lua, see <Design.Lua.txt>. + +When used in authentication, additional module *dovecot.auth* is added, which +contains constants for passdb and userdb. + +List of constants +----------------- + + * dovecot.auth.PASSDB_RESULT_INTERNAL_FAILURE + * dovecot.auth.PASSDB_RESULT_SCHEME_NOT_AVAILABLE - indicates password scheme + that cannot be understood + * dovecot.auth.PASSDB_RESULT_USER_UNKNOWN + * dovecot.auth.PASSDB_RESULT_USER_DISABLED + * dovecot.auth.PASSDB_RESULT_PASS_EXPIRED + * dovecot.auth.PASSDB_RESULT_NEXT - indicates that this passdb did not + authenticate user, next passdb should do it + * dovecot.auth.PASSDB_RESULT_PASSWORD_MISMATCH + * dovecot.auth.PASSDB_RESULT_OK + * dovecot.auth.USERDB_RESULT_INTERNAL_FAILURE + * dovecot.auth.USERDB_RESULT_USER_UNKNOWN + * dovecot.auth.USERDB_RESULT_OK + +Also, it registers object *struct auth_request** which lets access various +parts of the auth request. You should use the loggers associated with +auth_request when possible. + +Auth request methods +-------------------- + +Functions: + + * auth_request#log_debug(text) - logs debug message (if debug is enabled, noop + otherwise) + * auth_request#log_error(text) - logs error message + * auth_request#log_info(text) - logs informational message + * auth_request#log_warning(text) - logs warning message + * auth_request#response_from_template(template) - takes in key=value template + and expands it using var_expand and produces table suitable for passdb + result + * auth_request#var_expand(template) - performs var expansion on the template + using <Variables.txt> + * auth_request#password_verify(crypted_password, plain_password) - checks if + the plain password matches the crypted or hashed password + * auth_request#event() - Returns child event for the auth request, can be used + for logging and other events. Comes with a prefix. (Since v2.3.6+) + +Subtables: + + * auth_request#passdb + * auth_request#userdb + +Members: + +See <Variables.txt> for details + + * auth_request#auth_domain + * auth_request#auth_user + * auth_request#auth_username + * auth_request#cert + * auth_request#client_id + * auth_request#domain + * auth_request#domain_first + * auth_request#domain_last + * auth_request#home + * auth_request#lip + * auth_request#local_name + * auth_request#login_domain + * auth_request#login_user + * auth_request#login_username + * auth_request#lport + * auth_request#master_user + * auth_request#mech + * auth_request#orig_domain + * auth_request#orig_user + * auth_request#orig_username + * auth_request#password + * auth_request#pid + * auth_request#real_lip + * auth_request#real_lport + * auth_request#real_rip + * auth_request#real_rport + * auth_request#rip + * auth_request#rport + * auth_request#secured + * auth_request#service + * auth_request#session + * auth_request#session_pid + * auth_request#user + * auth_request#username + +Additionally you can access + + * skip_password_check - Set if the password has already been validated by + another passdb + * passdbs_seen_user_unknown - If some previous passdb has not found this user + * passdbs_seen_internal_failure - If some previous passdb has had internal + failure + * userdbs_seen_internal_failure - If some previous userdb has had internal + failure + +Password database +----------------- + +Lua passdb supports two modes of function. It can behave as lookup database, or +password verification database. + +Lookup function signature is *auth_passdb_lookup(request)* and the password +verification signature is *auth_password_verify(request, password)* + +Both functions must return a tuple, which contains a return code, and also +additionally string or table. Table must be in key-value format, it will be +imported into auth request. The string must be in key=value format, except if +return code indicates internal error, the second parameter can be used as error +string. + +If *auth_verify_password* is found, it's always used. + +To configure passdb in dovecot, use + +---%<------------------------------------------------------------------------- +passdb { + driver = lua + args = file=/path/to/lua blocking=yes # default is yes +} +---%<------------------------------------------------------------------------- + +By default, dovecot runs Lua scripts in auth-worker processes. If you do not +want this, you can disable blocking, and Lua script will be ran in auth +process. This can degrade performance if your script is slow or makes external +lookups. + +User database +------------- + +Lua userdb supports both single user lookup and iteration. Note that iteration +will hold the whole user database in memory during iteration. + +User lookup function signature is *auth_userdb_lookup(request)*. The function +must return a tuple, which contains a return code, and also additionally string +or table. Table must be in key-value format, it will be imported into auth +request. The string must be in key=value format, except if return code +indicates internal error, the second parameter can be used as error string. + +User iteration function signature is *auth_userdb_iterate*, which is expected +to return table of usernames. Key names are ignored. + +To configure userdb in dovecot, use + +---%<------------------------------------------------------------------------- +userdb { + driver = lua + args = file=/path/to/lua blocking=yes # default is yes +} +---%<------------------------------------------------------------------------- + +Examples +-------- + +Skeleton +-------- + +---%<------------------------------------------------------------------------- +function auth_passdb_lookup(req) + if req.user == "testuser1" then + return dovecot.auth.PASSDB_RESULT_OK, "password=pass" + end + return dovecot.auth.PASSDB_RESULT_USER_UNKNOWN, "no such user" +end + +function auth_userdb_lookup(req) + if req.user == "testuser1" then + return dovecot.auth.USERDB_RESULT_OK, "uid=vmail gid=vmail" + end + return dovecot.auth.USERDB_RESULT_USER_UNKNOWN, "no such user" +end + +function script_init() + return 0 +end + +function script_deinit() +end + +function auth_userdb_iterate() + return {"testuser1"} +end +---%<------------------------------------------------------------------------- + +Simple username password database (such as opensmtpd) +----------------------------------------------------- + +The example uses whitespace separated username and password. As a special +caution, the way Lua is used here means you can have multiple user password per +line, instead of just one.This can be extended to more complicated separators +or multiple fields per user. + +If you only want to autenticate users, and don't care about user listing, you +can use + +---%<------------------------------------------------------------------------- +function auth_passdb_lookup(req) + for line in io.lines("/path/to/file") do + for user, pass in string.gmatch(line, "(%w+)%s(.+)") do + if (user == req.username) then + -- you can add additional information here, like userdb_uid + return dovecot.auth.PASSDB_RESULT_OK, "password=" .. pass + end + end + end + return dovecot.auth.PASSDB_RESULT_USER_UNKNOWN, "" +end +---%<------------------------------------------------------------------------- + +If you also want to be able to list users, so that you could use doveadm cmd -A + +---%<------------------------------------------------------------------------- +local database = "/path/to/file" + +function db_lookup(username) + for line in io.lines(database) do + for user, pass in string.gmatch(line, "(%w+)%s(.+)") do + if (user == username) then + return {result=0, password=pass} + end + end + end + return {result=-1} +end + +function auth_passdb_lookup(req) + res = db_lookup(req.username) + if res.result == 0 then + -- you can add additional information here for passdb + return dovecot.auth.PASSDB_RESULT_OK, "password=" .. res.password + end + return dovecot.auth.PASSDB_RESULT_USER_UNKNOWN, "" +end + +function auth_userdb_lookup(req) + res = db_lookup(req.username) + if res.result == 0 then + -- you can add additional information here for userdb, like uid or home + return dovecot.auth.USERDB_RESULT_OK, "uid=vmail gid=vmail" + end + return dovecot.auth.USERDB_RESULT_USER_UNKNOWN, "" +end + +function auth_userdb_iterate() + users = {} + for line in io.lines(database) do + for user in string.gmatch(line, "(%w+)%s.+") do + table.insert(users, user) + end + end + return users +end +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.Passwd.txt b/doc/wiki/AuthDatabase.Passwd.txt new file mode 100644 index 0000000..d9e1071 --- /dev/null +++ b/doc/wiki/AuthDatabase.Passwd.txt @@ -0,0 +1,53 @@ +Passwd +====== + +User is looked up using 'getpwnam()' call, which usually looks into +'/etc/passwd' file, but depending on NSS +[http://en.wikipedia.org/wiki/Name_Service_Switch] configuration it may also +look up the user from eg. LDAP database. + +Most commonly used as a user database. + +The lookup is by default done in the auth worker processes. If you have only a +small local passwd file, you can avoid having extra auth worker processes by +disabling it: + +---%<------------------------------------------------------------------------- +userdb { + driver = passwd + args = blocking=no +} +---%<------------------------------------------------------------------------- + +Field overriding and extra fields (obsolete in v2.1+) +----------------------------------------------------- + +It's possible to override fields from passwd and add <extra fields> +[UserDatabase.ExtraFields.txt] with templates, but in v2.1+ it's done in a +better way by using override_fields. For example: + +---%<------------------------------------------------------------------------- +userdb { + driver = passwd + # Pre-v2.1: + #args = home=/var/mail/%u mail=maildir:/var/mail/%u/Maildir + # v2.1+: + override_fields = home=/var/mail/%u mail=maildir:/var/mail/%u/Maildir +} +---%<------------------------------------------------------------------------- + +This uses the UID and GID fields from passwd, but home directory is overridden. +Also the default <mail_location> [MailLocation.txt] setting is overridden. + +Passwd as a password database +----------------------------- + +Many systems use shadow passwords nowadays so passwd doesn't usually work as a +password database. BSDs are an exception to this, they still set the password +field even with shadow passwords. + +With FreeBSD, passwd doesn't work as a password database because the password +field is replaced by a '*'. But you can use <Passwd-file> +[AuthDatabase.PasswdFile.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.PasswdFile.txt b/doc/wiki/AuthDatabase.PasswdFile.txt new file mode 100644 index 0000000..aa25b6d --- /dev/null +++ b/doc/wiki/AuthDatabase.PasswdFile.txt @@ -0,0 +1,150 @@ +Passwd-file +=========== + +This file is compatible with a normal '/etc/passwd' file, and a password file +used by libpam-pwdfile <PAM> [PasswordDatabase.PAM.txt] plugin. It's in the +following format: + +---%<------------------------------------------------------------------------- +user:password:uid:gid:(gecos):home:(shell):extra_fields +---%<------------------------------------------------------------------------- + +For a password database it's enough to have only the user and password fields. +For a user database, you need to set also uid, gid and preferably also home +(see <VirtualUsers.txt>). (gecos) and (shell) fields are unused by Dovecot. + +The password field can be in four formats: + + * 'password': Assume CRYPT <password scheme> + [Authentication.PasswordSchemes.txt]. + * '{SCHEME}password': The password is in the given <scheme> + [Authentication.PasswordSchemes.txt]. + * 'password[13]': libpam-passwd file compatible format for CRYPT <scheme> + [Authentication.PasswordSchemes.txt]. + * 'password[34]': libpam-passwd file compatible format for MD5 <scheme> + [Authentication.PasswordSchemes.txt]. + +extra_fields is a space-separated list of key=value pairs which can be used to +set various <passdb settings> [PasswordDatabase.ExtraFields.txt] and <userdb +settings> [UserDatabase.ExtraFields.txt]. Keys which begin with a 'userdb_' +prefix are used for userdb, others are used for passdb. So for example if you +wish to override <mail_location> [MailLocation.txt] setting for one user, use +'userdb_mail=mbox:~/mail'. <Variable> [Variables.txt] expansion is done for +extra_fields. + +Empty lines and lines beginning with '#' character are ignored. + +Multiple passwd files +--------------------- + +You can use all the <variables> [Variables.txt] in the passwd-file filenames, +for example: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + # Each domain has a separate passwd-file: + args = /etc/auth/%d/passwd +} +---%<------------------------------------------------------------------------- + +Passwd-file args +---------------- + + * *scheme=<s>*: Allows you to specify the default <password scheme> + [Authentication.PasswordSchemes.txt]. The default is CRYPT. This is + available only for passdb. + * *username_format=<s>*: Look up usernames using this format instead of the + full username ('%u'). If you want to enable user@domain logins but have only + "user" in the file, set this to '%n'. + +Examples +-------- + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = scheme=plain-md5 username_format=%n /etc/imap.passwd +} +userdb { + driver = passwd-file + args = username_format=%n /etc/imap.passwd + default_fields = uid=vmail gid=vmail home=/home/vmail/%u +} +---%<------------------------------------------------------------------------- + + * The default_fields is explained in <UserDatabase#Userdb_settings.> + [UserDatabase.txt] They can be used to provide default userdb fields based + on templates in case they're not specified for everyone in the passwd file. + If you leave any of the standard userdb fields (uid, gid, home) empty, these + defaults will be used. + +This file can be used as a passdb: + +---%<------------------------------------------------------------------------- +user:{plain}password +user2:{plain}password2 +---%<------------------------------------------------------------------------- + +passdb with extra fields: + +---%<------------------------------------------------------------------------- +user:{plain}password::::::allow_nets=192.168.0.0/24 +---%<------------------------------------------------------------------------- + +This file can be used as both a passwd and a userdb: + +---%<------------------------------------------------------------------------- +user:{plain}pass:1000:1000::/home/user::userdb_mail=maildir:~/Maildir +allow_nets=192.168.0.0/24 +user2:{plain}pass2:1001:1001::/home/user2 +---%<------------------------------------------------------------------------- + +FreeBSD /etc/master.passwd as passdb and userdb +----------------------------------------------- + +On FreeBSD, '/etc/passwd' doesn't work as a password database because the +password field is replaced by a '*'. '/etc/master.passwd' can be converted into +a format usable by passwd-file. As <PasswordDatabase.PAM.txt> can access the +system-wide credentials on FreeBSD, what follows is generally needed only if +the mail accounts are different from the system accounts. + +If only using the result for 'name:password:uid:gid' and not using +<PasswordDatabase.ExtraFields.txt> you may be able to use the extract directly. +However, the Linux-style passwd file has fewer fields than that used by FreeBSD +and it will need to be edited if any fields past the first four are needed. In +particular, it will fail if used directly as a 'userdb' as the field used for +'home' is not in the same place as expected by the Dovecot parser. The +':class:change:expire' stanza in each line should be removed to be consistent +with the Linux-style format. While that stanza often is '::0:0' use of 'cut' is +likely much safer than 'sed' or other blind substitution. + +In '/etc/master.passwd', a password of '*' indicates that password +authentication is disabled for that user and the token '*LOCKED*' prevents all +login authentication, so you might as well exclude those: + +---%<------------------------------------------------------------------------- +# fgrep -v '*' /etc/master.passwd | cut -d : -f 1-4,8-10 > +/path/to/file-with-encrypted-passwords +# chmod 640 /path/to/file-with-encrypted-passwords +# chown root:dovecot /path/to/file-with-encrypted-passwords +---%<------------------------------------------------------------------------- + +or permissions and ownership that may be more appropriate for your install and +security needs. + +The following will work in many situations, after disabling the inclusion of +other 'userdb' and 'passdb' sections + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = username_format=%n /path/to/file-with-encrypted-passwords +} +userdb { + driver = passwd-file + args = username_format=%n /path/to/file-with-encrypted-passwords +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.SQL.txt b/doc/wiki/AuthDatabase.SQL.txt new file mode 100644 index 0000000..6176be7 --- /dev/null +++ b/doc/wiki/AuthDatabase.SQL.txt @@ -0,0 +1,243 @@ +SQL +=== + +SQL can be used for both passdb and userdb lookups. If the args parameter in +passdb sql and userdb sql contain the exact same filename, only one SQL +connection is used for both passdb and userdb lookups. + +Contents + + + 1. SQL + + 1. Dovecot configuration + + 2. Password database lookups + + 3. Password verification by SQL server + + 4. User database lookups + + 5. User iteration + + 6. Prefetching + + 7. High availability + + 8. Examples + +Dovecot configuration +--------------------- + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +Password database lookups +------------------------- + +'password_query' setting contains the SQL query to look up the password. It +must return a field named "password". If you have it by any other name in the +database, you can use the SQL's "AS" keyword ('SELECT pw AS password ..'). You +can use all the normal <variables> [Variables.txt] such as '%u' in the SQL +query. + +If all the passwords are in same format, you can use 'default_pass_scheme' to +specify it. Otherwise each password needs to be prefixed with +"{password-scheme}", for example "{plain}plaintext-password". See +<Authentication.PasswordSchemes.txt> for a list of supported password schemes. + +By default MySQL does case-insensitive string comparisons, so you may have a +problem if your users are logging with different as "user", "User" and "uSer". +To fix this, you can make the SQL database return a <"user" field> +[PasswordDatabase.ExtraFields.User.txt], which makes Dovecot modify the +username to the returned value. Note that if you're using separate user and +domain fields, a common problem is that you're returning only the "user" field +from the database.*This drops out the domain from the username*. So make sure +you're returning a concatenated user@domain string or username/domain fields +separately. See the examples below. + +The query can also return other <extra fields> +[PasswordDatabase.ExtraFields.txt] which have special meaning. + +You can't use multiple statements in one query, but you could use a stored +procedure. If you want something like a last login update, use +<PostLoginScripting.txt> instead. + +Password verification by SQL server +----------------------------------- + +If the passwords are in some special format in the SQL server that Dovecot +doesn't recognize, it's still possible to use them. Change the SQL query to +return NULL as the password and return the row only if the password matches. +You'll also need to return a non-NULL "nopassword" field. The password is in +'%w' variable. For example: + +---%<------------------------------------------------------------------------- +password_query = SELECT NULL AS password, 'Y' as nopassword, userid AS user \ + FROM users WHERE userid = '%u' AND mysql_pass = password('%w') +---%<------------------------------------------------------------------------- + +This of course makes the verbose logging a bit wrong, since password mismatches +are also logged as "unknown user". + +User database lookups +--------------------- + +Usually your SQL database contains also the userdb information. This means +user's UID, GID and home directory. If you're using only static UID and GID, +and your home directory can be specified with a template, you could use <static +userdb> [UserDatabase.Static.txt] instead. It is also a bit faster since it +avoids doing the userdb SQL query. + +'user_query' setting contains the SQL query to look up the userdb information. +The commonly returned userdb fields are uid, gid, home and mail. See +<UserDatabase.ExtraFields.txt> for more information about these and other +fields that can be returned. + +If you're using a single UID and GID for all users, you can set them in +dovecot.conf with: + +---%<------------------------------------------------------------------------- +mail_uid = vmail +mail_gid = vmail +---%<------------------------------------------------------------------------- + +User iteration +-------------- + +Some commands, such as 'doveadm -A' need to get a list of users. With SQL +userdb this is done with 'iterate_query' setting. You can either return + + * "user" field containing either user or user@domain style usernames, or + * "username" and "domain" fields + +Any other fields are ignored. + +Prefetching +----------- + +If you want to avoid doing two SQL queries when logging in with IMAP/POP3, you +can make the 'password_query' return all the necessary userdb fields and use +prefetch userdb to use those fields. If you're using Dovecot's deliver you'll +still need to have the 'user_query' working. + +See <UserDatabase.Prefetch.txt> for example configuration + +High availability +----------------- + +You can add multiple "host" parameters to the SQL connect string. Dovecot will +do round robin load balancing between them. If one of them goes down, the +others will handle the traffic. + +Examples +-------- + +Note that "user" can have a special meaning in some SQL databases, so we're +using "userid" instead. + +SQL table creation command: + +---%<------------------------------------------------------------------------- +CREATE TABLE users ( + userid VARCHAR(128) NOT NULL, + domain VARCHAR(128) NOT NULL, + password VARCHAR(64) NOT NULL, + home VARCHAR(255) NOT NULL, + uid INTEGER NOT NULL, + gid INTEGER NOT NULL +); +---%<------------------------------------------------------------------------- + +MySQL +----- + +Add to your 'dovecot-sql.conf' file: + +---%<------------------------------------------------------------------------- +driver = mysql +# The mysqld.sock socket may be in different locations in different systems. +# Use "host= ... pass=foo#bar" with double-quotes if your password has '#' +character. +# If you need SSL connection, you can add ssl_ca or ssl_ca_path +# You can also use ssl_cert/ssl_key, ssl_cipher, ssl_verify_server_cert +# or provide option_file and option_group +connect = host=/var/run/mysqld/mysqld.sock dbname=mails user=admin +password=pass +# Alternatively you can connect to localhost as well: +#connect = host=localhost dbname=mails user=admin password=pass # port=3306 + +password_query = SELECT userid AS username, domain, password \ + FROM users WHERE userid = '%n' AND domain = '%d' +user_query = SELECT home, uid, gid FROM users WHERE userid = '%n' AND domain = +'%d' + +# For using doveadm -A: +iterate_query = SELECT userid AS username, domain FROM users +---%<------------------------------------------------------------------------- + +PostgreSQL +---------- + +Add to your 'dovecot-sql.conf' file: + +---%<------------------------------------------------------------------------- +# You can also set up non-password authentication by modifying PostgreSQL's +pg_hba.conf +driver = pgsql +# Use "host= ... pass=foo#bar" if your password has '#' character +connect = host=localhost dbname=mails user=admin password=pass + +password_query = SELECT userid AS username, domain, password \ + FROM users WHERE userid = '%n' AND domain = '%d' +user_query = SELECT home, uid, gid FROM users WHERE userid = '%n' AND domain = +'%d' + +# For using doveadm -A: +iterate_query = SELECT userid AS username, domain FROM users +---%<------------------------------------------------------------------------- + +SQLite +------ + +Add to your 'dovecot-sql.conf' file: + +---%<------------------------------------------------------------------------- +driver = sqlite +connect = /path/to/sqlite.db + +password_query = SELECT userid AS username, domain, password \ + FROM users WHERE userid = '%n' AND domain = '%d' +user_query = SELECT home, uid, gid FROM users WHERE userid = '%n' AND domain = +'%d' + +# For using doveadm -A: +iterate_query = SELECT userid AS username, domain FROM users +---%<------------------------------------------------------------------------- + +PostgreSQL/Horde +---------------- + +I used the following in devocot-sql.conf file to authenticate directly against +the Horde user/password database (with static userdb) on PostgreSQL: + +---%<------------------------------------------------------------------------- +driver = pgsql +connect = host=localhost dbname=horde user=dovecot password= +default_pass_scheme = MD5-CRYPT +password_query = SELECT user_uid AS username, user_pass AS password \ + FROM horde_users WHERE user_uid = '%u' +iterate_query = SELECT user_uid AS username FROM users +---%<------------------------------------------------------------------------- + +Note that you will have to change the password encryption in Horde to +MD5-CRYPT. Also, the example above requires a 'dovecot' user in PostgreSQL with +read (SELECT) privileges on the 'horde_users' table. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.VPopMail.txt b/doc/wiki/AuthDatabase.VPopMail.txt new file mode 100644 index 0000000..30e7144 --- /dev/null +++ b/doc/wiki/AuthDatabase.VPopMail.txt @@ -0,0 +1,167 @@ +VPopMail +======== + +Dovecot supports authenticating against external VPopMail +[http://www.inter7.com/index.php?page=vpopmail] virtual domain manager. Dovecot +must have been configured with '--with-vpopmail' to enable this. You can check +this with 'dovecot --build-options'. See also <VMailMgr> [HowTo.VMailMgr.txt] +for another similar virtual domain manager. + +If the vpopmail database contains plaintext passwords, it can be used for +non-plaintext authentication as well. + +passdb parameters: + + * cache_key: If set, you can use 'auth_cache' with VPopMail. See <PAM> + [PasswordDatabase.PAM.txt] for more information about it. + * webmail=IP: If IP address is specified, connections from it are assumed to + come from webmail and VPopMail's webmail usage restrictions apply. + +userdb parameters: + + * cache_key: Like in passdb. + * quota_template=TEMPLATE: Template to specify quota rule, %q in value expands + to Maildir++ quota. + +Example +------- + +---%<------------------------------------------------------------------------- +passdb { + driver = vpopmail + args = webmail=127.0.0.1 +} +userdb { + driver = vpopmail + args = quota_template=quota_rule=*:backend=%q +} +---%<------------------------------------------------------------------------- + +VPopMail + MySQL +================ + +Alternatively, you can use the SQL backend with the following configuration: + +---%<------------------------------------------------------------------------- +driver = mysql +connect = host=/var/run/mysqld/mysqld.sock user=vpopmail +password=YOURPASSWORDHERE dbname=vpopmail + +default_pass_scheme = PLAIN +password_query = SELECT CONCAT(pw_name, '@', pw_domain) AS user, \ + pw_clear_passwd AS password \ + FROM vpopmail \ + WHERE pw_name = '%n' AND pw_domain = '%d' +user_query = SELECT pw_dir as home, \ + 89 AS uid, 89 AS gid \ + FROM vpopmail \ + WHERE pw_name = '%n' AND pw_domain = '%d' +---%<------------------------------------------------------------------------- + +VPopMail + MySQL + pw_gid (disable_imap, disable_webmail) and vlimits support +============================================================================= + +The above example doesn't support vpopmail's abilities to disable access to +services like IMAP, webmail etc. which is controlled by vmoduser and +vmoddomlimits. + +VPopMail uses pw_gid column in the database to store this information. It has a +binary format and every bit of the number stored in this column is responsible +for a different access limit. + +As defined in the vpopmail.h: + +---%<------------------------------------------------------------------------- +/* gid flags */ +#define NO_PASSWD_CHNG 0x01 +#define NO_POP 0x02 +#define NO_WEBMAIL 0x04 +#define NO_IMAP 0x08 +#define BOUNCE_MAIL 0x10 +#define NO_RELAY 0x20 +#define NO_DIALUP 0x40 +#define V_USER0 0x080 +#define V_USER1 0x100 +#define V_USER2 0x200 +#define V_USER3 0x400 +#define NO_SMTP 0x800 +#define QA_ADMIN 0x1000 +#define V_OVERRIDE 0x2000 +---%<------------------------------------------------------------------------- + ++ if vpopmail has been compiled with domain limits (--enable-mysql-limits) +domain wise limits will be defined in a table called "limits" where there are +fields like disable_imap and disable_webmail which values by default are NULL +and 1 if option is set. The use of NULLs in limits table is a bit problematic +because in order to properly handle this situation we're going to have replace +NULL with a numeric value. Of course we're going to join vpopmail table (the +table holding users) with limits table using LEFT JOIN. + +Here's the config taken directly from my install: + +---%<------------------------------------------------------------------------- +# +user_query = SELECT pw_name,89 as uid, 89 as gid, pw_dir as home FROM vpopmail +WHERE pw_name = '%n' AND pw_domain = '%d' +#The below passes all users and doesn't care for vpopmail limits (pw_gid column +or vlimits table) +#password_query = SELECT pw_passwd as password FROM vpopmail WHERE pw_name = +'%n' AND pw_domain = '%d' +# +#A little bit more complicated query to support vpopmail pw_gid flags and +vlimits for domain +#explanation: +#We're using bitwise operations on pw_gid. +#as defined in vpopmail.h: +#- 0x04 - disable webmail flag +#- 0x08 - disable imap flag +# +# !(pw_gid & 8) means - if 8th bit of pw_gid is not set +# !(pw_gid & 4) means - if 4th bit of pw_gid is not set +# (pw_gid & 8192) means - if 14th bit of pw_gid is set (ignore vlimits) +# +# additionally because we're using LEFT JOIN we have to take care of NULLs for +rows that don't return any records from the right table hence the use of +COALESCE() function +# !(pw_gid & 4) (disable webmail flag) is used in conjuntion with +'%r'!="127.0.0.1" which means that it will only apply to connections +originating from hosts other than localhost +# +# So the below query supports pw_gid and vlimits settings for user account and +domains but no domain limit overrides +# +#password_query = select pw_passwd as password FROM vpopmail LEFT JOIN limits +ON vpopmail.pw_domain=limits.domain WHERE pw_name='%n' and pw_domain='%d' and ( +!(pw_gid & 8) and ('%r'!='127.0.0.1' or !(pw_gid & 4)) and ( '%r'!='127.0.0.1' +or COALESCE(disable_webmail,0)!=1) and COALESCE(disable_imap,0)!=1); +# +# The below adds support for vlimits override on user account (vmoduser -o) +# +#logically this means: show password for user=%n at domain=%d when imap on the +account is not disabled and connection is not comming from localhost when +webmail access on the account is not disabled and if imap for the domain is not +disabled and (connection is not comming from localhost when webmail access for +the domain is not disabled) when vlimits are not overriden on the account +# +password_query = select pw_passwd as password FROM vpopmail LEFT JOIN limits ON +vpopmail.pw_domain=limits.domain WHERE pw_name='%n' and pw_domain='%d' and +!(pw_gid & 8) and ('%r'!='127.0.0.1' or !(pw_gid & 4)) and ( ('%r'!='127.0.0.1' +or COALESCE(disable_webmail,0)!=1) and COALESCE(disable_imap,0)!=1 or (pw_gid & +8192) ); +---%<------------------------------------------------------------------------- + +Please be aware that disable_webmail is strictly binded to the IP address hard +coded in the query. In this example webmail connections come from the same +machine that the IMAP server is running on using 127.0.0.1 IP address. So the +webmail client is configured with something like eg. $IMAP_SERVER="127.0.0.1". +If your webmail client is on a different machine you need to change 127.0.0.1 +to your webmail's server IP. + +Also - be aware that dovecot caches SQL results (configurable) so if you're +testing the above config on an account that has logged on succesfully within +the cache timeout period and you changed the settings on it using eg. vmoduser +-i test@example.com account which effectively disabled IMAP access for this +account dovecot can still log this user on because the result of the password +query has been stored in cache and used. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.txt b/doc/wiki/AuthDatabase.txt new file mode 100644 index 0000000..ccfc896 --- /dev/null +++ b/doc/wiki/AuthDatabase.txt @@ -0,0 +1,19 @@ +Authentication Databases +======================== + +These databases can be used as both <password databases> [PasswordDatabase.txt] +and <user databases> [UserDatabase.txt]: + + * <Passwd> [AuthDatabase.Passwd.txt]: System users (NSS, '/etc/passwd', or + similiar) + * <Passwd-file> [AuthDatabase.PasswdFile.txt]: '/etc/passwd'-like file in + specified location + * <LDAP> [AuthDatabase.LDAP.txt]: Lightweight Directory Access Protocol + * <SQL> [AuthDatabase.SQL.txt]: SQL database (PostgreSQL, MySQL, SQLite) + * <Dict> [AuthDatabase.Dict.txt]: Dict key-value database (Redis, memcached, + etc.) + * <VPopMail> [AuthDatabase.VPopMail.txt]: External software used to handle + virtual domains + * <Lua> [AuthDatabase.Lua.txt]: Lua script for authentication (v2.3.0+) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Caching.txt b/doc/wiki/Authentication.Caching.txt new file mode 100644 index 0000000..4f696d3 --- /dev/null +++ b/doc/wiki/Authentication.Caching.txt @@ -0,0 +1,107 @@ +Caching of authentication results +================================= + +Dovecot supports caching the results of password and user database lookups. The +following rules apply to using the authentication cache: + + * Data is used from the cache if it's not expired ('auth_cache_ttl' setting) + * If authentication fails this time, but it didn't fail last time, it's + assumed that the password has changed and a database lookup is done. + * If a database lookup fails because of some internal error, but data still + exists in the cache (even if expired), the cached data is used. This allows + Dovecot to log in some users even if the database is temporarily down. + +The authentication cache can be flushed by sending a SIGHUP to dovecot-auth. + +Sending SIGUSR2 to dovecot-auth makes it log the number of cache hits and +misses. You can use that information for tuning the cache size and TTL. + +Settings +-------- + +The settings related to the authentication cache are: + + * 'auth_cache_size': Authentication cache size, 0 disables caching (default). + A typical passdb cache entry is around 50 bytes and a typical userdb cache + entry is around 100-200 bytes, depending on the amount of information your + user and password database lookups return. + * 'auth_cache_ttl': Time to live in seconds for cache entries. A cache entry + is no longer used (except for internal failures) if it was created more than + this many seconds ago. Entries are removed from the cache only when the + cache is full and a new entry is to be added. + * 'auth_cache_negative_ttl': If a passdb or userdb lookup didn't return any + data (i.e. the user doesn't exist), it's also stored in the cache as a + negative entry. This setting allows you to give negative entries a different + TTL. 0 disables negative caching completely. + * 'auth_cache_verify_password_with_worker': Password hash verifications are + done by the auth master process by default. Setting this to "yes" moves the + verification to auth-worker processes. This allows distributing the hash + calculations to multiple CPU cores, which could make sense if strong hashes + are used. (v2.2.34+) + +It should be pretty safe to set very high TTLs, because the only field that +usually can change is the user's password, and Dovecot attempts to catch those +cases (see the rules above). + +Cache keys +---------- + +Usually only the username uniquely identifies a user, but in some setups you +may need something more, for example the remote IP address. For SQL and LDAP +lookups Dovecot figures this out automatically by using all the used +<%variables> [Variables.txt] as the cache key. For example if your SQL query +contains %s, %u and %r the cache entry is used only if all of them (service +name, username and remote IP) match for the new lookup. + +With other databases Dovecot doesn't know what could affect caching, so you +have to tell Dovecot manually. The following databases require specifying the +cache key: + + * vpopmail + * pam + * bsdauth + +For example if the PAM lookup depends on username and service, you can use: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = cache_key=%s%u * +} +---%<------------------------------------------------------------------------- + +Password changing scenarios +--------------------------- + +Normal scenario: + + 1. User logs in with password X. The password X is added to cache and login + succeeds. + 2. Password is changed to Y. + 3. User logs in with password Y. The cached password X doesn't match Y, but + since the previous authentication was successful Dovecot does another + backend passdb lookup to see if the password changed. It did, so the + password Y is cached and login succeeds. + +Using old cached password scenario: + + 1. User logs in with password X. The password X is added to cache and login + succeeds. + 2. Password is changed to Y. + 3. User logs in with password X. The cached password X matches X, so login + succeeds. + +Early change scenario: + + 1. User logs in with password X. The password X is added to cache and login + succeeds. + 2. User logs in with password Y. The cached password X doesn't match Y, but + since the previous authentication was successful Dovecot does another + backend passdb lookup to see if the password changed. It didn't, so the + login fails. + 3. Password is changed to Y. + 4. User logs in with password Y. The cached password X doesn't match Y and the + previous authentication was unsuccessful, so Dovecot doesn't bother doing + another backend passdb lookup (until cache TTL expires). The login fails. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Kerberos.txt b/doc/wiki/Authentication.Kerberos.txt new file mode 100644 index 0000000..d37dc3c --- /dev/null +++ b/doc/wiki/Authentication.Kerberos.txt @@ -0,0 +1,229 @@ +Kerberos +======== + +Dovecot supports Kerberos 5 using GSSAPI. The Kerberos authentication mechanism +doesn't require having a <passdb> [PasswordDatabase.txt], but you do need a +<userdb> [UserDatabase.txt] so Dovecot can lookup user-specific information, +such as where their mailboxes are stored. With centralized systems, such as +Microsoft Active Directory, LDAP is pretty good choice. + +*Note:* If you only wish to authenticate clients using their Kerberos +/passphrase/ (as opposed to ticket authentication), you will probably want to +use <PAM> [PasswordDatabase.PAM.txt] authentication with 'pam_krb5.so' instead. + +Pre-requisites +-------------- + +This document assumes that you already have a Kerberos Realm up and functioning +correctly at your site, and that each host in your realm also has a host +/keytab/ installed in the appropriate location. + +For Dovecot, you will need to install the appropriate /service/ keys on your +server. By default, Dovecot will look for these in the host's keytab file, +typically '/etc/krb5.keytab', but you can specify an alternate path using the +'auth_krb5_keytab' configuration entry in dovecot.conf. Anyway specified +keytab file should be readable by user "dovecot" (or whatever user the auth +process is running as). If you wish to provide an IMAP service, you will need +to install a service ticket of the form 'imap/hostname@REALM'. For POP3, you +will need a service ticket of the form 'pop/hostname@REALM'. When using +Dovecot's <SASL> [Sasl.txt] with MTA, you will need to install service ticket +of the form 'smtp/hostname@REALM'. + +Setting up (samba) +------------------ + +Create symlink for krb5.conf, if you do not have krb5.conf ready + +---%<------------------------------------------------------------------------- +ln -sf /usr/local/samba/private/krb5.conf /etc/krb5.conf +---%<------------------------------------------------------------------------- + +Create dovecot user to your samba instance (choose random password) + +---%<------------------------------------------------------------------------- +$ samba-tool user create dovecot +New Password: +Retype Password: +User 'dovecot' created successfully +---%<------------------------------------------------------------------------- + +Add Service Principal Names (SPNs) and create keytab + +---%<------------------------------------------------------------------------- +$ samba-tool spn add imap/host.domain.com dovecot +$ samba-tool domain exportkeytab --principal imap/host.domain.com +/etc/dovecot/dovecot.keytab +---%<------------------------------------------------------------------------- + +Dovecot needs to be able to read the keytab + +---%<------------------------------------------------------------------------- +$ chgrp dovecot /etc/dovecot/dovecot.keytab +$ chmod g+r /etc/dovecot/dovecot.keytab +---%<------------------------------------------------------------------------- + +Make sure your keytab has entry for imap/host.domain.name@REALM. + +---%<------------------------------------------------------------------------- +$ klist -Kek /etc/dovecot/dovecot.keytab +Keytab name: FILE:/etc/dovecot/dovecot.keytab +KVNO Principal +---- -------------------------------------------------------------------------- + 1 imap/host.domain.name@REALM (des-cbc-crc) + 1 imap/host.domain.name@REALM (des-cbc-md5) + 1 imap/host.domain.name@REALM (arcfour-hmac) +---%<------------------------------------------------------------------------- + +Example dovecot.conf configurations +----------------------------------- + +If you only want to use Kerberos ticket-based authentication: + +---%<------------------------------------------------------------------------- +auth_gssapi_hostname = "$ALL" +auth_mechanisms = gssapi +auth_krb5_keytab = /etc/dovecot/dovecot.keytab + +userdb { + driver = static + args = uid=vmail gid=vmail home=/var/vmail/%u +} +---%<------------------------------------------------------------------------- + +(In this virtual-hosting example, all mail is stored in /var/vmail/$username +with uid and gid set to 'vmail') + +If you also want to support plaintext authentication in addition to +ticket-based authentication, you will need something like: + +---%<------------------------------------------------------------------------- +auth_mechanisms = plain login gssapi +auth_gssapi_hostname = "$ALL" +auth_mechanisms = gssapi +auth_krb5_keytab = /etc/dovecot/dovecot.keytab +passdb { + driver = pam +} +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +(Note that in this example, you will also need to configure PAM to use +whichever authentication backends are appropriate for your site.) + +Enable plaintext authentication to use Kerberos +----------------------------------------------- + +This is needed when some of your clients don't support GSSAPI and you still +want them to authenticate against Kerberos. + +Install pam_krb5 module for PAM, and create '/etc/pam.d/dovecot': + +---%<------------------------------------------------------------------------- +auth sufficient pam_krb5.so +account sufficient pam_krb5.so +---%<------------------------------------------------------------------------- + +Then enable PAM passdb: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam +} +---%<------------------------------------------------------------------------- + +Check '/var/log/auth.log' if you have any problems logging in. The problem +could be that PAM is still trying to use pam_unix.so rather than pam_krb5.so. +Make sure pam_krb5.so is the first module for account or just change +pam_unix.so to sufficient. + +Cross-realm authentication +-------------------------- + +This seems to have all kinds of trouble. Search Dovecot mailing list for +previous threads about it. Some points about it: + + * krb5_kuserok() is used to check if access is allowed. It may try to do the + check by reading ~user/.k5login (good!) or ~dovecot/.k5login (bad!) + * Solaris uses _gss_userok() instead of krb5_kuserok() + * v2.2+ has "k5principals" <passdb extra field> + [PasswordDatabase.ExtraFields.txt], which is a comma separated list of + usernames that are allowed to log in. If it's set, it bypasses the + krb5_kuserok() check.*NOTE*: for this to work, you need a password database + which supports *credential lookups*. + * With 2.2.13, this excludes LDAP databases using authentication binds + (auth_bind = yes). However, a second LDAP passdb entry without "auth_bind + = yes" may be added for the sole purpose of Kerberos principals mapping. + This passdb doesn't need to return a password attribute (and usually + should'nt). + * UPDATE: code seems to have been reworked in later versions. With 2.2.24, + authentication-bind LDAP databases are able to provide k5principals + lookups if configured with "pass_filter". + +Client support +-------------- + +Mail clients that support Kerberos GSSAPI authentication include: + + * Evolution + * Mozilla Thunderbird + * SeaMonkey + * Mutt + * UW Pine + * Apple Mail + +Test that the server can access the keytab +------------------------------------------ + +This test demonstrates that the server can acquire its private credentials. +First telnet directly to the server + +---%<------------------------------------------------------------------------- +$ telnet localhost 143 +* OK Dovecot ready. +---%<------------------------------------------------------------------------- + +or, if you are using IMAPS then use openssl instead of telnet to connect: + +---%<------------------------------------------------------------------------- +$ openssl s_client -connect localhost:993 +CONNECTED(00000003) +... +* OK Dovecot ready. +---%<------------------------------------------------------------------------- + +Check that GSSAPI appears in the authentication capabilities: + +---%<------------------------------------------------------------------------- +a capability +* CAPABILITY ... AUTH=GSSAPI +---%<------------------------------------------------------------------------- + +Attempt the first round of GSS communication. The '+' indicates that the server +is ready + +---%<------------------------------------------------------------------------- +a authenticate GSSAPI ++ +---%<------------------------------------------------------------------------- + +Abort the telnet session by typing control-] and then 'close' + +---%<------------------------------------------------------------------------- +^] +telnet> close +---%<------------------------------------------------------------------------- + +The test: + + * Setup mutt in /etc/Muttrc to use kerberos using gssapi and imap + configuration + * this is done with 'set imap_authenticators="gssapi"' + * run kinit (type in password for kerb) + * run command mutt + * If you get error No Authentication Method + * run command klist (list all kerberos keys) should show imap/HOSTNAME + * DNS has to function correctly so that kerberos works. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.MasterUsers.txt b/doc/wiki/Authentication.MasterUsers.txt new file mode 100644 index 0000000..2374cf7 --- /dev/null +++ b/doc/wiki/Authentication.MasterUsers.txt @@ -0,0 +1,294 @@ +Master users/passwords +====================== + +It's possible to configure master users who are able to log in as other users. +It's also possible to directly log in as any user using a master password. + +Master users +------------ + +There are two ways for master users to log in as other users: + + 1. Give the login username in the <SASL mechanism's> + [Authentication.Mechanisms.txt] authorization ID field. + 2. Specify both the master username and the login username in the same + username field. The usernames are separated by a string configured by the + 'auth_master_user_separator' setting. UW-IMAP uses "*" as the separator, so + that could be a good choice. Using "*" as the separator, the master user + would log in as "login_user*master_user". + +Master users are configured by adding a new <passdb> [PasswordDatabase.txt] +with 'master=yes' setting. The users in the master passdb cannot log in as +themselves, only as other people. That means they don't need to exist in the +<userdb> [UserDatabase.txt], because the userdb lookup is done only for the +user they're logging in as. + +You should also add the 'pass=yes' setting to the master passdb if possible. It +means that Dovecot verifies that the login user really exists before allowing +the master user to log in. Without the setting if a nonexistent login username +is given, depending on the configuration, it could either return an internal +login error (the userdb lookup failed) or create a whole new user (with eg. +<static userdb> [UserDatabase.Static.txt]). 'pass=yes' doesn't work with PAM or +LDAP with 'auth_bind=yes', because both of them require knowing the user's +password. + +'pass=yes' is especially useful with a <Checkpassword> +[PasswordDatabase.CheckPassword.txt] passdb because the script gets both the +login and the master username as environment variables. Other passdbs see only +the login username in '%u'. In the future there will probably be another +setting to make the user verification to be done from userdb. + +If you want master users to be able to log in as themselves, you'll need to +either add the user to the normal passdb or add the passdb to 'dovecot.conf' +twice, with and without 'master=yes'. Note that if the passdbs point to +different locations, the user can have a different password when logging in as +other users than when logging in as himself. This is a good idea since it can +avoid accidentally logging in as someone else. + +Usually it's better to have only a few special master users that are used +*only* to log in as other people. One example could be a special "spam" master +user that trains the users' spam filters by reading the messages from the +user's spam mailbox. + +ACLs +---- + +If <ACL.txt> plugin is enabled, the Master user is still subject to ACLs just +like any other user, which means that by default the master user has no access +to any mailboxes of the user. The options for handling this are: + + 1. Adding a global <ACL.txt> for the master user. You can create a "default + ACL", that applies to all mailboxes. See example below. + 2. Set 'plugin { acl_user=%u } ' This preserves the master_user for other + purposes (e.g. %{master_user} variable). + 3. Set 'plugin { master_user=%u } ' This fully hides that master user login is + being used. + +Example configuration +--------------------- + +---%<------------------------------------------------------------------------- +auth_master_user_separator = * +passdb { + driver = passwd-file + args = /etc/dovecot/passwd.masterusers + master = yes + pass = yes +} +passdb { + driver = shadow +} +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +To grant the masteruser access to all Mailboxes, the 'dovecot-acl' file can +contain: + +---%<------------------------------------------------------------------------- +* user=masteruser lr +---%<------------------------------------------------------------------------- + +Where the 'passwd.masterusers' file would contain the master usernames and +passwords: + +---%<------------------------------------------------------------------------- +admin:{SHA1}nU4eI71bcnBGqeO0t9tXvY1u5oQ= +admin2:{SHA1}i+UhJqb95FCnFio2UdWJu1HpV50= +---%<------------------------------------------------------------------------- + +One way to create this master file is to use the htaccess program as follows: + +---%<------------------------------------------------------------------------- +htpasswd -b -c -s passwd.masterusers user password +---%<------------------------------------------------------------------------- + +SQL Example +----------- + +The master passdb doesn't have to be passwd-file, it could be an SQL query as +well: + +---%<------------------------------------------------------------------------- +auth_master_user_separator = * +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql-master.conf.ext + master = yes + pass = yes +} +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +} +---%<------------------------------------------------------------------------- + +'dovecot-sql-master.conf.ext' would contain all the normal connection settings +and a 'password_query': + +---%<------------------------------------------------------------------------- +password_query = SELECT password FROM users WHERE userid = '%u' and master_user += true +---%<------------------------------------------------------------------------- + +Testing +------- + +---%<------------------------------------------------------------------------- +# telnet localhost 143 +* OK Dovecot ready. +1 login loginuser*masteruser masterpass +1 OK Logged in. +---%<------------------------------------------------------------------------- + +If you had any problems, set 'auth_debug=yes' and look at the logs. + +Master passwords +---------------- + +You can configure a passdb which first performs authentication using the master +password. Then it continues to the primary passdb to verify that the user +exists and get other extra fields. + +---%<------------------------------------------------------------------------- +# master password passdb +passdb { + driver = static + default_fields = password=master-password + result_success = continue +} +# primary passdb +passdb { + driver = pam +} +---%<------------------------------------------------------------------------- + +Advanced SQL Examples +--------------------- + +In these example we will create 3 kinds of master users. The first will be +users who can read all email for all domains. The next example will be users +who can read all email for their domain only. The third example will be users +who can read email of domains listed in a separate ownership table. We will use +MySQL and create 2 tables with the following structure. + +---%<------------------------------------------------------------------------- +CREATE TABLE `users` ( + `uid` int(4) NOT NULL AUTO_INCREMENT, + `user_name` varchar(80) NOT NULL, + `domain_name` varchar(80) NOT NULL, + `password` varchar(60) DEFAULT NULL, + `last_login` datetime DEFAULT NULL, + `masteradmin` tinyint(1) NOT NULL DEFAULT '0', + `owns_domain` tinyint(1) NOT NULL DEFAULT '0', + UNIQUE KEY `emaillookup` (`domain_name`,`user_name`), + UNIQUE KEY `uid` (`uid`) +) ENGINE=MyISAM AUTO_INCREMENT=995 DEFAULT CHARSET=latin + +CREATE TABLE `ownership` ( + `login_id` varchar(128) NOT NULL, + `owned_object` varchar(128) NOT NULL, + UNIQUE KEY `login_id_full` (`login_id`,`owned_object`), + KEY `login_id` (`login_id`), + KEY `owned_object` (`owned_object`), + KEY `login_id_index` (`login_id`), + KEY `owned_object_index` (`owned_object`) +) ENGINE=MyISAM DEFAULT CHARSET=latin1 +---%<------------------------------------------------------------------------- + +The dovecot.conf file for all 3 master user configurations will be as follows: + +---%<------------------------------------------------------------------------- +passdb { + driver = sql + args = /etc/dovecot/ownership-sql.conf + master = yes + pass = yes +} + +passdb { + driver = sql + args = /etc/dovecot/domain-owner-sql.conf + master = yes + pass = yes +} + +passdb { + driver = sql + args = /etc/dovecot/masteradmin-sql.conf + master = yes + pass = yes +} +passdb { + args = /etc/dovecot/sql.conf + driver = sql +} +---%<------------------------------------------------------------------------- + +Before we get into the master user tricks, we start with normal email +authentication. The query for that is as follows: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users WHERE +user_name = '%n' AND domain_name = '%d' +---%<------------------------------------------------------------------------- + +In this first example master admin suppose you want to allow a few people to be +master users over all domains. These users will have the "masteradmin" field +set to 1. The query would be: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users WHERE +user_name = '%n' AND domain_name = '%d' AND masteradmin='1' +---%<------------------------------------------------------------------------- + +In the second example suppose you are hosting multiple domains and you want to +allow a few users to become master users of their domain only. + +Your query would be as follows: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users WHERE +user_name = '%n' \ +AND domain_name = '%d' AND owns_domain='1' AND '%d'='%{login_domain}' +---%<------------------------------------------------------------------------- + +This will allow you to log in using the following to read Joe's email if +master@dovecot.org is flagged as the domain_owner. + +---%<------------------------------------------------------------------------- +joe@dovecot.org*master@dovecot.org +---%<------------------------------------------------------------------------- + +In this third example we have a table of owners. There are a list of pairs +between owner email addresses and domains that are owned. That way if a person +controls a lot of domains then they can view all the users in all the domains +they control. The query would be as follows: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users, ownership +WHERE \ +user_name = '%n' AND domain_name = '%d' AND login_id='%u' AND +owned_object='%{login_domain}' +---%<------------------------------------------------------------------------- + +If you really want to get tricky and efficient you can combine all 3 queries +into one giant query that does everything. + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users, ownership +WHERE \ +user_name = '%n' AND domain_name = '%d' AND ( \ +(masteradmin='1') OR \ +(owns_domain='1' AND '%d'='%{login_domain}') OR \ +(login_id='%u' and owned_object='%{login_domain}')) \ +group by uid +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.DigestMD5.txt b/doc/wiki/Authentication.Mechanisms.DigestMD5.txt new file mode 100644 index 0000000..eda617e --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.DigestMD5.txt @@ -0,0 +1,80 @@ +Digest-MD5 Authentication Mechanism +=================================== + +Digest-MD5 has two things that make it special and which can cause problems: + + * Instead of using user@domain usernames, it supports *realms*. + * User name and realm are part of the MD5 hash that's used for authentication. + +For these and other reasons Digest-MD5 has been obsoleted +[http://tools.ietf.org/html/rfc6331] by SCRAM +[http://tools.ietf.org/html/rfc5802]. + +Realms +------ + +Dovecot v1.0 has problems handling user@domain style usernames with Digest-MD5 +and with passwords stored in plaintext in the password database. + +Instead, user@realm is used. Realms are an integral part of Digest-MD5. You +will need to specify realms you want to advertise to the client in the config +file: + +---%<------------------------------------------------------------------------- +auth_realms = example.com another.example.com foo +---%<------------------------------------------------------------------------- + +The realms don't have to be domains. All listed realms are presented to the +client and it can select to use one of them. Some clients always use the first +realm. Some clients use your domain name, whenever given more than one realm to +choose from. Even if this was NOT one of the choices you provided (KMail, +others?). In both cases the user never sees the advertised realms. + +You can also set an 'auth_default_realm' to use when a client gives an empty +realm. However the client is supposed to use the same realm it tells the server +for the calculation. So when this option helps, it is only for broken clients. + +DIGEST-MD5 scheme +----------------- + +Password must be stored in either plaintext or with DIGEST-MD5 scheme. See +<Authentication.PasswordSchemes.txt>. + +The Digest is the MD5 sum of the string "user:realm:password". So for example +if you want to log in as 'user' with password 'pass' and the realm should be +'example.com' (usually not provided by the user, see above), create the digest +with: + +---%<------------------------------------------------------------------------- +% echo -n "user:example.com:pass" | md5sum +c19c4c6e32f9d8026b26ba77c21fb8eb - +---%<------------------------------------------------------------------------- + +And save it as + +---%<------------------------------------------------------------------------- +user@example.com:c19c4c6e32f9d8026b26ba77c21fb8eb +---%<------------------------------------------------------------------------- + +Note that if you're using DIGEST-MD5 scheme to store the passwords, you can't +change the users' names or realms in any way or the authentication will fail +because the MD5 sums don't match. Also not that this is different from what +Apache does with HTTP AUTH Digest. There it would be +'user:example.com:c19c4c6e32f9d8026b26ba77c21fb8eb' and is created with +'htdigest'. + +Testing +------- + +You can use 'imtest' from Cyrus SASL +[http://asg.web.cmu.edu/sasl/sasl-library.html] library to test an IMAP +connection: + +---%<------------------------------------------------------------------------- +# With realm: +imtest -a user -r example.com +# Without realm: +imtest -a user@example.com +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.NTLM.txt b/doc/wiki/Authentication.Mechanisms.NTLM.txt new file mode 100644 index 0000000..46b5750 --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.NTLM.txt @@ -0,0 +1,34 @@ +NTLM +==== + +There are four authentication submethods inside the NTLM: + + 1. LM: server nonce only, highly vulnerable to MITM and rogue server attacks. + 2. NTLM: different algorithm, almost equally vulnerable as LM today. + 3. NTLM2: server and client nonce, but MITM can force downgrade to NTLM/LM. + 4. NTLMv2: server and client nonce, MITM can't force downgrade. + +NTLM <password scheme> [Authentication.PasswordSchemes.txt] is required for +NTLM, NTLM2 and NTLMv2. + +NTLMv2 can not be negotiated. It must be explicitly enabled on the client side +by setting registry key below to at least 3: + + * Win9x: + 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA\LMCompatibility' + * WinNT: + 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA\LMCompatibilityLevel' + +Dovecot's NTLM logic is: + + 1. If we have only LM password scheme, try LM authentication; + 2. If client sends LM response only (some very old clients do it), try LM too; + + 3. If NTLMv2 is guessed (using client response length), try NTLMv2; + 4. If NTLM2 was negotiated, try it; + 5. Otherwise try NTLM. + +For more information about NTLM internals, see http://ubiqx.org/cifs/ and +http://davenport.sourceforge.net/ntlm.html + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.Winbind.txt b/doc/wiki/Authentication.Mechanisms.Winbind.txt new file mode 100644 index 0000000..256a54d --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.Winbind.txt @@ -0,0 +1,35 @@ +Winbind mechanisms +================== + +Dovecot supports NTLM and GSS-SPNEGO authentication mechanisms using Samba +[http://www.samba.org]'s winbind daemon. It is useful when you need to +authenticate users against a Windows domain (either AD or NT). + +By default NTLM mechanism is handled internally. You can use winbind instead by +setting: + +---%<------------------------------------------------------------------------- +auth_use_winbind = yes +---%<------------------------------------------------------------------------- + +The usernames, returned by winbind, can contain some domain part (either +"DOMAIN\user" or "user@example.com"). Such usernames are always transformed to +the form of "user@domain". To strip domain part (to obtain corresponding local +username, for example), set: + +---%<------------------------------------------------------------------------- +auth_username_format = %n +---%<------------------------------------------------------------------------- + +Dovecot needs path to Samba's 'ntlm_auth' binary to perform the authentication. +You can change the path with: + +---%<------------------------------------------------------------------------- +auth_winbind_helper_path = /usr/bin/ntlm_auth +---%<------------------------------------------------------------------------- + +Dovecot currently does blocking lookups, so if 'ntlm_auth' is slow on +responding (e.g. network problems), Dovecot blocks all other authentication +requests until it's finished. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.txt b/doc/wiki/Authentication.Mechanisms.txt new file mode 100644 index 0000000..ed36580 --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.txt @@ -0,0 +1,82 @@ +Authentication Mechanisms +========================= + +Plaintext authentication +------------------------ + +The simplest authentication mechanism is PLAIN. The client simply sends the +password unencrypted to Dovecot. All clients support the PLAIN mechanism, but +obviously there's the problem that anyone listening on the network can steal +the password. For that reason (and some others) other mechanisms were +implemented. + +Today however many people use <SSL/TLS> [SSL.txt], and there's no problem with +sending unencrypted password inside SSL secured connections. So if you're using +SSL, you probably don't need to bother worrying about anything else than the +PLAIN mechanism. + +Another plaintext mechanism is LOGIN. It's typically used only by SMTP servers +to let Outlook clients perform SMTP authentication. Note that LOGIN mechanism +is not the same as IMAP's LOGIN command. The LOGIN command is internally +handled using PLAIN mechanism. + +Non-plaintext authentication +---------------------------- + +Non-plaintext mechanisms have been designed to be safe to use even without +<SSL/TLS> [SSL.txt] encryption. Because of how they have been designed, they +require access to the plaintext password or their own special hashed version of +it. This means that it's impossible to use non-plaintext mechanisms with +commonly used DES or MD5 password hashes. + +If you want to use more than one non-plaintext mechanism, the passwords must be +stored as plaintext so that Dovecot is able to generate the required special +hashes for all the different mechanisms. If you want to use only one +non-plaintext mechanism, you can store the passwords using the mechanism's own +<password scheme> [Authentication.PasswordSchemes.txt]. + +With <success/failure password databases> [PasswordDatabase.txt] (e.g. PAM) +it's not possible to use non-plaintext mechanisms at all, because they only +support verifying a known plaintext password. + +Dovecot supports the following non-plaintext mechanisms: + + * <CRAM-MD5.txt>: Protects the password in transit against eavesdroppers. + Somewhat good support in clients. + * <DIGEST-MD5> [Authentication.Mechanisms.DigestMD5.txt]: Somewhat stronger + cryptographically than CRAM-MD5, but clients rarely support it. + * SCRAM-SHA-1: Salted Challenge Response Authentication Mechanism (SCRAM) SASL + and GSS-API Mechanisms. Intended as DIGEST-MD5 replacement. + * APOP: This is a POP3-specific authentication. Similar to CRAM-MD5, but + requires storing password in plaintext. + * <NTLM> [Authentication.Mechanisms.NTLM.txt]: Mechanism created by Microsoft + and supported by their clients. + * Optionally supported <using Samba's winbind> + [Authentication.Mechanisms.Winbind.txt]. + * GSS-SPNEGO: A wrapper mechanism defined by RFC 4178 + [http://tools.ietf.org/html/rfc4178]. Can be accessed via either GSSAPI or + <Winbind> [Authentication.Mechanisms.Winbind.txt]. + * <GSSAPI> [Authentication.Kerberos.txt]: Kerberos v5 support. + * RPA: Compuserve RPA authentication mechanism. Similar to DIGEST-MD5, but + client support is rare. + * ANONYMOUS: Support for logging in anonymously. This may be useful if you're + intending to provide publicly accessible IMAP archive. + * OTP and SKEY: One time password mechanisms. + * EXTERNAL: EXTERNAL SASL mechanism. + * OAUTBEARER: OAuth2 bearer authentication + https://tools.ietf.org/html/rfc7628. See <OAuth2 password database> + [PasswordDatabase.oauth2.txt] (v2.2.29+) + * XOAUTH2: Google flavor OAUTHBEARER + [https://developers.google.com/gmail/xoauth2_protocol] (v2.2.29+) + +Configuration +------------- + +By default only PLAIN mechanism is enabled. To use more, edit your +'/etc/dovecot/conf.d/10-auth.conf' and set: + +---%<------------------------------------------------------------------------- +auth_mechanisms = plain login cram-md5 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.MultipleDatabases.txt b/doc/wiki/Authentication.MultipleDatabases.txt new file mode 100644 index 0000000..8a98692 --- /dev/null +++ b/doc/wiki/Authentication.MultipleDatabases.txt @@ -0,0 +1,112 @@ +Multiple Authentication Databases +================================= + +Dovecot supports defining multiple authentication databases, so that if the +password doesn't match in the first database, it checks the next one. This can +be useful if you want to easily support having both local system users in +'/etc/passwd' and virtual users. + +Currently the fallback works only with the PLAIN authentication mechanism. + +Often you also want a different mail location for system and virtual users. The +best way to do this would be to always have mails stored below the home +directory ( <virtual users should have a home directory too> +[VirtualUsers.Home.txt]): + + * System users' mails: /home/user/Maildir + * Virtual users' mails: /var/vmail/domain/user/Maildir + +This can be done by simply having both system and virtual userdbs return home +directory properly (i.e. virtual users''home=/var/vmail/%d/%n') and then set +'mail_location = maildir:~/Maildir'. + +If it's not possible to have a home directory for virtual users (avoid that if +possible), you can do this by pointing <mail_location> [MailLocation.txt] to +system users' mail location and have the virtual userdb override it by +returning 'mail' <extra field> [UserDatabase.ExtraFields.txt]. + +Example with home dirs +---------------------- + + * System users' mails: /home/user/Maildir + * Virtual users' mails: /var/vmail/domain/user/Maildir + +dovecot.conf: + +---%<------------------------------------------------------------------------- +# Mail location for both system and virtual users: +mail_location = maildir:~/Maildir + +# try to authenticate using SQL database first +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# fallback to PAM +passdb { + driver = pam +} + +# look up users from SQL first (even if authentication was done using PAM!) +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# if not found, fallback to /etc/passwd +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +dovecot-sql.conf.ext: + +---%<------------------------------------------------------------------------- +password_query = SELECT userid as user, password FROM users WHERE userid = '%u' +user_query = SELECT uid, gid, '/var/vmail/%d/%n' as home FROM users WHERE +userid = '%u' +---%<------------------------------------------------------------------------- + +Example with overriding mail location +------------------------------------- + + * System users' mails: /home/user/Maildir + * Virtual users' mails: /var/vmail/domain/user + +dovecot.conf: + +---%<------------------------------------------------------------------------- +# the default mail location for system users, this will be overridden in userdb +sql. +mail_location = maildir:~/Maildir + +# try to authenticate using SQL database first +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# fallback to PAM +passdb { + driver = pam +} + +# look up users from SQL first (even if authentication was done using PAM!) +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# if not found, fallback to /etc/passwd +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +dovecot-sql.conf.ext: + +---%<------------------------------------------------------------------------- +password_query = SELECT userid as user, password FROM users WHERE userid = '%u' +# returning mail overrides mail_location setting for SQL users. +user_query = SELECT uid, gid, 'maildir:/var/vmail/%u' as mail FROM users WHERE +userid = '%u' +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.PasswordSchemes.txt b/doc/wiki/Authentication.PasswordSchemes.txt new file mode 100644 index 0000000..c8516ac --- /dev/null +++ b/doc/wiki/Authentication.PasswordSchemes.txt @@ -0,0 +1,248 @@ +Password Schemes +================ + +Password scheme means the format in which the password is stored in <password +databases> [PasswordDatabase.txt]. The main reason for choosing a scheme other +than *PLAIN* is to prevent someone with access to the password database (such +as a hacker) from stealing users' passwords and using them to access other +services. + +What scheme to use? +------------------- + +You should choose the strongest crypt scheme that's supported by your system. +From strongest to weakest: + + * *ARGON2I/ARGON2ID*: Argon2 [https://en.wikipedia.org/wiki/Argon2] is the + winner of password hashing competition held at July 2015. The password will + start with $argon2i$ or $argon2id$. You can use -r to tune computational + complexity, minimum is 3. ARGON2ID is only available if your libsodium is + recent enough. ARGON2 can require quite a hefty amount of virtual memory, so + we recommend that you set 'service auth { vsz_limit = 2G }' at least, or + more. + * *BLF-CRYPT*: This is the Blowfish crypt (bcrypt) scheme. It is generally + considered to be very secure. The encrypted password will start with '$2y$' + (other generators can generate passwords that have other letters after ' $2 + ', those should work too.) (Note v2.2: bcrypt is not available on most Linux + distributions). Since v2.3.0 this is provided by dovecot. You can tune the + computational cost using -r parameter for doveadm. + * *SHA512-CRYPT*: A strong scheme. The encrypted password will start with + '$6$' + * *SHA256-CRYPT*: A strong scheme. The encrypted password will start with + '$5$' + * *MD5-CRYPT*: A weak but common scheme often used in '/etc/shadow'. The + encrypted password will start with '$1$' + +Note that the above schemes are implemented by the libc's 'crypt()' function. +Using them is especially useful when sharing the same passwords with other +software, because most of them support using 'crypt()' to verify the password. +However, not all libcs (especially older ones) implement all of the above +schemes. See below for other password schemes that are implemented by Dovecot +internally (instead of libc). + +A few articles about why choosing a good password scheme is important: + + * How To Safely Store A Password + [http://codahale.com/how-to-safely-store-a-password/] + * Speed Hashing [http://www.codinghorror.com/blog/2012/04/speed-hashing.html] + +It's not possible to easily switch from one password scheme to another. The +only practical way to do this is to wait until user logs in and change the +password during the login. <This HOWTO> [HowTo.ConvertPasswordSchemes.txt] +shows one way to do this. + +Generating encrypted passwords +------------------------------ + +You can generate passwords for a particular scheme easily with "doveadm pw" +utility. For example: + +---%<------------------------------------------------------------------------- +doveadm pw +---%<------------------------------------------------------------------------- + +Since v2.3.0+ the scheme defaults to BCRYPT, but you can use -s to override + +---%<------------------------------------------------------------------------- +doveadm pw -s SHA512-CRYPT +---%<------------------------------------------------------------------------- + +To provide password, for scripting purposes, you can use either + +---%<------------------------------------------------------------------------- +doveadm pw -p password +---%<------------------------------------------------------------------------- + +or + +---%<------------------------------------------------------------------------- +printf 'password\npassword\n' | doveadm pw +---%<------------------------------------------------------------------------- + +Default password schemes +------------------------ + +Password databases have a default password scheme: + + * <SQL> [AuthDatabase.SQL.txt]: See 'default_pass_scheme' setting in + 'dovecot-sql.conf.ext' + * <LDAP> [AuthDatabase.LDAP.txt]: See 'default_pass_scheme' setting in + 'dovecot-ldap.conf.ext' + * <PasswdFile> [AuthDatabase.PasswdFile.txt]: CRYPT is used by default, but + can be changed with 'scheme' parameter in passdb args. + * <Passwd> [AuthDatabase.Passwd.txt], <Shadow> [PasswordDatabase.Shadow.txt], + <VPopMail> [AuthDatabase.VPopMail.txt]: CRYPT is used by default and can't + be changed currently. + * <PAM> [PasswordDatabase.PAM.txt], <BSDAuth> [PasswordDatabase.BSDAuth.txt], + <CheckPassword> [PasswordDatabase.CheckPassword.txt]: Dovecot never even + sees the password with these databases, so Dovecot has nothing to do with + what password scheme is used. + +The password scheme can be overridden for each password by prefixing it with +{SCHEME}, for example:'{PLAIN}pass'. + +Non-plaintext authentication mechanisms +--------------------------------------- + +See <Authentication.Mechanisms.txt> for explanation of auth mechanisms. Most +installations use only plaintext mechanisms, so you can skip this section +unless you know you want to use them. + +The problem with non-plaintext auth mechanisms is that the password must be +stored either in plaintext, or using a mechanism-specific scheme that's +incompatible with all other non-plaintext mechanisms. In addition, the +mechanism-specific schemes often offer very little protection. This isn't a +limitation of Dovecot, it's a requirement for the algorithms to even work. + +For example if you're going to use CRAM-MD5 authentication, the password needs +to be stored in either PLAIN or CRAM-MD5 scheme. If you want to allow both +CRAM-MD5 and DIGEST-MD5, the password must be stored in plaintext. + +In future it's possible that Dovecot could support multiple passwords in +different schemes for a single user. + + * *LANMAN*: DES-based encryption. Used sometimes with NTLM mechanism. + * *NTLM*: MD4 sum of the password stored in hex. Used with NTLM mechanism. + * *RPA*: Used with RPA mechanism. + * *CRAM-MD5*: Used with CRAM-MD5 mechanism. + * *DIGEST-MD5*: Used with <DIGEST-MD5 mechanism> + [Authentication.Mechanisms.DigestMD5.txt]. The username is included in the + hash, so it's not possible to use the hash for different usernames. + * *SCRAM-SHA-1*: Used with SCRAM-SHA-1 mechanism. (v2.2+) + +Other supported password schemes +-------------------------------- + +Strong schemes and mechanism-specific schemes are listed above. + + * *PLAIN*: Password is in plaintext. + * *CRYPT*: Traditional DES-crypted password in '/etc/passwd' (e.g. "pass" = + 'vpvKh.SaNbR6s') + * Dovecot uses libc's 'crypt()' function, which means that CRYPT is usually + able to recognize MD5-CRYPT and possibly also other password schemes. See + all of the *-CRYPT schemes at the top of this page. + * The traditional DES-crypt scheme only uses the first 8 characters of the + password, the rest are ignored. Other schemes may have other password + length limitations (if they limit the password length at all). + +MD5 based schemes: + + * *PLAIN-MD5*: MD5 sum of the password stored in hex. + * *LDAP-MD5*: MD5 sum of the password stored in base64. + * *SMD5*: Salted MD5 sum of the password stored in base64. + +SHA based schemes (also see below for libc's SHA* support): + + * *SHA*: SHA1 sum of the password stored in base64. + * *SSHA*: Salted SHA1 sum of the password stored in base64. + * *SHA256*: SHA256 sum of the password stored in base64. (v1.1 and later). + * *SSHA256*: Salted SHA256 sum of the password stored in base64. (v1.2 and + later). + * *SHA512*: SHA512 sum of the password stored in base64. (v2.0 and later). + * *SSHA512*: Salted SHA512 sum of the password stored in base64. (v2.0 and + later). + +Other schemes + + * *ARGON2I*: ARGON2i password scheme (v2.3.0+), needs libsodium + * *ARGON2ID*: ARGON2id password scheme (v2.3.0+), needs libsodium + * *PBKDF2*: PKCS5 Password hashing algortihm + +For some schemes (e.g. PLAIN-MD5, SHA) Dovecot is able to detect if the +password hash is base64 or hex encoded, so both can be used.'doveadm pw' anyway +generates the passwords using the encoding mentioned above. + +3rd party password schemes +-------------------------- + +These plugins are provided by community members, we do not provide support or +help with them, please contact the developer(s) directly. Use at your own +discretion. Since v2.3.0 ARGON2 is provided by dovecot itself. + + * *SCRYPT* and *ARGON2*: See + https://github.com/LuckyFellow/dovecot-libsodium-plugin/ + +Encoding +-------- + +The base64 vs. hex encoding that is mentioned above is simply the default +encoding that is used. You can override it for any scheme by adding a ".hex", +".b64" or ".base64" suffix. For example: + + * '{SSHA.b64}986H5cS9JcDYQeJd6wKaITMho4M9CrXM' contains the password encoded + to base64 (just like {SSHA}) + * '{SSHA.HEX}3f5ca6203f8cdaa44d9160575c1ee1d77abcf59ca5f852d1' contains the + password encoded to hex + +This can be especially useful with plaintext passwords to encode characters +that would otherwise be illegal. For example in passwd-file you couldn't use a +":" character in the password without encoding it to base64 or hex. For +example:'{PLAIN}{\}:!"' is the same as '{PLAIN.b64}e1x9OiEiCg=='. + +You can also specify the encoding with doveadm pw. For example: 'doveadm pw -s +plain.b64' + +Salting +------- + +For the SHA512-CRYPT, SHA256-CRYPT and MD5-CRYPT schemes, the salt is stored +before the hash, e.g.:'$6$salt$hash'. For the BLF-CRYPT scheme, bcrypt stores +the salt as part of the hash. + +For most of the other salted password schemes (SMD5, SSHA*) the salt is stored +after the password hash and its length can vary. When hashing the password, +append the salt after the plaintext password, e.g.: SSHA256(pass, salt) = +SHA256(pass + salt) + salt. + +For example with SSHA256 you know that the hash itself is 32 bytes (256 bits/8 +bits per byte). Everything after that 32 bytes is the salt. For example if you +have a password: + +---%<------------------------------------------------------------------------- +{SSHA256}SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv +---%<------------------------------------------------------------------------- + +After base64 decoding it you'll see that its length is 36 bytes, so the first +32 bytes are the hash and the following 4 bytes are the salt: + + * length: 'echo SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv|base64 -d|wc + -c' -> 36 + * hash: 'echo SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv|base64 -d|dd + bs=1 count=32|hexdump -C' -> 4a 84 7f ef c4 f9 ab 45 0f 16 78 3c 50 25 d6 + 43 13 94 2a 1c eb 25 99 70 7c db 65 94 0b a9 01 e5 + * salt: 'echo SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv|base64 -d|dd + bs=1 skip=32|hexdump -C' -> 13 fa 44 2f + +Other common hash sizes are: + + * MD5: 16 bytes + * SHA: 20 bytes + * SHA256: 32 bytes + * SHA512: 64 bytes + +The web management gui VBoxAdm [http://developer.gauner.org/vboxadm/] has some +code dealing with creation and verification of salted hashes in Perl. However +not all password schemes provided by dovecotpw are supported. Have a look at +the module VBoxAdm::DovecotPW for more details. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Penalty.txt b/doc/wiki/Authentication.Penalty.txt new file mode 100644 index 0000000..dcbb3ea --- /dev/null +++ b/doc/wiki/Authentication.Penalty.txt @@ -0,0 +1,45 @@ +Authentication Penalty +====================== + +Dovecot anvil process tracks authentication penalties for different IPs to slow +down brute force login attempts. The algorithm works by: + + * First auth failure reply will be delayed for 2 seconds (this happens even + without auth penalty) + * 'AUTH_PENALTY_INIT_SECS' in 'src/auth/auth-penalty.h' + * The delay will be doubled for 4 -> 8 seconds, and then the upper limit of 15 + seconds is reached. + * 'AUTH_PENALTY_MAX_SECS' and AUTH_PENALTY_MAX_PENALTY in + 'src/auth/auth-penalty.h' + * If the IP is in login_trusted_networks (e.g. webmail), skip any + authentication penalties + * If the username+password combination is the same as one of the last 10 login + attempts, skip increasing authentication penalty. + * 'CHECKSUM_VALUE_PTR_COUNT' in 'src/anvil/penalty.c' + * The idea is that if a user has simply configured the password wrong, it + shouldn't keep increasing the delay. + * The username+password is tracked as the CRC32 of them, so there is a + small possibility of hash collisions + +Problems: + + * It is still possible to do multiple auth lookups from the same IP in + parallel. + * For IPv6 it currently blocks the entire /48 block, which may or may not be + what is wanted. + * PENALTY_IPV6_MASK_BITS in auth-penalty.c + +Authentication penalty tracking can be disabled completely with: + +---%<------------------------------------------------------------------------- +service anvil { + unix_listener anvil-auth-penalty { + mode = 0 + } +} +---%<------------------------------------------------------------------------- + +Also you can have similar functionality with fail2ban +[http://wiki2.dovecot.org/HowTo/Fail2Ban]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Policy.txt b/doc/wiki/Authentication.Policy.txt new file mode 100644 index 0000000..b902e48 --- /dev/null +++ b/doc/wiki/Authentication.Policy.txt @@ -0,0 +1,240 @@ +Authentication policy support +============================= + +Dovecot supports (v2.2.25+) external authentication policy server. This server +can be used to decide whether the connecting user is permitted, tarpitted or +outright rejected. While dovecot can do tarpitting and refusal on its own, this +adds support for making cluster-wide decisions to make it easier to deter and +defeat bruteforce attacks. + +Known issues +------------ + +Prior v2.2.27 request policy was mostly broken. + +Prior v2.2.34, the request attributes contained *orig_username* which was not +correct in all cases, especially with master login. + +Prior v2.3.5.2 / 2.2.36.3, invalid UTF-8 would crash auth server if auth policy +is used. + +Configuration +------------- + +The auth-policy server is a core feature and does not require plugin(s) to +work. To activate this feature, you need to configure it. + + * *auth_policy_server_url*: URL of the policy server, url is appended with + ?command=allow/report unless it ends with '&', at which just + command=allow/report is added. + * /Default/: None (*REQUIRED* configuration) + * Example: 'auth_policy_server_url = http://example.com:4001/' + * *auth_policy_server_api_header*: Header and value to add to request (for API + authentication) + * /Default/: None (No authentication is done) + * See https://en.wikipedia.org/wiki/Basic_access_authentication#Client_side + + * Example: 'Authorization: Basic <base64-encoded value>' + * *auth_policy_server_timeout_msecs*: Request timeout in milliseconds + * /Default/: 'auth_policy_server_timeout_msecs = 2000' + * *auth_policy_hash_mech*: Hash mechanism to use for password, you can use any + hash mechanism supported by Dovecot (md4,md5,sha1,sha256,sha512) + * /Default/: 'auth_policy_hash_mech = sha256' + * *auth_policy_hash_nonce*: Cluster-wide nonce to add to hash + * /Default/: None (*REQUIRED* configuration) + * Example: 'auth_policy_hash_nonce = localized_random_string' + * *auth_policy_request_attributes*: Request attributes specification (see + attributes section below) + * /Default/: 'auth_policy_request_attributes = login=%{requested_username} + pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} + protocol=%s' + * *auth_policy_reject_on_fail*: If policy request fails for some reason should + users be rejected + * /Default/: 'auth_policy_reject_on_fail = no' + * *auth_policy_hash_truncate*: How many *bits* to use from password hash. + * /Default/: 'auth_policy_hash_truncate = 12' + * *auth_policy_check_before_auth*: Whether to do policy lookup before + authentication is started + * /Default/: 'auth_policy_check_before_auth = yes' + * *auth_policy_check_after_auth*: Whether to do policy lookup after + authentication is completed + * /Default/: 'auth_policy_check_after_auth = yes' + * *auth_policy_report_after_auth*: Whether to report authentication result + * /Default/: 'auth_policy_report_after_auth = yes' + +Required Minimum Configuration +------------------------------ + +---%<------------------------------------------------------------------------- +auth_policy_server_url = http://example.com:4001/ +auth_policy_hash_nonce = localized_random_string +#auth_policy_server_api_header = Authorization: Basic <base64-encoded value> +#auth_policy_server_timeout_msecs = 2000 +#auth_policy_hash_mech = sha256 +#auth_policy_request_attributes = login=%{requested_username} +pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s +#auth_policy_reject_on_fail = no +#auth_policy_hash_truncate = 12 +#auth_policy_check_before_auth = yes +#auth_policy_check_after_auth = yes +#auth_policy_report_after_auth = yes +---%<------------------------------------------------------------------------- + +Password hash algorithm +----------------------- + +To generate the hash, you concatenate nonce, login name, nil byte, password and +run it through the hash algorithm once. The hash is truncated when truncation +is set to non-zero. The hash is truncated by first choosing bits from MSB to +byte boundary (rounding up), then right-shifting the remainding bits. + +---%<------------------------------------------------------------------------- +hash = H(nonce||user||'\x00'||password) +bytes = round8(bits*8) +hash = HEX(hash[0:bytes] >> (bytes-bits*8)) +---%<------------------------------------------------------------------------- + +Request attributes +------------------ + +Auth policy server requests are JSON requests. The JSON format can be specified +with auth_policy_request_attributes. The syntax is key=value pairs, and key can +contain one or more / to designate that a JSON object should be made. For +example: + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +---%<------------------------------------------------------------------------- + +produces + +---%<------------------------------------------------------------------------- +{"login":"john.doe","pwhash":"1234","remote":"127.0.0.1"} +---%<------------------------------------------------------------------------- + +And + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +attrs/cos=%{userdb:cos} +---%<------------------------------------------------------------------------- + +produces + +---%<------------------------------------------------------------------------- +{"login":"john.doe","pwhash":"1234","remote":"127.0.0.1", +"attrs":{"cos":"premium"}} +---%<------------------------------------------------------------------------- + +Since v2.2.29/v2.3 you can include IMAP ID command result in auth policy +requests, this is achieved with using %{client_id}, which will expand to IMAP +ID command arglist. You must set + +---%<------------------------------------------------------------------------- +imap_id_retain=yes +---%<------------------------------------------------------------------------- + +for this to work. + +Default values for auth_policy_request_attributes +------------------------------------------------- + +Since v2.2.25 + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +---%<------------------------------------------------------------------------- + +Since v2.2.30 + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.2.34 + +---%<------------------------------------------------------------------------- +login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.3.0 (note that 2.2 and 2.3 branches have been developed in parallel +for a while) + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.3.1 + +---%<------------------------------------------------------------------------- +login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.3.2 the request contains 'tls' attribute when TLS has been used. TLS +is also detected if it's offloaded by a load balancer that can provide this +information using HAProxy v2 protocol to dovecot. + +Response +-------- + +---%<------------------------------------------------------------------------- +{"status":-1,"msg":"go away"} +---%<------------------------------------------------------------------------- + +'status' values: see below + +Mode of operation +----------------- + +Auth policy check: Authentication ''before'' userdb/passdb +---------------------------------------------------------- + +First query is done *before* password and user databases are consulted. This +means that any userdb/passdb attributes are left empty. + +The command used here is 'allow' and will appear on the URL as command=allow. + +'status' result values: + + * *-1*: Reject + * *0*: Accept + * *(Any other positive value)*: Tarpit for this number of seconds. + +Auth policy check: Authentication ''after'' successful userdb/passdb lookup +--------------------------------------------------------------------------- + +Second lookup is done *after* authentication succeeds. + +The command used here is 'allow' and will appear on the URL as command=allow. + +'status' result values: + + * *-1*: Authentication fail + * *>= 0*: Authentication succeed + +Auth policy check: Reporting after authentication succeeds +---------------------------------------------------------- + +A report request is sent at end of authentication. + +The command used here is 'report' and will appear on the URL as command=report. + +The JSON request is sent with two additional attributes: + + * *success*: boolean true/false depending on whether the overall + authentication succeeded + * *policy_reject*: boolean true/false whether the failure was due to policy + server + +'status' result value is ignored. + +External Auth Policy Servers +---------------------------- + + * http://oxpedia.org/wiki/index.php?title=AppSuite:Dovecot_Antiabuse_Shield + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.RestrictAccess.txt b/doc/wiki/Authentication.RestrictAccess.txt new file mode 100644 index 0000000..b24be44 --- /dev/null +++ b/doc/wiki/Authentication.RestrictAccess.txt @@ -0,0 +1,85 @@ +Restricting Access +================== + +Restricting IMAP/POP3 access +---------------------------- + +Below examples show how you can give POP3 access to everyone, but IMAP access +only for some people. The exact solution you want depends on what passdb you +use. The solutions can also be modified for other types of IMAP/POP3/SMTP/etc. +access checks. + +PAM +--- + +Set PAM service name to '%s', ie.: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = %s +} +---%<------------------------------------------------------------------------- + +That way PAM uses '/etc/pam.d/imap' for IMAP, and '/etc/pam.d/pop3' for POP3. + +In '/etc/pam.d/imap' you could then use eg. the pam_listfile.so module: + +---%<------------------------------------------------------------------------- +# allow IMAP access only for users in /etc/imapusers file +auth required pam_listfile.so item=user sense=allow +file=/etc/imapusers onerr=fail +---%<------------------------------------------------------------------------- + +SQL +--- + +You can use the '%s' variable which expands to 'imap' or 'pop3' in +'password_query', eg: + +---%<------------------------------------------------------------------------- +password_query = SELECT password FROM users WHERE userid = '%u' and not +(imap_allowed = false and '%s' = 'imap') +---%<------------------------------------------------------------------------- + +LDAP +---- + +Just like with SQL, you can use '%s' in pass_filter, eg.: + +---%<------------------------------------------------------------------------- +pass_filter = (&(objectClass=posixAccount)(uid=%u)(service=%s)) +---%<------------------------------------------------------------------------- + +That would require setting both service=pop3 and service=imap attributes to the +user objects. + +passwd-file +----------- + +You can create a deny passwd-file based on the service: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = /etc/dovecot/deny.%s + deny = yes +} +---%<------------------------------------------------------------------------- + +This makes Dovecot look for '/etc/dovecot/deny.imap' and +'/etc/dovecot/deny.pop3' files. If the user exists in it, the access is denied. +The files don't need to have anything else than one username per line. + +Note that this deny passdb must be before other passdbs. It also means that it +can be used with any other passdb, not just with passwd-file passdbs. + +Restricting IP Access +--------------------- + +It's possible to allow a user to authenticate only from a specific IP or +network. This is especially useful for master users. This can be done by +returning <allow_nets> [PasswordDatabase.ExtraFields.AllowNets.txt] extra field +in passdb. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.txt b/doc/wiki/Authentication.txt new file mode 100644 index 0000000..d72c5d7 --- /dev/null +++ b/doc/wiki/Authentication.txt @@ -0,0 +1,37 @@ +Authentication +============== + +Authentication is split into four parts: + + 1. <Authentication mechanisms> [Authentication.Mechanisms.txt] + 2. <Password schemes> [Authentication.PasswordSchemes.txt] + 3. <Password databases> [PasswordDatabase.txt] + 4. <User databases> [UserDatabase.txt] + +See also <authentication penalty> [Authentication.Penalty.txt] handling for IP +addresses. See also <authentication policy support> [Authentication.Policy.txt] +for making policy based decisions. + +Authentication mechanisms vs. password schemes +---------------------------------------------- + +Authentication mechanisms and password schemes are often confused, because they +have somewhat similar values. For example there is a PLAIN auth mechanism and +PLAIN password scheme. But they mean completely different things. + + * *Authentication mechanism is a client/server protocol*. It's about how the + client and server talk to each others in order to perform the + authentication. Most people use only PLAIN authentication, which basically + means that the user and password are sent without any kind of encryption to + the server. SSL/TLS can then be used to provide the encryption to make PLAIN + authentication secure. + * *Password scheme is about how the password is hashed in your password + database*. If you use a PLAIN scheme, your passwords are stored in cleartext + without any hashing in the password database. A popular password scheme + MD5-CRYPT (also commonly used in '/etc/shadow') where passwords looks like + "$1$oDMXOrCA$plmv4yuMdGhL9xekM.q.I/". + * Plaintext authentication mechanisms work with ALL password schemes. + * Non-plaintext authentication mechanisms require either PLAIN password scheme + or a mechanism-specific password scheme. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/BasicConfiguration.txt b/doc/wiki/BasicConfiguration.txt new file mode 100644 index 0000000..a03590b --- /dev/null +++ b/doc/wiki/BasicConfiguration.txt @@ -0,0 +1,157 @@ +Basic Configuration +=================== + +This page tells you the basics that you'll need to get a working Dovecot +installation. + +Find Dovecot configuration file location using: + +---%<------------------------------------------------------------------------- +doveconf -n | head -n1 +---%<------------------------------------------------------------------------- + +Your configuration file doesn't exist if you installed Dovecot from sources. +The config directory should contain a 'README' file pointing to an example +configuration, which you can use as your basic configuration. For example: + +---%<------------------------------------------------------------------------- +cp -r /usr/share/doc/dovecot/example-config/* /etc/dovecot/ +---%<------------------------------------------------------------------------- + +The default configuration starts from 'dovecot.conf', which contains an +'!include conf.d/*.conf' statement to read the rest of the configuration. This +split of configuration files isn't a requirement to use, and it doesn't really +matter which .conf file you add any particular setting, just as long as it +isn't overridden in another file. You can verify with 'doveconf -n' that +everything looks as you intended. + +Authentication +-------------- + +By default Dovecot is set up to use system user authentication. If you're +planning on using system users, you can simply skip this section and read <PAM> +[PasswordDatabase.PAM.txt] (or <bsdauth> [PasswordDatabase.BSDAuth.txt]) for +configuring it. + +If you're planning on using virtual users, it's easier to first create a simple +passwd-like file to make sure that the authentication will work. Later when you +know Dovecot is working, you can do it differently (see <VirtualUsers.txt>). + +Run as your own non-root user: + +---%<------------------------------------------------------------------------- +echo "$USER:{PLAIN}password:$UID:$GID::$HOME" > users +sudo mv users /etc/dovecot/ + +# If SELinux is enabled: +restorecon -v /etc/dovecot/users +---%<------------------------------------------------------------------------- + +You can (and should) replace the "password" with whatever password you wish to +use, but don't use any important password here as we'll be logging in with +insecure plaintext authentication until <SSL.txt> is configured. + +(Remark: $GID is not set per default on <OpenSuse.txt> systems, replace by 'id +-g') + +If you used the example configuration files, switch to passwd-file by modifying +'conf.d/10-auth.conf': + +---%<------------------------------------------------------------------------- +# Add '#' to comment out the system user login for now: + +# Remove '#' to use passwd-file: +!include auth-passwdfile.conf.ext +---%<------------------------------------------------------------------------- + +In 'conf.d/auth-passwdfile.conf.ext' you should have: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = scheme=CRYPT username_format=%u /etc/dovecot/users +} +userdb { + driver = passwd-file + args = username_format=%u /etc/dovecot/users +} +---%<------------------------------------------------------------------------- + +Verify with 'doveconf -n passdb userdb' that the output looks like above (and +there are no other passdbs or userdbs). + +Plaintext Authentication +------------------------ + +To allow any Authentication without SSL, disable SSL in the +'conf.d/10-ssl.conf' file. This has to be done because Dovecot (now) uses SSL +as default. You probably want to switch this back to "yes" or other options +afterward. + +---%<------------------------------------------------------------------------- +ssl = no +---%<------------------------------------------------------------------------- + +Until SSL is configured, allow plaintext authentication in the +'conf.d/10-auth.conf' file. You probably want to switch this back to "yes" +afterward. + +---%<------------------------------------------------------------------------- +disable_plaintext_auth = no +---%<------------------------------------------------------------------------- + +If you didn't use the temporary passwd-file created above, don't do this if you +don't want your password to be sent in clear to network. Instead get SSL +configuration working and connect to Dovecot only using SSL. + +Mail Location +------------- + +Set the 'mail_location' in 'conf.d/10-mail.conf' as determined by the +instructions in <FindMailLocation.txt>. + +mbox +---- + +If you're using mboxes, it's important to have locking configuration correct. +See <MboxLocking.txt> for more information. + +If you're using '/var/mail/' or '/var/spool/mail/' directory for INBOXes, you +may need to give Dovecot additional permissions so it can create dotlock files +there. A failure to do so will result in errors like these: + +---%<------------------------------------------------------------------------- +open(/var/mail/.temp.host.1234.abcdefg) failed: Permission denied +file_lock_dotlock() failed with mbox file /var/mail/user: Permission denied +---%<------------------------------------------------------------------------- + +From here on I'm assuming the INBOX directory is '/var/mail'. + +First check what the permissions of '/var/mail' are: + +---%<------------------------------------------------------------------------- +# ls -ld /var/mail +drwxrwxrwt 2 root mail 47 2006-01-07 20:44 /var/mail/ +---%<------------------------------------------------------------------------- + +In this case everyone has write access there and the directory is marked +sticky. This allows Dovecot to create the dotlock files, so you don't need to +do anything. + +---%<------------------------------------------------------------------------- +# ls -ld /var/mail +drwxrwxr-- 2 root mail 47 2006-01-07 20:44 /var/mail/ +---%<------------------------------------------------------------------------- + +In this case only the root and the 'mail' group has write permission to the +directory. You'll need to give Dovecot's mail processes ability to use this +group by changing 'conf.d/10-mail.conf': + +---%<------------------------------------------------------------------------- +mail_privileged_group = mail +---%<------------------------------------------------------------------------- + +Note: Specifying the privileged user must be done as shown. Simply adding +'dovecot' user to the 'mail' group does /*not*/ grant write permission. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Chrooting.txt b/doc/wiki/Chrooting.txt new file mode 100644 index 0000000..8284dc3 --- /dev/null +++ b/doc/wiki/Chrooting.txt @@ -0,0 +1,65 @@ +Chrooting +========= + +Traditionally chrooting has been done to run the whole server within a single +chroot. This is also possible with Dovecot, but it requires manually setting up +the chroot and it can be a bit tricky. Dovecot however supports internally +running different parts of it in different chroots: + + * Login processes (imap-login, pop3-login) are chrooted by default into an + empty non-writable directory. + * Authentication process (dovecot-auth) can be chrooted by setting + 'chroot=<path>' inside 'service auth' and/or 'service auth-worker' sections. + This could be a good idea to change if you're not using a passdb or userdb + that needs to access files outside of the chroot. Also make sure not to run + the auth process as root then. + * Mail processes (imap, pop3) can be made to chroot in different ways. See + below. + +Security problems +----------------- + +If chrooting is used incorrectly, it allows local users to gain root +privileges. This is possible by hardlinking setuid binaries inside the chroot +jail and tricking them. There are at least two possibilities: + + 1. Hardlink '/bin/su' inside the chroot and create your own + '<chroot>/etc/passwd'. Then simply run 'su root'. + 2. Create your own '<chroot>/lib/libc.so' and run any setuid binary. + +Of course both of these require that the setuid binary can be run inside the +chroot. This isn't possible by default. Either user would have to find a +security hole from Dovecot, or the administrator would have had to set up +something special that allows running binaries. + +In any case it's a good idea not to allow users to hardlink setuid binaries +inside the chroots. The safest way to do this is to mount the filesystem with +"nosuid" option. + +Mail process chrooting +---------------------- + +Due to the potential security problem described above, Dovecot won't chroot +mail processes to directories which aren't listed in 'valid_chroot_dirs' +setting. For example if your users may be chrooting under '/var/mail/<user>/' +and '/home/<user>/', use: + +---%<------------------------------------------------------------------------- +valid_chroot_dirs = /var/mail:/home +---%<------------------------------------------------------------------------- + +You can chroot all users globally into the same directory by using +'mail_chroot' setting. For example: + +---%<------------------------------------------------------------------------- +mail_chroot = /home +---%<------------------------------------------------------------------------- + +You can also make userdb return a chroot. There are two ways to do that: + + 1. Make userdb return 'chroot=<path>' field. + 2. Insert "/./" inside the returned home directory, eg.: 'home=/home/./user' + to chroot into '/home', or 'home=/home/user/./' to chroot into + '/home/user'. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Clients.NegativeUIDs.txt b/doc/wiki/Clients.NegativeUIDs.txt new file mode 100644 index 0000000..28fc164 --- /dev/null +++ b/doc/wiki/Clients.NegativeUIDs.txt @@ -0,0 +1,53 @@ +Negative UIDs +============= + +---%<------------------------------------------------------------------------- +Invalid messageset: 1181461470:-1181461446. +---%<------------------------------------------------------------------------- + +IMAP uses unsigned 32bit integers for unique message identifiers. Unfortunately +a lot of IMAP clients use 32bit signed integers, which means that if the UIDs +go higher than 2147483647, they'll wrap to negative integers. This causes +errors such as above. + +However normally the UIDs should never go that high, so it's possible to avoid +this problem. + +mbox +---- + +Earlier Dovecot versions had bugs which could cause X-UID: headers in incoming +messages to grow the UIDs too high. Some spam messages especially contained +these intentionally broken X-UID: headers. + +With newer Dovecot versions these broken X-UID: headers aren't practically ever +used. It happens only if the mail has a valid X-IMAPbase: header, X-UID: header +and the mail is written to an empty mbox file. Note that this can happen only +new mboxes, because expunging all messages in a mailbox causes Dovecot to +create a metadata message at the beginning of the mbox file. + +In any case it's still a good idea to filter out X-UID: and other metadata +headers in your MDA. <Dovecot's deliver> [LDA.txt] does this internally. See +<MboxProblems.txt> for a list of headers to filter out. + +Fixing +------ + +Fixing is done by letting Dovecot update UIDVALIDITY value and recreate the +UIDs beginning from one. This means that client's local cache will be +invalidated and the client will be required to download all the messages again. + +mbox +---- + +Delete Dovecot's index files (eg. '.imap/INBOX/') and X-IMAP: and X-IMAPbase: +headers from the mbox file. + +Maildir +------- + +This should really never be a problem with Maildir. If however you have managed +to cause it somehow (by receiving 2 billion mails?), you can recreate the UIDs +by deleting 'dovecot-uidlist' file. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Clients.txt b/doc/wiki/Clients.txt new file mode 100644 index 0000000..f79d377 --- /dev/null +++ b/doc/wiki/Clients.txt @@ -0,0 +1,186 @@ +Client issues and configuration +=============================== + +It seems to be quite difficult to implement a working IMAP client. Best +Practices for Implementing an IMAP Client +[http://www.imapwiki.org/ClientImplementation] tries to help you with it. + +Contents + + + 1. Client issues and configuration + + 1. Apple Mail.app + + 2. Outlook + + 3. Outlook Express 6 + + 4. Netscape Mail + + 5. Evolution + + 6. Mulberry + + 7. Claws-mail + + 8. Thunderbird + + 9. Mutt + + 10. Pine + + 11. SquirrelMail + + 12. Horde IMP + + 13. RoundCube Webmail + + 14. @Mail Webmail + + 15. RainLoop Webmail + +Apple Mail.app +-------------- + +On Mac OS X Leopard 10.5 Mail.app appears to support subscribe/unsubscribe by +right clicking on a mailbox, selecting 'Get Account Info' and selecting +'Subscription List' from tabs. This however doesn't really work with any IMAP +server. + +Apple Mail 3.6 (that comes with OS X 10.5 Leopard) supports +subscribing/unsubscribing to folders in the public namespace. + +Outlook +------- + + * You should enable 'outlook-no-nuls' workaround with POP3. + * Outlook 2003 has problems with older Dovecot's default POP3 UIDL format, + which causes it to download the same mails over and over again if "leave + mails to server" option is enabled. See 'pop3_uidl_format' setting. + * Outlook might not hide or purge deleted items by default. Microsoft has a + how-to that shows how to fix this + [http://office.microsoft.com/en-us/outlook/HP100804201033.aspx] (Outlook + 2007, not Outlook 2003). + * If some Outlook users don't see new or sent mails in the appropriate folders + after a migration from UW IMAPd even if they are visible in other clients + (e.g. Roundcube, Thunderbird, or on the disk itself), and you get the error + message "BAD Error in IMAP command UID: Invalid UID messageset" in the log + or rawlog: It helps to remove the problematic IMAP account completely from + Outlook and recreating it again there. It speaks a different IMAP + afterwards, so there are reasons to believe it caches the details of some + server on the first connect and doesn't refresh them even if you change the + server's hostname in the account settings. + +Outlook Express 6 +----------------- + + * Using "Headers only" synchronization is buggy and can cause "Message is no + longer available on this server" error when opening a mail. This isn't + Dovecot specific problem, and I'm not aware of any possible workarounds at + the moment for this in server side. + * You should enable 'delay-newmail' workarounds for IMAP. + * You should enable 'outlook-no-nuls' and 'oe-ns-eoh' workarounds for POP3. + +Netscape Mail +------------- + +I'm not actually sure what version exactly this refers to. + + * You should enable 'oe-ns-eoh' workaround for POP3. + +Evolution +--------- + + * Some versions don't support creating subfolders with mbox format. Evolution + in Ubuntu Gutsy, 2.12.0-0ubuntu5, does support creating subfolders, at least + when the parent folder is empty. + +Mulberry +-------- + +Seems to be OK. + +Claws-mail +---------- + +Everything works perfectly with Dovecot. + +Thunderbird +----------- + + * If you're using <mbox> [MailboxFormat.mbox.txt], <dbox> + [MailboxFormat.dbox.txt] or <Maildir> [MailLocation.Maildir.txt] with + ':LAYOUT=fs' , + * You should enable 'tb-extra-mailbox-sep' workaround for IMAP. Bug report + [https://bugzilla.mozilla.org/show_bug.cgi?id=29926]. + * If you're using <mbox> [MailboxFormat.mbox.txt]: + * If you are not using a technique to allow folders that contain both + sub-folders and messages (e.g. see <MboxChildFolders.txt>) then you will + have to disable "Server supports folders that contain sub-folders and + messages" setting from Thunderbird.Enhancement request + [https://bugzilla.mozilla.org/show_bug.cgi?id=284933]. + * Versions of Thunderbird from at least 17 (possibly earlier) up to 24.0 + display incorrect new mail counts in the New Mail notification box. This is + due to a bug in Thunderbird's handling of the CONDSTORE extension. See Bug + Report [https://bugzilla.mozilla.org/show_bug.cgi?id=885220] for details and + a client-side workaround. + +Mutt +---- + + * New mutt versions supporting IDLE command will hang with Dovecot versions + earlier than v1.0beta3. Upgrade Dovecot or disable IDLE by setting + imap_idle=no in .muttrc. + * <Using mutt with IMAP> [mutt.txt] + +Pine +---- + +Seems to be OK. + +SquirrelMail +------------ + + * Configuration asks IMAP server name for some workarounds. There has been a + Dovecot option since 1.4.6 and 1.5.1. For olderSquirrelMail versions, select + the "other" option and remove the default INBOX-prefix. + +Horde IMP +--------- + +Dovecot namespace detection works automatically with any recent version of IMP +(4.1+). + +Quota support is now integrated into the 'imap' driver (as of horde-groupware +V1.2), an example config of /imp/config/servers.php is: + +---%<------------------------------------------------------------------------- +$servers['imap'] = array( + 'name' => 'IMAP Server', + 'server' => 'localhost', + 'hordeauth' => false, + 'protocol' => 'imap/notls', + 'port' => 143, + 'quota' => array('driver'=>'imap'), +); +---%<------------------------------------------------------------------------- + +RoundCube Webmail +----------------- + +Works fine. + +@Mail Webmail +------------- + +Uses the namespace returned via Dovecot, full support via IMAP/POP3 using @Mail +[http://atmail.com/]. Can also read mailbox quota via the getquotaroot IMAP +command. + +RainLoop Webmail +---------------- + +Works fine. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/CompilingSource.txt b/doc/wiki/CompilingSource.txt new file mode 100644 index 0000000..49555a3 --- /dev/null +++ b/doc/wiki/CompilingSource.txt @@ -0,0 +1,369 @@ +Contents + + + 1. Compiling Dovecot From Sources + + 2. Compiling Dovecot From Git + + 3. Compiling Dovecot with rpmbuild (Mandriva, RedHat, etc.) + + 4. SSL/TLS Support + + 1. Solaris and OpenSSL problems + + 5. Notify method + + 1. Linux + + 6. Optional Configure Options + + 1. SQL Driver Options + + 2. Authentication Backend Options + + 7. Dynamic IMAP and POP3 Modules + +Compiling Dovecot From Sources +============================== + +For most people it is enough to do: + +---%<------------------------------------------------------------------------- +./configure +make +sudo make install +---%<------------------------------------------------------------------------- + +That installs Dovecot under the '/usr/local' directory. The configuration file +is in '/usr/local/etc/dovecot.conf'. Logging goes to syslog's mail facility by +default, which typically goes to '/var/log/mail.log' or something similar. If +you are in a hurry, you can then jump to <QuickConfiguration.txt>. + + If you have installed some libraries into locations which require special +include or library paths, you can pass them in the 'CPPFLAGS' and 'LDFLAGS' +environment variables. For example: + +---%<------------------------------------------------------------------------- +CPPFLAGS="-I/opt/openssl/include" LDFLAGS="-L/opt/openssl/lib" ./configure +---%<------------------------------------------------------------------------- + +You'll need to create two users for Dovecot's internal use: + + * *dovenull*: Used by untrusted imap-login and pop3-login processes + (default_login_user setting). + * *dovecot*: Used by slightly more trusted Dovecot processes + (default_internal_user setting). + +Both of them should also have their own *dovenull* and *dovecot* groups. See +<UserIds.txt> for more information. + +Compiling Dovecot From Git +========================== + +If you got Dovecot from Git, for instance with + +---%<------------------------------------------------------------------------- +git clone https://github.com/dovecot/core.git dovecot +---%<------------------------------------------------------------------------- + +you will first need to run './autogen.sh' to generate the 'configure' script +and some other files. This requires that you have the following +software/packages installed: + + * 'autoconf' + * 'automake' + * 'libtool' + * 'pkg-config' + * 'gettext' + * 'pandoc' (not strictly required - you can avoid it by using: 'PANDOC=false + ./configure') + * GNU make. + +It is advisable to add '--enable-maintainer-mode' to the 'configure' script. +Thus: + +---%<------------------------------------------------------------------------- +./autogen.sh +./configure --enable-maintainer-mode +make +sudo make install +---%<------------------------------------------------------------------------- + +For later updates, you can use: + +---%<------------------------------------------------------------------------- +git pull +make +sudo make install +---%<------------------------------------------------------------------------- + +Compiling Dovecot with rpmbuild (Mandriva, RedHat, etc.) +======================================================== + +Fetch the source rpm from ftp://ftp.surfnet.nl/ +[ftp://ftp.surfnet.nl/vol/5/mandrakelinux/official/2007.0/SRPMS/contrib/release/] +or any other mirror. At the moment of this writing dovecot-10.rc26.src.rpm can +be found in the cooker subtree. If the current release is newer; updating the +source rpm is not difficult. Unpack the source rpm with 'rpm -ivh +dovecot-10.rc26.src.rpm' to a build environment (/usr/src/rpm...) Copy the +newer tarball from the dovecot site to the SOURCES directory of the build +environment. Change the dovecot.spec file in the SPECS directory to reflect the +new release and the new name of the tarball. The maintainer seems to work with +a bz2 tarball; a tar.gz tarball makes no difference Issue a rpmbuild -ba +dovecot.spec. The resulting rpm will be placed in RPMS/i586 Install with rpm or +urpmi. + +---%<------------------------------------------------------------------------- +rpm -ivh dovecot-1.0.rc26.src.rpm +cd /usr/src/rpm +mv ~/downloads/dovecot-1.0.rc28.tar.gz ./SOURCES +cd SPECS +vi dovecot.spec +...edit release and tarball name. Change default options if needed... +rpmbuild -ba dovecot.spec +cd ../RPMS/i586 +urpmi ./dovecot-1.0.rc28-1mdv2007.0.i586.rpm +---%<------------------------------------------------------------------------- + +During this process missing prerequisites may be detected. Install them and +rerun the build process. The spec file also need updating for the new add-ons +(idxview and logview). + +SSL/TLS Support +=============== + +Dovecot was initially built to support both OpenSSL and GNUTLS. GNUTLS has +however had some problems and nowadays it does not work any more. Patches to +fix it are welcome. + +OpenSSL is used by default now, and it should be automatically detected. If it +is not, you are missing some header files or libraries, or they are just in a +non-standard path. Make sure you have the 'openssl-dev' or a similar package +installed, and if it is not in the standard location, set 'CPPFLAGS' and +'LDFLAGS' as shown in <the first section above.> [CompilingSource.txt] + +By default the SSL certificate is read from '/etc/ssl/certs/dovecot.pem' and +the private key from '/etc/ssl/private/dovecot.pem'. The '/etc/ssl' directory +can be changed using the '--with-ssldir=DIR' configure option. Both can of +course be overridden from the configuration file. + +Solaris and OpenSSL problems +---------------------------- + +Solaris 10 includes a bundled OpenSSL that does not function correctly with +Dovecot when attempting to use SSL/TLS with the default dovecot config. This is +because the default setting of ssl_cipher_list in 'dovecot.conf' is HIGH:!ALL; +due to import restrictions in some countries (now apparently relaxed) the high +level routines are part of the unbundled SUNWcry package and are not available +if you don't have this package installed. This confuses the client as dovecot +announces support for high level crypto and then cannot deliver. In any case, +to resolve this you can alternatively (in decreasing order of simplicity): + + 1. Set 'ssl_cipher_list = MEDIUM:!LOW' in 'dovecot.conf' + 2. Find and install the missing SUNWcry package. + 3. Provide an alternate version of the openssl libraries that doesn't have the + high grade routines removed for your protection (sigh). The bundled version + of OpenSSL cannot be removed. Installing a newer OpenSSL from source or + package (for instance, from http://sunfreeware.com/) will enable Dovecot to + work correctly as long as you link against the new OpenSSL. Assuming you + are building with the built-in ld, make and gcc, then your build should go + something like this (notice the -R required by Sun's linker that sets the + runtime linking path in the resulting programs so the OpenSSL libraries + load from '/usr/local/ssl/lib'): + +---%<------------------------------------------------------------------------- +PATH=$PATH:/usr/sfw/bin:/usr/ccs/bin +export PATH +mv /usr/lib/pkgconfig/openssl.pc /usr/lib/pkgconfig/openssl.pc.orig +CPPFLAGS=-I/usr/local/ssl/include \ + LDFLAGS='-L/usr/local/ssl/lib -R/usr/local/ssl/lib' \ + ./configure --with-ssl=openssl +make +make install +---%<------------------------------------------------------------------------- + +Notify method +============= + +Linux +----- + +Note that current 'inotify' is in the Linux kernel since version 2.6.13 and it +is preferred over 'dnotify'. If your distribution does not have the required +'inotify' header file, you can get it from the inotify maintainer (this example +requires cURL [http://curl.haxx.se/]): + +---%<------------------------------------------------------------------------- +mkdir -p /usr/local/include/sys +cd /usr/local/include/sys +curl ftp://ftp.kernel.org/pub/linux/kernel/people/rml/inotify/headers/inotify.h +-O +curl +ftp://ftp.kernel.org/pub/linux/kernel/people/rml/inotify/headers/inotify-syscalls.h +>> inotify.h +---%<------------------------------------------------------------------------- + +/usr/local/include isn't in standard include lookup path, so you'll need to +specify that to configure: + +---%<------------------------------------------------------------------------- +CPPFLAGS=-I/usr/local/include ./configure --with-notify=inotify +---%<------------------------------------------------------------------------- + +Debian Etch ships 'sys/inotify.h' wrapped in the 'inotify-tools' package and +installs the header file into '/usr/include/inotifytools/'. To use the header +file use: + +---%<------------------------------------------------------------------------- +if ! test -e /usr/include/sys/inotify.h; then + aptitude install inotify-tools + ln -sf /usr/include/inotifytools/inotify.h /usr/include/sys/inotify.h +fi +---%<------------------------------------------------------------------------- + +Then pass 'CPPFLAGS' as in the example above: + +---%<------------------------------------------------------------------------- +CPPFLAGS=-I/usr/include/inotifytools ./configure --with-notify=inotify +---%<------------------------------------------------------------------------- + +Optional Configure Options +========================== + +--help: + gives a full list of available options + +--help=short: + just lists the options added by the particular package (= Dovecot) + +Options are usually listed as '--with-something' or '--enable-something'. If +you want to disable them, do it as '--without-something' or +'--disable-something'. There are many default options that come from autoconf, +automake or libtool. They are explained elsewhere. + +Here is a list of options that Dovecot adds. You should not usually have to +change these, but they are described here just for completeness: + +--enable-devel-checks: + Enables some extra sanity checks. This is mainly useful for developers. It + does quite a lot of unnecessary work but should catch some programming + mistakes more quickly. + +--enable-asserts: + Enable assertion checks, enabled by default. Disabling them may slightly save + some CPU, but if there are bugs they can cause more problems since they are + not detected as early. + +--without-shared-libs: + Link Dovecot binaries with static libraries instead of dynamic libraries. + +--disable-largefile: + Specifies if we use 32bit or 64bit file offsets in 32bit CPUs. 64bit is the + default if the system supports it (Linux and Solaris do). Dropping this to + 32bit may save some memory, but it prevents accessing any file larger than 2 + GB. + +--with-mem-align=BYTES: + Specifies memory alignment used for memory allocations. It is needed with + many non-x86 systems and it should speed up x86 systems too. Default is 8, to + make sure 64bit memory accessing works. + +--with-ioloop=IOLOOP: + Specifies what I/O loop method to use. Possibilities are 'select', 'poll', + 'epoll' and 'kqueue'. The default is to use the best method available on your + system. + +--with-notify=NOTIFY: + Specifies what file system notification method to use. Possibilities are + 'dnotify', 'inotify' (both on Linux), 'kqueue' (FreeBSD) and 'none'. The + default is to use the best method available on your system. See <Notify + method> [CompilingSource.txt] above for more information. + +--with-storages=FORMATS: + Specifies what mailbox formats to support. Note: Independent of this option, + the formats /raw/ and /shared/ will be always built. + +--with-solr: + Build with Solr full text search support + +--with-zlib: + Build with zlib compression support (default if detected) + +--with-bzlib: + Build with bzip2 compression support (default if detected) + +SQL Driver Options +------------------ + +SQL drivers are typically used only for authentication, but they may be used as +a lib-dict backend too, which can be used by plugins for different purposes. + +--with-sql-drivers: + Build with specified SQL drivers. Defaults to all that were found with + autodetection. + +--with-pgsql: + Build with PostgreSQL support (requires pgsql-devel, libpq-dev or similar + package) + +--with-mysql: + Build with MySQL support (requires mysql-devel, libmysqlclient15-dev or + similar package) + +--with-sqlite: + Build with SQLite3 driver support (requires sqlite-devel, libsqlite3-dev or + similar package) + +Authentication Backend Options +------------------------------ + +The basic backends are built if the system is detected to support them: + +--with-shadow: + Build with <shadow> [PasswordDatabase.Shadow.txt] password support + +--with-pam: + Build with <PAM> [PasswordDatabase.PAM.txt] support + +--with-nss: + Build with <NSS> [UserDatabase.NSS.txt] support + +--with-sia: + Build with Tru64 SIA support + +--with-bsdauth: + Build with <BSD authentication> [PasswordDatabase.BSDAuth.txt] support (if + supported by your OS) + +Some backends require extra libraries and are not necessarily wanted, so they +are built only if specifically enabled: + +--with-sql: + Build with generic SQL support (drivers are enabled separately) + +--with-ldap: + Build with LDAP support (requires openldap-devel, libldap2-dev or similar + package) + +--with-gssapi: + Build with GSSAPI authentication support (requires krb5-devel, libkrb5-dev or + similar package) + +--with-vpopmail: + Build with vpopmail support (requires vpopmail sources or a devel package) + +It's also possible to build these as plugins by giving e.g. --with-sql=plugin. + +Dynamic IMAP and POP3 Modules +============================= + +The 'mail_plugins' setting lists all plugins that Dovecot is supposed to load +from the 'mail_plugin_dir' directory at program start. These plugins can do +anything they want. They are only expected to contain the '<plugin name>_init' +and '<plugin name>_deinit' functions which are called at startup and at exit. + +The plugin filename is prefixed with a number which specifies the order in +which the plugins are loaded. This is important if one plugin depends on +another. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/ConfigFile.txt b/doc/wiki/ConfigFile.txt new file mode 100644 index 0000000..0f7ab52 --- /dev/null +++ b/doc/wiki/ConfigFile.txt @@ -0,0 +1,312 @@ +Dovecot Configuration File +========================== + +Contents + + + 1. Dovecot Configuration File + + 1. Basic syntax + + 2. Sections + + 3. Filters + + 4. Including config files + + 5. External config files + + 6. Long lines + + 7. Reading value from file + + 8. Variable expansion + +Basic syntax +------------ + +The syntax generally looks like this: + +---%<------------------------------------------------------------------------- +# this is a comment + +settings_key = settings_value +---%<------------------------------------------------------------------------- + +If Dovecot doesn't seem to be reading your configuration correctly, use +'doveconf -n' to check how Dovecot actually parses it. You can also check more +complex configurations by providing filters, for example:'doveconf -n -f +service=imap -f local=10.0.0.1 -f remote=1.2.3.4' + +Sections +-------- + +Sections look like this: + +---%<------------------------------------------------------------------------- +section optional_name { + section_setting_key = section_setting_value + subsection optional_subname { + subkey = subvalue + } +} +---%<------------------------------------------------------------------------- + +Note that sections must be currently written with the linefeeds as shown above. +So for example this doesn't work: + +---%<------------------------------------------------------------------------- +section optional_name { key = value } # DOES NOT WORK +---%<------------------------------------------------------------------------- + +The sections can be optionally named. This is especially useful if you want to +update the same section later on in the config. For example: + +---%<------------------------------------------------------------------------- +namespace inbox { + inbox = yes +} +# ... +# possibly included from another file: +namespace inbox { + mailbox Trash { + special_use = \Trash + } +} +# The namespaces get merged into the same inbox namespace. +---%<------------------------------------------------------------------------- + +Without naming the namespace it would have created a new namespace. The section +name may also sometimes be used as part of the settings instead of simply a +name. For example: + +---%<------------------------------------------------------------------------- +service auth { + unix_listener auth-master { + # .. + } +} +---%<------------------------------------------------------------------------- + +Above the "auth-master" both uniquely identifies the section name, but also it +names the UNIX socket path. + +Filters +------- + +There are a few different filters that can be used to apply settings +conditionally. The filters look exactly like sections, which may be a bit +confusing. The currently supported filters are: + + * protocol <name>: Name of the service/protocol that is reading the settings. + For example: imap, pop3, doveadm, lmtp, lda + * remote <ip/network>: Remote client's IP/network. For non-TCP connections + this will never match. For example 10.0.0.1 or 10.0.0.0/16. + * local_name <name>: Matches TLS connection's SNI + [https://en.wikipedia.org/wiki/Server_Name_Indication] name, if it's sent by + the client. Commonly used to <configure multiple TLS certificates> + [SSL.DovecotConfiguration.txt]. + * local <ip/range>: Locally connected IP/network. For non-TCP connections this + will never match. For example 127.0.0.1 or 10.0.0.0/16. + +These filters work for most of the settings, but most importantly auth settings +currently only support the protocol filter. Some of the other settings are also +global and can't be filtered, such as log_path. + +An example, which uses all of the filters: + +---%<------------------------------------------------------------------------- +local 127.0.0.1 { + local_name imap.example.com { + remote 10.0.0.0/24 { + protocol imap { + # ... + } + } + } +} +---%<------------------------------------------------------------------------- + +The nesting of the filters must be exactly in that order or the config parsing +will fail. + +When applying the settings, the settings within the most-specific filters +override the less-specific filter's settings, so the order of the filters in +config file doesn't matter. For example: + +---%<------------------------------------------------------------------------- +local 127.0.0.2 { + key = 127.0.0.2 +} +local 127.0.0.0/24 { + key = 127.0.0.0/24 +} +local 127.0.0.1 { + key = 127.0.0.1 +} +# The order of the above blocks doesn't matter: +# If local IP=127.0.0.1, key=127.0.0.1 +# If local IP=127.0.0.2, key=127.0.0.2 +# If local IP=127.0.0.3, key=127.0.0.0/24 +---%<------------------------------------------------------------------------- + +Similarly remote local filters override remote filters, which override +local_name filters, which override protocol filters. In some situations Dovecot +may also return an error if it detects that the same setting is being +ambiguously set by multiple matching filters. + +Including config files +---------------------- + +The main dovecot.conf file can also include other config files: + +---%<------------------------------------------------------------------------- +!include local.conf +!include /path/to/another.conf +!include conf.d/*.conf +---%<------------------------------------------------------------------------- + +The paths are relative to the currently parsed config file's directory. For +example: + +---%<------------------------------------------------------------------------- +# /etc/dovecot/dovecot.conf: +!include conf.d/imap.conf +# /etc/dovecot/conf.d/imap.conf: +!include imap2.conf +# /etc/dovecot/conf.d/imap2.conf is being included +---%<------------------------------------------------------------------------- + +If any of the includes fail (e.g. file doesn't exist or permission denied), it +results in an error. It's not an error if wildcards don't result in any +matching files. To avoid these errors, you can use !include_try instead: + +---%<------------------------------------------------------------------------- +!include_try passwords.conf +---%<------------------------------------------------------------------------- + +Including a file preserves the context where it's included from. For example: + +---%<------------------------------------------------------------------------- +protocol imap { + plugin { +!include imap-plugin-settings.conf + } +} +---%<------------------------------------------------------------------------- + +External config files +--------------------- + +Due to historical reasons there are still some config files that are external +to the main dovecot.conf, which are typically named '*.conf.ext'. For example: + + * passdb/userdb { args } for ldap/sql points to a dovecot-ldap.conf.ext and + dovecot-sql.conf.ext. + * dict { .. } points to dovecot-dict-*.conf.ext + +Although these external config files look similar to the main dovecot.conf +file, they have quite a lot of differences in details. Their parsing is done +with a completely different config parser, so things like filters, $variables, +!includes and<files don't work. + +The external config files are also not loaded by the config process at startup, +but instead they're parsed whenever the value is being used. So the external +passdb/userdb files are loaded by auth process at startup, while the dict +config is loaded by dict process at startup. + +Eventually these external config files will hopefully be removed. + +Long lines +---------- + +It's possible to split the setting values into multiple lines. Unfortunately +this was broken for a long time, so outside *.conf.ext files this works only in +v2.2.22+: + +---%<------------------------------------------------------------------------- +# This works in *.conf.ext files, but in the main dovecot.conf only with +v2.2.22+: +setting_key = \ + long \ + value +# equivalent to: "long value" +---%<------------------------------------------------------------------------- + +All the whitespace between lines is converted to a single space regardless of +how many spaces or tabs are at the beginning of the line or before the '\'. +Even if there is zero whitespace a single space is added. + +Reading value from file +----------------------- + +It's possible to read the value for a setting from a file: + +---%<------------------------------------------------------------------------- +key = </path/to/file +---%<------------------------------------------------------------------------- + +The value is read exactly as the entire contents of the file. This includes all +the whitespace and newlines. The paths are relative to the currently parsed +config file's directory, similar to how !include works. The file is read +immediately whenever parsing the configuration file, so if it changes +afterwards it requires a configuration reload to see the changes. This +functionality is especially useful for reading SSL certificates and keys. + +Variable expansion +------------------ + +It's possible to refer to other earlier settings as $name. For example: + +---%<------------------------------------------------------------------------- +key = value1 +key2 = $key value2 +# Equivalent to key2 = value1 value2 +---%<------------------------------------------------------------------------- + +This is commonly used with mail_plugins settings to easily add more plugins +e.g. inside imap protocol: + +---%<------------------------------------------------------------------------- +mail_plugins = acl quota +protocol imap { + mail_plugins = $mail_plugins imap_acl imap_quota +} +---%<------------------------------------------------------------------------- + +However, you must be careful with the ordering of these in the configuration +file, because the $variables are expanded immediately while parsing the config +file and they're not updated later. For example this is a common problem: + +---%<------------------------------------------------------------------------- +# NON-WORKING EXAMPLE +# Enable ACL plugin: +mail_plugins = $mail_plugins acl +protocol imap { + mail_plugins = $mail_plugins imap_acl +} +# Enable quota plugin: +mail_plugins = $mail_plugins quota +protocol imap { + mail_plugins = $mail_plugins imap_quota +} +# The end result is: +# mail_plugins = " acl quota" - OK +# protocol imap { +# mail_plugins = " acl imap_acl imap_quota" - NOT OK +# } +# v2.2.24+ also gives a warning about this: +# doveconf: Warning: /etc/dovecot/dovecot.conf line 8: Global setting +mail_plugins won't change the setting inside an earlier filter at +/etc/dovecot/dovecot.conf line 5 (if this is intentional, avoid this warning by +moving the global setting before /etc/dovecot/dovecot.conf line 5) +---%<------------------------------------------------------------------------- + +This is because the second mail_plugins change that added "quota" globally +didn't update anything inside the existing 'protocol { .. }' or other filters. + +Some variables exist in the plugin section only, such as sieve_extensions. +Those variables *cannot* be referred to, that is '$sieve_extensions' won't +work. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Debugging.Authentication.txt b/doc/wiki/Debugging.Authentication.txt new file mode 100644 index 0000000..6189f4d --- /dev/null +++ b/doc/wiki/Debugging.Authentication.txt @@ -0,0 +1,91 @@ +Debugging Authentication +======================== + +The most important thing to do is to set 'auth_debug=yes', and preferrably also +'auth_debug_passwords=yes'. After that you'll see in the logs exactly what +dovecot-auth is doing, and that should help you to fix the problem. + +PLAIN SASL mechanism +-------------------- + +With IMAP and POP3 it's easy to log in manually using the IMAP's LOGIN command +or POP3's USER and PASS commands (see <TestInstallation.txt> and +<TestPop3Installation.txt> for details), but with SMTP AUTH you'll need to use +PLAIN authentication mechanism, which requires you to build a base64-encoded +string in the correct format. The PLAIN authentication is also used internally +by both IMAP and POP3 to authenticate to dovecot-auth, so you see it in the +debug logs. + +The PLAIN mechanism's authentication format is: <authorization ID> NUL +<authentication ID> NUL <password>. Authorization ID is the username who you +want to log in as, and authentication ID is the username whose password you're +giving. If you're not planning on doing a <master user login> +[Authentication.MasterUsers.txt], you can either set both of these fields to +the same username, or leave the authorization ID empty. + +Encoding with mmencode +---------------------- + +printf(1) and mmencode(1) should be available on most Unix or GNU/Linux +systems. (If not, check with your distribution. GNU coreutils includes +printf(1), and metamail includes mmencode(1). In Debian, mmencode is called +mimencode(1).) + +---%<------------------------------------------------------------------------- +$ printf 'username\0username\0password' | mmencode +dXNlcm5hbWUAdXNlcm5hbWUAcGFzc3dvcmQ= +---%<------------------------------------------------------------------------- + +This string is what a client would use to attempt PLAIN authentication as user +"username" with password "password." With ''auth_debug_passwords=yes', it would +appear in your logs. + +Decoding with mmencode +---------------------- + +You can use mmencode -u to interpret the encoded string pasted into stdin as +follows: + +---%<------------------------------------------------------------------------- +# mmencode -u +bXl1c2VybmFtZUBkb21haW4udGxkAG15dXNlcm5hbWVAZG9tYWluLnRsZABteXBhc3N3b3Jk<CR> +myusername@domain.tldmyusername@domain.tldmypassword<CTRL-D> +# +---%<------------------------------------------------------------------------- + +You should see the correct user address (twice) and password. The null bytes +won't display. + +Encoding with Perl +------------------ + +Unfortunately, mmencode on FreeBSD chokes on "\0". As an alternate, if you +have MIME::Base64 on your system, you can use a perl statement to do the same +thing: + +---%<------------------------------------------------------------------------- +perl -MMIME::Base64 -e 'print +encode_base64("myusername\@domain.tld\0myusername\@domain.tld\0mypassword");' +---%<------------------------------------------------------------------------- + +As mmencode -u doesn't encounter any "\0" you can still do: + +---%<------------------------------------------------------------------------- +perl -MMIME::Base64 -e 'print +encode_base64("myusername\@domain.tld\0myusername\@domain.tld\0mypassword");' | +mmencode -u +---%<------------------------------------------------------------------------- + +to check that you have encoded correctly. + +Encoding with Python +-------------------- + +With python you can do: + +---%<------------------------------------------------------------------------- +python -c "import base64; +print(base64.encodestring('myusername@domain.tld\0myusername@domain.tld\0mypassword'));" +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Debugging.ProcessTracing.txt b/doc/wiki/Debugging.ProcessTracing.txt new file mode 100644 index 0000000..89ea41a --- /dev/null +++ b/doc/wiki/Debugging.ProcessTracing.txt @@ -0,0 +1,44 @@ +Process Tracing +=============== + +If a Dovecot's process hangs or is just really slow, the best way to debug it +is to see what it's really doing. Typically you'd be looking into imap or pop3 +processes. + +Linux +----- + +---%<------------------------------------------------------------------------- +strace -tt -o log -p <process pid> +---%<------------------------------------------------------------------------- + +BSDs, OS X <= 10.4 +------------------ + +---%<------------------------------------------------------------------------- +# enable process tracing +ktrace -f log -p <process pid> +# do whatever makes it break, then stop the process tracing: +ktrace -C +# and see what it's done: +kdump -T -f log +---%<------------------------------------------------------------------------- + +OS X >= 10.5 +------------ + +---%<------------------------------------------------------------------------- +dtruss -p <process id> +---%<------------------------------------------------------------------------- + +Solaris +------- + +---%<------------------------------------------------------------------------- +truss -d -r0 -w1 -o log -p <process pid> +---%<------------------------------------------------------------------------- + +'-r0' and '-w1' cause all IMAP input/output to be logged. '-d' adds timestamps +to the log. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Debugging.Rawlog.txt b/doc/wiki/Debugging.Rawlog.txt new file mode 100644 index 0000000..fc1ffdc --- /dev/null +++ b/doc/wiki/Debugging.Rawlog.txt @@ -0,0 +1,148 @@ +Rawlog +====== + +Dovecot supports logging IMAP/POP3/LMTP/SMTP(submission) traffic (also TLS/SSL +encrypted). There are several possibilities for this: + + 1. rawlog_dir setting (v2.2.26+) + 2. Using 'rawlog' binary, which is executed as post-login script. + 3. Pre-login imap/pop3-login process via -R parameter. + 4. For <lmtp> [LMTP.txt], you need to use lmtp_rawlog_dir and + lmtp_proxy_rawlog_dir settings (since v2.3.2) + 5. For <submission> [Submission.txt], you can use rawlog_dir setting and + submission_relay_rawlog_dir (since v2.3.2) + +rawlog_dir setting (v2.2.26+) +----------------------------- + +Dovecot creates *.in and *.out rawlogs to the specified directory if it exists. +For example: + +---%<------------------------------------------------------------------------- +protocol imap { + rawlog_dir = /tmp/rawlog/%u + # if you want to put files into user's homedir, use this, do not use ~ + #rawlog_dir = %h/rawlog +} +---%<------------------------------------------------------------------------- + +lmtp_rawlog_dir (v2.3.2+) +------------------------- + +You can use lmtp_rawlog_dir to generate rawlogs on lmtp backend server. Unlike +the rawlog_dir setting, this does not accept variables. + +lmtp_proxy_rawlog_dir (v2.3.2+) +------------------------------- + +You can use lmtp_proxy_rawlog_dir to generate rawlogs on lmtp proxy server. +Unlike the rawlog_dir setting, this does not accept variables. + +submission_relay_rawlog_dir (v2.3.2+) +------------------------------------- + +You can use submission_relay_rawlog_dir to generate relay rawlogs on the +dovecot submission server. + +rawlog binary +------------- + +It works by checking if 'dovecot.rawlog/' directory exists in the logged in +user's home directory, and writing the traffic to 'yyyymmdd-HHMMSS-pid.in' and +'.out' files. Each connection gets their own in/out files. Rawlog will simply +skip users who don't have the 'dovecot.rawlog/' directory and the performance +impact for those users is minimal. + +Home directory +-------------- + +Note that for rawlog to work, your <userdb> [UserDatabase.txt] must have +returned a home directory for the user.*IMPORTANT: The home directory must be +returned by userdb, mail_home setting won't work.* Verify that 'doveadm user -u +user@example.com' (with -u parameter) returns the home directory, for example: + +---%<------------------------------------------------------------------------- +% doveadm user -u user@example.com +userdb: user@example.com + user : user@example.com + uid : 1000 + gid : 1000 + home : /home/user@example.com +---%<------------------------------------------------------------------------- + +In above configuration rawlog would expect to find +'/home/user@example.com/dovecot.rawlog/' directory writable by uid 1000. + +If your userdb can't return a home directory directly, with v2.1+ you can add: + +---%<------------------------------------------------------------------------- +userdb { + # ... + default_fields = home=/home/%u + # or temporarily even e.g. default_fields = home=/tmp/temp-home +} +---%<------------------------------------------------------------------------- + +You can also set DEBUG environment to have rawlog log an info message why it's +not doing anything: + +---%<------------------------------------------------------------------------- +import_environment = $import_environment DEBUG=1 +---%<------------------------------------------------------------------------- + +Configuration +------------- + +To enable rawlog, you must use rawlog as a <post-login script> +[PostLoginScripting.txt]: + +---%<------------------------------------------------------------------------- +service imap { + executable = imap postlogin +} +service pop3 { + executable = pop3 postlogin +} + +service postlogin { + executable = script-login -d rawlog + unix_listener postlogin { + } +} +---%<------------------------------------------------------------------------- + +You can also give parameters to rawlog: + + * -b: Write IP packet boundaries (or whatever read() sees anyway) to the log + files. The packet is written between<<< and >>>. + * -t: Log a microsecond resolution timestamp at the beginning of each line. + * -I: Include IP address in the filename (v2.2.16+) + * v2.1 and newer: + * -f in: Log only to *.in files + * -f out: Log only to *.out files + * v2.0 and older: + * -i: Log only to *.in files + * -o: Log only to *.out files + +Pre-login rawlog (v2.1+) +------------------------ + +You can enable pre-login rawlog for all users by telling the login processes to +log to a rawlog directory, for example: + +---%<------------------------------------------------------------------------- +service imap-login { + executable = imap-login -R rawlogs +} +---%<------------------------------------------------------------------------- + +This tries to write the rawlogs under $base_dir/login/rawlogs directory. You +need to create it first with enough write permissions, e.g.: + +---%<------------------------------------------------------------------------- +mkdir /var/run/dovecot/login/rawlogs +chown dovenull /var/run/dovecot/login/rawlogs +chmod 0700 /var/run/dovecot/login/rawlogs +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Debugging.Thunderbird.txt b/doc/wiki/Debugging.Thunderbird.txt new file mode 100644 index 0000000..db82307 --- /dev/null +++ b/doc/wiki/Debugging.Thunderbird.txt @@ -0,0 +1,58 @@ +Debugging using Thunderbird's logging +------------------------------------- + +Thunderbird has the ability to log its client actions based on protocol; this +can be useful when you're experiencing a problem with Dovecot and want to trap +the commands that the client is sending. + +Windows batch file +------------------ + +Save the below as 'runtbird.bat' and place it on the desktop, then run this +instead of the Thunderbird icon. + +---%<------------------------------------------------------------------------- +set mydate=%date:~-4,4%%date:~-7,2%%date:~-10,2% +set mytime=%time:~0,2%%time:~+3,2% + +set NSPR_LOG_MODULES=IMAP:5 +set NSPR_LOG_FILE=%USERPROFILE%\thunderbird_%mydate%_%mytime%.log + +start /d "c:\program files\mozilla thunderbird" thunderbird.exe +---%<------------------------------------------------------------------------- + +Adjust the log location and Thunderbird install folder as appropriate. If you +want to log all modules instead of just IMAP (SMTP, e.g.) then replace "IMAP" +above with "all". The above will create a date/time stamped logfile for each +run, so you won't lose the previous logs. + +Linux/BSD/etc. shell script +--------------------------- + +Save the below as 'runtbird.sh', chmod it 0755, then run this instead of the +Thunderbird icon. + +---%<------------------------------------------------------------------------- +#!/bin/sh + +TB_PATH=`which thunderbird` +# or for MacOSX: +#TB_PATH="/Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin" + +MYDATE=`date "+%Y%m%d_%H%M%S"` +NSPR_LOG_MODULES=IMAP:5 +NSPR_LOG_FILE=/tmp/thunderbird_${MYDATE}.log +export NSPR_LOG_MODULES NSPR_LOG_FILE + +$TB_PATH & +exit $? +---%<------------------------------------------------------------------------- + +Adjust the log location and Thunderbird launch binary as appropriate. If you +want to log all modules instead of just IMAP (SMTP, e.g.) then replace "IMAP" +above with "all". You can also get a timestamp at the begining of all the log +lines, if you add ",timestamp" after "NSPR_LOG_MODULES=IMAP:5" for example. The +above will create a date/time stamped logfile for each run, so you won't lose +the previous logs. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Arrays.txt b/doc/wiki/Design.Arrays.txt new file mode 100644 index 0000000..2ecb787 --- /dev/null +++ b/doc/wiki/Design.Arrays.txt @@ -0,0 +1,109 @@ +Dynamic Arrays +============== + +'lib/array.h' and 'lib/array-decl.h' describes Dovecot's type-safe dynamic +arrays. Trying to add wrong typed elements gives a compiler warning. + +Declaring +--------- + +Arrays can be declared in two ways: + + 1. Directly: 'ARRAY_DEFINE(array_name, array_type);'. For example: + 'ARRAY_DEFINE(numbers, int);' or 'ARRAY_DEFINE(foos, struct foo);' + 2. Via predefined type: 'ARRAY_DEFINE_TYPE(foo, struct foo); ... + ARRAY_TYPE(foo) foos;' + +The main reason to define a type for an array is to be able to pass the array +as a function parameter, like: + +---%<------------------------------------------------------------------------- +void func(ARRAY_TYPE(foo) *foos) { .. } +---%<------------------------------------------------------------------------- + +Trying to do the same with 'ARRAY_DEFINE()' will generate a compiler warning. +'lib/array-decl.h' defines several commonly used types. + +Initializing +------------ + +Arrays are typically initialized by calling 'i_array_init()', 'p_array_init()' +or 't_array_init()' depending on where you want to allocate the memory from. +Arrays are internally handled as <buffers> [Design.Buffers.txt], so the initial +size is just multiplied by element size and passed to +'buffer_create_dynamic()'. + +Example: + +---%<------------------------------------------------------------------------- +ARRAY_DEFINE(foo, struct foo *); + +i_array_init(&foo, 32); /* initialize array with 32 elements until it needs to +be grown */ +---%<------------------------------------------------------------------------- + +Arrays can be freed with 'array_free()', but this isn't necessary if the memory +gets freed by other means (i.e. it was allocated from alloconly-pool or data +stack). + +Writing +------- + + * 'array_append(array, data, count)' is the most common way to add data to + arrays + * 'array_append_array(dest, src)' + * 'array_insert(array, idx, data, count)' + * 'array_delete(array, idx, count)' + * 'array_idx_set(array, idx, data)' replaces (or adds) data to given index + * 'array_idx_clear(array, idx)' clears given index by writing NULs to it + * 'array_append_space(array, count)' + +Reading +------- + +'array_idx(array, idx)' returns pointer to given index in array. The index must +already exist, otherwise the call assert-crashes. This call adds extra overhead +for accessing arrays though, so usually it's better to just get list of all +elements and access them directly: + +---%<------------------------------------------------------------------------- +data = array_get(&array, &count); +---%<------------------------------------------------------------------------- + +You can also iterate through the whole array easily: + +---%<------------------------------------------------------------------------- +const char *str; + +array_foreach(&string_array, str) { + /* str changes in each iteration */ +} +---%<------------------------------------------------------------------------- + +There's also 'array_foreach_modifiable()' to get the data without const. + +Unsafe Read/Write +----------------- + +Functions below have similar problems to [[Design/Buffer|buffer]'s '*_unsafe()' +functions. Memory returned by them must not be accessed after calls to other +'array_*()' modifying functions, because they may reallocate the array +elsewhere in memory. + + * 'array_append_space(array)' + * 'array_insert_space(array, idx)' + * 'array_get_modifiable(array, &count)' + * 'array_idx_modifiable(array, idx)' + +Others +------ + + * 'array_cmp(array1, array2)' compares two arrays + * 'array_reverse(array)' reverses all elements in an array + * 'array_sort(array, cmp_func)' is a wrapper for 'qsort()' adding also type + safety. The parameters in cmp_func should be the same type as the array, + instead of 'const void *'. + * 'array_bsearch(array, key, cmp_func)' is a wrapper for 'bsearch()' also + adding type safety, just like 'array_sort()'. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.AuthProcess.txt b/doc/wiki/Design.AuthProcess.txt new file mode 100644 index 0000000..dd4fc89 --- /dev/null +++ b/doc/wiki/Design.AuthProcess.txt @@ -0,0 +1,365 @@ +Authentication process design +============================= + +See <Design.Processes.txt> for an overview of how the authentication process +works. + +There are four major classes in the code: + + * 'struct mech_module': Authentication mechanism + * 'struct password_scheme': Password scheme + * 'struct passdb_module': Password database + * 'struct userdb_module': User database + +There are many implementations for each of these, and it's simple to add more +of them. They can also be added as plugins, although the current plugin loading +code doesn't allow loading authentication mechanisms cleanly, and it's not +possible to add new credentials (see below). + +The code flow usually goes like: + + * Dovecot-auth listens for new authentication client connections (the listener + socket is created by master process and passed in MASTER_SOCKET_FD -> + 'main.c:main_init()' -> + 'auth-master-connection.c:auth_master_listener_add()') + * A new authentication client connects via UNIX socket + ('auth-master-connection.c:auth_master_listener_accept()' -> + 'auth-client-connection.c:auth_client_connection_create()') + * Authentication client begins an authentication + ('auth-client-connection.c:auth_client_input()' -> + 'auth_client_handle_line()' -> + 'auth-request-handler.c:auth_request_handler_auth_begin()' [ -> + 'auth-request.c:auth_request_new()']) + * Authentication mechanism backend handles it ('mech->auth_initial()' and + 'mech->auth_continue()' in 'mech-*.c') + * The mechanism looks up the password from passdb + ('auth-request.c:auth_request_verify_plain()' and + 'auth_request_lookup_credentials()') and the password scheme code to + verifies it ('password-scheme.c:password_verify()' and + 'password_generate()') + * If user is logging in, the user information is looked up from the userdb + ('auth-master-connection.c:master_input()' -> 'master_input_request()' -> + 'auth-request-handler.c:auth_request_handler_master_request()' -> + 'auth-request.c:auth_request_lookup_user()') + * The authentication may begin new authentication requests even before the + existing ones are finished. + +It's also possible to request a userdb lookup directly, for example Dovecot's +<deliver> [LDA.txt] needs that. The code path for that goes +'auth-master-connection.c:master_input()' -> 'master_input_user()' -> +'auth-request.c:auth_request_lookup_user()'. + +Authentication mechanisms +------------------------- + +These are <SASL> [Sasl.txt] authentication mechanism implementations. See +<Authentication.Mechanisms.txt> for a list of mechanisms supported by Dovecot. + +A new mechanism is created by filling a 'struct mech_module' (in 'mech.h') and +passing it to 'mech_register_module()'. The struct fields are: + +mech_name: + The public name of the mechanism. This is shown to clients in the IMAP, POP3 + and SMTP capability lists. If you create a new non-standard mechanism, please + prefix it with "X-". + +flags: + Describes how secure the mechanism is. Also 'MECH_SEC_PRIVATE' flag specifies + that the mechanism shouldn't be advertised in the capability list. This is + currently used only for APOP mechanism, which is defined by the POP3 protocol + itself. + +passdb_need_plain: + This mechanism uses passdb's 'verify_plain()' function to verify the + password's validity. This means that the mechanism has access to the + plaintext password. This is true only for plaintext mechanisms such as PLAIN + and LOGIN. The main purpose of this flag is to make dovecot-auth complain at + startup if there are no passdbs defined in the configuration file. Note that + a configuration without any passdbs is valid with eg. GSSAPI mechanism which + doesn't need a passdb at all. + +passdb_need_credentials: + This mechanism uses passdb's 'lookup_credentials()' function. See below for + description of the credentials. + +auth_new(): + Allocates a new 'struct auth_request'. Typically with more complex mechanisms + it really allocates a 'struct <mech>_auth_request' which contains 'struct + auth_request' as the first field, followed by mechanism-specific fields. + +auth_initial(request, data, data_size): + This begins the authentication, data and data_size containing the initial + response sent by the client (decoded, not in base64). Call + 'request->callback()' once you're done (see below). + +auth_continue(request, data, data_size): + Continues the authentication. Works the same as 'auth_initial()'. + +auth_free(): + Free the request. Usually all the memory allocations for the request should + be allocated from 'request->pool', so you can use 'mech_generic_auth_free()' + which simply frees the pool. + +'auth_initial()' and 'auth_continue()' continue or finish the authentication by +calling 'request->callback()': + +---%<------------------------------------------------------------------------- +typedef void mech_callback_t(struct auth_request *request, + enum auth_client_result result, + const void *reply, size_t reply_size); +---%<------------------------------------------------------------------------- + +The 'reply' and 'reply_size' contain the server's mechanism-specific reply to +the client. If there is no need to return anything (which is usually the case +with the "success" reply), the 'reply_size' can be 0. The 'result' parameter is +one of: + + * AUTH_CLIENT_RESULT_CONTINUE: Client can continue the authentication. The + reply contains the mechanism-specific reply sent to the client. + * AUTH_CLIENT_RESULT_SUCCESS: Authentication successful. The reply is usually + empty. + * AUTH_CLIENT_RESULT_FAILURE: Authentication failed. The reply is always + ignored. + +The 'request->callback()' should actually be called directly only for +continuation requests (a new function should probably be added for this as +well). For success and failure replies, you should instead use one of these +functions: + + * 'auth_request_success()' + * 'auth_request_fail()' + * 'auth_request_internal_failure()': Use this if you couldn't figure out if + the authentication succeeded or failed, for example because passdb lookup + returned internal failure. + +SASL authentication in general works like: + + 1. Client begins the authentication, optionally sending an "initial response", + meaning some data that the mechanism sees in 'auth_initial()'. + * Note that not all protocols support the initial response. For example + IMAP supports it only if the server implements SASL-IR extension. + Because of this mechanisms, such as PLAIN, support doing the + authentication either in 'auth_initial()' or in 'auth_continue()'. + * If the client initiates the authentication (ie. server's initial reply + is empty, such as with PLAIN mechanism) you can use + 'mech_generic_auth_initial()' instead of implementing your own. + 2. Server processes the authentication request and replies back with + 'request->callback()'. + * If the authentication failed, it's placed into 'auth_failures_buf' + unless 'request->no_failure_delay=TRUE'. The failures are flushed from + the buffer once every 2 seconds to clients and 'mechanism->auth_free()' + is called. + * If the authentication succeeded and + * there is a master connection associated with the request (IMAP/POP3 + login), the authentication now waits for master connection to do a + verification request. If this for some reason doesn't happen in + 'AUTH_REQUEST_TIMEOUT' seconds (3,5 mins), it's freed. + * there isn't a master connection (SMTP AUTH), the authentication is + freed immediately. + 3. Client processes the reply: + * If the authentication continues, it sends back more data which is + processed in 'auth_continue()'. Goto 2. + * If the authentication failed, it's done. + 4. If the authentication succeeded, the client requests a login from the + master process, which in turn requests verification from the auth process. + * Besides verifying the authentication, dovecot-auth also does a userdb + lookup to return the userdb information to master. + * If the verification fails (normally because userdb lookup fails), the + client gets "internal authentication failure" + * If the verification succeeds, the user is now logged in + * In either case, 'mechanism->auth_free()' is called now. + +Credentials +----------- + +Most of the non-plaintext mechanisms need to verify the authentication by using +a special hash of the user's password. So either the passdb credentials lookup +returns a plaintext password from which the hash can be created, or the hash +directly. The plaintext to hash conversion is done by calling +'password_generate' function of the password scheme. + +Unfortunately the list of allowed credentials is currently hardcoded in 'enum +passdb_credentials'. The enum values are mapped to password scheme strings in +'passdb_credentials_to_str()'. Some day the enum will be removed so plugins can +add new mechanisms. Besides the mechanism-specific credentials, the enum +contains: + +_PASSDB_CREDENTIALS_INTERNAL: + I don't remember why this really exists. It should probably be called + _PASSDB_CREDENTIALS_INVALID or something and used only by some asserts.. + +PASSDB_CREDENTIALS_PLAINTEXT: + Request a plaintext password. + +PASSDB_CREDENTIALS_CRYPT: + Request the password in any scheme. This is especially useful if you only + want to verify a user's existence in a passdb. Used by <static userdb> + [UserDatabase.Static.txt] in userdb lookups. + +Password schemes +---------------- + +'struct password_scheme' has fields: + +name: + Name of the scheme. This only shows up in configuration files and maybe in + the passwords stored in passdb ("{scheme_name}password_hash"). + +password_verify(plaintext, password, user): + Returns TRUE if 'password' hash matches the plaintext password given in + 'plaintext' parameter. If the password hash depends on the username (eg. with + DIGEST-MD5), the 'user' parameter can also be used. + +password_generate(plaintext, user): + Returns the password hash for given plaintext password and username. + +You can create a new password scheme by simply creating a 'struct +password_scheme' named '<module_name>_scheme', compiling a shared object and +placing it to '$moduledir/auth/' directory. + +Password databases +------------------ + +See <PasswordDatabase.txt> for a description of passdbs and a list of already +implemented ones. + +'struct passdb_module' contains fields: + +cache_key: + A string containing <variables> [Variables.txt]. When expanded, it uniquely + identifies a passdb lookup. This is '%u' when the passdb lookup validity + depends only on the username. With more complex databases such as SQL and + LDAP this is created dynamically based on the password query in the + configuration file. If there are multiple variables, they should be separated + so that their contents don't get mixed, for example '%u<TAB>%r<TAB>%l'. + 'auth_cache_parse_key()' can be used to easily create a cache key from a + query string. + +default_pass_scheme: + The default scheme to use when it's not explicitly specified with a + "{scheme}" prefix. + +blocking: + If TRUE, the lookup is done in dovecot-auth worker process. This should be + used if the lookup may block. + +iface.preinit(auth_passdb, args): + Allocate 'struct passdb_module' and return it. This function is called before + chrooting and before privileges are dropped from dovecot-auth process, so if + should do things like read the configuration file.'auth_passdb' is typically + used for getting a memory pool and looking up some global settings such as + 'auth_passdb->auth->verbose_debug'. 'args' contains the args parameter in + configuration file. + +iface.init(module, args): + The privileges have been dropped before calling this. 'module' contains the + structure returned by 'preinit()'. 'args' is the same as in 'preinit()'. + Typically this function will do things like connect to the database. + +iface.deinit(module): + Close the connection to the password database and free all the memory you + used. + +iface.verify_plain(auth_request, password, callback): + Check if the given plaintext password matches. 'auth_request->credentials = + -1' always. When the verification is done, call the given callback with the + result in 'result' parameter. + +iface.lookup_credentials(auth_request, callback): + Look up the password credentials. 'auth_request->credentials' contains the + credentials that the mechanism wants. When the lookup is finished, call the + given callback with the result in 'result' parameter, and if the lookup was + successful the credentials in 'password' parameter. + +Plaintext authentication mechanisms typically call 'verify_plain()', which is +possible to implement with all the passdbs. Non-plaintext mechanisms typically +call 'lookup_credentials()', which isn't possible to implement always (eg. +PAM). If it's not possible to implement 'lookup_credentials()', you can leave +the pointer to it NULL. + +If the passdb uses connections to external services, it's preferred that they +use non-blocking connections. Dovecot does this whenever possible (PostgreSQL +and LDAP for example). If it's not possible, set 'blocking = TRUE'. + +With both functions 'auth_request->passdb->passdb' contains the passdb_module +returned by your 'preinit()' function. 'auth_request->user' contains the +username whose password we're verifying. You don't need to worry about <master +users> [MasterUsers.txt] here. It's also possible to use any other fields in +'auth_request' to do the lookup, such as 'service', 'local_ip' or 'remote_ip' +if they exist. Often you want to let user to configure the lookup with +<variables> [Variables.txt] (eg. SQL query). In that case you can use +'auth_request_get_var_expand_table()' to retrieve the variable table for +'var_expand()'. + +The passdb lookup can return one of the following results: + +PASSDB_RESULT_INTERNAL_FAILURE: + The lookup failed. For example SQL server is down. + +PASSDB_RESULT_SCHEME_NOT_AVAILABLE: + 'lookup_credentials()' requested a scheme which isn't in the passdb + +PASSDB_RESULT_USER_UNKNOWN: + The user doesn't exist in the database. + +PASSDB_RESULT_USER_DISABLED: + The user is disabled either entirely, or for this specific login (eg. only + POP3 logins allowed). This isn't commonly implemented in passdbs. + +PASSDB_RESULT_PASS_EXPIRED: + The user's password had expired. This isn't commonly implemented in passdbs. + +PASSDB_RESULT_PASSWORD_MISMATCH: + The password given in 'verify_plain()' wasn't valid. + +PASSDB_RESULT_OK: + Success. + +User databases +-------------- + +See <UserDatabase.txt> for a description of userdbs and a list of already +implemented ones. + +'struct userdb_module' is very similar to 'struct passdb_module'. The lookup +callback is a bit different though: + +---%<------------------------------------------------------------------------- +typedef void userdb_callback_t(enum userdb_result result, + struct auth_stream_reply *reply, + struct auth_request *request); +---%<------------------------------------------------------------------------- + +'result' contains one of: + +USERDB_RESULT_INTERNAL_FAILURE: + The lookup failed. For example SQL server is down. + +USERDB_RESULT_USER_UNKNOWN: + The user doesn't exist in the database. + +USERDB_RESULT_OK: + Success. + +There is no equivalent for PASSDB_RESULT_USER_DISABLED currently. Practically +the userdb result is used only by Dovecot's <deliver> [LDA.txt] to figure out +if the user exists or not. When logging in with IMAP or POP3, the user's +existence was already checked in passdb lookup, so only in rare conditions when +a user is logging in at the same time as it's being deleted, the userdb result +is USER_UNKNOWN. + +The 'reply' parameter contains the username (it's allowed to be different from +the looked up username) and a list of key=value pairs that were found from the +userdb. The userdb should make sure that at least "uid" and "gid" keys were +returned. Here's an example code based on passwd userdb: + +---%<------------------------------------------------------------------------- +reply = auth_stream_reply_init(auth_request); +auth_stream_reply_add(reply, NULL, pw->pw_name); +auth_stream_reply_add(reply, "uid", dec2str(pw->pw_uid)); +auth_stream_reply_add(reply, "gid", dec2str(pw->pw_gid)); +auth_stream_reply_add(reply, "home", pw->pw_dir); +callback(USERDB_RESULT_OK, reply, auth_request); +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.AuthProtocol.txt b/doc/wiki/Design.AuthProtocol.txt new file mode 100644 index 0000000..14ab2e6 --- /dev/null +++ b/doc/wiki/Design.AuthProtocol.txt @@ -0,0 +1,251 @@ +Dovecot Authentication Protocol v1.1 +==================================== + +General +------- + +This is a line based protocol. Each line is a command which ends with an LF +character. The maximum line length isn't defined, but it's currently expected +to fit into 8192 bytes. Authentication mechanism specific data transfers are +the largest single parameters. + +Each command is in format: + +---%<------------------------------------------------------------------------- +<command name> TAB <parameters separated with TAB> +---%<------------------------------------------------------------------------- + +Parameters are split into required and optional parameters. Required parameters +aren't in any specific format, but optional parameters are either booleans +without a value, or a name=value pair. If optional parameter name is unknown, +the parameter should just be ignored. + +Typical command looks like (without spaces): + +---%<------------------------------------------------------------------------- +command TAB param1 TAB param2 TAB optname=value TAB optboolean +---%<------------------------------------------------------------------------- + +There is no way to have TABs or LFs in parameters. + +Client <-> Server +----------------- + +Client is an untrusted authentication client process. It can serve one or more +users, so from user's point of view it's usually eg. IMAP or SMTP server +process. + +Server is an authentication server process. + +The connection starts by both client and server sending handshakes: + +---%<------------------------------------------------------------------------- +C: "VERSION" TAB <major> TAB <minor> +C: "CPID" TAB <pid> +S: "VERSION" TAB <major> TAB <minor> +S: "SPID" TAB <pid> +S: "CUID" TAB <pid> +S: "COOKIE" TAB <cookie> +S: "MECH" TAB <name> [TAB <parameters>] (multiple times) +S: "DONE" +---%<------------------------------------------------------------------------- + +Both client and server should check that they support the same major version +number. If they don't, the other side isn't expected to be talking the same +protocol and should be disconnected. Minor version can be ignored. This +document is version number 1.1. + + * CPID and SPID specify client and server Process Identifiers (PIDs). They + should be unique identifiers for the specific process. UNIX process IDs are + good choices. + * CUID is a server process-specific unique connection identifier. It's + different each time a connection is established for the server. + * CPID is used by master's REQUEST command. + * SPID can be used by authentication client to tell master which server + process handled the authentication. + * CUID is currently useful only for APOP authentication. + * COOKIE returns connection-specific 128 bit cookie in hex. It must be given + to REQUEST command. (Protocol v1.1+ / Dovecot v2.0+) + * DONE finishes the handshake from server. CPID finishes the handshake from + client. + +Authentication Mechanisms +------------------------- + +MECH command announces an available authentication SASL mechanism. Mechanisms +may have parameters giving some details about them: + +anonymous: + Anonymous authentication + +plaintext: + Transfers plaintext passwords + +dictionary: + Subject to passive (dictionary) attack + +active: + Subject to active (non-dictionary) attack + +forward-secrecy: + Provides forward secrecy between sessions + +mutual-auth: + Provides mutual authentication + +private: + Don't advertise this as available SASL mechanism (eg. APOP) + +Authentication Request +---------------------- + +---%<------------------------------------------------------------------------- +C: "AUTH" TAB <id> TAB <mechanism> TAB service=<service> [TAB <parameters>] +S1: "FAIL" TAB <id> [TAB <parameters>] +S2: "CONT" TAB <id> TAB <base64 data> +S3: "OK" TAB <id> [TAB <parameters>] +---%<------------------------------------------------------------------------- + +ID is a connection-specific unique request identifier. It must be a 32bit +number, so typically you'd just increment it by one. + +Service is the service requesting authentication, eg. POP3, IMAP, SMTP. + +AUTH and USER (see below) common parameters are: + +lip=<ip>: + Local IP - in standard string format, + +rip=<ip>: + Remote IP - ie. for IPv4 127.0.0.1 and for IPv6 ::1 + +lport=<port>: + Local port + +rport=<port>: + Remote port + +AUTH-only parameters are: + +secured: + Remote user has secured transport to auth client] (e.g. localhost, SSL, TLS) + +valid-client-cert: + Remote user has presented a valid SSL certificate. + +no-penalty: + Ignore auth penalty tracking for this request + +cert_username: + Username taken from client's SSL certificate. + +resp=<base64>: + Initial response for authentication mechanism. NOTE: This must be the last + parameter. Everything after it is ignored. This is to avoid accidental + security holes if user-given data is directly put to base64 string without + filtering out tabs. + +FAIL parameters may contain: + +reason=<str>: + <str> should be sent to remote user instead of the standard "Authentication + failed" messages. For example "invalid base64 data". It must NOT be used to + give exact reason for authentication failure (i.e. "user not found" vs. + "password mismatch"). + +code=temp_fail (v2.3+), temp (<v2.2): + This is a temporary internal failure, e.g. connection was lost to SQL + database. + +code=authz_fail (v2.3+), authz (v1.2..v2.2): + Authentication succeeded, but authorization failed (master user's password + was ok, but destination user was not ok). + +code=user_disabled (v2.3+), user_disabled (v2.2): + User is disabled (password may or may not have been correct) + +code=pass_expired (v2.3+), pass_expired (v2.2): + User's password has expired. + +A CONT response means that the authentication continues, and more data is +expected from client to finish the authentication. Given base64 data should be +sent to client. The client may continue the process issuing + +---%<------------------------------------------------------------------------- +C: "CONT" TAB <id> TAB <base64 data> +---%<------------------------------------------------------------------------- + +The <id> must match the <id> of the AUTH command. + +FAIL and OK may contain multiple unspecified parameters which authentication +client may handle specially. The only one specified here is "user=<userid>" +parameter, which should always be sent if the userid is known. + +Server <-> Master +----------------- + +Master is a trusted process which may query results of previous client +authentication or information about a specific user. Master is optional and in +SMTP AUTH case it's not needed. + +The connection starts by both server and master sending handshakes: + +---%<------------------------------------------------------------------------- +S: "VERSION" TAB <major> TAB <minor> +S: "SPID" TAB <pid> +M: "VERSION" TAB <major> TAB <minor> +---%<------------------------------------------------------------------------- + +Auth with client <-> server, both should check that the version numbers are +valid. + +SPID can be used to let master identify the server process. + +Master Requests +--------------- + +---%<------------------------------------------------------------------------- +M: "REQUEST" TAB <id> TAB <client-pid> TAB <client-id> TAB <cookie> +M: "USER" TAB <id> TAB <userid> TAB service=<service> [TAB <parameters>] +S: "NOTFOUND" TAB <id> +S: "FAIL" TAB <id> TAB <error message> +S: "USER" TAB <id> TAB <userid> [TAB <parameters>] +---%<------------------------------------------------------------------------- + +Master commands can request information about existing authentication request, +or about a specified user. + +USER command's service and parameters are the same as with AUTH client request. + +ID is a connection-specific unique request identifier. It must be a 32bit +number, so typically you'd just increment it by one. + +NOTFOUND reply means that the user wasn't found. (v1.x also reported unknown +request IDs with NOTFOUND.) + +FAIL reply means an internal error occurred. Usually either a configuration +mistake or temporary error caused by lost resource (eg. database down). Also +unknown request IDs are reported as FAILs (since v2.0). + +USER reply is sent if request succeeded. It can return parameters: + +uid=<uid>: + System user ID. + +gid=<gid>: + System group ID. + +home=<dir>: + Home directory. + +chroot=<dir>: + Chroot directory. + +mail=<data>: + Mail location. + +system_user=<user>: + System user name which can be used to get extra groups. This will probably + be replaced later by giving just multiple gid fields. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Buffers.txt b/doc/wiki/Design.Buffers.txt new file mode 100644 index 0000000..03542dc --- /dev/null +++ b/doc/wiki/Design.Buffers.txt @@ -0,0 +1,65 @@ +Buffers +======= + +'lib/buffers.h' describes Dovecot's buffer API. Unless your code happens to be +VERY performance critical, you shouldn't handle writing to buffers/arrays +manually, but instead use the buffer API's safe functions to guarantee that +your code can't write past the buffer and cause a security hole. + +Dovecot's buffers are the basic building block for <arrays> [Design.Arrays.txt] +and <strings> [Design.Strings.txt]. Use them instead if they make more sense +than buffers. + +There are a two different ways to create buffers: statically and dynamically +allocated. + +Static buffers +-------------- + +You can create statically allocated buffers with 'buffer_create_data()'. Trying +to write past the given buffer size will panic. The code to initialize this +looks like: + +---%<------------------------------------------------------------------------- +unsigned char buf_data[1024]; +buffer_t buf; + +buffer_create_data(&buf, buf_data, sizeof(buf_data)); +---%<------------------------------------------------------------------------- + +Trying to write more than 1024 bytes to the buffer will cause an assert-crash, +so these buffers shouldn't be used unless you know exactly what the maximum +buffer size is. + +To avoid accidental buffer overflows, don't use any more complex calculations +in the size parameter of 'buffer_create_data()'. It should always be +'sizeof(data_buffer)'. + +You can also create non-writable buffers with 'buffer_create_const_data()'. +Static buffers don't need to be freed. + +Dynamic buffers +--------------- + +Dynamically growing buffers can be created with 'buffer_create_dynamic(pool, +init_size)'. Memory for buffer is allocated from the given pool. When memory +needs to be grown, it's grown exponentially (2^n), with some exceptions to +avoid growing the given memory pool unless necessary. The initial buffer size +is always a guess - try to make it large enough that buffer wouldn't be grown +most of the time, but not so large that it wastes memory. + +You should be careful with memory returned by 'buffer_get_space_unsafe()' and +'buffer_append_space_unsafe()'. This returned memory should be accessed +immediately afterwards and it must not be accessed anymore after other +'buffer_*()' calls, because they may reallocate the buffer and move it +elsewhere in memory. + +Buffers always look like they're filled with NUL bytes. If you write past the +end of buffer, all the inserted bytes are filled with NULs. If you shrink the +buffer with 'buffer_set_used_size()' and again write past the end of used size, +all the old data is again gone and filled with NULs. If you for some reason +want to just temporarily shrink the buffer size and then change it back, you +can use 'buffer_set_used_size()' to grow it back to its original size (but no +larger). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Code.txt b/doc/wiki/Design.Code.txt new file mode 100644 index 0000000..5fbddba --- /dev/null +++ b/doc/wiki/Design.Code.txt @@ -0,0 +1,203 @@ +Code Design +=========== + +Generally Dovecot follows Linux kernel coding style +[https://www.kernel.org/doc/Documentation/CodingStyle]. + +Most of the coding style design is about getting as many compiler warnings and +errors as possible. Dovecot already has some patched-Clang-specific features to +get more warnings, and will likely have more in future. Issues found by static +analyzers should also be fixed in some way, e.g. adding extra asserts. + +Runtime errors should be caught by asserts or NULL pointer dereferences, which +cleanly crash the program instead of it continuing and possibly corrupting data +or causing other bad things. Dovecot's master process restarts the crashed +processes anyway. Obviously the master process should be very careful to avoid +crashing, but even then in some OSes a crashed Dovecot master gets restarted by +the init process. + +Dovecot's bottlenecks are primarily disk I/O and secondarily memory usage, so +using extra CPU for asserts and other extra checks costs practically nothing. + +Types +----- + + * Unsigned types are used whenever the value isn't allowed to be negative. + This makes it easier to do "value too large" checks when you don't also have + to check for negative values. Also in arithmetic it's better to have the + value wrap (and hopefully checked later!) than cause undefined behavior with + a signed integer overflow. + * 'char *' always points to a NUL-terminated string, 'unsigned char *' + doesn't. + * 'size_t' should generally be used when pointing to a large memory area, + especially for 'mmap()'. Since 'size_t' can be slower to access than + 'unsigned int' (or at least use memory), it's fine to use 'unsigned int' + when it's "very unlikely" that the size ever goes beyond 4 GB + (e.g.'string_t'). + * 'uoff_t' is used for file offsets/sizes. This is usually 64bit, even with + 32bit machines. + * 'uint32_t' vs. 'unsigned int': Use 'uint32_t' when the type really should be + 32bit, but don't spend too much energy trying to avoid mixing it with + 'unsigned int', since they are going to be the same types probably for the + rest of Dovecot's life.. + * 'uint8_t' vs. 'unsigned char': I doubt Dovecot will ever be compiled + anywhere where these differ from one another, but for readability use + 'uint8_t' for binary data and 'unsigned char' for text data. + +Function parameters +------------------- + + * Try to avoid using/allowing NULL pointers. For example for a public API + instead of having 'foo(struct bar *bar)' where bar can be NULL, you could + have both 'foo(void)' and 'foo_with_bar(struct bar *bar)'. Of course don't + try too hard if it makes the API otherwise ugly. + * Dovecot v2.2+ marks all such parameters with ATTR_NULL. These can be + verified with a patched clang. Unfortunately the API isn't very nice, it + would be better to mark each parameter separately with ATTR_NULL instead + of having a numbered list. This will likely change in future. + * Also unless you know that a parameter can be NULL, don't bother wasting + code on checking if it is. Ideally those are noticed by the patched clang + check, but even if not, it's not that bad to crash on NULL pointer + dereference. + * '_r' suffix in parameter is used for values that are returned (e.g. + 'foo(const char **error_r)'). + * Use const for pointers pretty much whenever possible. + * Some day in future I'd like compiler to give warnings if function parameters + are modified by the function itself. There's probably a lot of code that + does this currently, but try avoiding it in future. + +Function return values +---------------------- + + * 'int' type is commonly used for return values. Usually this is used to mean + one of: + * -1 = error, 0 = ok + * -1 = error, 0 = unfinished (e.g. non-blocking call), 1 = finished + * unfortunately there are also other uses. I've been wondering about using + different types for these some day, such as "err" and "err3", but haven't + figured out an easy enough to use design, especially one where compiler + verifies that types aren't accidentally mixed. + * 'bool' shouldn't be used as return value for ok/error, use int 0/-1 instead. + (Old code has some of these. Sometimes it's also not quite clear if + something is an "error" or not, so this rule isn't perfect.) + * Functions returning pointers shouldn't use NULL pointer to mean an error, + only for things like "not found". For example instead of 'if ((ctx = init()) + == NULL)' use 'if (init(&ctx) < 0)' + * Usually when calling a function, either save/check the return value or + explicitly say that you don't care about it by saying '(void)func();' + * Some functions where the return value can nearly always be ignored it's + annoying to add the '(void)' prefix. You can avoid these by adding + ATTR_NOWARN_UNUSED_RESULT to such function's prototype. A patched clang + can be used to find missing return value checks. + * If a system call fails unexpectedly, always log an error about it, possibly + even kill the process if it's vital for correct functionality + (e.g.'gettimeofday()'). '%m' in Dovecot's 'printf()'-style functions expands + to 'strerror(errno)'. + +Boolean expressions +------------------- + +Try to use boolean expressions the way they work in Java. C doesn't require +this, but I think it makes the code easier to understand and reduces bugs in +some cases (e.g.'if (!foo())' when thinking foo() returns bool/FALSE, but +actually returns int/-1 on error). We've a clang patch +[http://dovecot.org/patches/clang/] to give warnings in these cases. As +expected, it found quite a lot of bugs (some real bugs and a lot of "it just +accidentally worked" +[https://github.com/dovecot/core/commit/d9a7e950a9cd21f2b4a90ec7759fca9e8fcc7995]). + + * 'bool x' and 'bool x:1' are the boolean types + * TRUE and FALSE are the only valid explicit boolean values (not 0 or 1, and + currently also not true/false although that could be changed) + * !=, ==, <, >, etc. comparisons create a boolean + * if, for, while, etc. require a boolean + +So: + + * 'if (!ptr)' -> 'if (ptr == NULL)' + * 'if (!num)' -> 'if (num == 0)' + * 'if (flags & FLAG_FOO)' -> 'if ((flags & FLAG_FOO) != 0)' + +Memory +------ + +Memory is always allocated through one of Dovecot's wrappers, e.g. 'i_malloc()' +or 'i_new()'. All of Dovecot's memory allocations always succeed or kill the +process. There's no point in writing a lot of code to check for memory +allocation failures that happen just about never. The only reason some memory +allocations fail in Dovecot currently is because a process VSZ limit is +reached, which usually indicates either a memory leak or trying to access a +mailbox that is too large. In either of these cases it's better to just +completely restart the process than try to limp along without getting anything +useful done anymore. + +Memory allocations can be assumed to be zero-initialized. All of the memory +allocation functions do it, except 't_malloc()' and 't_buffer_get()', which you +should almost never use directly anyway. The code currently also assumes that +pointers in zero-initialized memory area are NULL, which isn't guaranteed by +ANSI-C, but practically Dovecot isn't going to be run in systems where it's not +true and you're not going to remember to NULL-initialize all of your pointers +anyway without compiler/runtime failure. + +When using a struct, always zero-initialize it with 'memset()' instead of +setting each field separately to 0. It's too easy to cause bugs by adding a new +field to the struct and forgetting to initialize it. + +Double-frees in C are bad. Better to crash with NULL pointer dereference than +possibly allow an attacker to exploit heap corruption and run executable code. +Most of the pointers are set to NULL when they are freed: + + * 'i_free_and_null()' is a macro to free the memory and set the pointer to + NULL. + * 'i_free()' also currently sets the pointer to NULL, but another thought was + to set this pointing to some other invalid pointer. But arguably this should + be defined to be exactly the same as 'i_free_and_null()'. + * Most deinit() functions take a pointer-to-pointer parameter and set the + original one to NULL. There's no need to explicitly set the same pointer to + NULL afterwards. + +Buffers +------- + +Use dynamically growing strings/buffers wherever necessary instead of a static +sized buffer, where on larger input the function fails or truncates the data. +It's of course not good to allow users to infinitely grow memory usage, so +there should be some limits added, but it shouldn't fail even if the limit is +set to infinite. + +Avoid explicitly calculating memory usage for allocations. If you do, mark it +with '/* @UNSAFE */' comment unless it's the calculation is "so obvious that +you see it's correct at the first glance". If in doubt, just mark it UNSAFE. +The idea is that anyone can easily grep for these and verify their correctness. + +Type safety +----------- + + * Try to avoid using void pointers. + * Try to avoid casting types to other types, especially if the cast isn't + necessary to avoid a compiler warning. + +It's better if compiler can give a warning when something is accidentally used +wrong. + +Callback functions +------------------ + +Callback functions make the code more difficult to follow, especially when a +callback calls another callback, or when using function pointers to jump to +different callbacks depending on state. Of course with asynchronous C code it's +pretty much impossible to avoid callbacks. Still, try to avoid them where +possible to keep the code readable. lib-fs/fs-api.h is an example API which +supports async operations but with a single common "do more now" callback +rather than every single operation having its own callback parameter. This +makes it similar to async network IO with read()/write() EAGAIN handling. + +Often callback functions can be avoided by creating iterator functions instead. +For example instead of 'parse(callback, context' use 'ctx = parse_init(); while +(parse_next(ctx)) { .. } parse_deinit(&ctx);' + +Dovecot has some helper macros to make callbacks' context parameters type-safe. +In v2.2+ see 'CALLBACK_TYPECHECK()' macro and for example 'io_add()' for +example usage. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Dcrypt.txt b/doc/wiki/Design.Dcrypt.txt new file mode 100644 index 0000000..e17f017 --- /dev/null +++ b/doc/wiki/Design.Dcrypt.txt @@ -0,0 +1,113 @@ +lib-dcrypt +========== + +lib-dcrypt is component for abstracting asymmetric and symmetric cryptographic +operations. It can be used for public/private key handling too. Currently we +support OpenSSL backing, but it is possible to write alternative backends for +dcrypt. + +ECDH algorithm +-------------- + +ECDH (Elliptic curve Diffie-Hellman) is widely used in lib-dcrypt for both key +and data storage. This algorithm is also known as ECIES +[https://en.wikipedia.org/wiki/ECIES] (Elliptic curve Integrated Encryption +Scheme). + +When encrypting data, we perform following steps, this is the currently used +algorithm. There is also a legacy algorithm, but since that has not been used +publicly, we do not describe it here. You can deduce it from the code if you +want to. + +ENCRYPT(RECIPIENT-KEY, DATA): + + 1. Ensure recipient key is not point at infinity + 2. Generate new keypair from same group + 3. Choose ephemeral public key as R + 4. Calculate P = R * RECIPIENT-KEY + 5. From P = (x,y) choose x as S + 6. Generate random salt + 7. Use PBKDF2(SHA256, S, salt, 2000) to produce iv+key, and hmac seed or aad + 8. Encrypt data + 9. OUTPUT R, salt and encrypted data. + +In dcrypt-openssl.c, we use EVP_PKEY_derive_* for the actual derivations. +ephemeral public key is exported with EC_POINT_point2oct in compressed form. + +DECRYPT(PRIVATE-KEY, R, SALT, DATA): + + 1. Ensure R is not point at infinity + 2. Calculate P = R * PRIVATE-KEY + 3. From P = (x,y) choose x as S + 4. Use PBKDF2(SHA256, S, salt, 2000) to produce iv+key, and hmac seed or aad + 5. Decrypt data + 6. OUTPUT decrypted data + +Key formats +----------- + +lib-dcrypt can consume keys in PEM format [https://tools.ietf.org/html/rfc1421] +(with or without password), and in Dovecot's special format intended for dict +storage. + +Dovecot's format consists from unencrypted and encrypted keys. You can encrypt +keys using password or another key. There are also two version, version 1 +(deprecated) and version 2 (current). Both versions support either tab or : +separated fields. ECC keys are stored always in compressed form. Version 1 +format is not described as it's deprecated and should not be used. + +Version 2 format +---------------- + +---%<------------------------------------------------------------------------- +public key id: HEX(SHA256(public key in DER format)) +key data: + RSA: i2d_PrivateKey + ECC: BN_bn2mpi using compressed form + +public key: 2:HEX(public key in DER format):public key ID +private key (unencrypted) : 2:key algo oid:0:key data:public key ID +private key (encrypted, key) : 2:key algo oid:1:symmetric algo:salt:digest algo +(for pbkdf2):rounds:encrypted key data:ephemeral public key:digest of +encryption key:public key ID +private key (encrypted, pwd) : 2:key algo oid:2:symmetric algo:salt:digest algo +(for pbkd2f):rounds:encrypted key data:public key ID +---%<------------------------------------------------------------------------- + +File format +----------- + +This library can also generate encrypted files that are encrypted using +asymmetric key pair. File encryption can be done using whatever algorithm(s) +the underlying library supports. For integrity support, either HMAC based or +AEAD based system is used when requested. + +File format is described below + +---%<------------------------------------------------------------------------- +000 - 008 CRYPTED\x03\x07 (MAGIC) +009 - 009 \x02 (VERSION FIELD, 2) +010 - 013 MSB flags +014 - 017 MSB total header length (starting from 000) +018 - cod cipher oid in DER format +cod - mod MAC algorithm oid in DER format +mod - +4 MSB PBKDF2 rounds ++5 - +8 MSB length of key data ++9 - +9 number of key blocks +----- key block ----- ++10 - +10 key type (1 = RSA, 2 = ECC) ++11 - +43 public key id (SHA256 of public key in DER format, point compressed) ++44 - +48 MSB length of ephemeral key ++49 - epk ephemeral key +epk - +4 MSB length of encrypted key ++4 - ek encrypted key +----- end of key block (this can then repeat) ----- +eokb - +4 MSB length of encryption key hash ++4 - ekh encryption key hash +ekh - (eof-maclen) payload data +---%<------------------------------------------------------------------------- + +There is a small script for decrypting these files, see +[attachment:decrypt.rb]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.DoveadmProtocol.HTTP.txt b/doc/wiki/Design.DoveadmProtocol.HTTP.txt new file mode 100644 index 0000000..a9718e4 --- /dev/null +++ b/doc/wiki/Design.DoveadmProtocol.HTTP.txt @@ -0,0 +1,113 @@ +Doveadm HTTP API +================ + +Doveadm HTTP API is available since v2.2.22. It is considered *experimental* in +v2.2.22. Can be considered as stable since 2.2.23. It lets you perform doveadm +commands over HTTP transport. + +Configuration +------------- + +To enable HTTP API, add following to your config file: + +---%<------------------------------------------------------------------------- +service doveadm { + inet_listener http { + port = 8080 + #ssl = yes # uncomment to enable https + } +} +---%<------------------------------------------------------------------------- + +To enable SSL make sure 'ssl=yes' or 'ssl=required' in global settings, and set +'ssl=yes' in the listener. + +You can use unix listener too, and define host to listen on. You also need to +either define 'doveadm_password', or 'doveadm_api_key'. With +'doveadm_password', the username is doveadm. This is going to change in future +release. With API Key you are expected to send + +---%<------------------------------------------------------------------------- +Authorization: X-Dovecot-API Base64(apikey) +---%<------------------------------------------------------------------------- + +header to access the API. (In v2.2.22-2.2.23, this key was incorrectly +"X-Doveadm-API".) + +Usage +----- + +You can see valid commands and their parameters by accessing +'http://host:port/doveadm/v1'. + +To send command(s), you can send following with 'Content-Type: +application/json' + +---%<------------------------------------------------------------------------- +[ + ["command", {"parameter":"value"}, "optional identifer"] +] +---%<------------------------------------------------------------------------- + +In following examples, you can either use Basic or X-Dovecot-API authorization. +X-Dovecot-API usage: + +---%<------------------------------------------------------------------------- +curl -H "Authorization: X-Dovecot-API <base64 dovecot_api_key>" +---%<------------------------------------------------------------------------- + +Basic authorization uses "doveadm" as the username, and doveadm_password +setting as the password: + +---%<------------------------------------------------------------------------- +curl -H "Authorization: Basic <base64 doveadm:doveadm_password>" +---%<------------------------------------------------------------------------- + +To get acceptable routes + +---%<------------------------------------------------------------------------- +curl -H "Authorization: Basic <base64 doveadm:doveadm_password>" +http://server:8080/ +---%<------------------------------------------------------------------------- + +To get acceptable commands and their parameters + +---%<------------------------------------------------------------------------- +curl -H "Authorization: Basic <base64 doveadm:doveadm_password>" +http://server:8080/doveadm/v1 +---%<------------------------------------------------------------------------- + +an example command would be + +---%<------------------------------------------------------------------------- +curl -H "Content-Type: application/json" -H "Authorization: Basic <base64 +doveadm:doveadm_password>" -d +'[["fetch",{"user":"username","field":["uid"],"query":["mailbox","INBOX"]},"c01"]]' +http://server:8080/doveadm/v1 +---%<------------------------------------------------------------------------- + +You can have multiple commands in the array, but for now it is safest *not* to +do so, as some commands may kill the server in certain error conditions and +leaving you without any response. + +Responses are in same format, command is replaced with either "error" or +"doveadmResponse" and parameters with array of response variables. In case of +*successful* command which has *no* output, response is going to be []. + +Errors are indicated with "error" and type, exit-code in response part. + +Logging +------- + +The logs will indicated execute command(s) and also Apache access.log format +strings on requests. In case of fatal errors, the access.log string might be +missing. + +Example clients +--------------- + + * A PoC API client/library written in Python is available at + https://github.com/hnsk/doveadm-http-cli + * See also internal doveadm protocol clients in <Design.DoveadmProtocol.txt> + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.DoveadmProtocol.txt b/doc/wiki/Design.DoveadmProtocol.txt new file mode 100644 index 0000000..10d8bd5 --- /dev/null +++ b/doc/wiki/Design.DoveadmProtocol.txt @@ -0,0 +1,95 @@ +Doveadm protocol +================ + +See also <doveadm HTTP protocol> [Design.DoveadmProtocol.HTTP.txt]. + +doveadm-server can be accessed via UNIX sockets or TCP protocol (by adding +inet_listener to doveadm service). The protocol looks like: + +Initial handshake from client to server: + +---%<------------------------------------------------------------------------- +C: VERSION doveadm-server 1 0 +---%<------------------------------------------------------------------------- + +Note that the spaces you see are TABs. All the fields are TAB-separated. The +server will send you back either: + + * "+" means you are preauthenticated and can start sending commands. This + happens when connecting to the UNIX socket. + * "-" means you need to authenticate first. + +Authentication +-------------- + +The authentication is done with a regular SASL PLAIN authentication, i.e. +"PLAIN<tab>base64(\0username\0password)". Currently the username must be +"doveadm". For example for user=doveadm, password=secret use: + +---%<------------------------------------------------------------------------- +C: PLAIN AGRvdmVhZG0Ac2VjcmV0 +S: + +---%<------------------------------------------------------------------------- + +Running commands +---------------- + +The actual commands are in format: flags<tab>username<tab>command +name[<tab>parameter[<tab>parameter2...]], where the flags can be either empty, +"v" (verbose) or "D" (debug). Note that if the command name has spaces, they +are sent as spaces instead of as tabs (e.g. "quota get", not "quota<tab>get"). +So for example to get a quota for user tss: + +---%<------------------------------------------------------------------------- +C: tss quota get +S: user STORAGE 1814 - 0 user MESSAGE 6 - 0 + +S: + +---%<------------------------------------------------------------------------- + +The storage values are all given in kilobytes. + +The server replies using the same fields TAB-separated as what a regular +doveadm command sends. The reply itself ends with LF. So if the reply is large, +it may return a very long line as a reply. After the reply follows a status +line: + + * "+" = success. + * In future the "+" may be followed by more text, for now you should just + ignore those. + * "-" = failed (the error was probably logged to Dovecot's error log) + * "-NOUSER" = the user doesn't exist + * Other "-SOMETHING" errors may be added in future. + +Available commands +------------------ + +The command names and output are exactly the same as what regular doveadm +commands on command line do. Currently only "mail commands" are available via +doveadm protocol, but this will change in future. + +You can use the doveadm itself to find out what the output format will look +like. For example: + +---%<------------------------------------------------------------------------- +doveadm -f tab search mailbox inbox 1:2 +mailbox-guid uid +fa8cb722dfad9c52b62600007049b30b 125159 +fa8cb722dfad9c52b62600007049b30b 125160 +---%<------------------------------------------------------------------------- + +There are two fields, "mailbox-guid" and "uid" in the output. The title names +won't be sent via doveadm protocol, but everything else will be sent in one +line. So in the above case the protocol output will be: + +---%<------------------------------------------------------------------------- +fa8cb722dfad9c52b62600007049b30b<tab>125159<tab>fa8cb722dfad9c52b62600007049b30b<tab>125160 +---%<------------------------------------------------------------------------- + +Example Clients +--------------- + + * Perl: Net::Doveadm [https://metacpan.org/pod/Net::Doveadm] + * See also HTTP protocol-based clients in <Design.DoveadmProtocol.HTTP.txt> + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Dsync.txt b/doc/wiki/Design.Dsync.txt new file mode 100644 index 0000000..7b51c93 --- /dev/null +++ b/doc/wiki/Design.Dsync.txt @@ -0,0 +1,114 @@ +Dsync Design +============ + +FIXME: This describes the design for v2.0/v2.1. The v2.2 design is somewhat +different. + +Two-way synchronization +----------------------- + +dsync attempts to preserve all changes done by both sides of the synced +mailboxes. + +Mailbox list +------------ + +Mailboxes have 128 bit globally unique IDs, which are used for figuring out +when two mailboxes should actually be synchronized. This solves two major +problems: + + * If mailbox has been renamed in one side, dsync finds it because its GUID + hasn't changed. + * If mailbox has been deleted and recreated, dsync doesn't attempt to sync it + because it's a different mailbox. + +Then there's the problem of how to correctly sync mailbox renames and +deletions. How do you know which side of the sync has the most recent name for +the mailbox? How do you know if one side had deleted mailbox, or if the other +side had created it? To solve these problems, Dovecot v2.0 created a "mailbox +log", which adds a record with mailbox GUID and timestamp whenever mailbox is +renamed or deleted. So: + + * If mailbox has different names on two sides, its "last renamed" timestamp is + looked up from the mailbox list index. The side with the most recent + timestamp is assumed to contain the newer name and the other side's mailbox + is renamed to it. + * If neither side has a "last renamed" timestamp, one side is picked. This + shouldn't happen, except when mailbox log is deleted for some reason or + if the renaming is done outside Dovecot. + * If mailbox exists only on one side, the other side checks if mailbox log + contains a delete record for its GUID. If there is one, the mailbox is + deleted from the other side. If there's not, the mailbox is created and + synced. + * Subscriptions and unsubscriptions are synced in a similar way. But because + it's possible to be subscribed to nonexistent mailboxes, mailbox log can't + contain mailbox GUIDs for them. Instead the first 128 bits of SHA1 of + mailbox name are used. Collisions for mailbox names are highly unlikely, but + even if one happens, the worst that can happen is that user gets + unsubscribed from wrong mailbox. + +dsync writes timestamps to changelog using the original timestamps, so that +dsync's changes won't override changes done by user during sync. + +Mailbox +------- + +When saving new mails, dsync preserves all of their immutable state: + + * GUID + * Received date + * Save date + * Message contents + +It also attempts preserve IMAP UID. This works as long as the other side hasn't +already used the UID for another mail. If it has, dsync doesn't attempt to +preserve the UID, because an IMAP client might have already seen the UID and +cached another mail's contents for it. IMAP requires that message's contents +must never change, so UIDs can't be reused. So whenever an UID conflict +happens, dsync gives messages in both sides a new UID, because it can't know +which message the client had seen, or perhaps user used two clients and both +saw a different message. (This assumes a master/slave replication use case for +dsync.) + +The mutable metadata that dsync preserves is: + + * Message flags and keywords + * Modification sequences (modseqs) + +Flags and keywords are synced based on modseqs. Whichever side has a higher +modseq for the message, its flags and keywords are synced to the other side. +Currently there's no per-flag or per-keyword synchronization, so that if one +side had added \Seen flag and other side had added \Answered flag, one of them +would be dropped. + +Finding what to sync +-------------------- + +dsync can run in full mode or fast mode. Full mode means it goes through all +messages in all mailboxes, making sure everything is fully synchronized. In +fast mode it relies on uidvalidity, uid-next and highest-modseq values to find +out changes. If any of the values changed, the mailbox is included in sync. + +FIXME: A superfast mode should still be implemented, where once a mailbox is +selected for syncing, it should sync only mails whose modseq is higher than a +given one. This would improve performance and network traffic with large +mailboxes. + +Copy optimizations +------------------ + +Before dsync actually starts syncing anything, it first fetched a list of all +to-be-synced messages and adds them to a GUID -> message hash table. Whenever +dsync needs to sync a new message to the other side, it first checks if the +message's GUID already exists on the other side. If it does, it starts a +message copy operation instead of a full save. It's possible that this copy +operation fails if the message just gets expunged from the other side, so there +needs to be fallback handling for this. If the message exists in multiple +mailboxes, a copy from the next mailbox is attempted. If all of them fail, +dsync fallbacks to saving the message. + +FIXME: This optimization currently works only in full sync mode. If this were +to work in fast sync mode, the full mailbox list would have to be looked up +from local side. And this would slow it down.. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Events.txt b/doc/wiki/Design.Events.txt new file mode 100644 index 0000000..c6df740 --- /dev/null +++ b/doc/wiki/Design.Events.txt @@ -0,0 +1,264 @@ +Events (v2.3+) +============== + +Dovecot v2.3 introduces "events", which improves both logging and statistics. +See <Events.txt> for list of all events. + +Each logging call can be attached to a specific event, which can provide more +metadata and context than just the log message string. This will eventually +allow implementing things like machine-parseable (e.g. JSON) log lines +containing key=value pairs, while still keeping the human readable text +available. Each logging event can also be captured and sent to stats, even if +it's not actually logged. Commonly statistics-related events are logged with +debug level. + +Events have: + + * Categories, such as "storage", "mailbox" or "auth". + * Fields, such as 'user=foo@example.com' or 'service=imap'. + * Creation timestamp with microsecond precision. + * Source code file and line number location when sending the event. + * It may have an easy human-readable name. This is important for events that + are expected to be used for statistics, so they can be easily referred to. + * "Forced debug"-flag. Debug logging is enabled for this event regardless of + the global debug log filters. A child event will inherit this flag. + +Events are hierarchical, so they can have parent events. The events always +inherit all of their parents' categories and fields. A child event can replace +a parent's field, and it can also remove a parent's field with +'event_field_clear()'. Ideally most events would have a parent hierarchy that +reaches the top event that was created for the current user/session. This +allows statistics to track which events happened due to which users. In some +cases this may not really be possible, such as an HTTP connection that is +shared across multiple users in the same process. Generic libraries should take +the parent event in function parameters or in a settings struct or similar. + +An event's lifetime is usually the same as the "object" it attaches to. For +example an IMAP client connection should have a single event created at the +beginning of the connection and destroyed at disconnection. The IMAP client +connection event could be used for logging things like "Client connected" and +"Client disconnected" and perhaps some other connection-specific events. +However, most of the logging should be done by new events that have the IMAP +client connection event as their parent. For example IMAP command event should +exist during the execution of the IMAP command, and its parent should be the +IMAP client connection event. Note that there's an automatic "duration" +statistics field that is calculated from the creation of the event to the +(last) sending of the event, so for it to make sense the event lifetime and its +logging also needs to make sense. So for example if the IMAP client connection +event was used for logging many things throughout the session, the "duration" +field would make little sense for most of those events. + +Events are "sent" by logging it. Any e_debug(), e_info(), e_warning() or +e_error() call will also send the event, which may be redirected to the stats +process. Often events that are intended for statistics are sent using the +e_debug() call. The event can be sent to statistics even if it's not actually +logged. Avoid sending events excessively. For example an e_debug() call every +time connection reads or writes something will likely result in a huge amount +of unnecessary debug logging. + +Event names +----------- + +Events that are expected to be used in statistics should have a name. Be +consistent when naming the events. The name's prefix should be the subsystem +that is logging the event. Usually this would be the primary category of the +event. For example imap related events should begin with "imap_" and mailbox +related events begin with "mailbox_". + +The name should consist of only [a-z], [0-9] and '_' characters. + +Current naming conventions for name suffixes: + + * _connected (for connections) + * _disconnected (for connections) + * _finished (when some operation finishes, e.g. IMAP command or HTTP request) + * e.g. http_request_finished, dns_request_finished, imap_command_finished + * This should be used regardless of whether the operation succeeded or + failed. The details would be in fields. + * _retried (if an operation is internally retried one or more times before + it's finished) + +Categories +---------- + +The event categories are hierarchical. For example "mail" category has parent +"mailbox", which has parent "storage". If an event filter contains +"category:storage", it will match the "mail" and "mailbox" child categories as +well. + +Note that a category isn't the same as a service/process name. So for example +imap process has an "imap" category for its IMAP-related events, such as IMAP +client connection and IMAP command related events. Because most events would be +child events under these IMAP events, they would all inherit the "imap" +category. So it would appear that using "category:imap" filter would match most +of the logging from imap process. However, there would likely be some events +that wouldn't have the IMAP client as their parent event, so these wouldn't +match the imap category. + +The same category name must not be duplicated within the process. This is +because event handling is optimized and performs category checking by comparing +the categories' pointers, not names' strings. (Then again, if the struct +event_category variable names were consistent, you'd get duplicate symbol +errors from linker as well.) + +Be careful naming events that go through client and server boundaries. For +example if both lib-dns and dns service use "dns" as their category and also +have identically named "dns_lookup" event, there's no easy way to differentiate +in event filters between these two. So a statistics filter could end up +counting each DNS lookup twice. Since it's more difficult to remember to check +for event naming conflicts, it would be safer to use different category names +entirely. + +The category name should consist of only [a-z], [0-9] and '_' characters. + +Fields +------ + +Each event can have any number of key=value fields. Parent event's fields are +inherited by the child event. + +There are 3 types of fields: + + * strings + * numbers (intmax_t = signed 64bit usually) + * timestamp (struct timeval) + +The fields can be used for various purposes: + + * Filtering events with field_name=value matching + * Counting fields in statistics (most commonly number fields) + * They can include metadata that are internally used by the code. For example + passing data from one plugin to others. + * Later on these fields can be used by the logging system. + +Field names should be consistent across the code. Besides making it easier for +admins to configure the events, this allows statistics code to sum up fields +from different unrelated events. For example if all the networking events +include "ip", "bytes_in" and "bytes_out" fields, statistics can globally track +how much network traffic Dovecot is doing from its own point of view, +regardless of whether it's HTTP traffic or IMAP traffic or something else. + +Current naming conventions: + + * The name should consist of only [a-z], [0-9] and '_' characters. + * Timestamps should have "_time" suffix + * Durations should have "_usecs" suffix and be in microseconds. + * Try to avoid adding extra duration fields for most events. There's the + automatic "duration" field already that contains how long the event has + existed. So usually the event lifetime should be the same as the wanted + duration field. + * Incoming TCP/IP connections should have "remote_ip", "remote_port", + "local_ip" and "local_port" fields + * Outgoing TCP/IP connections should have "ip" and "port" for the remote side. + + * For local side "client_ip" and "client_port" may optionally be used + * NOTE: These are all different from incoming connection's IP/port fields. + This is because often everything starts from an incoming connection, + which will be used as the root event. So we may want to filter e.g. + outgoing HTTP events going to port 80 which were initiated from IMAP + clients that connected to port 993 (port=80 local_port=993) + * Connection reads/writes should be counted in "bytes_in" and "bytes_out" + fields + * These fields were chosen over e.g. network_in/out because a lot of code + is rather generic and can work over TCP/IP or UNIX sockets, or maybe even + any other kind of iostreams. Using a generic bytes_in/out makes it + simpler to count these. If further differentiation is wanted on + statistics side, networking events can be filtered out with "ip". + * These fields are usually easiest updated with 'event_add_int(event, + "bytes_in", istream->v_offset)' and 'event_add_int(event, "bytes_out", + ostream->offset)'. If iostreams aren't used, 'event_inc_int()' maybe be + easier. + * (Local) disk reads should have "disk_read" and "disk_write" fields + * With remote filesystems like NFS it may be difficult to differentiate + between disk IO and network IO. Generally the disk_read/write should be + used for POSIX read() and write() calls from filesystem. + * Counting only read()s and write()s doesn't necessarily translate to + actual disk IO since it may only be accessing the kernel page cache. + Still, this may be useful. + * There is a lot of disk IO performed all over the code, so Dovecot will + likely never include events for all disk reads/writes. + * error=<value> : The operation failed. The <value> may be simply "y" or + contain more details. This field shouldn't exist at all for successful + operations. + * error_code=<value> : Machine-readable error code for a failed operation. If + set, the "error" field must also be set. + +Note that events shouldn't be sent every time when receiving/sending network +traffic. Instead, the bytes_in/out fields should be updated internally so that +whenever the next event is sent it will have an updated traffic number. + +Generally it's not useful for events to be counting operations. Rather each +operation should be a separate event, and the statistics code should be the one +counting them. This way statistics can only be counting e.g. operations with +duration> 1 sec. If the statistics code was seeing only bulk operation counts +this wouldn't be possible. The bytes_in/out and such fields are more of an +exception, because it would be too inefficient to send individual events each +time those were updated. + +Note that even though internally updating a field for an event's parent will be +immediately visible to its children, the update won't be automatically sent to +the stats process. We may need to fix this if it becomes a problem. + +Field inheritance may become problematic also when multiple nested ioloops are +used. For example an outgoing imapc connection could receive a reply, which +synchronously triggers an outgoing quota SQL connection. The quota SQL +connection's parent event likely shouldn't be the imapc connection's event, +because otherwise they could be mixing the IP/port fields and perhaps others. +This isn't necessarily a problem though, but this is why when connection.c +performs outgoing UNIX socket connection it clears the IP/port fields to make +sure they don't exist for the connection event due to inheritance from a parent +event. + +Passthrough events +------------------ + +Passthrough events' main purpose is to make it easier to create temporary +events as part of the event parameter in e_error(), e_warning(), e_info() or +e_debug(). These passthrough events are automatically freed when the e_*() call +is finished. Because this makes the freeing less obvious, it should be avoided +outside e_*()'s event parameter. + +A passthrough event's creation timestamp is the same as the parent event's +timestamp, because its intention is to only complement it with additional +fields. This way the generated event "duration" field is preserved properly. + +The passthrough events also change the API to be more convenient towards being +used in a parameter. Instead of having to use e.g. + +---%<------------------------------------------------------------------------- +event_add_str(event_set_name(event_create(parent), "name"), "key", "value") +---%<------------------------------------------------------------------------- + +The event_passthrough API can be a bit more readable as: + +---%<------------------------------------------------------------------------- +event_create_passthrough(parent)->set_name("name")->add_str("key", +"value")->event(). +---%<------------------------------------------------------------------------- + +The passthrough event is converted to a normal event at the end with the +event() call. Note that this API works by modifying the last created +passthrough event, so it's not possible to have multiple passthrough events +created in parallel. + +Log prefixes +------------ + +Events allow replacing the current log prefix or appending to it. This way for +example opening a mailbox can add a "Mailbox<name>: " prefix and then use +'e_debug(box->event, ...)' without having to specify the mailbox name in every +log message. + +Global events +------------- + +Sometimes there's not really any specific event that a log message would belong +to, or it would be difficult to transfer the event there. In these cases the +old i_debug(), i_info(), i_error(), etc. logging calls can still be used. These +will be using the global event and its logging prefix. + +The global events are pushed/popped in a stack. For example with IMAP the +initial global event is the user's event. During IMAP command execution the +global event is the IMAP command event. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.Cache.txt b/doc/wiki/Design.Indexes.Cache.txt new file mode 100644 index 0000000..45e75a7 --- /dev/null +++ b/doc/wiki/Design.Indexes.Cache.txt @@ -0,0 +1,137 @@ +Cache file +========== + +Cache file is used for storing immutable data. It supports several different +kinds of fields: + +MAIL_CACHE_FIELD_FIXED_SIZE: + The field size doesn't need to be stored in the cache file. It's always the + same. + +MAIL_CACHE_FIELD_BITMASK: + A fixed size bitmask field. It's possible to add new bits by updating this + field. All the added fields are ORed together. + +MAIL_CACHE_FIELD_VARIABLE_SIZE: + Variable sized binary data. + +MAIL_CACHE_FIELD_STRING: + Variable sized string. + +MAIL_CACHE_FIELD_HEADER: + Variable sized message header. The data begins with a 0-terminated 'uint32_t + line_numbers[]'. The line number exists only for each header, header + continuation lines in multiline headers don't get listed. After the line + numbers comes the list of headers, including the "header-name: " prefix for + each line, LFs and the TABs or spaces for continued lines. + +The last 3 variable sized fields are treated identically by the cache file +code. Their main purpose is to make it easier for "dump cache file's contents" +programs ('src/util/idxview') to do their job. + +Locking +------- + +Because cache file is typically used in potentially long-running operations, +such as with IMAP command 'FETCH 1:* (BODY.PEEK[] ENVELOPE BODYSTRUCTURE)' it's +important that updating the cache file doesn't block out any other readers. +Also because the readers are often also writers (if something isn't cached, +it's added there), it's important that they don't block writers either. + +Reading cache files requires no locking. Writing is done by first locking the +file, reserving some space to write to, and immediately after that unlocking +the file. This way the transaction can keep writing to the cache file as long +as it wants to without blocking other writers. When the transaction is +committed, the updated cache offsets are written to the transaction log which +makes them visible to other processes. + +This also means that it's possible for two processes to write the same cached +fields twice to the cache file. Because the data written to the cache file are +really just cached data, the fields' contents are identical. Having the data +exist twice (or even more times) means wasting some disk space, but otherwise +it isn't a problem. The duplicates are dropped the next time the file is +compressed. + +Cache decisions +--------------- + +Dovecot tries to be smart about what it keeps in the cache file. If the client +never fetches the cached data, it's just waste of disk space and disk I/O. + +The caching decisions are: + +MAIL_CACHE_DECISION_NO: + This field isn't cached currently. + +MAIL_CACHE_DECISION_TEMP: + This field is cached for new mails. + +MAIL_CACHE_DECISION_YES: + This field is cached for all mails. + +Normally Dovecot changes the decisions based on what fields are fetched and for +what messages. A specific decision can be forced by ORing it with +'MAIL_CACHE_DECISION_FORCED'. + +'mail-cache-decisions.c' file contains the rules how Dovecot changes the +decisions. The following is copied from the file: + +Users can be divided to three groups: + + 1. Most users will use only a single IMAP client which caches everything + locally. For these users it's quite pointless to do any kind of caching as + it only wastes disk space. That might also mean more disk I/O. + 2. Some users use multiple IMAP clients which cache everything locally. These + could benefit from caching until all clients have fetched the data. After + that it's useless. + 3. Some clients don't do permanent local caching at all. For example Pine and + webmails. These clients would benefit from caching everything. Some locally + caching clients might also access some data from server again, such as when + searching messages. They could benefit from caching only these fields. + +After thinking about these a while, I figured out that people who care about +performance most will be using Dovecot optimized LDA anyway which updates the +indexes/cache immediately. In that case even the first user group would benefit +from caching the same way as second group. LDA reads the mail anyway, so it +might as well extract some information about it and store them into cache. + +So, group 1. and 2. could be optimally implemented by keeping things cached +only for a while. I thought a week would be good. When cache file is +compressed, everything older than week will be dropped. + +But how to figure out if user is in group 3? One quite easy rule would be to +see if client is accessing messages older than a week. But with only that rule +we might have already dropped useful cached data. It's not very nice if we have +to read and cache it twice. + +Most locally caching clients always fetch new messages (all but body) when they +see them. They fetch them in ascending order. Noncaching clients might fetch +messages in pretty much any order, as they usually don't fetch everything they +can, only what's visible in screen. Some will use server side sorting/threading +which also makes messages to be fetched in random order. Second rule would then +be that if a session doesn't fetch messages in ascending order, the fetched +field type will be permanently cached. + +So, we have three caching decisions: + + 1. Don't cache: Clients have never wanted the field + 2. Cache temporarily: Clients want this only once + 3. Cache permanently: Clients want this more than once + +Different mailboxes have different decisions. Different fields have different +decisions. + +There are some problems, such as if a client accesses message older than a +week, we can't know if user just started using a new client which is just +filling its local cache for the first time. Or it might be a client user hasn't +just used for over a week. In these cases we shouldn't have marked the field to +be permanently cached. User might also switch clients from non-caching to +caching. + +So we should re-evaluate our caching decisions from time to time. This is done +by checking the above rules constantly and marking when was the last time the +decision was right. If decision hasn't matched for two months, it's changed. I +picked two months because people go to at least one month vacations where they +might still be reading mails, but with different clients. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.MailIndexApi.txt b/doc/wiki/Design.Indexes.MailIndexApi.txt new file mode 100644 index 0000000..e7f597b --- /dev/null +++ b/doc/wiki/Design.Indexes.MailIndexApi.txt @@ -0,0 +1,47 @@ +Mail Index API +============== + +'lib-index/mail-index.h' contains the functions to access the index files. +'mail-cache.h' contains the functions to access the cache file. + +The purpose of the main structures are: + + * 'struct mail_index': Global state of the index. + * 'struct mail_index_view': You can have multiple views to the index. The + views see new messages come and expunged messages go only when it's being + explicitly synchronized. With mmaped indexes you can't really trust the + record data (flags, keywords, extensions) not to change. This doesn't matter + with IMAP. + * 'struct mail_index_map': Index file is accessed via maps. Views can point to + one or more maps. Maps can be shared by different views. Maps can contain + either mmap()ed memory areas pointing to the index file, or a in-memory copy + of it. + * 'struct mail_index_transaction': In-memory list of changes to be written to + the transaction log. The writing is done only when the transaction is + committed. + +Views and maps +-------------- + +In general you access all the data in the index files via views. The mails are +accessed using sequence numbers, which change only when the view is +synchronized. + +For accessing messages with their UIDs, you'll first need to convert them to +sequences with either 'mail_index_lookup_uid()' or +'mail_index_lookup_uid_range()'. + +'mail_index_lookup()' can be used to look up a single record's UID and flags. +The returned record points to the latest map, so that it contains the latest +flag changes. If the message was already expunged from the latest map, it +returns 0. + +'mail_index_lookup_full()' can be used to get also the map where the message +was found. This can be important with extensions. If extension record's state +depends on the extension header, they must be looked up from the same map. For +this reason there exists 'mail_index_map_get_header_ext()' and +'mail_index_lookup_ext_full()' functions which take the map as parameter. The +non-map versions return the data from the latest map if the message hasn't been +expunged. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.MainIndex.txt b/doc/wiki/Design.Indexes.MainIndex.txt new file mode 100644 index 0000000..db0fb0f --- /dev/null +++ b/doc/wiki/Design.Indexes.MainIndex.txt @@ -0,0 +1,284 @@ +Main index +========== + +The main index can be used to quickly look up messages' UIDs, flags, keywords +and extension-specific data, such as cache file or mbox file offsets. + +Reading, writing and locking +---------------------------- + +Reading 'dovecot.index' file requires locking, unfortunately. Shared read +locking is done using the standard index locking method specified in +'lock_method' setting ('lock_method' parameter for 'mail_index_open()'). + +Writing to index files requires transaction log to be exclusively locked first. +This way the index locking only has to worry about existing read locks. The +locking works by first trying to lock the index with the standard locking +method, but if it couldn't acquire the lock in two seconds, it'll fallback to +copying the index file to a temporary file, and when unlocking it'll 'rename()' +the temporary file over the 'dovecot.index' file. Note that this is safe only +because of the exclusive transaction log lock. This way the writers are never +blocked by readers who are allowed to keep the shared lock as long as they +want. + +The copy-locking is used always when doing anything that could corrupt the +index file if it crashed in the middle of an operation. For example if the +header or record size changes, or if messages are expunged. New messages can be +appended however, because the message count in the header is updated last. +Expunging the last messages would probably be safe also (because only the +header needs updating), but it's not done currently. + +The index file should never be directly modified. Everything should go through +the transaction log, and the only time the index needs to be write-locked is +when transactions are written to it. + +Currently the index file is updated whenever the backend mailbox is +synchronized. This isn't necessary, because an old index file can be updated +using the transaction log. In future there could be some smarter decisions +about when writing to the index isn't worth the extra disk writes. + +Header +------ + +---%<------------------------------------------------------------------------- +struct mail_index_header { + uint8_t major_version; + uint8_t minor_version; + + uint16_t base_header_size; + uint32_t header_size; + uint32_t record_size; + + uint8_t compat_flags; + uint8_t unused[3]; + + uint32_t indexid; + uint32_t flags; + + uint32_t uid_validity; + uint32_t next_uid; + + uint32_t messages_count; + uint32_t unused_old_recent_messages_count; + uint32_t seen_messages_count; + uint32_t deleted_messages_count; + + uint32_t first_recent_uid; + uint32_t first_unseen_uid_lowwater; + uint32_t first_deleted_uid_lowwater; + + uint32_t log_file_seq; + uint32_t log_file_tail_offset; + uint32_t log_file_head_offset; + + uint64_t unused_old_sync_size; + uint32_t unused_old_sync_stamp; + + uint32_t day_stamp; + uint32_t day_first_uid[8]; +} +---%<------------------------------------------------------------------------- + +Fields that won't change without recreating the index: + +major_version: + If this doesn't match 'MAIL_INDEX_MAJOR_VERSION', don't try to read the + index. Dovecot recreates the index file then. + +minor_version: + If this doesn't match 'MAIL_INDEX_MINOR_VERSION' there are some backwards + compatible changes in the index file (typically header fields). Try to + preserve the headers and the minor version when updating the index file. + +base_header_size: + Extension headers begin after the base headers. This is normally the same as + 'sizeof(struct mail_index_header)'. + +header_size: + Records begin after base and extension headers. + +record_size: + Size of each record and its extensions. Initially the same as 'sizeof(struct + mail_index_record)'. + +compat_flags: + Currently there is just one compatibility flag: + 'MAIL_INDEX_COMPAT_LITTLE_ENDIAN'. Dovecot doesn't try to bother to read + different endianess files, they're simply recreated. + +indexid: + Unique index file ID. This is used to make sure that the main index, + transaction log and cache file are all part of the same index. + +Header flags: + +MAIL_INDEX_HDR_FLAG_CORRUPTED: + Set whenever the index file is found to be corrupted. If the reader notices + this flag, it shouldn't try to continue using the index. + +MAIL_INDEX_HDR_FLAG_HAVE_DIRTY: + This index has records with 'MAIL_INDEX_MAIL_FLAG_DIRTY' flag set. + +MAIL_INDEX_HDR_FLAG_FSCK: + Call 'mail_index_fsck()' as soon as possible. This flag isn't actually set + anywhere currently. + +Message UIDs and counters: + +uid_validity: + IMAP UIDVALIDITY field. Initially can be 0, but after it's set we don't + currently try to even handle the case of UIDVALIDITY changing. It's done by + marking the index file corrupted and recreating it. That's a bit ugly, but + typically the UIDVALIDITY never changes. + +next_uid: + UID given to the next appended message. Only increases. + +messages_count: + Number of records in the index file. + +recent_messages_count: + Number of records with 'MAIL_RECENT' flag set. + +seen_messages_count: + Number of records with 'MAIL_SEEN' flag set. + +deleted_messages_count: + Number of records with 'MAIL_DELETED' flag set. + +first_recent_uid_lowwater: + There are no UIDs lower than this with 'MAIL_RECENT' flag set. + +first_unseen_uid_lowwater: + There are no UIDs lower than this *without* 'MAIL_SEEN' flag set. + +first_deleted_uid_lowwater: + There are no UIDs lower than this with 'MAIL_DELETE' flag set. + +The lowwater fields are used to optimize searching messages with/without a +specific flag. + +Fields related to syncing: + +log_file_seq: + Log file the log_*_offset fields point to. + +log_file_int_offset, log_file_ext_offset: + All the internal/external transactions before this offset in the log file are + synced to the index. External transactions are synced more often than + internal, so 'log_file_int_offset' <= 'log_file_ext_offset'. + +sync_size, sync_stamp: + Used by the mailbox backends to store their synchronization information. Some + day these should be removed and replaced with extension headers. + +Then there are day fields: + +day_stamp: + UNIX timestamp to the beginning of the day when new records were last added + to the index file. + +day_first_uid[8]: + These fields are updated when 'day_stamp' < today. The [0..6] are first moved + to [1..7], then [0] is set to the first appended UID. So they contain the + first UID of the day for last 8 days when messages were appended. + +The 'day_first_uid[]' fields are used by cache file compression to decide when +to drop 'MAIL_CACHE_DECISION_TEMP' data. + +Extension headers +----------------- + +After the base header comes a list of extensions and their headers. The first +extension begins from 'mail_index_header.base_header_size' offset. The second +begins after the first one's 'data[]' and so on. The extensions always begin +64bit aligned however, so you may need to skip a few bytes always. Read the +extensions as long as the offset is smaller than +'mail_index_header.header_size'. + +---%<------------------------------------------------------------------------- +struct mail_index_ext_header { + uint32_t hdr_size; /* size of data[] */ + uint32_t reset_id; + uint16_t record_offset; + uint16_t record_size; + uint16_t record_align; + uint16_t name_size; + /* unsigned char name[name_size] */ + /* unsigned char data[hdr_size] (starting 64bit aligned) */ +}; +---%<------------------------------------------------------------------------- + +'reset_id', record offset, size and alignment is explained in +<Design.Indexes.TransactionLog.txt>'s 'struct mail_transaction_ext_intro'. + +Records +------- + +There are 'hdr.messages_count' records in the file. Each record contains at +least two fields: Record UID and flags. The UID is always increasing for the +records, so it's possible to find a record by its UID with binary search. The +record size is specified by 'mail_index_header.record_size'. + +The flags are a combination of 'enum mail_flags' and 'enum +mail_index_mail_flags'. There exists only one index flag currently: +'MAIL_INDEX_MAIL_FLAG_DIRTY'. If a record has this flag set, it means that the +mailbox syncing code should ignore the flag in the mailbox and use the flag in +the index file instead. This is used for example with mbox and +'mbox_lazy_writes=yes'. It also allows having modifiable flags for read-only +mailboxes. + +The rest data is stored in record extensions. + +Keywords +-------- + +The keywords are stored in record extensions, but for better performance and +lower disk space usage in transaction logs, they are quite tightly integrated +to the index file code. + +The list of keywords is stored in "keywords" extension header: + +---%<------------------------------------------------------------------------- +struct mail_index_keyword_header { + uint32_t keywords_count; + /* struct mail_index_keyword_header_rec[] */ + /* char name[][] */ +}; +struct mail_index_keyword_header_rec { + uint32_t unused; /* for backwards compatibility */ + uint32_t name_offset; /* relative to beginning of name[] */ +}; +---%<------------------------------------------------------------------------- + +The unused field originally contained 'count' field, but while writing this +documentation I noticed it's not actually used anywhere. Apparently it was +added there accidentally. It'll be removed in later versions. + +So there exists 'keywords_count' keywords, each listed in a NUL-terminated +string beginning from 'name_offset'. + +Since crashing in the middle of updating the keywords list pretty much breaks +the keywords, adding new keywords causes the index file to be always copied to +a temporary file and be replaced. + +The keywords in the records are stored in a "keywords" extension bitfield. So +the nth bit in the bitfield points to the nth keyword listed in the header. + +It's not currently possible to safely remove existing keywords. + +Extensions +---------- + +The extensions only specify their wanted size and alignment, the index file +syncing code is free to assign any offset inside the record to them. The +extensions may be reordered at any time. + +Dovecot's current extension ordering code works pretty well, but it's not +perfect. If the extension size isn't the same as its alignment, it may create +larger records than necessary. This will be fixed later. + +The records size is always divisible by the maximum alignment requirement. This +isn't strictly necessary either, so it could be fixed later as well. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.TransactionLog.txt b/doc/wiki/Design.Indexes.TransactionLog.txt new file mode 100644 index 0000000..e86751b --- /dev/null +++ b/doc/wiki/Design.Indexes.TransactionLog.txt @@ -0,0 +1,265 @@ +Transaction log +=============== + +The transaction log is a bit similar to transaction logs in databases. All the +updates to the main index files are first written to the transaction log, and +only after that the main index file is updated. There are several advantages to +this: + + * It provides atomic transactions: The transaction either succeeds, or it + doesn't. For example if a transaction sets a flag to one message and removes + it from another, it's guaranteed that both changes happen. + * When updating the changes to the main index file, the last thing that's + done is to update the "transaction log position" in the header. So if + Dovecot crashes after having updated only the first flag, the next time + the mailbox is opened both of the changes are done all over again. + * It allows another process to quickly see what changes have been made. For + example IMAP needs to get a list of external changes after each command. + * This is also important when storing the index files in NFS or in a + clustered filesystem. Instead of re-reading the whole index file after + each external change, Dovecot can simply read the new changes from the + transaction log and apply them to the in-memory copy of the main index. + In-memory caching of 'dovecot.index.cache' file also relies on the + transaction log telling what parts of the file has changed. + * In future the transaction logs can be somewhat easily used to implement + replication. + +Internal vs. external +--------------------- + +Transactions are either internal or external. The difference is that external +transactions describe changes that were already made to the mailbox, while +internal transactions are commands to do something to the mailbox. When +beginning to synchronize a mailbox with index files, the index file is first +updated with all the external changes, and the uncommitted internal +transactions are applied on top of them. + +When synchronizing the mailbox, using the synchronization transaction writes +only external transactions. Also if the index file is updated when saving new +mails to the mailbox, the append transactions must be external. This is because +the changes are already in the mailbox at the time the transaction is read. + +Reading and writing +------------------- + +Reading transaction logs doesn't require any locking at all. Writing is +exclusively locked using the index files' default lock method (as specified by +the 'lock_method' setting). + +A new log is created by first creating a 'dovecot.index.log.newlock' dotlock +file. Once you have the dotlock, check again that the 'dovecot.index.log' +wasn't created (or recreated) by another process. If not, go ahead and write +the log header to the dotlock file and finally 'rename()' it to +'dovecot.index.log'. + +Currently there doesn't exist actual transaction boundaries in the log file. +All the changes in a transaction are simply written as separate records to the +file. Each record begins with a 'struct mail_transaction_header', which +contains the record's size and type. The size is in <lockless integer format> +[Design.Indexes.txt]. + +The first transaction record is written with the size field being 0. Once the +whole transaction has been written, the 0 is updated with the actual size. This +way the transaction log readers won't see partial transactions because they +stop at the size=0 if the transaction isn't fully written yet. + +Note that because there are no transaction boundaries, there's a small race +condition here with mmap()ed log files: + + 1. Process A: write() half of the transaction + 2. Process B: mmap() the file. + 3. Process A: write() the rest of the transaction, updating the size=0 also + 4. Process B: parse the log file. it'll go past the original size=0 because + the size had changed in the mmap, but it stops in the middle of the + transaction because the mmap size doesn't contain the whole transaction + +This probably isn't a big problem, because I've never seen this happen even +with stress tests. Should be fixed at some point anyway. + +Header +------ + +The transaction log's header never changes, except the indexid field may be +overwritten with 0 if the log is found to be corrupted. The fields are: + +major_version: + If this doesn't match 'MAIL_TRANSACTION_LOG_MAJOR_VERSION', don't try to + parse it. If Dovecot sees this, it'll recreate the log file. + +minor_version: + If this doesn't match 'MAIL_TRANSACTION_LOG_MINOR_VERSION', the log file + contains some backwards compatible changes. Currently you can just ignore + this field. + +hdr_size: + Size of the log file's header. Use this instead of 'sizeof(struct + mail_transaction_log_header)', so that it's possible to add new fields and + still be backwards compatible. + +indexid: + This field must match to main index file's indexid field. + +file_seq: + The file's creation sequence. Must be increasing. + +prev_file_seq, prev_file_offset: + Contains the sequence and offset of where the last transaction log ended. + When transaction log is rotated and the reader's "sync position" still points + to the previous log file, these fields allow it to easily check if there had + been any more changes in the previous file. + +create_stamp: + UNIX timestamp when the file was created. Used in determining when to rotate + the log file. + +Record header +------------- + +The transaction record header ('struct mail_transaction_header') contains +'size' and 'type' fields. The 'size' field is in <lockless integer format> +[Design.Indexes.txt]. A single transaction record may contain multiple changes +of the same type, although some types don't allow this. Because the size of the +transaction record for each type is known (or can be determined from the +type-specific record contents), the 'size' field can be used to figure out how +many changes need to be done. So for example a record can contain: + + * 'struct mail_transaction_header { type = MAIL_TRANSACTION_APPEND, size = + sizeof(struct mail_index_record) * 2 }' + * 'struct mail_index_record { uid = 1, flags = 0 } ' + * 'struct mail_index_record { uid = 2, flags = 0 } ' + +UIDs +---- + +Many record types contain 'uint32_t uid1, uid2' fields. This means that the +changes apply to all the messages in uid1..uid2 range. The messages don't +really have to exist in the range, so for example if the first messages in the +mailbox had UIDs 1, 100 and 1000, it would be possible to use uid1=1, uid2=1000 +to describe changes made to these 3 messages. This also means that it's safe to +write transactions describing changes to messages that were just expunged by +another process (and already written to the log file before our changes). + +Appends +------- + +As described above, the appends must be in external transactions. The append +transaction's contents is simply the 'struct mail_index_record', so it contains +only the message's UID and flags. The message contents aren't written to +transaction log. Also if the message had any keywords when it was appended, +they're in a separate transaction record. + +Expunges +-------- + +Because expunges actually destroy messages, they deserve some extra protection +to make it less likely to accidentally expunge wrong messages in case of for +example file corruption. The expunge transactions must have +'MAIL_TRANSACTION_EXPUNGE_PROT' ORed to the transaction type field. If an +expunge type is found without it, assume a corrupted transaction log. + +Flag changes +------------ + +The flag changes are described in: + +---%<------------------------------------------------------------------------- +struct mail_transaction_flag_update { + uint32_t uid1, uid2; + uint8_t add_flags; + uint8_t remove_flags; + uint16_t padding; +}; +---%<------------------------------------------------------------------------- + +The 'padding' is ignored completely. A single flag update structure can add new +flags or remove existing flags. Replacing all the files works by setting +'remove_flags = 0xFF' and the 'add_flags' containing the new flags. + +Keyword changes +--------------- + +Specific keywords can be added or removed one keyword at a time: + +---%<------------------------------------------------------------------------- +struct mail_transaction_keyword_update { + uint8_t modify_type; /* enum modify_type : MODIFY_ADD / MODIFY_REMOVE +*/ + uint8_t padding; + uint16_t name_size; + /* unsigned char name[]; + array of { uint32_t uid1, uid2; } + */ +}; +---%<------------------------------------------------------------------------- + +There is padding after 'name[]' so that uid1 begins from a 32bit aligned +offset. + +If you want to replace all the keywords (eg. IMAP's 'STORE 1:* FLAGS (keyword)' +command), you'll first have to remove all of them with +'MAIL_TRANSACTION_KEYWORD_RESET' and then add the new keywords. + +Extensions +---------- + +Extension records allow creating and updating extension-specific header and +message record data. For example messages' offsets to cache file or mbox file +are stored in extensions. + +Whenever using an extension, you'll need to first write +'MAIL_TRANSACTION_EXT_INTRO' record. This is a bit kludgy and hopefully will be +replaced by something better in future. The intro contains: + +---%<------------------------------------------------------------------------- +struct mail_transaction_ext_intro { + /* old extension: set ext_id. don't set name. + new extension: ext_id = (uint32_t)-1. give name. */ + uint32_t ext_id; + uint32_t reset_id; + uint32_t hdr_size; + uint16_t record_size; + uint16_t record_align; + uint16_t unused_padding; + uint16_t name_size; + /* unsigned char name[]; */ +}; +---%<------------------------------------------------------------------------- + +If the extension already exists in the index file (it can't be removed), you +can use the 'ext_id' field directly. Otherwise you'll need to give a name to +the extension. It's always possible to just give the name if you don't know the +existing extension ID, but this uses more space of course. + +'reset_id' contains kind of a "transaction validity" field. It's updated with +'MAIL_TRANSACTION_EXT_RESET' record, which also causes the extension records' +contents to be zeroed. If an introduction's 'reset_id' doesn't match the last +EXT_RESET, it means that the extension changes are stale and they must be +ignored. For example: + + * 'dovecot.index.cache' file's 'file_seq' header is used as a 'reset_id'. + Initially it's 1. + * Process A: Begins a cache transaction, updating some fields in it. + * Process B: Decides to compress the cache file, and issues a 'reset_id = 2' + change. + * Process A: Commits the transaction with 'reset_id = 1', but the cache file + offsets point to the old file, so the changes must be ignored. + +'hdr_size' specifies the number of bytes the extension wants to have in the +index file's header.'record_size' specifies the number of bytes it wants to use +for each record. The sizes may grow or shrink any time.'record_align' contains +the required alignmentation for the field. For example if the extension +contains a 32bit integer, you want it to be 32bit aligned so that the process +won't crash in CPUs which require proper alignmentation. Then again if you want +to access the field as 4 bytes, the alignmentation can be 1. + +Extension record updates typically are message-specific, so the changes must be +done for each message separately: + +---%<------------------------------------------------------------------------- +struct mail_transaction_ext_rec_update { + uint32_t uid; + /* unsigned char data[]; */ +}; +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.txt b/doc/wiki/Design.Indexes.txt new file mode 100644 index 0000000..00b55a2 --- /dev/null +++ b/doc/wiki/Design.Indexes.txt @@ -0,0 +1,70 @@ +Dovecot's index files +===================== + +Dovecot's index files consist of three different files: + + * <Main index file> [Design.Indexes.MainIndex.txt] ('dovecot.index') + * <Transaction log> [Design.Indexes.TransactionLog.txt] ('dovecot.index.log' + and 'dovecot.index.log.2') + * <Cache file> [Design.Indexes.Cache.txt] ('dovecot.index.cache') + +See <IndexFiles.txt> for more generic information about what they contain and +why. + +The index files can be accessed using <mail-index.h API> +[Design.Indexes.MailIndexApi.txt]. + +Locking +------- + +The index files are designed so that readers cannot block a writer, and write +locks are always short enough not to cause other processes to wait too long. +Dovecot v0.99's index files didn't do this, and it was common to get lock +timeouts when using multiple connections to the same large mailbox. + +The main index file is the only file which has read locks. They can however +block the writer only for two seconds (and even this could be changed to not +block at all). The writes are locked only for the duration of the mailbox +synchronization. + +Transaction logs don't require read locks. The writing is locked for the +duration of the mailbox synchronization, and also for single transaction +appends. + +Cache files doesn't require read locks. They're locked for writing only for the +duration of allocating space inside the file. The actual writing inside the +allocated space is done without any locks being held. + +In future these could be improved even further. For example there's no need to +keep any index files locked while synchronizing, as long the mailbox backend +takes care of the locking issues. Also writing to transaction log could work in +a similar way to cache files: Lock, allocate space, unlock, write. + +Lockless integers +----------------- + +Dovecot uses several different techniques to allow reading files without +locking them. One of them uses fields in a "lockless integer" format. Initially +these fields have "unset" value. They can be set to a wanted value in range +0..2^28 (with 32bit fields) once, but they cannot be changed. It would be +possible to set them back to "unset", but setting them the second time isn't +safe anymore, so Dovecot never does this. + +The lockless integers work by allocating one bit from each byte of the value to +"this value is set" flag. The reader then verifies that the flag is set for the +value's all bytes. If all of them aren't set, the value is still "unset". +Dovecot uses the highest bit for this flag. So for example: + + * 0x00000000: The value is unset + * 0xFFFF7FFF: The value is unset, because one of the bytes didn't have the + highest bit set + * 0xFFFFFFFF: The value is 2^28-1 + * 0x80808080: The value is 0 + * 0x80808180: The value is 0x80 + +Dovecot contains 'mail_index_uint32_to_offset()' and +'mail_index_offset_to_uint32()' functions to translate values between integers +and lockless integers. The "unset" value is returned as 0, so it's not possible +to differentiate between "unset" and "set" 0 values. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.InputStreams.txt b/doc/wiki/Design.InputStreams.txt new file mode 100644 index 0000000..c64d4cb --- /dev/null +++ b/doc/wiki/Design.InputStreams.txt @@ -0,0 +1,207 @@ +Input Streams +============= + +'lib/istream.h' describes Dovecot's input streams. Input streams can be stacked +on top of each others as many times as wanted. + +Input streams actually reading data: + + * file: Read data from fd using 'pread()' for files and 'read()' for + non-files. + * unix: Read data from UNIX socket. Similar to file, but supports receiving + file descriptors. + * mmap: Read data from file using 'mmap()'. This usually seems to be slower + than just using it with 'read()', so this input stream is probably quite + unnecessary. + * data: Read data from memory. + +Input stream filters: + + * concat: Concatenate multiple input streams together + * chain: Chain multiple input streams together. Similar to istream-concat, but + more istreams can be added after initialization and EOF needs to be + explicitly added. + * seekable: Make a number of (possibly non-seekable) input streams into a + single seekable input stream. If all of the input streams are already + seekable, a concat stream is created instead. + * Usually the only non-seekable input streams are non-file fds, such as + pipes or sockets. + * crlf: Change all newlines to either LFs or CRLFs, by adding or removing CRs + as necessary. + * limit: Limit input stream's length, so after reading a given number of bytes + it returns EOF. + * sized: Require istream's length to be exactly the given size, or the last + read returns error. + * timeout: Fail the read when given timeout is reached. + * try: Read from the first input stream that doesn't fail with EINVAL. + * tee: Fork an input stream to multiple streams that can be read + independently. + * multiplex: Multiplex-iostreams support multiple iostream channels inside a + single parent istream. + * callback: Build an input stream by calling callback functions that return + the data. + * base64-encoder, base64-decoder: Encode/decode base64. + * failure-at: Insert a failure at the specified offset. This can be useful for + testing. + * hash: Calculate hash of the istream while it's being read. + * lib-mail/dot: Read SMTP-style DATA input where the input ends with an empty + "." line. + * lib-mail/header-filter: Add/remove/modify email headers. + * lib-compression/*: Read zlib/bzlib/lz4/lzma compressed data. + +Reading +------- + +'i_stream_read()' tries to read more data into the stream's buffer. It returns: + + * -2: Nothing was read, because buffer is full. + * -1: Either input reached EOF, or read failed and stream_errno was set. + * 0: Input stream is non-blocking, and no more input is available now. + * >0: Number of bytes read. + +Reading from a stream doesn't actually go forward in the stream, that needs to +be done manually with 'i_stream_skip()'. This makes it easy to read full data +records into the stream directly instead of creating separate buffers. For +example when reading line-based input you can keep reading input into the +stream until you find LF and then just access the string directly from the +input buffer. There are actually helper functions for +this:'i_stream_next_line()' attempts to return the next line if available, +'i_stream_read_next_line()' does the same but does a read to try to get the +data. + +Because more and more data can be read into the buffer, the buffer size is +typically limited, and once this limit is reached read returns -2. The buffer +size is usually given as parameter to the 'i_stream_create_*()', filters use +their parent stream's buffer size. The buffer size can be also changed with +'i_stream_set_max_buffer_size()'. Figuring out what the buffer size should be +depends on the situation. It should be large enough to contain all valid input, +but small enough that users can't cause a DoS by sending a too large record and +having Dovecot eat up all the memory. + +Once read returns -1, the stream has reached EOF. 'stream->eof=TRUE' is also +set. In this situation it's important to remember that there may still be data +available in the buffer. If 'i_stream_have_bytes_left()' returns FALSE, there +really isn't anything left to read. + +Whenever i_stream_read() returns >0, all the existing pointers are potentially +invalidated. v2.3+: When i_stream_read() returns<= 0, the data previously +returned by i_stream_get_data() are still valid, preserved in "snapshots". +(<v2.3 may or may not have invalidated them.) + +Example: + +---%<------------------------------------------------------------------------- +/* read line-based data from file_fd, buffer size has no limits */ +struct istream *input = i_stream_create_fd(file_fd, (size_t)-1, FALSE); +const char *line; + +/* return the last line also even if it doesn't end with LF. + this is generally a good idea when reading files (but not a good idea + when reading commands from e.g. socket). */ +i_stream_set_return_partial_line(input, TRUE); +while ((line = i_stream_read_next_line(input)) != NULL) { + /* handle line */ +} +i_stream_destroy(&input); +---%<------------------------------------------------------------------------- + +Internals +--------- + +'lib/istream-internal.h' describes the internal API that input streams need to +implement. The methods that need to be implemented are: + + * 'read()' is the most important function. It can also be tricky to get it + completely bug-free. See the existing unit tests for other istreams and try + to test the edge cases as well (such as ability to read one byte at a time + and also with max buffer size of 1). When it needs to read from parent + streams, try to use 'i_stream_read_memarea(parent)' if possible so a new + snapshot isn't unnecessarily created (see the snapshot discussion below). + * 'seek(v_offset, mark)' seeks to given offset. The 'mark' parameter is + necessary only when it's difficult to seek backwards in the stream, such as + when reading compressed input. + * 'sync()' removes everything from internal buffers, so that if the underlying + file has changed the changes get noticed immediately after sync. + * 'get_size(exact)' returns the size of the input stream, if it's known. If + 'exact=TRUE', the returned size must be the same how many bytes can be read + from the input. If 'exact=FALSE', the size is mainly used to compare against + another stat to see if the underlying input had changed. For example with + compressed input the size could be the compressed size. + * 'stat(exact)' stats the file, filling as much of the fields as makes sense. + 'st_size' field is filled the same way as with 'get_size()', or set to -1 if + it's unknown. + * 'snapshot(prev_snapshot)' creates a snapshot of the data that is currently + available via i_stream_get_data(), merges it with prev_snapshot (if any) and + returns the merged snapshot (see below more more details). + +There are some variables available: + + * 'buffer' contains pointer to the data. + * First 'skip' bytes of the buffer are already skipped over (with + 'i_stream_skip()' or seeking). + * Data up to 'pos' bytes (beginning after 'skip') in the buffer are available + with 'i_stream_get_data()'. If pos=skip, it means there is no available data + in the buffer. + +If your input stream needs a write buffer, you can use some of the common +helper functions and variables: + + * 'w_buffer' contain the pointer where you can write data. It should be kept + in sync with 'buffer'. + * 'buffer_size' specifies the buffer's size, and 'max_buffer_size' the max. + size the buffer can be grown to. + * 'i_stream_try_alloc(wanted_size, size_r)' can be used when you want to store + 'wanted_bytes' into 'w_buffer'. If the buffer isn't large enough for it, + it's grown if possible. The buffer isn't grown above the stream's max buffer + size. The returned 'size_r' specifies how many bytes are actually available + for writing at 'stream->w_buffer + stream->pos'. + * 'i_stream_alloc(size) is like 'i_stream_try_alloc()', except it always + succeeds allocating'size` bytes, even if it has to grow the buffer larger + then the stream's max buffer size. + * Lower-level memory allocation functions: + * 'i_stream_w_buffer_realloc(old_size)' reallocates 'w_buffer' to the + current 'buffer_size'. If memarea's refcount is 1, this can be done with + 'i_realloc()', otherwise new memory is allocated. + * 'i_stream_grow_buffer(bytes)' grows the 'w_buffer' by the given number of + bytes, if possible. It won't reach the stream's current max buffer size. + The caller must verify from 'buffer_size' how large the buffer became as + a result of this call. + * 'i_stream_compress()' attempts to compress the current 'w_buffer' by + removing already-skipped data with 'memmove()'. If 'skip' is 0, it does + nothing. Note that this function must not be called if 'memarea' has + refcount>1. Otherwise that could be modifying a snapshotted memarea. + +The snapshots have made implementing slightly more complicated than earlier. +There are a few different ways to implement istreams: + + * Always point 'buffer=w_buffer' and use 'i_stream_try_alloc()' and/or + 'i_stream_alloc()' to allocate the 'w_buffer'. The generic code will handle + all the snapshotting. Use 'i_stream_read_memarea()' to read data from parent + stream so multiple snapshots aren't unnecessarily created. + * Guarantee that if 'read()' returns <=0, the existing 'buffer' will stay + valid. Use 'ISTREAM_CREATE_FLAG_NOOP_SNAPSHOT' flag in 'i_stream_create()' + so your filter stream isn't unnecessarily snapshotted (or causing a panic + due to missing 'snapshot()' implementation). + * One way of doing this with filter streams is to read from the parent + stream via 'i_stream_read(parent)' and always use + 'buffer=i_stream_get_data(parent)'. The parent's snapshotting guarantees + that the buffer will stay valid. + * Implement the 'snapshot()' yourself in the stream. You'll need to create a + new memarea of the current data available via 'i_stream_get_data()' and it + must not change, i.e. most likely you'll need to duplicate the allocated + memory. Create a new 'struct istream_snapshot' and assign the allocated + memarea to its 'old_memarea'. Fill 'prev_snapshot' field and return your new + snapshot. The snapshot will be freed by the generic istream code either when + the next 'read()' returns >0 or when the istream is destroyed. + * Filter streams that only pass through parent stream's contents without + changes can just point to the parent stream. The default snapshotting causes + the parent to be snapshotted, so the filter stream can simply use + 'i_stream_read_memarea()' and point to the parent's buffer. + +When Dovecot is configured with '--enable-devel-checks', 'i_stream_read()' will +verify that the first and the last two bytes of the buffer didn't unexpectedly +change due to a 'read()'. While developing istream changes you should use this +to make sure the istream is working properly. Running the istream unit test +also via valgrind can also be used to verify that the buffer wasn't freed. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Lua.txt b/doc/wiki/Design.Lua.txt new file mode 100644 index 0000000..805e00b --- /dev/null +++ b/doc/wiki/Design.Lua.txt @@ -0,0 +1,249 @@ +Dovecot Lua support +=================== + +Since v2.3.0 dovecot supports Lua scripting. Dovecot supports lua 5.0 or newer. +See also: + + * <Lua based authentication> [AuthDatabase.Lua.txt]. Since v2.3.0 + * <Lua push notifications> [Plugins.PushNotification.txt]. Since v2.3.4 + +lib-lua +------- + +Dovecot provides a lib-lua internal helper as part of libdovecot.so. It has +facilities for loading scripts from various sources, and also helps with +reusing scripts by keeping track of which scripts are loaded. Each script has +it's own memory pool, which is guaranteed to be released when script is +unloaded. + +When script is loaded, *script_load* function is called if found. This can +return non-zero to indicate that the script has a problem. + +C API +----- + + * 'dlua_register_dovecot(script)' + +Register dovecot variable. This item can also be extended by context specific +tables, like authentication database adds *dovecot.auth*. + + * 'dlua_push_event(event)' + +Pushes an Dovecot Event to stack. + +Lua API +------- + + * 'i_debug(text)' + * 'i_error(text)' + * 'i_info(text)' + * 'i_warning(text)' + +Functions within dovecot variable that do generic logging + +Event functions are available from v2.3.4 on + + * 'dovecot.event()' + * 'dovecot.event(parent)' + +Creates a new event for dovecot + +object event +------------ + +Note: + + * object event_passthrough has same API, except the 'passthrough_event' method + is not present. + +Functions: + + * 'append_log_prefix("prefix")' - set prefix to append + * 'replace_log_prefix("prefix")' - replace append prefix + * 'set_name("name")' - set name for event + * 'add_str("key","value")' - add a key-value pair to event + * 'add_int("key",1)' - add a key-value pair to event + * 'add_timeval("key",seconds)' - add a key-value pair to event + * 'inc_int("key",1)' - increment key-value pair + * 'log_debug("message")' - emit debug message + * 'log_info("message")' - emit info message + * 'log_warning("message")' - emit warning message + * 'log_error("message")' - emit error message + * 'passthrough_event()' - returns an passthrough event. A log message *must + be* logged or else a panic will occur. + +mail-lua +-------- + +Available from v2.3.4 on + +mail-lua is a plugin that can be loaded to provide API for mail storage Lua +plugins. Mail-lua provides a common script to be used in mail storage instead +of per-plugin scripts. + +C API +----- + + * 'dlua_register_mail_storage(script)' + +Register storage Lua interface to script context + + * 'bool mail_lua_plugin_get_script(user, script_r)' + +Returns script context if available. If FALSE is returned, no Lua script has +been loaded, and you should optionally deal this yourself. + + * 'dlua_push_mail_user(script, user)' + +Pushes a mail user on top of stack. + + * 'dlua_push_mailbox(script, box)' + +Pushes a mailbox on top of stack. + + * 'dlua_push_mail(script, mail)' + +Pushes a mail on top of stack. + +Lua API +------- + +When mail user is created, a script is loaded if present as *mail_lua_script* +and *mail_user_created* is called if present in script. + +On deinitialization, *mail_user_deinit_pre* is called first, if present, +followed by *mail_user_deinit*. + +dovecot.storage +--------------- + +Following constants are specified + + * 'STATUS_MESSAGES' + * 'STATUS_RECENT' + * 'STATUS_UIDNEXT' + * 'STATUS_UIDVALIDITY' + * 'STATUS_UNSEEN' + * 'STATUS_FIRST_UNSEEN_SEQ' + * 'STATUS_KEYWORDS' + * 'STATUS_HIGHESTMODSEQ' + * 'STATUS_PERMANENT_FLAGS' + * 'STATUS_FIRST_RECENT_UID' + * 'STATUS_HIGHESTPVTMODSEQ' + * 'MAILBOX_FLAG_READONLY' + * 'MAILBOX_FLAG_SAVEONLY' + * 'MAILBOX_FLAG_DROP_RECENT' + * 'MAILBOX_FLAG_NO_INDEX_FILES' + * 'MAILBOX_FLAG_KEEP_LOCKED' + * 'MAILBOX_FLAG_IGNORE_ACLS' + * 'MAILBOX_FLAG_AUTO_CREATE' + * 'MAILBOX_FLAG_AUTO_SUBSCRIBE' + * 'MAILBOX_SYNC_FLAG_FULL_READ' + * 'MAILBOX_SYNC_FLAG_FULL_WRITE' + * 'MAILBOX_SYNC_FLAG_FAST' + * 'MAILBOX_SYNC_FLAG_NO_EXPUNGES' + * 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT' + * 'MAILBOX_SYNC_FLAG_EXPUNGE' + * 'MAILBOX_SYNC_FLAG_FORCE_RESYNC' + +object mail_user +---------------- + +Meta: + + * has tostring + * is comparable (by username) + +Functions: + + * 'plugin_getenv(key)' - returns key from user plugin settings or userdb + environment + * 'var_expand(template)' - expands mail user variables (see <Variables.txt>) + * 'mailbox(name, flags)' - allocates a mailbox, flags optional + +Variables: + + * 'home' - home directory (if available) + * 'username' - user's name + * 'uid' - system uid + * 'gid' - system gid + * 'service' - IMAP/POP3/LMTP/LDA/... + * 'session_id' - Current session ID + * 'session_create_time' - When session was created + * 'nonexistent' - If user does not really exist + * 'anonymous' - If user is anonymous + * 'autocreated' - If user was automatically created internally for some + operation + * 'mail_debug' - If debugging is turned on + * 'fuzzy_search' - FIXME: Undocumented + * 'dsyncing' - If user is being dsync'd + * 'session_restored' - If this is a restored session + +object mailbox +-------------- + +Meta: + + * has tostring + * is comparable (by full mailbox name) + +Functions: + + * 'open()' - opens the mailbox + * 'close()' - closes the mailbox + * 'free()' - releases mailbox (must be done) + * 'sync(flags)' - synchronizes the mailbox (should usually be done, flags + optional) + * 'status(item,item,item...)' - returns requested mailbox status items as + table + +Variables: + + * 'vname' - Full mailbox name + * 'name' - Mailbox name + +table mailbox status +-------------------- + +Variables: + + * 'mailbox' - full name of mailbox + * 'messages' - number of messages + * 'recent' - number of \Recent messages + * 'unseen' - number of \Unseen messages + * 'uidvalidity' - current UID validity + * 'uidnext' - next UID + * 'first_unseen_seq' - first seqno of unseen mail + * 'first_recent_uid' - first UID of unseen mail + * 'highest_modseq' - highest modification sequence + * 'highest_pvt_modseq' - highest private modification sequence + * 'permanent_flags' - supported permanent flags as a bitmask + * 'flags' - supported flags as a bitmask + * 'permanent_keywords' - if permanent keywords are supported + * 'allow_new_keywords' - if new keywords can be added + * 'nonpermanent_modseqs' - whether non-permanent keywords are allowed + * 'no_modseq_tracking' - no modification sequence tracking + * 'have_guids' - whether GUIDs exist + * 'have_save_guids' - whether GUIDs can be saved + * 'have_only_guid128' - whether GUIDs are 128 bit always + * 'keywords' - table of current keywords + +object mail +----------- + +Meta: + + * has tostring + * is comparable (within same mailbox, by UID) + +Functions: + + * None yet + +Variables: + + * 'mailbox' - mailbox object + * 'seq' - Sequence number (can change) + * 'uid' - UID number (immutable) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.MailProcess.txt b/doc/wiki/Design.MailProcess.txt new file mode 100644 index 0000000..b583d46 --- /dev/null +++ b/doc/wiki/Design.MailProcess.txt @@ -0,0 +1,12 @@ +Mail Process Design +=================== + +FIXME: + + * IMAP/POP3 protocol specific code + * Mail storage API + * Index files under everything + * Plugins + * etc. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Memory.txt b/doc/wiki/Design.Memory.txt new file mode 100644 index 0000000..64ad013 --- /dev/null +++ b/doc/wiki/Design.Memory.txt @@ -0,0 +1,193 @@ +Memory Allocations +================== + +C language requires explicitly allocating and freeing memory. The main two +problems with this are: + + 1. A lot of allocations and frees cause memory fragmentation. The longer a + process runs, the more it could have leaked memory because there are tiny + unused free spots all around in heap. + 2. Freeing memory is easy to forget, causing memory leaks. Sometimes it can be + accidentally done multiple times, causing a potential security hole. A lot + of free() calls all around in the code also makes the code more difficult + to read and write. + +The second problem could be solved with Boehm garbage collector, which Dovecot +can use optionally (prior to 2.3), but it's not very efficient. It also doesn't +help with the first problem. + +To reduce the problems caused by these issues, Dovecot has several ways to do +memory management. + +Common Design Decisions +----------------------- + +All memory allocations (with some exceptions in data stack) return memory +filled with NULs. This is also true for new memory when growing an allocated +memory with realloc. The zeroing reduces accidental use of uninitialized memory +and makes the code simpler since there is no need to explicitly set all fields +in allocated structs to zero/NULL. (I guess assuming that this works correctly +for NULLs isn't strictly ANSI-C compliant, but I don't see this assumption +breaking in any system anyone would really use Dovecot.) The zeroing is cheap +anyway. + +In out-of-memory situations memory allocation functions die internally by +calling 'i_fatal_status(FATAL_OUTOFMEM, ..)'. There are several reasons for +this: + + * Trying to handle malloc() failures explicitly would add a lot of error + handling code paths and make the code much more complex than necessary. + * In most systems malloc() rarely actually fails because the system has run + out of memory. Instead the kernel will just start killing processes. + * Typically when malloc() does fail, it's because the process's address space + limit is reached. Dovecot enforces these limits by default. Reaching it + could mean that the process was leaking memory and it should be killed. It + could also mean that the process is doing more work than anticipated and + that the limit should probably be increased. + * Even with perfect out-of-memory handling, the result isn't much better + anyway than the process dying. User isn't any happier by seeing "out of + memory" error than "server disconnected". + +When freeing memory, most functions usually also change the pointer to NULL. +This is also the reason why most APIs' deinit functions take pointer-to-pointer +parameter, so that when they're done they can change the original pointer to +NULL. + +malloc() Replacements +--------------------- + +'lib/imem.h' has replacements for all the common memory allocation functions: + + * 'malloc', 'calloc' -> 'i_malloc()' + * 'realloc()' -> 'i_realloc()' + * 'strdup()' -> 'i_strdup()' + * 'free()' -> 'i_free' + * etc. + +All memory allocation functions that begin with 'i_' prefix require that the +memory is later freed with 'i_free()'. If you actually want the freed pointer +to be set to NULL, use 'i_free_and_null()'. Currently 'i_free()' also changes +the pointer to NULL, but in future it might change to something else. + +Memory Pools +------------ + +'lib/mempool.h' defines API for allocating memory through memory pools. All +memory allocations actually go through memory pools. Even the 'i_*()' functions +get called through 'default_pool', which by default is 'system_pool' but can be +changed to another pool if wanted. All memory allocation functions that begin +with 'p_' prefix have a memory pool parameter, which it uses to allocate the +memory. + +Dovecot has many APIs that require you to specify a memory pool. Usually (but +not always) they don't bother freeing any memory from the pool, instead they +assume that more memory can be just allocated from the pool and the whole pool +is freed later. These pools are usually alloconly-pools, but can also be data +stack pools. See below. + +Alloc-only Pools +---------------- + +'pool_alloconly_create()' creates an allocate-only memory pool with a given +initial size. + +As the name says, alloconly-pools only support allocating more memory. As a +special case its last allocation can be freed.'p_realloc()' also tries to grow +the existing allocation only if it's the last allocation, otherwise it'll just +allocates new memory area and copies the data there. + +Initial memory pool sizes are often optimized in Dovecot to be set large enough +that in most situations the pool doesn't need to be grown. To make this easier, +when Dovecot is configured with --enable-devel-checks, it logs a warning each +time a memory pool is grown. The initial pool size shouldn't of course be made +too large, so usually I just pick some small initial guessed value and later if +I get too many "growing memory pool" warnings I start growing the pool sizes. +Sometimes there's just no good way to set the initial pool size and avoid the +warnings, in that situation you can prefix the pool's name with MEMPOOL_GROWING +and it doesn't log warnings. + +Alloconly-pools are commonly used for an object that builds its state from many +memory allocations, but doesn't change (much of) its state. It's a lot easier +when you can do a lot of small memory allocations and when destroying the +object you'll just free the memory pool. + +Data Stack +---------- + +'lib/data-stack.h' describes the low-level data stack functions. Data stack +works a bit like C's control stack.'alloca()' is quite near to what it does, +but there's one major difference: In data stack the stack frames are explicitly +defined, so functions can return values allocated from data +stack.'t_strdup_printf()' call is an excellent example of why this is useful. +Rather than creating some arbitrary sized buffer and using snprintf(), which +might truncate the value, you can just use t_strdup_printf() without worrying +about buffer sizes being large enough. + +Try to keep the allocations from data stack small, since the data stack's +highest memory usage size is kept for the rest of the process's lifetime. The +initial data stack size is 32kB, which should be enough in normal use. If +Dovecot is configured with --enable-devel-checks, it logs a warning each time +the data stack needs to be grown. + +Stack frames are preferably created using T_BEGIN/T_END block, for example: + +---%<------------------------------------------------------------------------- +T_BEGIN { + string_t *str1 = t_str_new(256); + string_t *str2 = t_str_new(256); + /* .. */ +} T_END; +---%<------------------------------------------------------------------------- + +In the above example two strings are allocated from data stack. They get freed +once the code goes past T_END, that's why the variables are preferably declared +inside the T_BEGIN/T_END block so they won't accidentally be used after they're +freed. + +T_BEGIN and T_END expand to 't_push()' and 't_pop()' calls and they must be +synchronized. Returning from the block without going past T_END is going to +cause Dovecot to panic in next T_END call with "Leaked t_pop() call" error. + +Memory allocations have similar disadvantages to alloc-only memory pools. +Allocations can't be grown, so with the above example if str1 grows past 256 +characters, it needs to be reallocated, which will cause it to forget about the +original 256 bytes and allocate 512 bytes more. + +Memory allocations from data stack often begin with 't_' prefix, meaning +"temporary". There are however many other functions that allocate memory from +data stack without mentioning it. Memory allocated from data stack is usually +returned as a const pointer, so that the caller doesn't try to free it (which +would cause a compiler warning). + +When should T_BEGIN/T_END used and when not? This is kind of black magic. In +general they shouldn't be used unless it's really necessary, because they make +the code more complex. But if the code is going through loops with many +iterations, where each iteration is allocating memory from data stack, running +each iteration inside its own stack frame would be a good idea to avoid +excessive memory usage. It's also difficult to guess how public APIs are being +used, so I've tried to make such API functions use their own private stack +frames. Dovecot's ioloop code also wraps all I/O callbacks and timeout +callbacks into their own stack frames, so you don't need to worry about them. + +You can create temporary memory pools from data stack too. Usually you should +be calling 'pool_datastack_create()' to generate a new pool, which also tries +to track that it's not being used unsafely across different stack frames. Some +low-level functions can also use 'unsafe_data_stack_pool' as the pool, which +doesn't do such tracking. + +Data stack's advantages over malloc(): + + * FAST, most of the time allocating memory means only updating a couple of + pointers and integers. Freeing memory all at once also is a fast operation. + * No need to 'free()' each allocation resulting in prettier code + * No memory leaks + * No memory fragmentation + +It also has some disadvantages: + + * Allocating memory inside loops can accidentally allocate a lot of memory + * Memory allocated from data stack can be accidentally stored into a permanent + location and accessed after it's already been freed. + * Debugging invalid memory usage may be difficult using existing tools + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.OutputStreams.txt b/doc/wiki/Design.OutputStreams.txt new file mode 100644 index 0000000..97bc741 --- /dev/null +++ b/doc/wiki/Design.OutputStreams.txt @@ -0,0 +1,96 @@ +Output Streams +============== + +'lib/ostream.h' describes Dovecot's output streams. Output streams can be +stacked on top of each others as many times as wanted. + +Output streams actually writing data: + + * file: Write to given fd using 'pwrite()' for files and 'write()' for + non-files. + * unix: Write to given UNIX socket. Similar to file, but supports sending file + descriptors. + * buffer: Write to <buffer> [Design.Buffers.txt]. + +Output stream filters: + + * hash: Calculate hash of the ostream while it's being written. + * escaped: Write output escaped via callback. Built-in support for HEX and + JSON escaping. + * multiplex: Multiplex-iostreams support multiple iostream channels inside a + single parent istream. + * null: All the output is discarded. + * failure-at: Insert a failure at the specified offset. This can be useful for + testing. + * lib-mail/ostream-dot: Write SMTP-style DATA input where the output ends with + an empty "." line. + * lib-dcrypt/encrypt: Write encrypted data. + * lib-compression/*: Write zlib/bzlib/lz4/lzma compressed data. + +A typical life cycle for an ostream can look like: + + * create + * 'o_stream_cork()' + * 'o_stream_nsend*()' one or more times + * 'o_stream_uncork()' + * If necessary, check errors with 'o_stream_flush()' + * 'o_stream_cork()' + * 'o_stream_nsend*()' one or more times + * 'o_stream_uncork()' + * finalize the ostream with 'o_stream_finish()' + * optionally close the ostream with 'o_stream_close()' + * unref or destroy + +Once the ostream is finished, it can't be written to anymore. The +'o_stream_finish()' call writes any potential trailer that the ostream may have +(e.g. ostream-gz, ostream-encrypt, ostream-dot) while still allowing the caller +to check if the trailer write failed. After 'o_stream_finish()' is called, any +further write will panic. The ostreams that require a trailer will panic if +'o_stream_finish()' hasn't been called before the stream is destroyed, but +other ostreams don't currently require this. Still, it's not always easy to +know whether there might be ostreams that require the trailer, so if there's +any doubt, it's preferred to call 'o_stream_finish()' just before destroying +the ostream. + +Usually calling 'o_stream_finish()' will also finish its parent ostream. This +may or may not be wanted depending on the situation. For example ostream-dot +must be finished to write the last "." line, but ostream-dot is always a +sub-stream of something else that must not be finished yet. This is why +ostream-dot by default has called 'o_stream_set_finish_also_parent(FALSE)', so +finishing the ostream-dot won't finish the parent stream. Similarly +'connection.c' API sets 'o_stream_set_finish_via_child(FALSE)' so none of the +socket connections created via it will be finished even though one of their +sub-streams is finished. These functions may need to be called explicitly in +other situations. + +When doing a lot of writes, you can simplify the error handling by delaying the +error checking. Use the 'o_stream_nsend*()' functions and afterwards check the +error with 'o_stream_flush()' or 'o_stream_finish()'. If you forgot to do this +check before the ostream is destroyed, it will panic with:'output stream %s is +missing error handling' regardless of whether there is an error or not. If you +don't care about errors for the ostream (e.g. because it's a client socket and +there's nothing you can do about the write errors), you can use +'o_stream_set_no_error_handling()' to fully disable error checks. You can also +use 'o_stream_ignore_last_errors()' to ignore the errors so far, but not for +future writes. + +Writes are non-buffered by default. To add buffering, use 'o_stream_cork()' to +start buffering and 'o_stream_uncork()' to stop/flush. When output buffer gets +full, it's automatically flushed even while the stream is corked. The term +"cork" is used because with TCP connections the call actually sets/removes TCP +cork option. It's quite easy to forget to enable the corking with files, making +the performance worse. The corking/uncorking is done automatically when flush +callbacks are called. Using 'o_stream_uncork()' will trigger an automatic +'o_stream_flush()' but the error is ignored. This is why it acts similarly to +'o_stream_nsend*()', i.e. it requires another explicit 'o_stream_flush()', +'o_stream_finish()' or error ignoring before the ostream is destroyed. + +If output buffer's size isn't unlimited, the writes can also fail or be partial +because there's no more space in the buffer and 'write()' syscall is returning +'EAGAIN'. This of course doesn't happen with blocking fds (e.g. files), but you +need to handle this in some way with non-blocking network sockets. A common way +in Dovecot to handle this is to just use unlimited buffer sizes and after each +write check if the buffer size becomes too large, and when it does it stops +writing until more space is available. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.ParameterForwarding.txt b/doc/wiki/Design.ParameterForwarding.txt new file mode 100644 index 0000000..4413b58 --- /dev/null +++ b/doc/wiki/Design.ParameterForwarding.txt @@ -0,0 +1,65 @@ +Forwarding parameters in IMAP/POP3/LMTP/SMTP proxying +===================================================== + +Dovecot supports proxying various pieces of information and even variables for +various protocols when forwarding connection.It requires that the sender is +listed under 'trusted_networks'. For IMAP, it uses the 'ID' command, for other +protocols,'XCLIENT' is used. + +This feature is supported since v1.2, except for x-forward which was added in +v2.2.29. + +IMAP protocol +------------- + +For IMAP protocol, this is done by extending the ID command. The maximum length +of parameters to parse is 255 bytes.The parameters are forwarded as part of the +ID command key-value list. + +'5 ID (x-originating-ip "127.0.0.1" x-originating-port "143"... )' + +Supported parameters +-------------------- + + * 'x-originating-ip' - Client IP + * 'x-originating-port' - Client port + * 'x-connected-ip' - Server IP + * 'x-connected-port' - Server port + * 'x-proxy-ttl' - TTL which is reduced by each hop, loop prevention. When TTL + drops to 0, the connection is dropped. + * 'x-session-id' / 'x-session-ext-id' - Session ID to be used. + * 'x-forward-<variable_name>' - Forwarded variable, see <Variables.txt> + +POP3 protocol +------------- + +For POP3 protocol, this is done with custom 'XCLIENT' command which accepts a +space separated list of key=value parameters. + +Supported parameters +-------------------- + + * 'ADDR' - Client IP + * 'PORT' - Client port + * 'SESSION' - Session ID + * 'TTL' - TTL which is reduced by each hop, loop prevention. When TTL drops to + 0, the connection is dropped. + * 'FORWARD' - Base64 encoded key=value parameter to be stored. + +SMTP/LMTP protocol +------------------ + +See http://www.postfix.org/XCLIENT_README.html + + * 'ADDR' - Client IP, IPV6: prefix is needed for IPv6. + * 'PORT' - Client port + * 'SESSION' - Session ID + * 'HELO' - Original HELO/EHLO + * 'LOGIN' - Original LOGIN value + * 'TIMEOUT' - Original TIMEOUT + * 'TTL' - TTL which is reduced by each hop, loop prevention. When TTL drops to + 0, the connection is dropped. + * 'PROTO' - Forwarded protocol, can be one of SMTP, ESTMP, LMTP. + * 'FORWARD' - Base64 encoded key=value parameter to be stored. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Plugins.txt b/doc/wiki/Design.Plugins.txt new file mode 100644 index 0000000..7d28af4 --- /dev/null +++ b/doc/wiki/Design.Plugins.txt @@ -0,0 +1,62 @@ +Plugins +======= + +Plugins in Dovecot are really simple. They basically have two functions: + + * <plugin_name>_init(module) is called when 'module_dir_init()' is called. + * <plugin_name>_deinit() is called when 'module_dir_deinit()' or + 'module_dir_unload()' is called. + +The <plugin_name> is the short version of the plugin name, based on the +filename. For example if the filename is 'lib11_imap_quota_plugin.so', the +<plugin_name> is "imap_quota" and the init function to be called is +imap_quota_plugin_init(). + +Versioning +---------- + +Since different Dovecot versions can have different APIs, your plugin should +usually also define<plugin_name>_version, like: + +---%<------------------------------------------------------------------------- +const char *imap_quota_plugin_version = DOVECOT_ABI_VERSION; +---%<------------------------------------------------------------------------- + +If the version string in plugin doesn't match the version of the running +binary, the plugin loading fails. The DOVECOT_ABI_VERSION is defined in +Dovecot's 'config.h', which you're typically including. + +Dependencies +------------ + +Some plugins depend on another one. In some systems (but not all) it's possible +to handle this by giving a nicer error message than "symbol xyz not found". +There are two steps for this: + +First create <plugin_name>_dependencies array listing plugin names that the +plugin depends on, like: + +---%<------------------------------------------------------------------------- +const char *imap_quota_plugin_dependencies[] = { "quota", NULL }; +---%<------------------------------------------------------------------------- + +Then you'll also have to make the plugin .so binary link to the other plugins: + +---%<------------------------------------------------------------------------- +if PLUGIN_DEPS +lib11_imap_quota_plugin_la_LIBADD = \ + ../quota/lib10_quota_plugin.la +endif +---%<------------------------------------------------------------------------- + +PLUGIN_DEPS is set only if plugin dependencies are actually supported. +Otherwise the build might fail or plugin loading might fail. + +Once all this is done, trying to load imap_quota plugin without quota plugin +gives a nice error message: + +---%<------------------------------------------------------------------------- +Error: Can't load plugin imap_quota_plugin: Plugin quota must be loaded also +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Processes.txt b/doc/wiki/Design.Processes.txt new file mode 100644 index 0000000..189960c --- /dev/null +++ b/doc/wiki/Design.Processes.txt @@ -0,0 +1,137 @@ +Dovecot processes +================= + +Dovecot is split into multiple processes where each process does only one +thing. This is partially because it makes the code cleaner, but alsobecause it +allows setting up different privileges for each process. Themost important +processes are: + + * Master process (dovecot) + * Login processes (imap-login, pop3-login) + * Authentication process (dovecot-auth) + * Mail processes (imap, pop3) + +Master process +-------------- + +This process keeps all the other processes running. If a child process dies, +another one is restarted automatically. It always runs as root,unless you're +specifically running everything under a single normal UID. + +The master process reads the configuration file and exports the settings to +other processes via environment variables. + +All logging also goes through master process. This avoids problems with +rotating log files, as there's only a single process to send a signal toreopen +the log file. Also writing to the same log file (if not using syslog)isn't +necessarily safe to do in multiple processes concurrently. + +Making the logging go through master process also gives a couple of advantages +from security and reliability point of view: All log lines canbe prefixed with +the process's name and the username of the user who was logged in, withoutthe +possibility for the process itself to forge them. Flooding logs canalso be +prevented. By default Dovecot allows non-privileged processes towrite 10 lines +per second before it begins to delay reading their input,which finally causes +the badly behaving process to start blocking onwriting to stderr instead of +eating all the CPU and disk space. + +In the Dovecot 2.0 design, the master process is split to three parts: the +Masterprocess which does nothing more than keep the processes running, the +configprocess which handles reading the configuration file (supporting also eg. +SQL storages!) and the log process which handles the logging. + +Login processes +--------------- + +The login processes implement the required minimum of the IMAP and POP3 +protocolsbefore a user logs in successfully. There are separate processes (and +binaries) to handle IMAP and POP3 protocols. + +These processes are run with least possible privileges. Unfortunately the +default UNIX security model still allows them to do much more than theywould +have to: Accept new connections on a socket, connect to new UNIXsockets and +read and write to existing file descriptors. Still, the loginprocess is by +default run under a user account that has no special accessto anything, and +inside a non-writable chroot where only a couple of filesexist. Doing any +damage inside there should be difficult. + +When a new connection comes, one of the login processes accept()s it. After +that the client typically does nothing more than ask the server'scapability +list and then log in. The client may also start TLS sessionbefore logging in. + +Authentication is done by talking to the authentication process. The login +process is completely untrusted by the authentication process, so even if +anattacker is able to execute arbitrary code inside a login process, they won't +be able tolog in without a valid username and password. + +After receiving a successful authentication reply from the authentication +process, the login process sends the file descriptor to the master processwhich +creates a new mail process and transfers the fd into it. Before doingthat, the +master process verifies from the authentication process that theauthentication +really was successful. + +By default each login process will handle only a single connection and +afterwards kill itself (but see SSL proxying below). This way attacker can't +see other people'sconnections. This can however be disabled +('login_process_per_connection=no'), in which case the security of the design +suffers greatly. + +The login processes handle SSL/TLS connections themselves completely. They keep +proxying the connection to mail processes for the entire lifetime ofthe +connection. This way if a security hole is found from the SSL library,an +authenticated user still can't execute code outside the login process. + +See <LoginProcess.txt> for more information about different settings related to +login processes. + +Authentication process +---------------------- + +The authentication process handles everything related to the actual +authentication: SASL authentication mechanisms, looking up and verifyingthe +passwords and looking up user information. + +It listens for two different kinds of connections: untrusted authentication +client connections (from login processes) and master connections (frommaster +process, but also from Dovecot LDA). The client connections are onlyallowed to +try to authenticate. The master connections are allowed to askif an +authentication request with a given ID was successful, and also to lookup user +information based on a username. This user lookup feature is usedby Dovecot +LDA. + +Each client connection tells their process ID to the authentication process in +a handshake. If a connection with the same PID already exists, an erroris +logged and the new connection is refused. Although this makes DoSattacks +possible, it won't go unnoticed for long and I don't see this as areal issue +for now. + +Having the authentication process know the PID of the client connection allows +all authentication requests to be mapped to one specific client +connection.Since the master process knows the login process's real PID, it's +used whenasking from authentication process if the request was successful. This +makes it impossible for a login process to try to fake another login +process'slogin requests. Faking PIDs will also be quite pointless. + +Once the master process has done the verification request for a successful +authentication request, the request is freed from memory. The requests arealso +freed about 2 minutes after their creation, regardless of the statethey +currently are in. + +For blocking password and user database backends (eg. MySQL) separate "worker +processes" are used. Initially only one of them exists, butmore are created as +needed. <PAM> [PasswordDatabase.PAM.txt] can be configured to use worker +processes instead of doing the forking itself, but this isn'tcurrently done by +default and there may be problems related to it. Also <checkpassword> +[PasswordDatabase.CheckPassword.txt] currently does the forking itself. + +Mail processes +-------------- + +These processes handle the actual post-login mail handling using the privileges +of the logged in user. It's possible to chroot these processes,but practically +it's usually more trouble than worth. + +See <mail process design> [Design.MailProcess.txt] for their internal design +documentation. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.ErrorHandling.txt b/doc/wiki/Design.Storage.ErrorHandling.txt new file mode 100644 index 0000000..7497598 --- /dev/null +++ b/doc/wiki/Design.Storage.ErrorHandling.txt @@ -0,0 +1,63 @@ +Lib-storage Error Handling +========================== + +'src/lib-storage/mail-error.h' describes different types of errors and has some +other error-related functions and macros. + +Only errors returning "int" can actually return a failure. + + * Functions that return a pointer will never return failure with NULL. Only + "find" type of functions can return NULL, which means "not found". + * Iterators usually work by the init() returning iterator pointer and next() + returning a boolean. If there were any errors in either init() or next(), + deinit() finally returns a failure. + +Getting lib-storage errors +-------------------------- + + * 'mailbox_list_*()' functions set their errors to the given mailbox_list + structure. You can get these errors with 'mailbox_list_get_last_error()'. + * All other functions that have some way of accessing mail_storage (mailbox, + mail, transactions, etc.) set their errors to the storage. You can get these + errors with 'mail_storage_get_last_error()'. + * Mail user and namespace functions have their own error handling, + typically by returning error strings as parameters. + * Both '*_get_last_error()' functions should be called soon after the error is + noticed, before other failing lib-storage calls overwrite the error. + * In deinit failures it usually doesn't matter if you get the first or the + last error, so it's easier to just call all the different deinit + functions and finally look up what the last failure was. + +Setting lib-storage errors +-------------------------- + +Errors can be set with two calls: + + * 'mail_storage_set_error()' and 'mailbox_list_set_error()' should be used + when the error is user's fault in some way. For example invalid mailbox + name, out of quota, etc. The error string will be shown to user. It won't be + written to a log file. + * 'mail_storage_set_critical()' and 'mailbox_list_set_critical()' should be + used when the error is a problem in the system and sysadmin should be + notified. For example out of disk space or just in general an unexpected + syscall failure. The error string that will be shown to user is the + "Internal error occurred", but it will be logged as an error. + * The reason for the separation of these two is: + 1. Only log errors that sysadmin can do something about. + 2. Never show user anything even potentially sensitive about the system, + such as path names. + +There are also a few other calls that aren't used as often, but can be helpful: + + * 'mail_storage_set_internal_error()' and 'mailbox_list_set_internal_error()' + simply set the user-visible error message to "Internal error occurred". + These can be used if the actual error was already logged. + * 'mail_storage_set_error_from_errno()' and + 'mailbox_list_set_error_from_errno()' set the user-visible error message + based on some common 'errno' values. Currently: + * EACCESS, EPERM, EROFS: Permission denied + * ENOSPC, EDQUOT: Not enough disk space + * ENOENT, ENOTDIR: Not found + * ELOOP: Directory structure is broken + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mail.txt b/doc/wiki/Design.Storage.Mail.txt new file mode 100644 index 0000000..a86d722 --- /dev/null +++ b/doc/wiki/Design.Storage.Mail.txt @@ -0,0 +1,100 @@ +Mail +==== + +Mail is first allocated with 'mail_alloc()'. Mails always belong to a +<transaction> [Design.Storage.Mailbox.Transaction.txt]. Even if mail is treated +read-only, Dovecot might write data to cache file, so whenever possible, mail +transactions should be committed. When mail is allocated, you can specify a +list of fields and headers that you're (most likely) going to need. This allows +Dovecot to optimize the later 'mail_get_*()' lookups so that it doesn't need to +parse the message multiple times. These fields are also added to cache file, so +you shouldn't list fields unless you're fairly certain you need them. + +If you need to jump around in the mailbox based on already known sequences or +UIDs, you can do this with 'mail_set_seq()' and 'mail_set_uid()'. This way you +don't have to go through all the trouble of setting up a mailbox search. + +Getting data +------------ + +Some of the mail fields can be accessed directly: + + * 'box': Mail's mailbox, same as the transaction's. + * 'transaction': Mail's transaction, the same that was given to + 'mail_alloc()'. + * 'seq': Currently selected message's sequence number. + * 'uid': Currently selected message's UID. + * 'expunged': We already detected that the message is expunged and can't be + accessed. This may also be set (and looked up) later when some + 'mail_get_*()' function fails. + * 'has_nuls' and 'has_no_nuls': Message body is known to (not) have NUL + characters. + +The final field is 'lookup_abort', which is write-only. Normally when doing a +'mail_get_*()' operation for a field that isn't in a cache, the field is +generated and added to cache. If you don't want this, but instead have figured +out some better optimized way to do non-cached lookups, you can change this +field so that 'mail_get_*()' lookups fail instead. This is primarily used by +searching code internally. + + * 'MAIL_LOOKUP_ABORT_NEVER': The default - do whatever it takes to return the + value. + * 'MAIL_LOOKUP_ABORT_READ_MAIL': If returning the value would require reading + message headers/body, abort. + * 'MAIL_LOOKUP_ABORT_NOT_IN_CACHE': If the value isn't already in cache, + abort. For example if looking up message's physical size would require + 'stat()'ing the file this wouldn't be done. + +Most of the 'mail_get_*()' should be fairly obvious in how they work. Only +functions returning int can fail, others don't. + + * Keywords can be looked up either by getting an array of keyword strings or + keyword indexes. The index lookups are slightly faster. Keyword indexes can + be converted to strings by using the 'status.keywords' array returned by + 'mailbox_get_status()'. + * 'mail_get_first_header()' return 0 if header wasn't found, 1 if it was. + * 'mail_get_special()' can return various special fields. If a special isn't + implemented by some backend, the call returns success and sets the value to + "". + * 'mail_get_stream()' returns an input stream that can be used to access the + mail. If this function is called multiple times, each call seeks the stream + to beginning. Don't unreference the stream, it's destroyed automatically. + * Typically the input contains the raw data in disk, lines may end with LF + or CRLF depending on how they're on disk. + * mbox drops Dovecot's internal headers from the stream (X-UID:, Status:, + etc.). + * Plugins (e.g. zlib) can also hook into this call and modify the input + stream. + +Sometimes you might notice that some looked up field is actually corrupted. For +example you might notice that input stream returns EOF earlier than previous +'mail_get_physical_size()' said. This might have been caused by various +different things, but in any case all you can really do then is to just call +'mail_set_cache_corrupted()' and try again. + +Setting data +------------ + +Some of the messages' metadata can be updated: + + * 'mail_update_flags()' and 'mail_update_keywords()' changes flags/keywords. + * Usually you should set 'modify_type' parameter to 'MODIFY_ADD' or + 'MODIFY_REMOVE', instead of replacing all the flags. That allows + concurrent flag updates not to overwrite each others changes. + * 'mail_expunge()' expunges a message. + +Other functions are mainly intended for mailbox replication or restoring an +existing mailbox (e.g. dsync): + + * 'mail_update_modseq()' can be used to increase message's modseq + * 'mail_update_uid()' can be used to give a new (higher) UID for the message. + If such UID can't be given, this call is just ignored. + * 'mail_update_pop3_uidl()' can be used to give a specific POP3 UIDL for the + message. This is used internally when 'pop3_save_uidl=yes'. + +Other metadata can't be changed. IMAP protocol requires that messages are +immutable, so it's not possible to change message's received date, headers or +body. If you wish to modify any of them, you need to create a new message and +expunge the old one. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailNamespace.txt b/doc/wiki/Design.Storage.MailNamespace.txt new file mode 100644 index 0000000..1613a87 --- /dev/null +++ b/doc/wiki/Design.Storage.MailNamespace.txt @@ -0,0 +1,91 @@ +Mail Namespace +============== + +'src/lib-storage/mail-namespace.h' describes mail namespaces. See +<Namespaces.txt> for more information about what they are actually about. + +Hierarchy separators and namespace prefixes +------------------------------------------- + +A namespace and <mailbox list> [Design.Storage.MailboxList.txt] has 1:1 +relationship. A namespace is mainly about dealing with hierarchy separators and +namespace prefixes, which mailbox list doesn't know or care much about. + +Mailbox lists have their native hierarchy separators. For example with FS +layout the separator is '/', because child mailboxes are physically in +subdirectories and '/' is the separator for directories. With Maildir++ layout +the separator is (currently) hardcoded to '.' in the maildir directory name, so +that's its native hierarchy separator. + +Dovecot allows separators to be configurable, so namespaces have two +separators: + + * 'ns->sep' is the configured virtual separator, which defaults to same as + native separator. + * 'ns->real_sep' is the native separator. + +Namespaces also have prefixes. The prefixes are visible for users/clients and +they appear to be part of the actual mailbox name. One commonly used namespace +prefix is "INBOX.", so all mailboxes (other than INBOX itself) appear to be +children of the INBOX. + +So the same mailbox can be visible in three different forms: + + * Virtual name uses the virtual separator and namespace prefix. For example + "INBOX/foo/bar". + * Storage name uses the native separator and doesn't have a namespace prefix. + For example "foo.bar". + * Physical directory name on disk can be different again. For example with + Maildir++ it could be ".../Maildir/.foo.bar" (not the leading dot before + "foo"). + +Users and owners +---------------- + +When accessing other users' shared mailboxes, there's a difference between a +namespace's user and owner: + + * 'ns->user' points to the mail user actually accessing the mailbox (e.g. the + IMAP connection's mail user). + * 'ns->owner' points to the mail user who shared the mailbox. + +The distinction can be important sometimes. For example if user adds or removes +messages from the shared mailbox, the owner's quota must be updated instead of +the user's. + +Functions +--------- + +Functions about finding namespaces: + + * 'mail_namespace_find()' returns namespace for given virtual name and changes + the virtual name -> storage name. It also has a few variations: + * 'mail_namespace_find_visible()' skips hidden=yes namespaces. + * 'mail_namespace_find_subscribable()' skips subscriptions=no namespaces. + * 'mail_namespace_find_unsubscribable()' skips subscriptions=yes + namespaces. + * 'mail_namespace_find_inbox()' returns the namespace with inbox=yes. (There + can be only one.) + * 'mail_namespace_find_prefix()' returns namespace that has the given prefix. + * 'mail_namespace_find_prefix_nosep()' does the same, but ignores the + trailing separator in prefix (e.g. "foo" would find namespace with + prefix=foo/). + +Functions about translating between virtual and storage names when the +namespace is known: + + * 'mail_namespace_fix_sep()' changes virtual separators -> native separators. + * 'mail_namespace_get_storage_name()' changes virtual name -> storage name. + * 'mail_namespace_get_vname()' changes storage name -> virtual name. + * 'mail_namespace_update_name()' returns FALSE if virtual name doesn't + actually match the given namespace. Otherwise returns TRUE and changes + virtual name -> storage name. + +A single namespace can currently point to only a single storage, but there is +already some code that attempts to make the transition to multiple storages per +namespace easier. In general you shouldn't try to access 'ns->storage' +directly. When creating new mailboxes,'mail_namespace_get_default_storage()' +returns the storage that should be used. For other purposes you should find the +storage via <mailbox list> [Design.Storage.MailboxList.txt] functions. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailStorage.txt b/doc/wiki/Design.Storage.MailStorage.txt new file mode 100644 index 0000000..3052d8f --- /dev/null +++ b/doc/wiki/Design.Storage.MailStorage.txt @@ -0,0 +1,41 @@ +Mail Storage +============ + +'src/lib-storage/mail-storage.h' and 'mail-storage-private.h' describes mail +storage. Mail storage is mainly about being a common container for its +mailboxes. For example with <multi-dbox> [MailboxFormat.dbox.txt] each storage +has one directory where all the message bodies are written to, while the +per-mailbox directories only contain index files. With other mailbox formats +mail storage doesn't do much else than allow allocating <mailboxes> +[Design.Storage.Mailbox.txt]. + +The only public functions for mail storage are: + + * 'mail_storage_purge()' frees disk space used by expunged messages. Currently + the only mailbox format that uses this is multi-dbox. + * 'mail_storage_get_settings()' returns mail storage settings. + * 'mail_storage_set_callbacks()' can be used to specify "OK" and "NO" + callbacks, which are called when a long running operation wants to send a + status update. For example "OK Stale mailbox lock file detected, will + override in n seconds" or "NO Mailbox is locked, will abort in n seconds". + +Methods that mail storage backends need to implement are: + + * 'get_setting_parser_info()': Returns storage-specific settings parser + information. + * 'alloc()': Allocate memory for a storage and set its virtual functions. + * 'create(ns)': Initialize the storage based on given namespace settings. The + same storage can be used by other namespaces, but they don't call 'create()' + again. This function typically shouldn't fail, except when storage can't + handle the wanted namespace settings. + * 'destroy()': Destroys the storage. + * 'add_list(list)': Called every time the storage is attached to a new + namespace / mailbox list. + * 'get_list_settings(ns, set)': Used to get storage's default settings. + * 'autodetect(ns, set)': Returns TRUE if based on the given settings it looks + like this storage should be handling the namespace. This is done when + mail_location doesn't explicitly specify the mailbox format. + * 'mailbox_alloc()': Allocate memory for <mailbox> + [Design.Storage.Mailbox.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailUser.txt b/doc/wiki/Design.Storage.MailUser.txt new file mode 100644 index 0000000..1656aef --- /dev/null +++ b/doc/wiki/Design.Storage.MailUser.txt @@ -0,0 +1,25 @@ +Mail User +========= + +'src/lib-storage/mail-user.h' describes mail user. The struct contains all +kinds of useful information about the user that can be accessed directly. Some +of the most useful things you can do with a user are: + + * 'user->username' gives you the actual username string (e.g. + 'user@domain.org'). + * 'user->set' gives you access to user's settings. + * 'user->namespaces' points to a linked list of user's namespaces. + * 'mail_user_get_home()' returns user's home directory, if there's one. + * 'mail_user_home_expand()' expands '~/' at the beginning of given path to + user's actual home directory. + * 'mail_user_plugin_getenv()' returns value for a setting defined in 'plugin + {}' section. + +Typically each new IMAP/POP3/etc. connection creates a single mail user. +Currently multiple connections for same user don't even try to share the mail +user, but this may change in future. If a user has shared mailboxes from other +users (not public namespaces), a mail user is also created whenever necessary +to list/access the user's mailboxes. Again there is no attempt to share the +created mail user with other connections. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Save.txt b/doc/wiki/Design.Storage.Mailbox.Save.txt new file mode 100644 index 0000000..69ee754 --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Save.txt @@ -0,0 +1,51 @@ +Mailbox Saving +============== + +Both saving and copying messages begins by calling 'mailbox_save_alloc()'. +After that you can set message's metadata fields: + + * 'mailbox_save_set_flags()' sets flags and keywords. + * 'mailbox_save_set_received_date()' sets message's received date (IMAP + INTERNALDATE). It also supports specifying timezone, but most backends don't + support saving it. + * 'mailbox_save_set_dest_mail()' specifies a mail that can be used to access + the partially saved mail after save/copy is finished (but not committed). + You should be able to do pretty much anything with the mail, but its UID is + usually 0 at this point. + * 'mailbox_save_set_from_envelope()' sets the envelope sender. Currently this + is only used by mbox format to specify the address in From_-line. + +When copying, most of the metadata fields are automatically copied from the +source message. The only exception is message's flags and keywords. If you want +to preserve them, the easiest way is to call 'mailbox_save_copy_flags()'. + +Some metadata fields are mainly useful when you're replicating or restoring an +existing mailbox and want to preserve metadata: + + * 'mailbox_save_set_min_modseq()' sets message's modseq to at least the + specified modseq. If the modseqs are already higher in the mailbox, the + resulting modseq is what it would have been without this call. + * 'mailbox_save_set_uid()' sets message's UID. If mailbox's next_uid is + already higher than the specified UID, the UID request is ignored. + * 'mailbox_save_set_guid()' sets message's globally unique ID. A new GUID is + generated by default, and if there already exists a message with the same + GUID a different one may or may not be given. For example with maildir the + GUID is same as the base filename, while dbox has an explicit GUID metadata + field. + * 'mailbox_save_set_pop3_uidl()' sets POP3 UIDL value. Not all backends + support this. + * 'mailbox_save_set_save_date()' sets "message saved" date, i.e. the date/time + when the message was saved to this mailbox. The default is the current time. + +Once you're done with setting the metadata fields, you can either copy an +existing mail with 'mailbox_copy()' or provide message body with: + + * 'mailbox_save_begin()' starts saving from given input stream. + * 'mailbox_save_continue()' saves all data from input stream. If input stream + is blocking, typically a single call to this function should be enough. If + input stream is non-blocking, you need to call this function until you're + done. In any case call this until 'i_stream_is_eof()' returns TRUE. + * 'mailbox_save_finish()' finishes saving the mail, or 'mailbox_save_cancel()' + aborts it. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Search.txt b/doc/wiki/Design.Storage.Mailbox.Search.txt new file mode 100644 index 0000000..d1438ba --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Search.txt @@ -0,0 +1,101 @@ +Mailbox Searching +================= + +'mailbox_search_*()' functions should be used always when you're iterating +through multiple messages. The search queries can be complex or as simple as +"all messages". Without searching there's also a way to directly switch to a +specific message by its sequence number or UID, but this should be avoided +usually. + +Initializing +------------ + +Search is initialized with 'mailbox_search_init()' on top of a <transaction> +[Design.Storage.Mailbox.Transaction.txt]. Although it would appear that a +search is a read-only operation, it can actually write data to Dovecot's cache +file. For example if you search for a specific header not yet in cache file, +the results are saved to cache file so the next search will be fast. This is +why you should always commit search transactions, even if the rest of your +operation fails (you should use separate transactions for search and for +updates if necessary). + +You'll need a search query. 'src/lib-storage/mail-search.h' and +'mail-search-build.h' contain all the functions and structures related to it. +Usually you should start with 'mail_search_build_init()' and then start adding +the search parameters, either manually or with one of the existing +'mail_search_build_add_*()' helper functions. The same search query structures +can be saved and used for other searches later, but because search state is +stored inside the structs you need to be careful that multiple searches aren't +using them at the same time. This is usually more trouble than worth, so avoid +doing it. + +Search results can be sorted by giving a sort program. Dovecot optimizes this +by keeping sort indexes in the index files. + +Reading search results +---------------------- + +While 'mailbox_search_next()' returns TRUE, a new search result is found and it +changes the given <mail> [Design.Storage.Mail.txt] to point to the search +result. The mail's "wanted fields/headers" parameters don't need to include +anything needed by the search query, Dovecot optimizes them internally. + +If the search needs to parse message bodies and the mailbox is large, this call +can take a long time. If you want to do other things while searching, you can +use 'mailbox_search_next_nonblock()' that does only a bit of work and then +returns either with a result or "try again later" status. Dovecot attempts to +keep each non-matching call to this function between 200 and 250 milliseconds, +although the upper bound can't be guaranteed. + +It's possible that messages are being expunged while Dovecot is searching them, +so it can't determine if they would have matched the search rule or not. In +this case it skips over them, but if you want to know if this has happened, you +can see if 'mailbox_search_seen_lost_data()' returns TRUE. + +Deinitializing +-------------- + +'mailbox_search_deinit()' finishes the search. If it returns -1, some error +occurred and some search results might not have been returned. + +Example +------- + +Iterating through all messages in a mailbox goes like: + +---%<------------------------------------------------------------------------- +/* build search query */ +search_args = mail_search_build_init(); +mail_search_build_add_all(search_args); + +search_ctx = mailbox_search_init(trans, search_args, NULL); +/* search context keeps args referenced */ +mail_search_args_unref(&search_args); + +mail = mail_alloc(trans, 0, NULL); +while (mailbox_search_next(ctx, mail)) { + printf("matched uid %u\n", mail->uid); +} +mail_free(&mail); +if (mailbox_search_deinit(&search_ctx) < 0) + i_error("search failed"); +---%<------------------------------------------------------------------------- + +Saving search results +--------------------- + +Search results can be saved for future use by calling +'mailbox_search_result_save()' just after initializing the search. The results +as returned as UIDs with 'mailbox_search_result_get()' and may contain UIDs +that are already expunged. Once you're done with the saved result, free it with +'mailbox_search_result_free()'. + +The search result can also be automatically updated whenever mailbox is synced +if 'MAILBOX_SEARCH_RESULT_FLAG_UPDATE' is set. The update is optimized, so +Dovecot doesn't do a full re-search, but matches only new and changed messages. +If 'MAILBOX_SEARCH_RESULT_FLAG_QUEUE_SYNC' is also set, search result additions +and removals are also tracked and can be retrieved with +'mailbox_search_result_sync()', i.e. with this you can implement "what changed +in search results since last time I checked". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Sync.txt b/doc/wiki/Design.Storage.Mailbox.Sync.txt new file mode 100644 index 0000000..a7d295a --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Sync.txt @@ -0,0 +1,128 @@ +Mailbox Synchronization +======================= + +The idea behind synchronization is to find out what changes other sessions have +done to the mailbox and to finalize our own changes to the mailbox. + +For example if you expunge a message in a transaction and commit it, the commit +will only write a "please expunge UID n" record to Dovecot's transaction log +file. The message still exists on the disk. The next time Dovecot syncs the +mailbox (either the session that wrote the record or another one), it goes +through all the non-synchronized records in transaction log and applies the +requested changes to the backend mailbox. Syncing can be a bit heavyweight +operation, so it's possible to commit multiple transactions and perform a +single sync for all of them. Dovecot attempts to do this with IMAP protocol +when pipelining commands. + +The other important job of syncing is to refresh mailbox's state: + + * Finding out about external modifications to mailbox (e.g. a new mail + delivered to Maildir/new/). + * Updating in-memory view of what messages exist, what their flags are, etc. + +When a mailbox is opened, its state starts with what index files contain at the +time. Since the backend mailbox may have already changed, and syncing an +up-to-date mailbox is usually really cheap, there isn't much point in not +syncing mailbox immediately after opening. The mailbox state stays the same +until you synchronize the mailbox again, before that no new messages show up +and no messages get expunged. + +Typically you would sync the mailbox + + * after committing a transaction that modifies backend mailbox in any way + (instead of just internal index data), such as after changing message flags + or expunging a message. + * whenever you want to find out if there are any changes. With IMAP protocol + this is done every time after running a command. + +Initializing +------------ + +'mailbox_sync_init()' initializes syncing. + +There are some flags that control how much effort is spent on syncing: + + * 'MAILBOX_SYNC_FLAG_FAST' can be given when you're ready for mailbox to be + refreshed, but don't care much if it actually is or not. When this flag is + set, Dovecot still notices all internal changes, but external changes are + checked only once every few seconds or so. + * 'MAILBOX_SYNC_FLAG_FULL_READ' is mainly useful with mboxes. If + 'mbox_dirty_syncs=yes' and a new mail gets appended to mbox by an external + program, Dovecot assumes that the only change was the added mail, even + though the program may have also modified existing messages' flags by + rewriting Status: headers. If 'mbox_very_dirty_syncs=no', these changes are + noticed after the next time mailbox is opened. So when this flag is enabled, + it means Dovecot should try harder to find out if there were any external + unexpected changes. It's currently used only with IMAP SELECT and CHECK + commands and POP3 startup. Probably unnecessary elsewhere. + * 'MAILBOX_SYNC_FLAG_FULL_WRITE' is again mainly useful with mboxes. If + 'mbox_lazy_writes=no', Dovecot delays writing flag changes to mbox file + until mailbox is closed or IMAP CHECK command is issued. Using this + elsewhere is probably unnecessary, except as an optimization if mailbox is + in any case synced just before closing it, you might as well give this flag + to it to avoid double-syncing with mbox. + * 'MAILBOX_SYNC_FLAG_FORCE_RESYNC' is used to force resyncing indexes. The + only time this should be done is when manually triggered by administrator. + +Then there are also other syncing flags: + + * 'MAILBOX_SYNC_FLAG_NO_EXPUNGES': No expunged messages are removed from the + in-memory mailbox view. Their removal is delayed until syncing is done + without this flag. Attempting to access the expunged messages may or may not + work, depending on what information is accessed and what storage backend is + used. + * 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT': Normally when the internal mailbox + state can't be consistently updated (typically due to index file + corruption), the syncing fails. When this flag is set, it means that the + caller doesn't care about mailbox's previous state and just wants to get it + accessible again. Typically this is used when the mailbox is being opened, + but not afterwards. + * 'MAILBOX_SYNC_FLAG_EXPUNGE' is mainly intended for virtual plugin with IMAP + protocol. You probably shouldn't use it. + +Reading changes +--------------- + +While 'mailbox_sync_next()' returns TRUE, it fills out sync record: + + * seq1, seq2: Message sequence numbers that were affected + * type: expunge, flag change or modseq change. + +Expunge records don't immediately change the view's sequence numbers. After +seeing an expunge record you can still fetch the expunged messages' flags and +possibly other information. Only after syncing is deinitialized, the sequences +change. + +Message flag change records don't actually show what the changes were. You can +find the new flags just by fetching them ('mail_get_flags()', etc.), they're +available immediately. You'll need to create a <transaction> +[Design.Storage.Mailbox.Transaction.txt] and a <mail> [Design.Storage.Mail.txt] +for that. For example: + +---%<------------------------------------------------------------------------- +sync_ctx = mailbox_sync_init(box, flags); +trans = mailbox_transaction_begin(box, 0); +mail = mail_alloc(trans, MAIL_FETCH_FLAGS, 0); +---%<------------------------------------------------------------------------- + +If you don't actually care about sync records, you don't necessarily have to +even call 'mailbox_sync_next()'. In that case it's actually easiest to perform +the whole sync using a one-step 'mailbox_sync()' function. This function also +sets 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT' flag automatically. + +Deinitializing +-------------- + +'mailbox_sync_deinit()' finalizes the syncing. If any errors occurred during +sync, it'll return -1. + +If 'MAILBOX_SYNC_FLAG_NO_EXPUNGES' was used and some expunges were actually +delayed,'status_r->sync_delayed_expunges' is set to TRUE. + +Implementing sync for a storage backend +--------------------------------------- + +FIXME: talk about mail_index_sync_*() and how to change stuff and how to update +internal state. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Transaction.txt b/doc/wiki/Design.Storage.Mailbox.Transaction.txt new file mode 100644 index 0000000..63a37e1 --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Transaction.txt @@ -0,0 +1,52 @@ +Mailbox Transactions +==================== + +Before you can read any mails or do any changes to mails, you need to create a +transaction with 'mailbox_transaction_begin()'. It has a few flags: + + * 'MAILBOX_TRANSACTION_FLAG_HIDE': Mark the changes in a way that when later + <syncing> [Design.Storage.Mailbox.Sync.txt] the mailbox in this session, + 'mailbox_sync_next()' won't return sync records for the changes done by this + transaction. This is primarily meant for flag and keyword changes, you can't + hide expunges. For example IMAP's 'STORE FLAGS.SILENT' command is + implemented by setting this flag for the transaction. + * 'MAILBOX_TRANSACTION_FLAG_EXTERNAL': Changes done by this transaction should + be marked as external changes. Internal changes can be thought of as "change + requests" that syncing later finishes, while external changes are done + immediately and syncing ignores them. Normally you would use this flag when + you want to save or copy messages, nothing else. + * 'MAILBOX_TRANSACTION_FLAG_ASSIGN_UIDS': Require assigning UIDs to + saved/copied messages immediately. Normally this is done only when it's easy + (maildir: if dovecot-uidlist can be locked without waiting, mbox: if mbox is + already fully synced). + * 'MAILBOX_TRANSACTION_FLAG_REFRESH': Do a quick index refresh, so any recent + flag/modseq changes done by other Dovecot sessions will be visible. You + shouldn't usually need this, because usually you should have recently done a + mailbox sync. + +Changes for a transaction are kept in memory until the transaction is +committed. If you want to cancel the changes, you can call +'mailbox_transaction_rollback()'. Transaction can be commited with +'mailbox_transaction_commit()'. If you want to know a bit more about the +results of the transaction, use 'mailbox_transaction_commit_get_changes()' +instead. It returns a change structure: + + * uid_validity: UIDVALIDITY used by returned UIDs + * saved_uids: UIDs assigned to saved/copied mails. Typically they're in an + ascending order, unless you explicitly requested some specific UIDs for + mails while saving them (e.g. dsync does this). + * ignored_uid_changes: Number of UIDs that couldn't be changed by + 'mail_update_uid()' calls, because the UIDs were less than next_uid's value. + + * ignored_modseq_changes: Number of modseqs that couldn't be changed by + 'mail_update_modseq()' calls, because they would have lowered the modseq. + +Once you're done with reading the change structure, be sure to free the memory +used by it with 'pool_unref(&changes->pool)'. + +'mailbox_transaction_set_max_modseq()' can be used to implement atomic +conditional flag changes. If message's modseq is higher than the given +max_modseq while transaction is being committed, the change isn't done and the +message's sequence number is added to the given array. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.txt b/doc/wiki/Design.Storage.Mailbox.txt new file mode 100644 index 0000000..5af8263 --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.txt @@ -0,0 +1,34 @@ +Mailbox +======= + +'src/lib-storage/mail-storage.h' and 'mail-storage-private.h' describes mailbox +API, among others. Mailbox life cycle often goes like: + + * 'mailbox_alloc()' allocates memory for the mailbox and initializes some + internal settings, but doesn't actually try to open it. + * 'mailbox_open()' opens the mailbox. Instead of opening a mailbox, you can + also create it with 'mailbox_create()'. + * If you're immediately syncing the mailbox, you don't need to open it, + because it's done implicitly. This reduces your code and error handling a + bit. + * 'mailbox_close()' closes the mailbox, so that it needs to be opened again if + it's wanted to be accessed. This is rarely needed. + * 'mailbox_free()' closes and frees the mailbox. + +There are a lot of functions to deal with mailboxes. The most important ones +are: + + * 'mailbox_get_status()' to get a summary of mailbox, such as number of + messages in it. + * <Syncing> [Design.Storage.Mailbox.Sync.txt]: 'mailbox_sync_*()' to + synchronize changes from the backend to memory. + * <Transactions> [Design.Storage.Mailbox.Transaction.txt]: + 'mailbox_transaction_*()' for transaction handling. All message reads and + writes are done in a transaction. + * <Searching> [Design.Storage.Mailbox.Search.txt]: 'mailbox_search_*()' is + used for searching messages. Even simple operations like "get all messages" + go through this API, it'll then simply do "search all". + * <Saving> [Design.Storage.Mailbox.Save.txt]: 'mailbox_save_*()' and + 'mailbox_copy()' is used for saving/copying new messages to mailbox. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailboxList.txt b/doc/wiki/Design.Storage.MailboxList.txt new file mode 100644 index 0000000..6c27883 --- /dev/null +++ b/doc/wiki/Design.Storage.MailboxList.txt @@ -0,0 +1,159 @@ +Mailbox List +============ + +'src/lib-storage/mailbox-list.h' and 'mailbox-list-private.h' describes mailbox +list. The purpose of mailbox list is to manage mailbox storage name<-> physical +directory path mapping. Its most important functions are: + + * listing existing mailboxes, + * creating directories for new mailboxes (but not the mailboxes themselves, + that's storage's job), + * deleting mailboxes, + * renaming mailboxes and + * managing mailbox subscriptions. + +Mailbox list code also internally creates and updates mailbox changelog (in +'dovecot.mailbox.log' file), which keeps track of mailbox deletions, renames +and subscription changes. This is primarily useful for dsync utility. + +Mailbox list is configured by <mail_location> [MailLocation.txt] setting, which +fills 'struct mailbox_list_settings': + + * root_dir: The root mail directory (e.g. with + 'mail_location=maildir:~/Maildir' it would be the '~/Maildir'). + * index_dir: Directory under which index files are written to. Empty string + means in-memory indexes. Defaults to root_dir. + * control_dir: Directory under which control files are written to. Control + files are files that contain some important metadata information about + mailbox so (unlike index files) they should never be deleted. For example + subscriptions file is a control file. Defaults to root_dir. + * alt_dir: This is currently <dbox> [MailboxFormat.dbox.txt]-specific setting. + + * inbox_path: Path to INBOX mailbox. This exists mainly because with mbox + format INBOX is often in a different location than other mailboxes. + * subscription_fname: Filename used by subscriptions file. + * dir_guid_fname: Filename used to store directories' (not mailboxes') global + UIDs. Directory GUIDs are mainly useful for dsync. + * maildir_name: Directory name under which the actual mailboxes are stored in, + such as dbox-Mails/ with dbox. See the .h file for more detailed + description. + * mailbox_dir_name: If non-empty, store all mailboxes under + root_dir/mailbox_dir_name/. + +Listing mailboxes +----------------- + +First the list operation is initialized with one of the init functions: + + * 'mailbox_list_iter_init()' lists mailboxes that match the given pattern. + * 'mailbox_list_iter_init_multiple()' lists mailboxes that match any of the + given patterns list. + * 'mailbox_list_iter_init_namespaces()' lists matching mailboxes from all + namespaces. + * 'MAILBOX_LIST_ITER_SKIP_ALIASES' flag skips namespaces that have + 'alias_for' set. You usually want to set this flag to avoid processing + the same mailbox multiple times. + +The patterns are IMAP-style patterns with '%' and '*' wildcards as described by +RFC 3501: '%' matches only up to next hierarchy separator, while '*' matches +the rest of the string. + +These flags control what mailboxes are returned: + + * 'MAILBOX_LIST_ITER_NO_AUTO_INBOX' doesn't list INBOX unless it physically + exists. Normally INBOX is listed, because INBOX doesn't need to be (and + cannot be) explicitly created. It can always be opened and messages can be + saved to it, it's just automatically created when it doesn't exist. + * 'MAILBOX_LIST_ITER_SELECT_SUBSCRIBED' lists only subscribed mailboxes. + * 'MAILBOX_LIST_ITER_SELECT_RECURSIVEMATCH' is currently only useful when + combined with '_SELECT_SUBSCRIBED' flag. Then it adds + 'MAILBOX_CHILD_SUBSCRIBED' flags for mailboxes whose children are + subscribed. It also lists mailboxes that aren't themselves subscribed, but + have children that do. + +These flags control what is returned for matching mailboxes: + + * 'MAILBOX_LIST_ITER_RETURN_NO_FLAGS' can be set when you don't care about + mailbox flags. They're then set only if it can be done without any + additional disk I/O. + * 'MAILBOX_LIST_ITER_RETURN_SUBSCRIBED' returns mailbox's subscription state. + * 'MAILBOX_LIST_ITER_RETURN_CHILDREN' sets "has child mailboxes" or "doesn't + have child mailboxes" flag. + +Other flags: + + * 'MAILBOX_LIST_ITER_RAW_LIST' should usually be avoided. It ignores ACLs and + just returns everything. + * 'MAILBOX_LIST_ITER_VIRTUAL_NAMES' enables listing to use virtual names + instead of storage names in patterns and returned mailbox names. + +Once listing is initialized, 'mailbox_list_iter_next()' can be called until it +returns NULL. The returned mailbox_info struct contains: + + * 'name': Mailbox's name, either virtual or storage name depending on + '_VIRTUAL_NAMES' flag. + * 'ns': Mailbox's namespace. This is useful only when mailboxes are listed + using 'mailbox_list_iter_init_namespaces()'. + * 'flags': Mailbox flags: + * 'MAILBOX_NOSELECT': Mailbox exists, but can't be selected. It's possible + that it can be created and then it becomes selectable. For example with + mbox and FS layout the directories aren't selectable mailboxes. + * 'MAILBOX_NONEXISTENT': Mailbox doesn't exist. It's listed only because it + has child mailboxes that do exist but don't match the pattern. + * Example: "foo/bar" exists, but "foo" doesn't. "%", "foo" or "*o" + pattern would list "foo", because it matches the pattern but its child + doesn't. Then again "*", "*bar" or "%/%" wouldn't list "foo", because + "foo/bar" matches the pattern (and is also listed). Something like + "*asd*" wouldn't match either "foo" or "foo/bar" so neither is + returned. + * 'MAILBOX_CHILDREN' and 'MAILBOX_NOCHILDREN': Mailbox has or doesn't have + children. If neither of these flags are set, it's not known if mailbox + has children. + * 'MAILBOX_NOINFERIORS': Mailbox doesn't have children and none can ever be + created. For example with mbox and FS layout the mailboxes have this flag + set, because files can't be created under files. + * 'MAILBOX_MARKED' and 'MAILBOX_UNMARKED': Mailbox has or doesn't have + messages with \Recent flags. If neither is set, the state is unknown. + Because this check is done in a very cheap way, having 'MAILBOX_MARKED' + doesn't always mean that there are \Recent flags. However, if + 'MAILBOX_UNMARKED' is returned it is guaranteed to be correct. (False + positives are ok, false negatives are not ok.) + * 'MAILBOX_SUBSCRIBED': Mailbox is subscribed. + * 'MAILBOX_CHILD_SUBSCRIBED': Mailbox has a child that is subscribed (and + '_SELECT_RECURSIVEMATCH' flag was set). + +Finally the listing is deinitalized with 'mailbox_list_iter_deinit()'. If it +returns -1, it means that some mailboxes perhaps weren't listed due to some +internal error. + +If you wish to get mailbox_info flags only for a single mailbox, you can use +'mailbox_list_mailbox()'. + +Directory permissions +--------------------- + +'mailbox_list_get_permissions()' and 'mailbox_list_get_dir_permissions()' can +be used to get wanted permissions for newly created files and directories. + + * For global files, give NULL as the mailbox name. The permissions are then + based on the root_dir. If root_dir doesn't exist, it returns 0700/0600 mode. + * For per-mailbox files, give the mailbox name. The permissions are then based + on the mailbox's directory. + +The returned permissions are: + + * mode: Creation mode, like 0600. + * gid: Group that should be set, unless it's '(gid_t)-1'. There are 3 reasons + why it could be that: + * directory has g+s bit set, so the wanted group is set automatically + * group is the same as process's effective GID, so it gets set + automatically + * mode's group permissions are the same as world permissions, so group + doesn't matter. + * gid_origin: This string points to the directory where the group (and + permissions in general) was based on, or "defaults" for internal defaults. + +If changing the group fails with EPERM, 'eperm_error_get_chgrp()' can be used +to log a nice and understandable error message. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Plugins.txt b/doc/wiki/Design.Storage.Plugins.txt new file mode 100644 index 0000000..349ab2e --- /dev/null +++ b/doc/wiki/Design.Storage.Plugins.txt @@ -0,0 +1,124 @@ +Mail Plugins +============ + +Typically plugins add hooks in their init() function by calling +'mail_storage_hooks_add()', and remove the hooks at deinit() with +'mail_storage_hooks_remove()'. Hooks that are currently supported: + + * mail_user_created: A new mail user was created. It doesn't yet have any + namespaces. + * mail_storage_created: A new mail storage was created. It's not connected to + any namespaces/mailbox lists yet. + * mailbox_list_created: A new mailbox list was created. It's not connected to + any storages yet. Because of this, some internal virtual methods haven't + been overridden by the storage yet, so plugins rarely want to use this hook. + Instead they should use: + * mail_namespace_storage_added: Storage was connected to its first + namespace/mailbox list. This hook should usually be used if plugin wants to + override mailbox_list's methods. + * mail_namespaces_created: User's all namespaces have been created. This hook + is called only per user at startup. More internal namespaces may be created + later when using shared mailboxes. + * mailbox_allocated: 'mailbox_alloc()' was called. + * mailbox_opened: Mailbox (and its index) was actually opened, either + explicitly with 'mailbox_open()' or implicitly by some other function. + +Overriding methods +------------------ + +When the hook gets called, you usually want to override some method of the +created object. This is the easy part, for example: + +---%<------------------------------------------------------------------------- +static void plugin_mailbox_allocated(struct mailbox *box) +.. + box->v.transaction_begin = plugin_transaction_begin; +---%<------------------------------------------------------------------------- + +The problem is that once 'plugin_transaction_begin()' is called, it should call +the original 'transaction_begin()'. There may also be multiple plugins that +want to override the same method, so the idea is to just have each plugin call +the previous 'transaction_begin()'. The next problem is where do you save the +previous value? Most objects have a 'module_contexts' array for storing +per-plugin pointers for this purpose. There are several helper functions to +make setting and accessing them in a quite safe way. + +Easiest way to set up the module context is to just copy&paste code from an +existing plugin that sets the same context. Here's some documentation about it +anyway: + +First you start by creating register for the plugin. There are different +registers for different types of objects: + + * mail_user_module_register: For mail_user. + * mailbox_list_module_register: For mailbox_list. + * mail_storage_module_register: For mail_storage, mailbox, mailbox_transaction + and mail_search. + * mail_module_register: For mail. + +We'll assume you want to use mail_storage_module_register: + +---%<------------------------------------------------------------------------- +static MODULE_CONTEXT_DEFINE_INIT(plugin_storage_module, +&mail_storage_module_register); +---%<------------------------------------------------------------------------- + +If you need to make it external, use: + +---%<------------------------------------------------------------------------- +extern MODULE_CONTEXT_DEFINE(plugin_storage_module, +&mail_storage_module_register); +struct plugin_storage_module plugin_storage_module = + MODULE_CONTEXT_INIT(&mail_storage_module_register); +---%<------------------------------------------------------------------------- + +Next you'll need to allocate memory for the structure you want to place in the +context. If you only want to override some methods, you can use: + +---%<------------------------------------------------------------------------- +union mailbox_module_context *mbox; +struct mailbox_vfuncs *v = box->vlast; + +mbox = p_new(box->pool, union mailbox_module_context, 1); +mbox->super = *v; +box->vlast = &mbox->super; + +v->transaction_begin = plugin_transaction_begin; +MODULE_CONTEXT_SET_SELF(box, plugin_storage_module, mbox); +---%<------------------------------------------------------------------------- + +If you want to store some more plugin-specific data to the object instead of +just the super methods, you can do: + +---%<------------------------------------------------------------------------- +struct plugin_mailbox { + /* must be called module_ctx */ + union mailbox_module_context module_ctx; +}; +/* .. */ + +struct plugin_mailbox *mbox; +struct mailbox_vfuncs *v = box->vlast; + +mbox = p_new(box->pool, struct plugin_mailbox, 1); +mbox->module_ctx.super = *v; +box->vlast = &mbox->super; + +v->transaction_begin = plugin_transaction_begin; +MODULE_CONTEXT_SET(box, plugin_storage_module, mbox); +---%<------------------------------------------------------------------------- + +Note that when using union directly you use 'MODULE_CONTEXT_SET_SELF()', while +when it's inside a struct you use 'MODULE_CONTEXT_SET()'. + +Once all this initialization is done, you can look up the module context with: + +---%<------------------------------------------------------------------------- +#define PLUGIN_CONTEXT(obj) MODULE_CONTEXT(obj, plugin_storage_module) +/* .. */ +struct plugin_mailbox *mbox = PLUGIN_CONTEXT(box); +---%<------------------------------------------------------------------------- + +(Yes, this API seems a bit too difficult to use and could use a redesign.) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Strings.txt b/doc/wiki/Design.Strings.txt new file mode 100644 index 0000000..72ca055 --- /dev/null +++ b/doc/wiki/Design.Strings.txt @@ -0,0 +1,70 @@ +Dynamic Strings +=============== + +'lib/str.h' describes Dovecot's dynamically growing strings. Strings are +actually only a simple wrapper on top of <buffers> [Design.Buffers.txt]. Even +the 'string_t' type is only a typedef of 'buffer_t', so it's possible to use +'buffer_*()' functions with strings (although it's ugly so it should be +avoided). The decision of whether to use a string_t or a buffer_t is mainly for +human readability: if the buffer's contents are (ASCII/UTF8) text use string_t, +otherwise for binary data use buffer_t. + +Once you're done modifying a string with 'str_*()' functions, you can get it +out as a NUL-terminated string with 'str_c()' or 'str_c_modifiable()'. These +pointers shouldn't be accessed after modifying the string again, they could +have moved elsewhere in memory and they're no longer guaranteed to be +NUL-terminated. + +Example: + +---%<------------------------------------------------------------------------- +T_BEGIN { + string_t *str = t_str_new(64); + + str_append(str, "hello world"); + str_printfa(str, "\nand %u", str_len(str)); + + printf("%s\n", str_c(str)); +} T_END; +---%<------------------------------------------------------------------------- + +String Handling Functions +========================= + +'lib/strfuncs.h' contains a lot of functions intended to make string handling +easier. They use C's NUL-terminated strings instead of Dovecot's dynamic +strings. + + * '[pt]_strdup_printf()' and '[pt]_strconcat()' are the most commonly used + functions.'*_strconcat' is slightly faster than '*_strdup_printf()', so use + it if you simply need to concatenate strings. + * Various functions for doing a 'strdup()' from wanted input. + * 'i_snprintf()' is a wrapper to 'snprintf()' that makes it easier to check if + result was truncated. It also adds few other safety checks. This should be + avoided in general, except in situations where you just don't want to use + data stack and there's no way for the result to get truncated. + * 'i_strocpy()' is similar to 'strlcpy()', but makes it easier to check if + result was truncated. This has the same problems as 'i_snprintf()'. + * Functions for uppercasing and lowercasing strings. + * Functions you can pass to 'bsearch()' and 'qsort()' for handling string + arrays. + * '[pt]_strsplit()' is an easy way to split a string into an array of strings + from given separator. + * 't_strarray_join()' reverses this. + * There are also some other functions to handle array of strings, like + getting its length or finding a given string. + * 'dec2str()' can be used to convert a number to a string. This can be useful + if you don't know the correct type and don't want to add casting (that could + potentially truncate the string). For example:'print("pid = %s\n", + dec2str(getpid()));' + +String Escaping +=============== + +'lib/strescape.h' contains functions to escape and unescape <">. <'> and <\> +characters in strings using<\> character.\ + +Dovecot's internal protocols are often line-based with TAB as the field +separator. This file also contains functions to escape and unescape such data. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.txt b/doc/wiki/Design.txt new file mode 100644 index 0000000..c27f4ec --- /dev/null +++ b/doc/wiki/Design.txt @@ -0,0 +1,66 @@ +Dovecot Design +============== + + * <Overview of Dovecot processes> [Design.Processes.txt] + * <Design of index files> [Design.Indexes.txt] + * <API for accessing the index files> [Design.Indexes.MailIndexApi.txt] + * <Design of authentication process> [Design.AuthProcess.txt] + * <Authentication protocol> [Design.AuthProtocol.txt] + * <Design of IMAP/POP3 processes> [Design.MailProcess.txt] + * <Doveadm server protocol> [Design.DoveadmProtocol.txt] and <Doveadm HTTP + server protocol> [Design.DoveadmProtocol.HTTP.txt] + * <Doveadm synchronization> [Design.Dsync.txt] + * <Dovecot Lua support> [Design.Lua.txt] + +Protocol extensions +------------------- + + * <Forwarding parameters in IMAP/POP3/LMTP/SMTP proxying> + [Design.ParameterForwarding.txt] + +Code APIs +--------- + + * <Code design> [Design.Code.txt] - explanations how and why the coding style + is the way it is + +Look at the *.h files for the actual API documentation. The documentation below +doesn't attempt to list full API documentation. + +liblib: + + * <Memory allocations> [Design.Memory.txt] + * <Static/dynamic buffers> [Design.Buffers.txt] + * <Dynamic arrays> [Design.Arrays.txt] + * <String handling> [Design.Strings.txt] + * <Input streams> [Design.InputStreams.txt] + * <Output streams> [Design.OutputStreams.txt] + * <Events> [Design.Events.txt] + * <Plugins> [Design.Plugins.txt] + +lib-dcrypt: + + * <lib-dcrypt data formats> [Design.Dcrypt.txt] + +lib-storage: + + * <Mail user> [Design.Storage.MailUser.txt] contains everything related to a + single user. + * <Mail namespace> [Design.Storage.MailNamespace.txt]: A single user can + contain multiple <namespaces> [Namespaces.txt]. + * <Mailbox list> [Design.Storage.MailboxList.txt] is used to list/manage a + list of mailboxes for a single namespace (1:1 relationship). + * <Mail storage> [Design.Storage.MailStorage.txt] is used to access mails in a + specific location with a specific mailbox format. Multiple namespaces can + point to the same storage. A single namespace may in future (but not + currently) point to multiple storages (e.g. a mixed mbox and Maildir + directory). + * <Mailbox> [Design.Storage.Mailbox.txt] is used to access a specific mailbox + in a storage. + * <Mail> [Design.Storage.Mail.txt] is used to access a specific mail in a + mailbox. + * <Error handling> [Design.Storage.ErrorHandling.txt]. + * <Plugins> [Design.Storage.Plugins.txt] - how to hook into lib-storage + functions. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Dict.txt b/doc/wiki/Dict.txt new file mode 100644 index 0000000..082fb4a --- /dev/null +++ b/doc/wiki/Dict.txt @@ -0,0 +1,45 @@ +Dict Proxy Process +================== + +Dict server is used for providing <dictionary> [Dictionary.txt] access via +server processes instead of doing it directly from whichever process wants to +access the dictionary. This is useful for some backends with relatively high +connection cost (e.g. SQL), but not necessarily for others (e.g. Redis). + +When a mail process uses the dict proxy, it needs to have access the dict UNIX +socket. By default only the "dovecot" user has access to the dict socket, which +doesn't typically work in any installation. However, giving too wide +permissions by default might allow untrusted users to access the dict and cause +problems. + +If all users share a single UNIX UID (e.g. "vmail"), you could make the dict +socket accessible only to it: + +---%<------------------------------------------------------------------------- +service dict { + unix_listener dict { + mode = 0600 + user = vmail + } +} +---%<------------------------------------------------------------------------- + +If you use multiple UNIX UIDs, you can add an extra group for all Dovecot mail +processes. This works even if you have untrusted system users who have shell +access to the server: + +---%<------------------------------------------------------------------------- +mail_access_groups = dovecot + +service dict { + unix_listener dict { + mode = 0660 + group = dovecot + } +} +---%<------------------------------------------------------------------------- + +However, it works with <LDA.txt> only if it's started as root. If this isn't +possible, look into using <LMTP.txt> instead. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Dictionary.txt b/doc/wiki/Dictionary.txt new file mode 100644 index 0000000..1eb078b --- /dev/null +++ b/doc/wiki/Dictionary.txt @@ -0,0 +1,268 @@ +Dovecot Dictionaries +==================== + +Dovecot's lib-dict can be used to access simple key-value databases. This is +used by for example <quota-dict> [Quota.Dict.txt], <passdb&userdb> +[AuthDatabase.Dict.txt], <last-login plugin> [Plugins.LastLogin.txt], +<METADATA> [ImapMetadata.txt], etc. The dictionaries can be accessed either +directly by the mail processes or they can be accessed via <dict proxy> +[Dict.txt] processes. + +Currently supported dict backends are: + + * Flat files + * FS (lib-fs wrapper) + * Memcached (ASCII protocol) + * Memcached (Binary protocol) + * Redis + * Proxy + * SQL + * LDAP (read only) + +Flat Files +---------- + +---%<------------------------------------------------------------------------- +file:<path> +---%<------------------------------------------------------------------------- + +The file will simply contain all the keys that are used. Not very efficient for +large databases, but good for small ones such as a single user's quota. + +FS (v2.2.11+) +------------- + +---%<------------------------------------------------------------------------- +fs:<driver>:<driver args> +---%<------------------------------------------------------------------------- + +This is a wrapper for lib-fs, which most importantly has the "posix" backend. +So using: + +---%<------------------------------------------------------------------------- +fs:posix:prefix=/var/lib/dovecot/dict +---%<------------------------------------------------------------------------- + +Would create a separate file under /var/lib/dovecot/dict for each key. + +Memcached (Binary Protocol) (v2.2.9+) +------------------------------------- + +This driver uses the "new" Memcache binary protocol. + +https://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol + +---%<------------------------------------------------------------------------- +memcached:param=value:param2=value2:... +---%<------------------------------------------------------------------------- + +Supported parameters are: + + * host: Memcached server host (default: 127.0.0.1) + * port: Memcached server port (default: 11211) + * prefix: Prefix to add to all keys (default: empty) + * timeout_msecs: Abort lookups after specified number of milliseconds + (default: 30000) + +Memcached (ASCII Protocol) (v2.2.9+) +------------------------------------ + +This driver uses the "legacy" Memcache ASCII protocol. + +https://github.com/memcached/memcached/blob/master/doc/protocol.txt + +---%<------------------------------------------------------------------------- +memcached_ascii:param=value:param2=value2:... +---%<------------------------------------------------------------------------- + +Supported parameters are: + + * host: Memcached server host (default: 127.0.0.1) + * port: Memcached server port (default: 11211) + * prefix: Prefix to add to all keys (default: empty) + * timeout_msecs: Abort lookups after specified number of milliseconds + (default: 30000) + +Redis (v2.2.9+) +--------------- + +---%<------------------------------------------------------------------------- +redis:param=value:param2=value2:... +---%<------------------------------------------------------------------------- + +Supported parameters are: + + * host: Redis server host (default: 127.0.0.1) + * port: Redis server port (default: 6379) + * prefix: Prefix to add to all keys (default: empty) + * db: Database number (default: 0) + * expire_secs=<n>: Set expiration value to all the keys + * timeout_msecs: Abort lookups after specified number of milliseconds + (default: 30000) + +Proxy +----- + +---%<------------------------------------------------------------------------- +proxy:[<dict path>]:<destination dict> +---%<------------------------------------------------------------------------- + +Proxying is used to perform all dictionary accessing via the dict processes. +(The dict processes exist only if dict proxying is used.) This is especially +useful with backends where their initialization is relatively expensive, such +as SQL. The dict processes will then perform also connection pooling. + +If <dict path> is specified, it points to the socket where the dict server is +answering. The default is to use $base_dir/dict. Usually this is changed to +"dict-async" if the dict backend support asynchronous lookups (e.g. ldap, +pgsql, cassandra). The dict-async service allows more than one client, so this +configuration prevents creating unnecessarily many dict processes. + +The <destination dict> contains the dict name in the dict { .. } settings. For +example:'proxy:dict-async:quota' + +See <Dict.txt> for more information about the dict server. + +SQL +--- + +---%<------------------------------------------------------------------------- +<sql driver>:<path to dict-sql config> +---%<------------------------------------------------------------------------- + +The <sql driver> contains the SQL driver name, such as "mysql", "pgsql", +"sqlite" or "cassandra". + +The dict-sql config file consists of SQL server configuration and mapping of +keys to SQL tables/fields. + +SQL Connect String +------------------ + +---%<------------------------------------------------------------------------- +connect = host=localhost dbname=mails user=sqluser password=sqlpass +---%<------------------------------------------------------------------------- + +The connect setting is exactly the same as used for <SQL passdb/userdb> +[AuthDatabase.SQL.txt]. See 'example-config/dovecot-sql.conf.ext' for detailed +information. + +SQL Mapping +----------- + +SQL mapping is done with a dict key pattern and fields. When a dict lookup or +update is done, Dovecot goes through all the maps and uses the first one whose +pattern matches the dict key. + +For example when using dict for a per-user quota value the map looks like: + +---%<------------------------------------------------------------------------- +map { + pattern = priv/quota/storage + table = quota + username_field = username + value_field = quota_bytes +} +---%<------------------------------------------------------------------------- + +This means that: + + * The dict key must match exactly "priv/quota/storage". The dict keys are + hardcoded in the Dovecot code, so depending on what functionality you're + configuring you need to know the available dict keys used it. + * This is a private dict key ("priv/" prefix), which means that there must be + a username_field. The username_field is assumed to be (at least part of) the + primary key. In this example we don't have any other primary keys. + * With MySQL the above map translates to SQL queries: + * 'SELECT quota_bytes FROM quota WHERE username = '$username_field'' + * 'INSERT INTO quota (username, quota_bytes) VALUES ('$username_field', + '$value') ON DUPLICATE KEY UPDATE quota_bytes='$value'' + +You can also access multiple SQL fields. For example acl_shared_dict can +contain: + +---%<------------------------------------------------------------------------- +map { + pattern = shared/shared-boxes/user/$to/$from + table = user_shares + value_field = dummy + + fields { + from_user = $from + to_user = $to + } +} +---%<------------------------------------------------------------------------- + + * The acl_shared_dict always uses "1" as the value, so here the value_field is + called "dummy". + * The SQL from_user and to_user fields are the interesting ones. Typically the + extra fields would be part of the primary key. + * With MySQL the above map translates to SQL queries: + * 'SELECT dummy FROM user_shares WHERE from_user = '$from' AND to_user = + '$to'' + * 'INSERT INTO user_shares (from_user, to_user, dummy) VALUES ('$from', + '$to', '$value') ON DUPLICATE KEY UPDATE dummy='$value'' + +LDAP (v2.2.24+) +--------------- + +LDAP support is very similar to SQL support, but there is no write support. + +Configuration +------------- + +---%<------------------------------------------------------------------------- +dict { + somedict = ldap:/path/to/dovecot-ldap-dict.conf.ext +} +---%<------------------------------------------------------------------------- + +Then in ext file put + +---%<------------------------------------------------------------------------- +uri = ldap://hostname +bind_dn = optional bind dn +password = optional password +timeout = optional timeout +debug = 0 or 1 (optional, as well) +tls = yes|try|no (default is try) +---%<------------------------------------------------------------------------- + + * uri - LDAP connection URI as expected by OpenLDAP. + * bind_dn - DN or upn to use for binding + * password - password to use, only SIMPLE auth is supported at the moment + * timeout - How long to wait for reply, default is 30 seconds + * debug - 0 off, 1 on, will produce metric ton of output + * tls - yes = require either ldaps or successful start TLS, try = send start + TLS if necessary, no = do not send start TLS + +To map some key to a search do + +---%<------------------------------------------------------------------------- +map { + pattern = priv/test/mail + filter = (mail=*) # the () is required + base_dn = ou=container,dc=domain + username_attribute = uid # default is cn + value_attribute = mail +} +---%<------------------------------------------------------------------------- + +To do some more complex search + +---%<------------------------------------------------------------------------- +map { + pattern = priv/test/mail/$location + filter = (&(mail=*)(location=%{location}) # the () is required + base_dn = ou=container,dc=domain + username_attribute = uid # default is cn + value_attribute = mail + + fields { + location=$location + } +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Director.txt b/doc/wiki/Director.txt new file mode 100644 index 0000000..1ffa364 --- /dev/null +++ b/doc/wiki/Director.txt @@ -0,0 +1,366 @@ +Director +======== + +Director can be used by <Dovecot's IMAP/POP3/LMTP proxy> +[PasswordDatabase.ExtraFields.Proxy.txt] to keep a temporary user -> mail +server mapping. As long as user has simultaneous connections, the user is +always redirected to the same server. Each proxy server is running its own +director process, and the directors are communicating the state to each others. +Directors are mainly useful for setups where all of the mail storage is seen by +all servers, such as with NFS or a cluster filesystem. + +First test non-director proxying +-------------------------------- + +The director is simply a small add-on for Dovecot proxy. Before configuring +director, you should test that a simple proxying setup with static destination +server works. See the <Proxy> [PasswordDatabase.ExtraFields.Proxy.txt] page for +more information about how to configure it. If you have a simple setup, you can +test this easily using a static passdb: + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = proxy=y host=10.2.0.20 nopassword=y +} +---%<------------------------------------------------------------------------- + +Once finished testing, remember to remove the "host" field. + +Servers +------- + +You need one or more servers assigned for Dovecot proxies. The same servers +could also act as backends handling the mails, but you need to run <two +separate Dovecot configurations> [RunningDovecot.txt] in different ports. This +may get a bit confusing, so it's not recommended (although v2.1 makes it easier +with 'instance_name' setting). + +The directors are going to connect to each others in a ring. For example if you +have servers called A, B and C, director will create connections A->B, B->C and +C->A. + +Director configuration +---------------------- + +In example configuration you can configure director from +'conf.d/10-director.conf'. + +Listeners +--------- + +Configure the listeners that director requires: + +---%<------------------------------------------------------------------------- +service director { + unix_listener login/director { + mode = 0666 + } + fifo_listener login/proxy-notify { + mode = 0600 + user = $default_login_user + } + # NOTE: director-userdb socket is actually used only for passdb lookups, not +userdb lookups + unix_listener director-userdb { + mode = 0600 + } + inet_listener { + port = 9090 + } + ## uncomment this if you want to use + ## doveadm director status -a ip:9091 + #inet_listener director-doveadm { + # port = 9091 + #} +} +---%<------------------------------------------------------------------------- + +The port 9090 will be used for listening and connecting to other directors. +You're free to use any port number you want. + +Configuring list of director servers +------------------------------------ + +List all of your directors in 'director_servers' setting separated by spaces. +You can use: + + * IP addresses + * hostnames + * hostnames that expand to multiple IPs (e.g. you could have a "directors-all" + DNS entry that expands to all directors' IPs) + +You can also add :port after the IP/host. The default port is the same as what +director service's inet_listener is using (the port 9090 above). + +Note that the same director must not be listed multiple times with different +IPs. This especially means that a hostname can't expand to both IPv4 and IPv6 +address. Otherwise Dovecot becomes confused about what directors actually +exist. This also means that a single director ring must use either IPv4 or IPv6 +addresses, but not both at the same time. + +For example if you have 3 directors, you could set: + +---%<------------------------------------------------------------------------- +director_servers = 10.1.0.2 10.1.0.3 10.1.0.4 +---%<------------------------------------------------------------------------- + +Configuring list of mail servers +-------------------------------- + +List all of your backend mail servers in 'director_mail_servers' setting +separated by spaces. You can use: + + * IP addresses + * IP ranges (e.g. 10.2.0.10-10.2.0.30) + * hostnames + * hostnames that expand to multiple IPs + +For example if you had 20 mail servers with consecutive IPs: + +---%<------------------------------------------------------------------------- +director_mail_servers = 10.2.0.11-10.2.0.30 +---%<------------------------------------------------------------------------- + +Enabling director +----------------- + +Enable director for the wanted login services by telling them to connect to +director socket instead of the default login socket: + +---%<------------------------------------------------------------------------- +service imap-login { + executable = imap-login director +} +service pop3-login { + executable = pop3-login director +} +---%<------------------------------------------------------------------------- + +Consistent hashing should be enabled for new director clusters. It's not +possible to change this setting without a complete director cluster shutdown. +Using it reduces users being moved around when doing backend changes. + +---%<------------------------------------------------------------------------- +director_consistent_hashing = yes # Supported by v2.2.16+ +---%<------------------------------------------------------------------------- + +If you want to enable director for LMTP, also set: + +---%<------------------------------------------------------------------------- +# LMTP first does a passdb lookup to to see if there's a proxy field returned. +# If not, it fallbacks to doing userdb lookup. +lmtp_proxy = yes + +protocol lmtp { + # NOTE: director-userdb socket is actually used only for passdb lookups, not +userdb lookups + auth_socket_path = director-userdb +} + +# If you want lmtp-proxy listening on the network, uncomment the following: +#service lmtp { +# inet_listener lmtp { +# port = 24 +# } +#} +---%<------------------------------------------------------------------------- + +By default LMTP proxy connects to the same port in backend as what was used for +the incoming connection. + +Other settings +-------------- + +Directors redirect a user to the same server always the user has active +connections. The redirection is also done for a while after the last connection +already disconnected. This is mainly to avoid trouble with NFS caches that +haven't yet expired. You can configure this setting from: + +---%<------------------------------------------------------------------------- +director_user_expire = 15 min +---%<------------------------------------------------------------------------- + +'doveadm director kick' and 'doveadm director move' need to be able to connect +to the 'ipc' socket. Make sure the director process can do it: + +---%<------------------------------------------------------------------------- +service ipc { + unix_listener ipc { + user = dovecot # This is already the default in v2.3.1+ + } +} +---%<------------------------------------------------------------------------- + +Passdb configuration +-------------------- + +Your passdb must return "proxy" <extra field> +[PasswordDatabase.ExtraFields.txt], otherwise director doesn't do anything. + +Director works by adding a "host" extra field to the auth reply, which contains +the temporary destination mail server. This "host" field isn't added if the +passdb lookup already returns "host". This allows configuring some users to be +always proxied to a specific server. + +If the backend servers verify password, you can use static passdb for director: + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = proxy=y nopassword=y +} +---%<------------------------------------------------------------------------- + +Note that while this is the simplest director configuration, users will be +assigned to a backend before they have been authenticated. A director +configured this way can be attacked by sending it a large number of unknown +users. To prevent this, the director should be configured to authenticate the +user and might make use of a master password to log into the backend servers. + +Doveadm server +-------------- + +Use these settings for both director and backends: + +---%<------------------------------------------------------------------------- +service doveadm { + inet_listener { + # any port you want to use for this: + port = 24245 + } +} + +local 10.10.10.0/24 { + # password to use for client authentication + doveadm_password = secret + # allow client to only use specified list of commands (default is all): + #doveadm_allowed_commands = +} +---%<------------------------------------------------------------------------- + +The director also needs the following configuration: + +---%<------------------------------------------------------------------------- +# same port as doveadm's inet_listener +doveadm_proxy_port = 24245 + +protocol doveadm { + # NOTE: director-userdb socket is actually used only for passdb lookups, not +userdb lookups + auth_socket_path = director-userdb +} +---%<------------------------------------------------------------------------- + +Now you can run doveadm commands on the director, and it'll run them +automatically on the correct backend server. + +Health monitoring of backend servers +------------------------------------ + +Brad Davidson has written a small daemon for monitoring backend servers, and +disable/enable them on demand. +Ref:https://dovecot.org/list/dovecot/2010-August/051946.html + +Forcefully moving users to a different backend +---------------------------------------------- + +This is useful if you need to do maintenance on one of the backend servers and +want (active) clients to move to a different backend: + + 1. Disable any watchdog system that will undo changes you make to backend + server weights, such as poolmon. + * Not needed if the watchdog is new enough to use HOST-UP/HOST-DOWN + commands rather than change weights. + 2. Set the weight of the backend server to be worked on to 0: 'doveadm + director add <backend server ip> 0' + 3. Flush current assignments to disable new connections to this backend: + 'doveadm director flush <backend server ip>' + * This will also kick the existing connections to the backend in v2.2.19+. + +Most IMAP clients will silently just reconnect to the (new backend) server +after being kicked (at least Apple Mail 6.0 and Thunderbird 14.0). + +For moving specific users to other servers (e.g. because there are too many +"heavy users" assigned to the same backend), you can use 'doveadm director +move' command in v2.0.14+. This requires the ipc permissions to be configured +correctly (see above). + +Tags +---- + +(Requires v2.2.16+) + +*WARNING*: This feature isn't working perfectly in v2.2.26.1 and older. If two +users with different tags have the same 32bit hash, they may end up going to +the wrong tag's backend. + +With tags you can use a single director ring to serve multiple backend +clusters. Each backend cluster is assigned a tag name, which can be anything +you want. By default everything has an empty tag. A passdb lookup can return +"director_tag" field containing the wanted tag name. If there aren't any +backend servers with the wanted tag, it's treated the same as if there aren't +any backend servers available (= wait for 30 secs for a backend and then return +temporary failure). + +Tags can be added to configuration by adding @tag suffix to IPs/hosts. For +example: + +---%<------------------------------------------------------------------------- +director_mail_servers = 10.0.0.100-10.0.0.110@name1 10.0.0.120@name2 +---%<------------------------------------------------------------------------- + +"doveadm director add" can also add tags either with @tag suffix or with -t +parameter. "doveadm director status user@domain" requires giving the user's +correct tag with -t parameter or the results won't be correct (empty tag's +results are shown). Tags can't currently be changed for an existing host +without removing it first. + +Director and Backend in same server (broken) +-------------------------------------------- + +NOTE: This feature never actually worked. It would require further development +to fix (director would need to add "proxy" field to extra fields and notify +auth that the auth_request can be freed). + +Have the passdb lookup return 'director_proxy_maybe=y'. LMTP however doesn't +currently support mixing recipients to both being proxied and store locally. + +Flush socket +------------ + +(Requires v2.2.26+) + +This allows calling a script for each user (hash) that is moved between +backends. This is triggered by "doveadm director move" and "doveadm director +flush" commands. What happens is: + + * User's connections are kicked from the director cluster + * Flush socket is called and waited on. + * User logins are delayed until the flush socket is finished, or the user move + times out after 30 seconds (hardcoded). + * Only the director that initiated the doveadm command will call the flush + socket. + * director_user_kick_delay is ignored by the initiating director, but used + by the other directors. + +Configuration: + +---%<------------------------------------------------------------------------- +director_flush_script = user-flush +service user-flush { + executable = script /usr/local/bin/user-flush.sh + unix_listener user-flush { + user = dovecot + } +} +---%<------------------------------------------------------------------------- + +The user-flush.sh will receive as parameters: + + * "FLUSH" + * username hash + * old host's IP + * new host's IP + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/DomainLost.txt b/doc/wiki/DomainLost.txt new file mode 100644 index 0000000..d7d24a5 --- /dev/null +++ b/doc/wiki/DomainLost.txt @@ -0,0 +1,60 @@ +Domain (%d) is empty +==================== + +IMAP or POP3 protocol doesn't have explicit support for domains. The usernames +are commonly in user@domain format, and that is also where Dovecot gets the +domain from. If the username doesn't have @domain, then the domain is also +usually empty (unless 'auth_default_realm' setting is used). + +If you login as user@domain, but the %d is still empty, the problem is that +your configuration lost the domain part by changing the username. Dovecot +doesn't keep track of the domain separately from username, so if something +changes username from "user@domain" to just plain "user", the domain is lost +and %d returns nothing. If you have 'auth_debug=yes', this shows up in logs +like: + +---%<------------------------------------------------------------------------- +Info: auth(user@domain.org): username changed user@domain.org -> user +---%<------------------------------------------------------------------------- + +Below are some of the most common reasons for this. + +Settings +-------- + +'auth_username_format = %Ln' lowercases the username but also drops the domain. +Use 'auth_username_format = %Lu' instead. + +'auth_username_format' changes the username permanently, currently it's not +possible to make it affect only the authentication part. + +SQL +--- + +'password_query' gets often misconfigured to drop the domain if username and +domain are stored separately. For example: + +---%<------------------------------------------------------------------------- +# BROKEN: +password_query = SELECT username AS user, password FROM users WHERE username = +'%n' AND domain = '%d' +---%<------------------------------------------------------------------------- + +The "username AS user" changes the username permanently and the domain is +dropped. You can instead use: + +---%<------------------------------------------------------------------------- +# MySQL: +password_query = SELECT concat(username, '@', domain) AS user, password FROM +users WHERE username = '%n' AND domain = '%d' +---%<------------------------------------------------------------------------- + +Or you can return username and domain fields separately and Dovecot will merge +them into a single user field: + +---%<------------------------------------------------------------------------- +password_query = SELECT username, domain, password FROM users WHERE username = +'%n' AND domain = '%d' +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Errors.ChgrpNoPerm.txt b/doc/wiki/Errors.ChgrpNoPerm.txt new file mode 100644 index 0000000..721c672 --- /dev/null +++ b/doc/wiki/Errors.ChgrpNoPerm.txt @@ -0,0 +1,28 @@ +Operation Not Permitted +======================= + +---%<------------------------------------------------------------------------- +imap(user): Error: chown(/home/user/mail/.imap/INBOX, group=12(mail)) failed: +Operation not permitted (egid=1000(user), group based on /var/mail/user - see +http://wiki2.dovecot.org/Errors/ChgrpNoPerm) +---%<------------------------------------------------------------------------- + +This means that Dovecot tried to copy '/var/mail/user' file's group (mail) to +the index file directory it was creating ('/home/user/mail/.imap/INBOX'), but +the process didn't belong to the mail group, so it failed. This is important +for preserving access permissions with <shared mailboxes> +[SharedMailboxes.txt]. Group copying is done only when it actually changes the +access permissions; for example with 0600 or 0666 mode the group doesn't matter +at all, but with 0660 or 0640 it does. + +To solve this problem you can do only one of two things: + + 1. If the group doesn't actually matter, change the permissions so that the + group isn't copied (e.g.'chmod 0600 /var/mail/*', see + <MailLocation.mbox.txt>) + 2. Give the mail process access to the group (e.g. 'mail_access_groups=mail' + setting). However, this is dangerous.It allows users with shell access to + read other users' INBOXes + [http://dovecot.org/list/dovecot-news/2008-March/000060.html]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Events.txt b/doc/wiki/Events.txt new file mode 100644 index 0000000..39e0a5a --- /dev/null +++ b/doc/wiki/Events.txt @@ -0,0 +1,288 @@ +Events +====== + +List of all events that can be used in <Statistics.txt> and elsewhere. Note +that in v2.3 these are added on version basis, so not all events are available +in all v2.3 releases. + +Contents + + + 1. Events + + 1. Authentication Server (v2.3.6) + + 2. Authentication Client (v2.3.6) + + 3. Connection + + 4. HTTP + + 5. IMAP + + 6. DNS + + 7. SQL + +Authentication Server (v2.3.6) +------------------------------ + +These events are generated by authentication in auth process(es). + +*auth_request_finished* + + * user: full username + * original_username: original username used + * translated_username: username after translations + * login_user: when doing login using master_user, this is the user we are + logging in as + * master_user: master username + * error: set when error happens + * successful: yes, when succeeded + * transport: insecure, trusted, TLS + * mechanism: name of used mechanism + * credentials_scheme: type of credential (SHA256-CRYPT, PLAIN, ...) + * policy_penalty: time of penalty added by policy server + * policy_result: ok, delayed, refused + +*auth_passdb_request_started* + + * passdb_name: name of passdb + * passdb: driver + +*auth_passdb_request_finished* + + * passdb_name: name of passdb + * passdb: driver + * user: full username + * master_user: master user name + * username: username without domain + * domain: domain if present + * result: authentication result + +*auth_userdb_request_started* + + * userdb_name: name of passdb + * userdb: driver + +*auth_userdb_request_finished* + + * userdb_name: name of passdb + * userdb: driver + * user: full username + * master_user: master user name + * username: username without domain + * domain: domain if present + * result: authentication result + +*auth_policy_request_finished* + + * mode: allow, report + * policy_response: number + +Authentication Client (v2.3.6) +------------------------------ + +These events are generated by authentication clients (lib-auth). + +*auth_client_request_started* + + * id: event id + +*auth_client_request_continue* + + * id: event id + +*auth_client_request_finished* + + * id: event id + * error: reason + +*auth_client_request_challenged* + + * id: event id + +*auth_client_userdb_lookup_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_userdb_lookup_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +*auth_client_passdb_lookup_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_passdb_lookup_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +*auth_client_userdb_list_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_userdb_list_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +*auth_client_cache_flush_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_cache_flush_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +Connection +---------- + +These events apply only for connections using the "connection API". +Unfortunately not all connections currently use this, so these events work for +some types of connections, but not for others. + +*client_connection_connected* + + * client_ip: source IP + * client_port: source port + * ip: target IP + * port: target port + +*client_connection_disconnected* + + * fields from *client_connection_connected* + * bytes_in: amount of data read + * bytes_out: amount of data written + * reason: disconnection reason + +*server_connection_connected* + + * client_ip: source IP + * client_port: source port + * ip: target IP + * port: target port + * bytes_in: amount of data read + * bytes_out: amount of data written + +*server_connection_disconnected* + + * fields from *server_connection_connected* + * bytes_in: amount of data read + * bytes_out: amount of data written + * reason: disconnection reason + +HTTP +---- + +*http_request_finished* + + * status_code: HTTP result status code + * attempts: Number of individual HTTP request attempts that were done (i.e. + number of retries after failures + 1) + * redirects: Number of redirects that were done while processing this request. + + * bytes_in, bytes_out: Number of bytes received/sent for this request. + +*http_request_redirected* + + * Intermediate event sent while HTTP request is being redirected. The + *http_request_finished* is still being sent as well. + * Same fields as *http_request_finished* + +*http_request_retried* + + * Intermediate event sent while HTTP request is being retried. The + *http_request_finished* is still being sent as well. + * Same fields as *http_request_finished* + +IMAP +---- + +*imap_command_finished* + +NOTE: These are currently not sent for pre-login IMAP commands. + + * tag: Command tag + * name: Command name + * args: Command's full parameters + * human_args: Command's parameters as more human-readable output + * tagged_reply_state: OK, NO, BAD + * tagged_reply: Full tagged reply, e.g. "OK SELECT finished." + * last_run_time: Timestamp when the command was running last time (it's + followed by "mailbox sync" that can take some time) + * running_usecs: How many usecs this command itself has spent running + * lock_wait_usecs: How many usecs this command itself has spent waiting for + locks. + * bytes_in, bytes_out: How many bytes of client input/output command has used. + +DNS +--- + +*dns_worker_request_finished* and *dns_request_finished* + + * error: Human readable error + * error_code: Error code usable with net_gethosterror() + +*dns_worker_request_started* and *dns_request_started* + +SQL +--- + +*sql_query_finished* + + * error: Human readable error + * error_code: Error code (if available) + * query_first_word: First word of the query, such as SELECT + +*sql_transaction_finished* + + * error: Human readable error + * error_code: Error code (if available) + +*sql_connection_finished* + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/FindMailLocation.txt b/doc/wiki/FindMailLocation.txt new file mode 100644 index 0000000..ed8d62c --- /dev/null +++ b/doc/wiki/FindMailLocation.txt @@ -0,0 +1,65 @@ +Finding Your Mail +================= + +Before configuring Dovecot, you'll need to know where your mails are located. +You should already have an SMTP server installed and configured to deliver +mails somewhere, so the easiest way to make Dovecot work is to just use the +same location. Otherwise you could create '~/Maildir' directory and configure +your SMTP server to use the Maildir format. + +First send a test mail to yourself (as your own non-root user): + +---%<------------------------------------------------------------------------- +echo "Hello me" | mail -s "Dovecot test" $USER +---%<------------------------------------------------------------------------- + +Now, find where the mail went. Here's a simple script which checks the most +common locations: + +---%<------------------------------------------------------------------------- +for mbox in /var/mail/$USER /var/spool/mail/$USER ~/mbox ~/mail/* ~/*; do + grep -q "Dovecot test" "$mbox" && echo "mbox: $mbox" +done +grep -q "Dovecot test" ~/Maildir/new/* 2>/dev/null && echo "Maildir: ~/Maildir" +---%<------------------------------------------------------------------------- + +mbox +---- + +In most installations your mail went to '/var/mail/username' file. This file is +called *INBOX* in IMAP world. Since IMAP supports multiple mailboxes, you'll +also have to have a directory for them as well. Usually '~/mail' is a good +choice for this. For installation such as this, the mail location is specified +with (typically in 'conf.d/10-mail.conf'): + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +Where '%u' is replaced with the username that logs in. Similarly if your INBOX +is in '~/mbox', use: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=~/mbox +---%<------------------------------------------------------------------------- + +Maildir +------- + +Maildir exists almost always in '~/Maildir' directory. The mail location is +specified with (typically in 'conf.d/10-mail.conf'): + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +---%<------------------------------------------------------------------------- + +Problems? +--------- + +If you can't find the mail, you should check your SMTP server logs and +configuration to see where it went or what went wrong. + +If you can find the mail, but it's in more exotic location, see if +<MailLocation.txt> can help you to configure it. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/FinishBasicConfiguration.txt b/doc/wiki/FinishBasicConfiguration.txt new file mode 100644 index 0000000..42a63ad --- /dev/null +++ b/doc/wiki/FinishBasicConfiguration.txt @@ -0,0 +1,12 @@ +Finishing Basic Configuration +============================= + +Unless you're going to have only virtual users and you don't care about their +passwords,*switch back to disable_plaintext_only = yes* and <set up SSL> +[SSL.txt]. + +Plaintext authentication is still allowed from localhost, so you can have your +webmail application to connect to Dovecot without using SSL or even having to +configure it. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HAProxy.txt b/doc/wiki/HAProxy.txt new file mode 100644 index 0000000..e7194c7 --- /dev/null +++ b/doc/wiki/HAProxy.txt @@ -0,0 +1,90 @@ +HAProxy Support +=============== + +For high availability, client connections can be spread across multiple backend +servers using HAProxy [http://www.haproxy.org/]. This solution is often +employed for large Dovecot installations as a replacement for a hardware load +balancer. A common problem with such a reverse proxy +[http://en.wikipedia.org/wiki/Reverse_proxy] is that Dovecot does not talk to +the client directly. This means that the connection information (IP:port) that +Dovecot uses for logging and authentication purposes points to the proxy, +rather than the real client. To solve this, the proxy needs to convey the +connection information to the backend somehow, so that that information is +available there. The developers of HAProxy defined a custom Proxy Protocol +[http://blog.haproxy.com/haproxy/proxy-protocol/] for this purpose. By this +protocol, the proxy sends the connection information immediately after +connection setup in a special initial header. Note that this isn't normally +needed after the initial Dovecot proxies, because Dovecot internally uses +IMAP/POP3/LMTP extensions to forward the original IP address. Dovecot supports +both versions of the Proxy Protocol since Dovecot version 2.2.19. + +Dovecot Configuration +--------------------- + +The following global settings relate to HAProxy: + +haproxy_trusted_networks = : + A space-separated list of trusted network ranges for HAProxy connections. + Connections from networks outside these ranges to ports that are configured + for HAProxy are aborted immediately. + +haproxy_timeout = 3 : + The time in seconds after which a HAPRoxy connection is aborted when no + complete header is received. + +The HAPRoxy protocol can be enabled for specific <TCP listeners> +[Services.txt]. This way, a service such as IMAP or POP3 can accept both normal +and HAProxy connections. A TCP listener is configured for HAProxy by setting +'haproxy=yes' for that listener. If 'haproxy=yes' is set for a listener, its +use is mandatory on that port; i.e., if the client is not a proper proxy (its +omits the PROXY header), the connection will be aborted. + +For example, to enable normal IMAP connections on port 143, SSL connections on +port 993 and HAProxy connections on port 10143, the 'imap-login' service is +configured as follows: + +---%<------------------------------------------------------------------------- +service imap-login { + inet_listener imap { + port = 143 + } + inet_listener imaps { + port = 993 + ssl = yes + } + inet_listener imap_haproxy { + port = 10143 + haproxy = yes + } +} +---%<------------------------------------------------------------------------- + +HAProxy Configuration +--------------------- + +The documentation of this feature on the HAProxy side is a bit fragmented +between the HAProxy Configuration Manual +[http://www.haproxy.org/download/1.6/doc/configuration.txt] and the Proxy +Protocol documentation +[http://www.haproxy.org/download/1.6/doc/proxy-protocol.txt]. In summary, it is +enabled by including the 'send-proxy' setting in the 'server' lines. It is +therefore only enabled on a per-server basis. + +For example, the 'frontend' and 'backend' configuration of HAProxy could look +as follows: + +---%<------------------------------------------------------------------------- +frontend ft_imap + bind :143 + mode tcp + default_backend bk_imap + +backend bk_imap + mode tcp + balance leastconn + stick store-request src + stick-table type ip size 200k expire 30m + server s1 backend.example.com:10143 send-proxy-v2 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.AntispamWithSieve.txt b/doc/wiki/HowTo.AntispamWithSieve.txt new file mode 100644 index 0000000..e8aea06 --- /dev/null +++ b/doc/wiki/HowTo.AntispamWithSieve.txt @@ -0,0 +1,270 @@ +Replacing antispam plugin with IMAPSieve +======================================== + +Contents + + + 1. Replacing antispam plugin with IMAPSieve + + 1. Caveats and possible pitfalls + + 2. Dovecot configuration + + 3. Sieve scripts + + 4. Shell scripts + + 1. For spamassassin + + 2. For dspam + + 3. For rspamd + + 5. Debugging + + 6. RoundCube + +You will need at least pigeonhole v0.4.14 for this. If you have already +configured sieve, please adjust the following to match your setup. + +Caveats and possible pitfalls +----------------------------- + + * INBOX name is case-sensitive + * <IMAP Sieve> [Pigeonhole.Sieve.Plugins.IMAPSieve.txt] will *only* apply to + IMAP. It *will not* apply to LDA or LMTP. Use <Sieve> [Pigeonhole.Sieve.txt] + normally for LDA/LMTP. + * With this configuration, moving mails will slow down due to learn being done + per email. If you want to avoid this, you need to think of something else. + Probably piping things into a FIFO or perhaps using a socket based worker + might work better. + * Please read <Pigeonhole.Sieve.txt> and <Pigeonhole.Sieve.Configuration.txt> + to understand sieve configuration better. + * Please read <Pigeonhole.Sieve.Plugins.txt> for more information about sieve + extensions. + * If you run Spamassassin trough Amavis and you use a virtual users setup, you + should instead configure Spamassassin to use MySQL/PostgreSQL as a backend, + unless you want a headache with file permissions and lock files. You can + find instructions here + [http://www.iredmail.org/docs/store.spamassassin.bayes.in.sql.html]. In this + case, the '-u' parameter passed to 'sa-learn' (and the relevant sieve + variables) is obsolete and can be safely removed. + * Reloading dovecot doesn't activate changes in this configuration, you'll + need to perform a full restart. + +Changes: + + * 2017/11/20 - Possibility of using spamc with <SpamAssassin.txt> to mitigate + multi-message delays + * 2017/05/05 - Recommendation about Virtual Users and using an SQL Backend. + Added brief info about <RoundCube.txt>. + * 2017/04/01 - Pass imap user to scripts. + * 2017/03/19 - Added rspamd scripts and mention about sieve plugins. + * 2017/02/13 - Improved documentation and added instructions for Spam->Trash. + (Thanks for everyone who commented on mailing list) + * 2017/02/10 - Removed imap_stats (it's not needed). + * 2018/04/11 - Added notes about sa-learn/spamc and warning about sieve script + location. + +Dovecot configuration +--------------------- + +---%<------------------------------------------------------------------------- +protocol imap { + mail_plugins = $mail_plugins imap_sieve +} + +plugin { + sieve_plugins = sieve_imapsieve sieve_extprograms + + # From elsewhere to Spam folder + imapsieve_mailbox1_name = Spam + imapsieve_mailbox1_causes = COPY + imapsieve_mailbox1_before = file:/usr/lib/dovecot/sieve/report-spam.sieve + + # From Spam folder to elsewhere + imapsieve_mailbox2_name = * + imapsieve_mailbox2_from = Spam + imapsieve_mailbox2_causes = COPY + imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/report-ham.sieve + + sieve_pipe_bin_dir = /usr/lib/dovecot/sieve + + sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment +} +---%<------------------------------------------------------------------------- + +Sieve scripts +------------- + +*You cannot run scripts anywhere you want* + +Sieve allows you to only run scripts under sieve_pipe_bin_dir. You can't use +/usr/local/bin/my-sieve-filter.sh, you have to put the script under +sieve_pipe_bin_dir and use my-sieve-filter.sh instead. + +Create directory /usr/lib/dovecot/sieve and put following files to that: + +report-spam.sieve + +---%<------------------------------------------------------------------------- +require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; + +if environment :matches "imap.user" "*" { + set "username" "${1}"; +} + +pipe :copy "sa-learn-spam.sh" [ "${username}" ]; +---%<------------------------------------------------------------------------- + +report-ham.sieve + +---%<------------------------------------------------------------------------- +require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; + +if environment :matches "imap.mailbox" "*" { + set "mailbox" "${1}"; +} + +if string "${mailbox}" "Trash" { + stop; +} + +if environment :matches "imap.user" "*" { + set "username" "${1}"; +} + +pipe :copy "sa-learn-ham.sh" [ "${username}" ]; +---%<------------------------------------------------------------------------- + +Shell scripts +------------- + +For spamassassin +---------------- + +*Untested* + +spamc interaction scripts are not tested yet. + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +# you can also use tcp/ip here, consult spamc(1) 2 +exec /usr/bin/spamc -u ${1} -L spam -C report 3 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +# you can also use tcp/ip here, consult spamc(1) 2 +exec /usr/bin/spamc -u ${1} -L ham -C report 3 +---CodeArea------------------------------------------------------------------- + +You can also use sa-learn. + +Note that using sa-learn often incurs significant start-up time for every +message. This can cause "lockout" of the user until all the processes +sequentially complete, potentially tens of seconds or minutes. If spamd is +being used and the administrator is willing to accept the potential security +issues of allowing unauthenticated learning of spam/ham, spamd can be envoked +with the --allow-tell option and spamc with the --learntype= option. Please +consult the man pages of spamd and spamc for further details. + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/sa-learn -u ${1} --spam 2 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/sa-learn -u ${1} --ham 2 +---CodeArea------------------------------------------------------------------- + +For dspam +--------- + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/dspam --client --user ${1} --class=spam --source=error 2 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/dspam --client --user ${1} --class=innocent --source=error 2 +---CodeArea------------------------------------------------------------------- + +*CRLF handling* + +dspam may fail to read the mail if it contains CRLF line endings, add the +*Broken lineStripping* option in dspam.conf if needed. + +For rspamd +---------- + +By default, rspamd does global learning. If you want per-user classification, +or something more complex, see +https://rspamd.com/doc/configuration/statistic.html + +Alternative scripts can be found from +https://github.com/darix/dovecot-sieve-antispam-rspamd/ + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/rspamc -h /run/rspamd/worker-controller.socket -P <secret> +learn_spam 2 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/rspamc -h /run/rspamd/worker-controller.socket -P <secret> +learn_ham 2 +---CodeArea------------------------------------------------------------------- + +Before running following commands, make sure dovecot.conf has all the sieve +configuration you want. Then run following commands: + +---%<------------------------------------------------------------------------- +sievec /usr/lib/dovecot/sieve/report-spam.sieve +sievec /usr/lib/dovecot/sieve/report-ham.sieve +chmod +x /usr/lib/dovecot/sieve/sa-learn-ham.sh +/usr/lib/dovecot/sieve/sa-learn-spam.sh +---%<------------------------------------------------------------------------- + +Now your learn scripts should be invoked when you move mails between folders. + +Debugging +--------- + +To debug, you need to import "vnd.dovecot.debug" extension. Then you can put, +when required + +---%<------------------------------------------------------------------------- +debug_log "something" +---%<------------------------------------------------------------------------- + +variables are supported in this. + +RoundCube +--------- + +Recent versions of RoundCube [https://roundcube.net/] include a markasjunk2 +plugin [https://plugins.roundcube.net/packages/johndoh/markasjunk2] for +allowing users to mark Spam/Ham in a convenient way. Please make sure the +Junk/Spam folder matches your configuration. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.EximAndDovecotSASL.txt b/doc/wiki/HowTo.EximAndDovecotSASL.txt new file mode 100644 index 0000000..99d178c --- /dev/null +++ b/doc/wiki/HowTo.EximAndDovecotSASL.txt @@ -0,0 +1,54 @@ +Exim and Dovecot SASL +===================== + +Exim v4.64+ users can use Dovecot SASL instead of Cyrus SASL for authenticating +SMTP clients. + +conf.d/10-master.conf +--------------------- + +---%<------------------------------------------------------------------------- +service auth { +... +#SASL + unix_listener auth-client { + mode = 0660 + user = mail + } +... +} +---%<------------------------------------------------------------------------- + +conf.d/10-auth.conf +------------------- + +---%<------------------------------------------------------------------------- +auth_mechanisms = plain login +---%<------------------------------------------------------------------------- + +exim.conf +--------- + +Create authenticators for Dovecot: + +---%<------------------------------------------------------------------------- +dovecot_login: + driver = dovecot + public_name = LOGIN + server_socket = /var/run/dovecot/auth-client +# setting server_set_id might break several headers in mails sent by +authenticated smtp. So be careful. + server_set_id = $auth1 + +dovecot_plain: + driver = dovecot + public_name = PLAIN + server_socket = /var/run/dovecot/auth-client + server_set_id = $auth1 +---%<------------------------------------------------------------------------- + +If you are having problems with this not working ensure that you are using +version 4.72 or greater of exim. Previous versions of exim have trouble with +the version of the protocol used in Dovecot 2 + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.ImapcProxy.txt b/doc/wiki/HowTo.ImapcProxy.txt new file mode 100644 index 0000000..805c989 --- /dev/null +++ b/doc/wiki/HowTo.ImapcProxy.txt @@ -0,0 +1,174 @@ +Dovecot imapc proxy +=================== + +Using Dovecot as a secure IMAP Proxy in front of Exchange, using Exchange +Authentication and IMAPC. This requires Dovecot 2.1.rc1 or newer. + +Many thanks to Timo on the Dovecot mailing list for all his help! + +This " <HowTo.txt>" is based on already having Dovecot already compiled and +installed. + + 1. Create an unprivileged, non-system account user and group for the proxy, + with a home directory. This needs to have a writable home directory, but no + other privileges. + + ---%<--------------------------------------------------------------------- + [root@localhost]# useradd imapproxy + ---%<--------------------------------------------------------------------- + + 2. Verify that the user can not login: + + ---%<--------------------------------------------------------------------- + [root@localhost]# grep imapproxy /etc/shadow + ---%<--------------------------------------------------------------------- + + You should see something like: + + ---%<--------------------------------------------------------------------- + imapproxy:!!:nnnn:0:nn:n::: + ---%<--------------------------------------------------------------------- + + The important part is the "!!". This indicates that the account is locked. + If you don't see this, lockout the account (check man passwd) + 3. Create '/etc/dovecot/dovecot.conf' or + ('/usr/local/etc/dovecot/dovecot.conf') as appropriate: + + ---%<--------------------------------------------------------------------- + ## Dovecot configuration file + + mail_uid = imapproxy + mail_gid = imapproxy + + protocols = imap + + listen = *, :: + + mail_location = imapc:~/imapc + # Change the line below to reflect the IP address of your Exchange Server. + imapc_host = 10.1.2.3 + imapc_port = 143 + + passdb { + driver = imap + # Change the line below to reflect the IP address of your Exchange + Server. + args = host=10.1.2.3 + default_fields = userdb_imapc_user=%u userdb_imapc_password=%w + } + userdb { + driver = prefetch + } + + # /home/imapproxy is the home directory for the imapproxy user, and + # %u is a subdir that will be automatically created for each IMAP user when + they connect + + mail_home = /home/imapproxy/%u + + auth_mechanisms = plain login + + # This is the auth service used by Postfix to do dovecot auth. + service auth { + unix_listener auth-userdb { + } + inet_listener { + port = 12345 + } + } + + ## + ## SSL settings + ## + + # These will need to ba adjusted to point to *your* certificates, not mine + 8-) + # The ssl_ca line refers to the intermediate certificate bundle which may + or may not be required by your SSL provider + + ssl_cert = </etc/pki/tls/certs/machine.example.org.crt + ssl_key = </etc/pki/tls/private/machine.example.org.key + ssl_ca = </etc/pki/tls/certs/gd_bundle.crt + ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL + ---%<--------------------------------------------------------------------- + +Start dovecot and test it with openssl as: + +---%<------------------------------------------------------------------------- +openssl s_client -connect machine.example.org:143 -starttls imap +---%<------------------------------------------------------------------------- + +You should see a whole bunch of SSL information, and the last line should say: + +---%<------------------------------------------------------------------------- +. OK Pre-login capabilities listed, post-login capabilities have more. +---%<------------------------------------------------------------------------- + +Next, type: + +---%<------------------------------------------------------------------------- +01 LOGIN username badpassword +---%<------------------------------------------------------------------------- + +You should then see: + +---%<------------------------------------------------------------------------- +01 NO [AUTHENTICATIONFAILED] Authentication failed +---%<------------------------------------------------------------------------- + +And should see a list similar to this: + +---%<------------------------------------------------------------------------- +* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT +SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN +NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT +SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS FUZZY +02 OK Logged in +---%<------------------------------------------------------------------------- + +If you get this far, the proxy is working and is authenticating against your +exchange server. + +Postfix Configuration +--------------------- + +My configuration is for a closed server that will never allow inbound SMTP from +unauthenticated clients, and authenticates inbound SMTP TLS connections against +the above Dovecot auth service, which in turn authenticates against Exchange, +which authenticates against Active Directory. + +This means that disabling an account in Active Directory, also disables inbound +and outbound mail access. + +If this is what you want, add the following to you /etc/postfix/main.cf file.: + +---%<------------------------------------------------------------------------- +smtpd_sasl_type = dovecot +smtpd_sasl_path = inet:127.0.0.1:12345 +smtpd_sasl_auth_enable = yes + +smtpd_client_restrictions = permit_sasl_authenticated, reject +## Don't relay for anybody from or to anywhere, unless they authenticated + +smtpd_recipient_restrictions = permit_sasl_authenticated reject + +broken_sasl_auth_clients = yes +# Talk to outlook <= 2003 and O Express <=6 + +smtpd_tls_security_level = encrypt +smtpd_tls_received_header = yes + +smtpd_tls_cert_file = /etc/pki/tls/certs/machine.example.org.crt +smtpd_tls_key_file = /etc/pki/tls/private/machine.example.org.key + +smtpd_tls_CAfile = /etc/pki/tls/certs/gd_bundle.crt +# If your Certification Authority requires intermediate certificates, the +bundle goes here. + +tls_random_source = dev:/dev/urandom + +smtpd_tls_auth_only = yes +# only allow auth if it's encrypted +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.PopBSMTPAndDovecot.txt b/doc/wiki/HowTo.PopBSMTPAndDovecot.txt new file mode 100644 index 0000000..edacc19 --- /dev/null +++ b/doc/wiki/HowTo.PopBSMTPAndDovecot.txt @@ -0,0 +1,363 @@ +Contents + + + 1. POP3 (IMAP) before SMTP + + 1. Are you sure you want this? + + 1. Problems with POP-before-SMTP + + 2. Advantages of POP-before-SMTP over SMTP AUTH + + 2. Pop-before-smtp.pl + + 3. DRAC + + 4. SQL + + 1. Example for postgresql, postfix + + 2. Example for MySQL, postfix + + 5. relay-ctrl + +POP3 (IMAP) before SMTP +======================= + +/sometimes also called SMTP-after-POP3 or SMTP-after-IMAP/ + +Are you sure you want this? +--------------------------- + +POP-before-SMTP is generally considered a kludge, originally invented to make +up for the lack of authentication in the original SMTP +[http://en.wikipedia.org/wiki/Smtp] specification for clients on dynamic IP +addresses.ESMTP [http://en.wikipedia.org/wiki/Extended_SMTP] resolved that +shortcoming long ago, and all modern mail clients and servers support it by +now. You should consider implementing ESMTP AUTH +[http://en.wikipedia.org/wiki/SMTP-AUTH] in your mail transport/submission +agent, and using it in your clients, rather than using POP-before-SMTP. See +also <HowTo.PostfixAndDovecotSASL.txt> or <HowTo.EximAndDovecotSASL.txt>. + +Problems with POP-before-SMTP +----------------------------- + + * *Shared IP addresses* are in widespread use. You are opening your server not + only to your user, but to anyone else who might be sharing the same IP + address, other users, other computers in the same NAT. If you lose the + connection, the next one who is assigned your IP also inherits your relay + permit. This might include virus-infected spambot machines. Or consider a + public wireless hotspot or an Internet cafe: both types of establishments + are known to be frequented by spammers. + * *Not properly implemented* in all mail clients: it only works right if the + client checks for new mail immediately before attempting to send. And it can + be very unsafe if longer timeouts are used, such that the user has time to + write an email. + * Probably others. <I> [RobMcGee.txt] ( <Rob McGee> [RobMcGee.txt]) just + thought it was wrong to have a HOWTO page here without a warning about why + /not/ to. Know what you are doing. If you are setting up a new mail service + from scratch, by all means, do it right! + +Advantages of POP-before-SMTP over SMTP AUTH +-------------------------------------------- + + * Likely to be relatively easier to implement in your mail submission agent. + What's easier is a matter of opinion, and it varies, of course, but probably + all MTA/MSA servers support some form of access lists without patching or + recompiling. + * Simple non-technical instructions for users: /"Remember to check for new + mail before you try to send mail."/ + +Pop-before-smtp.pl +================== + +If you want to use pop-before-smtp.pl (from http://popbsmtp.sourceforge.net/) +together with Dovecot, you can use this regular expression to match successful +POP3 and IMAP logins: + +---%<------------------------------------------------------------------------- +$pat = '^(... .. ..:..:..) \S+ (?:pop3|imap)-login: Login: .+ +\[(\d+\.\d+\.\d+\.\d+)\]'; +---%<------------------------------------------------------------------------- + +v1.0RC2 seems to need this format to work properly: + +---%<------------------------------------------------------------------------- +$pat = '^dovecot: (... .. ..:..:..) \S+ (?:pop3|imap)-login: Login: \S+ \S+ \S+ +lip=(\d+\.\d+\.\d+\.\d+)'; +---%<------------------------------------------------------------------------- + +Note: This only works with IPv4, anyone who wants to fix it for IPv6, please do +so:) + +worked for me on Fedora: <drak at navel.gr> + +---%<------------------------------------------------------------------------- +$pat = '(?:pop3|imap)-login: (... .. ..:..:..) Info: Login: \S+ +\[(\d+\.\d+\.\d+\.\d+)\]'; +---%<------------------------------------------------------------------------- + +With v1.0 Alpha 4, the following pattern works: + +---%<------------------------------------------------------------------------- +$pat = '^(... .. ..:..:..) \S+ (?:dovecot: )?(?:imap|pop3)-login: Login: \S+ +\S+ rip=(\d+\.\d+\.\d+\.\d+)' +---%<------------------------------------------------------------------------- + +This works with RHEL 4.3 (at least until IPv6 really catches): + +---%<------------------------------------------------------------------------- +$pat = '(?:pop3|imap)-login: (... .. ..:..:..) Info: Login: \S+ +\[::ffff:(\d+\.\d+\.\d+\.\d+)\]'; +---%<------------------------------------------------------------------------- + +DRAC +==== + +The DRAC historical plugin for Dovecot 1.x, located here +[http://mail.cc.umanitoba.ca/drac/], doesn't work with Dovecot 2.x, since it +relies on the "IP" environment variable, not set anymore by Dovecot 2.x + +a more recent version of this plugin is available here: DRAC Plugin for Dovecot +2.x [http://sourceforge.jp/projects/dovecot2-drac/]. The README file explains +how to compile it. Change the path to your Dovecot 2.x source code into the +Makefile to compile it. + +DRAC runs as a separate daemon, maintaining a BerkeleyDB database of IPs that +have successfully authenticated via POP3 or IMAP, expiring them after 30 +minutes. Installing it therefore requires that both your POP3/IMAP server and +your SMTP daemon (Postfix/Sendmail/qmail) be set up to support it. +DRAC-PLUGIN.c is a small C program, and accessing BerkeleyDB databases is +efficient so it works pretty well. + +By following the instructions you will install a file drac_plugin.so in your +dovecot 'lib/' directories for IMAP and/or POP3 loadable modules. + +To turn on the new DRAC plugin in dovecot, you must set up these lines in your +dovecot.conf. There is a separate section for ''protocol imap'' and another +under ''protocol pop3''; make sure you enable both. + +---%<------------------------------------------------------------------------- + # Support for dynamically loadable modules + mail_plugin_dir = /usr/lib/dovecot/imap # not mandatory + mail_plugins = drac # provide a list of all +plugins you want to load here +---%<------------------------------------------------------------------------- + +Permissions note: the directory containing the drac_plugin.so file has to be +readable by ordinary users. Check your Dovecot error log for help. + +To get DRAC working on your machine, download the main DRAC +[http://mail.cc.umanitoba.ca/drac/] daemon, edit the makefile as directed in +the instructions, and make and install it. You will also want to ensure that +you register the rpcs by executing rpcgen. See the Makefile for more details. + +SQL +=== + +Advantage: you do not have a multi-megabyte Perl daemon reading your logs + +Disadvantage: for each login you need the time and space to execute this script + + 1. tell your MTA to look up IPs authorized to relay in an SQL table + 2. delete old IPs from the table regularly (cron job for example, or a + modification to the script below) + 3. tell dovecot to update the SQL table upon successful login + +Dovecot 1.0 (and probably 0.99) can update a SQL table with the script below. + +/!\ *Note* that *you* must set up a script that deletes old IPs separately, and +*you* also must configure your MTA properly. The script *only* performs the +'update on successful login' step, which alone is insecure without expiring +older IPs!/Add your working examples to this section. This Wiki depends on your +help!/ + +---%<------------------------------------------------------------------------- +#!/bin/sh +# This script created 2005-08-21 by Lorens Kockum +# Released into the Public Domain +# Changes: +# 2006-06-06 Matthias Andree +# - changed $* to "$@" for more robust argument quoting +# Action: when called by dovecot 1.0 as described below, updates an SQL table +# with logged-in IP and current time, and then executes the relevant process. +# Output: normally nothing +# dovecot.conf should be modified with these lines (where +# /usr/lib/dovecot/popbsmtp.sh represents this script): +# protocol pop3 { +# mail_executable = /usr/lib/dovecot/popbsmtp.sh /usr/lib/dovecot/pop3 +# } +# protocol imap { +# mail_executable = /usr/lib/dovecot/popbsmtp.sh /usr/lib/dovecot/imap +# } +# The HOME= lines are necessary to find $HOME/.my.cnf containing login info, +# because mail_executable is executed as root, but without a home directory. +# Of course this script must not be writable by anyone else than root. +( + # drop out IPs from local networks that can relay anyway + IP=`echo $IP | grep -v '^192\.168\.'` + if [ -n "$IP" ] + then + export HOME=/root/ + echo "replace into popbsmtp VALUES('$IP',now());" | mysql mail + export HOME=/ + fi +) >> /var/log/dovecot3 2>&1 +exec "$@" +---%<------------------------------------------------------------------------- + +Example for postgresql, postfix +------------------------------- + +/usr/lib/dovecot/popbsmtp.sh + +---%<------------------------------------------------------------------------- +#!/bin/sh +( + if [ -n "$IP" ] + then + /usr/bin/psql -U popbsmtp -d popbsmtp -c "begin;update auth set +accessed=now() where host=substring('$IP' from 8);commit;insert into auth(host, +accessed) values(substring('$IP' from 8),now());" + fi + +) >> /var/log/dovecot3 2>&1 +exec "$@" +---%<------------------------------------------------------------------------- + +The substring call was necessary because $IP has '::ffff:' or something like +that in front of the IP address on my system. The update followed by an insert, +with the update in a transaction is necessary to replicate mysql's REPLACE INTO +functionality. The INSERT will produce an error if the IP already exists but it +doesn't matter as the UPDATE will have committed by then. + +/etc/postfix/main.cf + +---%<------------------------------------------------------------------------- +smtpd_recipient_restrictions = + permit_mynetworks + permit_sasl_authenticated + permit_tls_clientcerts + check_client_access pgsql:/etc/postfix/popbsmtp.cf + reject_unauth_destination + check_policy_service unix:private/policy +---%<------------------------------------------------------------------------- + +/etc/postfix/popbsmtp.cf + +---%<------------------------------------------------------------------------- +hosts = localhost +user = username +password = secret +dbname = popbsmtp +query = SELECT 'OK' as result FROM auth WHERE host = '%s' +---%<------------------------------------------------------------------------- + +/etc/cron.hourly/popbsmtp_purge + +---%<------------------------------------------------------------------------- +#!/bin/bash +/usr/bin/psql -U popbsmtp -d popbsmtp -c "DELETE FROM auth WHERE (now() - +accessed) > '30 minutes'::interval" +---%<------------------------------------------------------------------------- + +Example for MySQL, postfix +-------------------------- + +Note that you can use this even if pop/imap and smtp are not on the same host +as it is the case in my setup. + +First you have to create a table (in this example named "popbsmtp") with 2 +fields: + + * address (varchar 39, primary) + * last_seen (datetime) + +varchar size 39 is for IPv6 addresses.You should definitely consider adding +IPv6 support to your popbsmtp solution because postfix and dovecot do well with +IPv6. + +/!\ *address field* must be *primary* for "REPLACE into" to work. + +/opt/dovecot-popbsmtp.sh + +---%<------------------------------------------------------------------------- +#!/bin/sh +( + if [ -n "$IP" ] + then + echo "REPLACE INTO virtual_mail.popbsmtp (address,last_seen) +VALUES ('$IP', NOW( ))" \ + | mysql -u user -p secret -h host > /dev/null 2>&1 + fi +) +exec "$@" +---%<------------------------------------------------------------------------- + +mail_executable in dovecot.conf looks something like this: + +---%<------------------------------------------------------------------------- +mail_executable = /opt/dovecot-popbsmtp.sh /usr/libexec/dovecot/imap +---%<------------------------------------------------------------------------- + +postfix map (/etc/postfix/mysql_popbsmtp_access_maps.cf): + +---%<------------------------------------------------------------------------- +hosts = mysqlhost +user = user +password = secret +dbname = virtual_mail +query = SELECT 'OK' FROM popbsmtp WHERE last_seen >= DATE_SUB(NOW(),INTERVAL 30 +MINUTE) AND address = '%s' +---%<------------------------------------------------------------------------- + +In postfix main.cf add the following access map to your recipient restrictions +(/!\ *before* "reject_unauth_destination"): + +---%<------------------------------------------------------------------------- +check_client_access mysql:$config_directory/mysql_popbsmtp_access_maps.cf +---%<------------------------------------------------------------------------- + +The 30 minute relay access period is handled by the INTERVAL in DATE_SUB. So +it's safe anyway, but you should definitely run a cron job daily that deletes +older records. That's to keep the table clean and speed up lookups. You might +also need to run "OPTIMIZE TABLE" via the cron job to free up allocated space. + +relay-ctrl +========== + +relay-ctrl [http://untroubled.org/relay-ctrl/] consists of a few small programs +designed to fit in qmail-like command chains. The most important: + + * 'relay-ctrl-allow' runs after a successful POP/IMAP login, recording the + client IP and timestamp + * 'relay-ctrl-check' runs before the SMTP server, enabling relaying if the + client IP has authenticated recently + +'relay-ctrl-allow' expects to find the client IP in the environment as +'$TCPREMOTEIP'. Dovecot provides it as '$IP', so you'll need this tiny +'dovecot-settcpremoteip' wrapper script: + +---%<------------------------------------------------------------------------- +#!/bin/sh +# +# Wrapper for relay-ctrl-allow that sets TCPREMOTEIP. +TCPREMOTEIP="${IP}"; export TCPREMOTEIP +exec "$@" +---%<------------------------------------------------------------------------- + +Edit 'dovecot.conf' and set 'mail_executable' appropriately, e.g., for IMAP +(this is one long line): + +---%<------------------------------------------------------------------------- +mail_executable = /usr/local/bin/envdir /etc/relay-ctrl +/usr/local/bin/relay-ctrl-chdir /usr/local/bin/dovecot-settcpremoteip +/usr/local/bin/relay-ctrl-allow /usr/local/libexec/dovecot/imap +Dove +---%<------------------------------------------------------------------------- + +Restart Dovecot. Verify that your IMAP client still works. Verify that +relay-ctrl has recorded your client IP. Hook 'relay-ctrl-check' into your SMTP +service, as documented in the relay-ctrl README, and you're done. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.PopRelay.txt b/doc/wiki/HowTo.PopRelay.txt new file mode 100644 index 0000000..e8e9532 --- /dev/null +++ b/doc/wiki/HowTo.PopRelay.txt @@ -0,0 +1,139 @@ +Poprelay is a service that allows your Dovecot pop users to also send e-mails +through your sendmail SMTP server on the same machine.When they check their +e-mails, their IP address is logged in a database. Once the same user wants to +send an e-mail using your SMTP port, then sendmail checks this database and if +the IP is present, then the relay is allowed. This makes it completely +transparent to the user and the server will still be very secure and blocking +all other IP addresses for relaying.The IP address is kept in the database for +about 15 minutes only, so there is no chance for anyone else to use the same IP +address and server. + +Dated 2005-08-19 you can now use this poprelay service with Dovecot. + +The main page of poprelayd is the following: +http://sourceforge.net/projects/poprelay + +Download the latest source there. You have an install script that does almost +everything for you. You still need some manual intervention to add some +required lines in your sendmail.mc file. This is described in the install +documentation.Once this is done, you will have to replace your +/etc/mail/poprelay.conf file by the lines below.Once this is done restart the +poprelay service and it should be working. + +Please report any problem on the poprelay forum. You will need to leave the +v6-mapped ip4 address in the /etc/mail/maillog for the moment (e.g. you should +see '[::ffff:IP_ADDRESS]' in the pop user's login logs). + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +You might have to adapt the first configuration lines if you are using +something else than CentOS 4. + +OpenBSD (using ip4 addresses) produces syslog lines such as: + +Jun 7 09:54:53 mail dovecot: pop3-login: Login: user=<user>, method=PLAIN, +rip=[IP Address], lip=[IP Address] + +To fix the extra word in the split use this line: + +($crap, $info, $string)=split(/\: /,$line); + +To accommodate ip4 address in the form rip=w.x.y.z, use this line: + +($ip) = $line=~/\, rip\=(\d+\.\d+\.\d+\.\d+)/; + +---%<------------------------------------------------------------------------- +#This file is interpreted by perl +#you can do a quick syntax check by doing "perl poprelay.conf" + +#=======================Standard configuration options===================== + +# where POP3/IMAP daemon connections get logged +$logfile = "/var/log/maillog"; + +# Where we put our PID. (dieing output +# will be dumped here too) +$pidfile = "/var/run/poprelayd.pid"; + +# Sendmail map to update. +$dbfile = "/etc/mail/popip.db"; + +#Change this to match the type of db file sendmail needs +#Your perl must support that type of file as well +$dbtype = "DB_HASH"; + +# Minutes an entry lasts. (3000 = ~ 2 days) +# IMAP connections can last a very long time so I like to keep this long. +# The odds that someone will hop onto one of your valid user's old IP's and +# spam from it are so small I wouldn't worry about it. I recommend making +# this long to avoid complications. +$timeout_minutes = 15; + +# Number of seconds to sleep between checks +$log_wait_interval = 5; + +#=======================Advanced configuration options===================== + +#Alternate log line parsers: + +#There can be only one log parser. +#the standard one should work for most systems. The other +#ones may be slightly out of date. I don't have any systems +#that run these servers so I can't update or test the routines. +#If you fix anything with them please let me know and I will +#roll the changes into the main version. +# +#$log_parser = \&log_parse_standard; +#$log_parser = \&log_parse_berkeley; +#$log_parser = \&log_parse_qpopper; +#$log_parser = \&log_parse_qpopper_old; +#$log_parser = \&log_parse_cucipop; +$log_parser = \&log_parse_custom; + +#Custom log line parsing scripts: + +#If you want to create your own log parsing routine, do it here in +#the config file so you can update the poprelayd without losing your +#custom parsing routine. The routine below does the same thing as +#log_parse_standard. It should be a good starting point for any +#customization. It parses lines in many stages so it can be easily +#customized. It will even do dns lookups of hostnames using +#gethostbyname if the program logs the hostname instead of the ip. +# +#If you get something working post it to the forums at +#http://sourceforge.net/projects/poprelay so the next guy doesn't have +#to go through the same headache. I'll try and roll new routines into +#the main program so that poprelayd can work out of the box for all +#the log formats. + +# Dovecot maillog parser: +sub log_parse_custom ($) { + my $s = $_[0]; + my @paddrs; # Packed IP addresses. + my @addrs; # ASCII addresses. + my ($junk,$info,$string,$service,$ip,$host); + ($info, $string)=split(/\: /,$line); + ($service) = $info=~/(\S+)$/; + $service=~s/\[\d+\]//; + return () unless $service=~/^(pop2|pop3|imap)-login$/; + return () unless $string=~/^(Login|Authenticated)/; + ($ip) = $line=~/.*\:\:ffff\:(\d+\.\d+\.\d+\.\d+)\]/; + if ($ip) { + print "$service: $ip\n"; + return ($ip); + } else { + ($host) = $string=~/^(\S+)/; + print "$service: $host\n"; + ($junk, $junk, $junk, $junk, @paddrs) = gethostbyname($host); + while (@paddrs) { + push(@addrs, join('.', unpack('C4', shift(@paddrs)))); + } + return (@addrs); + } +} + +#leave this alone: +1; +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.PostfixAndDovecotSASL.txt b/doc/wiki/HowTo.PostfixAndDovecotSASL.txt new file mode 100644 index 0000000..48aaf15 --- /dev/null +++ b/doc/wiki/HowTo.PostfixAndDovecotSASL.txt @@ -0,0 +1,113 @@ +Postfix and Dovecot SASL +------------------------ + +Since version 2.3, Postfix supports SMTP AUTH through <Dovecot SASL> [Sasl.txt] +as introduced in the Dovecot 1.0 series. If using Postfix obtained from a +binary (such as a .rpm or .deb file), you can check if Postfix was compiled +with support for Dovecot SASL by running the command: + +---%<------------------------------------------------------------------------- +postconf -a +---%<------------------------------------------------------------------------- + +Once you have verified that your installation of Postfix supports Dovecot SASL, +it's very simple to configure: + +Example conf.d/10-master.conf excerpt +------------------------------------- + +---%<------------------------------------------------------------------------- +service auth { +... + unix_listener /var/spool/postfix/private/auth { + mode = 0660 + # Assuming the default Postfix user and group + user = postfix + group = postfix + } + ... +} + +# Outlook Express and Windows Mail works only with LOGIN mechanism, not the +standard PLAIN: +auth_mechanisms = plain login +---%<------------------------------------------------------------------------- + +Example Postfix main.cf excerpt +------------------------------- + +---%<------------------------------------------------------------------------- +smtpd_sasl_type = dovecot + +# Can be an absolute path, or relative to $queue_directory +# Debian/Ubuntu users: Postfix is setup by default to run chrooted, so it is +best to leave it as-is below +smtpd_sasl_path = private/auth + +# On Debian Wheezy path must be relative and queue_directory defined +#queue_directory = /var/spool/postfix + +# and the common settings to enable SASL: +smtpd_sasl_auth_enable = yes +# With Postfix version before 2.10, use smtpd_recipient_restrictions +smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, +reject_unauth_destination +---%<------------------------------------------------------------------------- + +Using SASL with Postfix submission port +--------------------------------------- + +When Dovecot is used as the authentication backend for Postfix it is good +practice to use a dedicated submission port for the MUAs (TCP 587). Not only +can you specify individual parameters in *master.cf* overriding the global ones +but you will not run into internet mail rejection while the Dovecot Auth +Mechanism is unavailable. In this example Postfix is configured to accept TLS +encrypted sessions only, along with several other sanity checks: + + * Verification of alias ownership via Login Maps + * Domainname and recipient plausibility + +'master.cf' + +---%<------------------------------------------------------------------------- +submission inet n - n - - smtpd + -o smtpd_tls_security_level=encrypt + -o smtpd_sasl_auth_enable=yes + -o smtpd_sasl_type=dovecot + -o smtpd_sasl_path=private/auth + -o smtpd_sasl_security_options=noanonymous + -o smtpd_sasl_local_domain=$myhostname + -o smtpd_client_restrictions=permit_sasl_authenticated,reject + -o smtpd_sender_login_maps=hash:/etc/postfix/virtual + -o smtpd_sender_restrictions=reject_sender_login_mismatch + -o +smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject +---%<------------------------------------------------------------------------- + +Dovecot authentication via TCP +------------------------------ + +If Postfix and Dovecot are running on separate servers, you can also +authenticate via TCP. For Dovecot set up an inet_listener: + +---%<------------------------------------------------------------------------- +service auth { + inet_listener { + port = 12345 + } +} +---%<------------------------------------------------------------------------- + +And configure Postfix to use it: + +---%<------------------------------------------------------------------------- +smtpd_sasl_path = inet:dovecot.example.com:12345 +smtpd_sasl_type = dovecot +---%<------------------------------------------------------------------------- + +See also: +--------- + + * http://www.postfix.org/SASL_README.html#server_dovecot + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.Rootless.txt b/doc/wiki/HowTo.Rootless.txt new file mode 100644 index 0000000..b9f23f7 --- /dev/null +++ b/doc/wiki/HowTo.Rootless.txt @@ -0,0 +1,119 @@ +Rootless Installation +===================== + +It's possible to make Dovecot run under a single system user without requiring +root privileges at any point. This shouldn't be thought of as a security +feature, but instead simply as a way for non-admins to run Dovecot in their +favorite mail server. It's also useful if you just wish to test Dovecot without +messing up your system. + +If you think of this as a good way to achieve security, ask yourself which is +worse: + +a) + + * A very small possibility to get root privileges through Dovecot. + * A small possibility without logging in to get into system as a + non-privileged *dovecot* user, chrooted into an empty directory. + * A small possibility to get user's privileges after logging in, but no + possibility to read others' mails since they're saved with different UIDs + (plus you might also be chrooted to your own mailbox). + +b) + + * Absolutely zero possibility to get root privileges through Dovecot. + * A small possibility to get into system as a mail user, possibly even without + logging in, and being able to read everyone's mail (and finally gaining + roots by exploiting some just discovered local vulnerability, unless you + bothered to set up a special chrooted environment). + +Installation +------------ + +Install somewhere under home directory: + +---%<------------------------------------------------------------------------- +./configure --prefix=$HOME/dovecot +make +make install +---%<------------------------------------------------------------------------- + +Dovecot is then started by running '~/dovecot/sbin/dovecot'. The example +configuration file exists in '~/dovecot/share/doc/dovecot/example-config/' and +needs to be copied to '~/dovecot/etc/dovecot/'. + +Configuration +------------- + +The important settings to change for rootless installation are: + + * Set usernames and group to the user which dovecot will be run under: + + ---%<---------------------------------------------------------------------- + default_internal_user = user + default_login_user = user + default_internal_group = group + ---%<---------------------------------------------------------------------- + + * Remove default chrooting from all services: + + ---%<---------------------------------------------------------------------- + service anvil { + chroot = + } + service imap-login { + chroot = + } + service pop3-login { + chroot = + } + ---%<---------------------------------------------------------------------- + + * Change listener ports: + + ---%<---------------------------------------------------------------------- + service imap-login { + inet_listener imap { + port = 10143 + } + inet_listener imaps { + port = 10993 + } + } + service pop3-login { + inet_listener pop3 { + port = 10110 + } + inet_listener pop3s { + port = 10995 + } + } + ---%<---------------------------------------------------------------------- + + * Change logging destination: + + ---%<---------------------------------------------------------------------- + log_path = /home/user/dovecot.log + ---%<---------------------------------------------------------------------- + + * Instead of <passdb PAM> [PasswordDatabase.PAM.txt] use for example + <passwd-file> [AuthDatabase.PasswdFile.txt]: + + ---%<---------------------------------------------------------------------- + passdb { + driver = passwd-file + args = /home/user/dovecot/etc/passwd + } + userdb { + driver = passwd + } + ---%<---------------------------------------------------------------------- + + Where the 'passwd' file contains the username and password for your login + user: + + ---%<---------------------------------------------------------------------- + user:{PLAIN}pass + ---%<---------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.SimpleVirtualInstall.txt b/doc/wiki/HowTo.SimpleVirtualInstall.txt new file mode 100644 index 0000000..cdc21fe --- /dev/null +++ b/doc/wiki/HowTo.SimpleVirtualInstall.txt @@ -0,0 +1,188 @@ +Simple Virtual User Installation +================================ + + * Virtual users configured in '/etc/dovecot/passwd' file + * Assuming an unmodified Dovecot v2.x installation + * Assuming you're not using NFS. See <NFS.txt> for problems related to it. + +Contents + + + 1. Simple Virtual User Installation + + 2. System configuration + + 3. dovecot.conf + + 4. /etc/dovecot/passwd + + 1. Passwords + + 5. SMTP server configuration + + 1. Delivering mails + + 2. SMTP AUTH + + 6. Quota + +System configuration +==================== + + * Create *dovecot* and *dovenull* users and groups if they don't exist yet. + These are unprivileged users for Dovecot's internal use. They doen't need a + home directory or a shell. + * Create *vmail* user and *vmail* group. This is the user/group that's used to + access the mails. + * Create '/home/vmail' directory owned by vmail:vmail. The mails for all users + are stored under this directory. + * Create '/var/log/dovecot.log' and '/var/log/dovecot-info.log' files owned by + vmail:vmail, so that <dovecot-lda> [LDA.txt] can write to them. + +dovecot.conf +============ + +Below is a fully working 'dovecot.conf' file. You can use it directly, but it +might be better to instead use the included example-config as the base and make +the same modifications to it. + +If you want to configure SSL, see <SSL.txt>. + +---%<------------------------------------------------------------------------- +protocols = imap pop3 + +# It's nice to have separate log files for Dovecot. You could do this +# by changing syslog configuration also, but this is easier. +log_path = /var/log/dovecot.log +info_log_path = /var/log/dovecot-info.log + +# Disable SSL for now. +ssl = no +disable_plaintext_auth = no + +# We're using Maildir format +mail_location = maildir:~/Maildir + +# If you're using POP3, you'll need this: +pop3_uidl_format = %g + +# Authentication configuration: +auth_verbose = yes +auth_mechanisms = plain +passdb { + driver = passwd-file + args = /etc/dovecot/passwd +} +userdb { + driver = static + args = uid=vmail gid=vmail home=/home/vmail/%u +} +---%<------------------------------------------------------------------------- + +/etc/dovecot/passwd +=================== + +See <AuthDatabase.PasswdFile.txt> for the full file format. Here we're +interested only having usernames and passwords in it. Below's an example file: + +---%<------------------------------------------------------------------------- +test:{PLAIN}pass:::::: +bill:{PLAIN}secret:::::: +timo@example.com:{PLAIN}hello123:::::: +dave@example.com:{PLAIN}world234:::::: +joe@elsewhere.org:{PLAIN}whee:::::: +jane@elsewhere.org:{PLAIN}mypass:::::: +---%<------------------------------------------------------------------------- + +As you can see, you can use multiple domains in the file, or no domains at all. +Dovecot doesn't care about domains. The extra colons are needed for <userdb> +[UserDatabase.txt] passwd-file format, and can be omitted if you are using the +static user database in the example above. + +Users can be added by editing this file. Dovecot automatically notices the new +users immediately after they're added. It also creates their home directories +when the user logs in. + +Passwords +--------- + +The passwords in the example passwd file are listed using plaintext scheme. +It's possible to use other <password schemes> +[Authentication.PasswordSchemes.txt] as well. For example SSHA256 would be a +pretty strong scheme. You can create them using 'doveadm pw' utility, for +example: + +---%<------------------------------------------------------------------------- +doveadm pw -s ssha256 +Enter new password: foo +Retype new password: foo +{SSHA256}ZpgszeowIcHdoxe3BNqvUTtPxFd6fMsyQxEWyY0Qlobaacjk +---%<------------------------------------------------------------------------- + +Note that you won't get the same output after {SSHA256} as above, because +Dovecot uses random salts when creating the SSHA256 hash. This means that even +if multiple users have the same password, you won't know that because their +hashes are different. + +The passwd file entry would be: + +---%<------------------------------------------------------------------------- +{SSHA256}ZpgszeowIcHdoxe3BNqvUTtPxFd6fMsyQxEWyY0Qlobaacjk +---%<------------------------------------------------------------------------- + +Joe would now have "foo" as his password. + +SMTP server configuration +========================= + +Delivering mails +---------------- + +You can configure the SMTP server to deliver mails internally, or you can use +<dovecot-lda> [LDA.txt]. Using dovecot-lda gives you better performance because +it updates Dovecot's index files while saving the mails. See <LDA.txt> for how +to configure this. Alternatively you can also use <LMTP.txt>. In config you +should have: + +---%<------------------------------------------------------------------------- +protocol lda { + postmaster_address = postmaster@example.com +} +---%<------------------------------------------------------------------------- + +SMTP AUTH +--------- + +If you're using Postfix v2.3+ or Exim v4.64+ you can use Dovecot SASL instead +of Cyrus SASL. + + * <Postfix configuration> [HowTo.PostfixAndDovecotSASL.txt] + * <Exim configuration> [HowTo.EximAndDovecotSASL.txt] + +Quota +===== + +If you need to have quota, add this to 'dovecot.conf': + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins quota +protocol imap { + mail_plugins = $mail_plugins imap_quota +} +plugin { + quota = maildir +} +---%<------------------------------------------------------------------------- + +Then configure quota by adding 'userdb_quota_rule' <extra field> +[UserDatabase.ExtraFields.txt] to '/etc/dovecot/passwd', for example: + +---%<------------------------------------------------------------------------- +joe:{PLAIN}pass::::::userdb_quota_rule=*:storage=100M +jane:{PLAIN}pass::::::userdb_quota_rule=*:storage=200M +---%<------------------------------------------------------------------------- + +Joe has now 100MB quota and Jane has 200MB quota. See <Quota.txt> for more +information about quota settings. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.WriteConfiguration.txt b/doc/wiki/HowTo.WriteConfiguration.txt new file mode 100644 index 0000000..72960b8 --- /dev/null +++ b/doc/wiki/HowTo.WriteConfiguration.txt @@ -0,0 +1,56 @@ +local / remote blocks +===================== + +Use 'local' and 'remote' blocks in this order: + +---%<------------------------------------------------------------------------- +local 127.0.0.1 { +# next block is used only, if TLS SNI extension is used. It expands to the TLS +SNI hostname. +# Typically this is only used to configure per-host TLS certificates. + local_name foo { + remote 127.0.0.1 { + protocol imap { + } + } + } +} +---%<------------------------------------------------------------------------- + +The protocol block is the innermost block always, you can leave some blocks +out. + +Change a Setting +================ + +Typically you can change each setting in another block, like so: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +mail_plugins = quota +protocol imap { + # overwrites a setting + mail_location = mbox:~/mbox + # adds other values to the setting + mail_plugins = $mail_plugins imap_quota +} +---%<------------------------------------------------------------------------- + +This way, you can keep all settings for a feature, e.g. ACL in one .conf file. + +Unfortunately, you cannot access variables from the plugin section, e.g.: + +---%<------------------------------------------------------------------------- +plugin { + sieve_plugins = extdata +} + +# other file or later + +plugin { + # this won't work !!! + sieve_plugins = $sieve_plugins extdata +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.txt b/doc/wiki/HowTo.txt new file mode 100644 index 0000000..d2be24c --- /dev/null +++ b/doc/wiki/HowTo.txt @@ -0,0 +1,93 @@ +HOWTOs / Examples / Tutorials +============================= + +<Virtual users> [VirtualUsers.txt] with <passwd-files> +[AuthDatabase.PasswdFile.txt]: + + * <A simple virtual passwd file installation> [HowTo.SimpleVirtualInstall.txt] + + * <Virtual passwd file and Exim> [HowTo.VirtualhostingWithExim.txt] + * <Virtual passwd file and Postfix> [HowTo.VirtualUserFlatFilesPostfix.txt] + +<Virtual users> [VirtualUsers.txt] with <LDAP> [AuthDatabase.LDAP.txt]: + + * <OpenLDAP> [HowTo.DovecotOpenLdap.txt] ( <Cheat sheet> + [HowTo.DoveLdapCheatSheet.txt]) + * Postfix and Active Directory or OpenLDAP [http://www.linuxmail.info] + +<System users> [SystemUsers.txt] and/or <Virtual users> [VirtualUsers.txt] with +<LDAP> [AuthDatabase.LDAP.txt]: + + * Dovecot, ManageSieve, Exim, OpenLDAP and getmail + [http://www.effinger.org/blog/2009/03/22/dovecot-exim-openldap-und-getmail-unter-ubuntu-1-openldap/] + (Instructions in German) - LDAP users (can be both <System users> + [SystemUsers.txt] and <Virtual users> [VirtualUsers.txt] depending on how + you use <LDAP> [AuthDatabase.LDAP.txt]) with the possibility to add + subaccounts for each user. For example if you have a LDAP user named peter, + you can add a separate subordinate mailbox to retrieve mail from an external + mail account like peter[at]gmail.com + +<Virtual users> [VirtualUsers.txt] with <SQL> [AuthDatabase.SQL.txt]: + + * MySQL + * <Using Dovecot with XAMS> [HowTo.DovecotXAMS.txt] + * <Dovecot, Postfix with Dovecot LDA transport and Dovecot SASL Auth, + Postfix Admin, MySQL and SquirrelMail> + [HowTo.DovecotLDAPostfixAdminMySQL.txt] + * MySQL, Exim, SpamAssassin and ClamAV + [http://struction.de/projects/HOWTO_VirtualMail_Exim-MySQL-Spamassassin-ClamAV-Dovecot/] + * Postfix and Dovecot with MySQL and TLS/SSL, Postgrey and DSPAM + [http://johnny.chadda.se/2007/04/15/mail-server-howto-postfix-and-dovecot-with-mysql-and-tlsssl-postgrey-and-dspam/] + * ISP-style Email Server with Debian-Etch and Postfix (MySQL, Dovecot, + Postfix etc.) [http://workaround.org/ispmail] + * PostgreSQL + * <PostgreSQL and Postfix> [HowTo.DovecotPostgresql.txt] + * PostgreSQL, Postfix (Dovecot LMTP and Dovecot SASL), Dovecot and vmm + (command line tool) [http://vmm.localdomain.org/] + * Installing a fully fledged, ready to use mailserver on Centos 6 with + Postfix, PostgreSQL, Amavis, ClamAV, Spamassassin and Dovecot + [http://shisaa.jp/postset/mailserver-1.html] + * SQLite + * Postfix+Dovecot with SQLite3 backend [http://rob0.nodns4.us/howto/] (also + implements system users) + +Others: + + * SMTP AUTH + * <With Postfix> [HowTo.PostfixAndDovecotSASL.txt] + * <With Exim> [HowTo.EximAndDovecotSASL.txt] + * With HALON [http://wiki.halon.se/SASL] + * <With chasquid> [HowTo.ChasquidAndDovecotSASL.txt] + * <Default Debian stable setup with Exim, modified to use Maildir> + [HowTo.DebianStable.txt] + * <IMAPC: Configuring Dovecot as an IMAP Proxy in front of Exchange (Dovecot + >= 2.1)> [HowTo.ImapcProxy.txt] + * <CRAM-MD5 authentication HOWTO> [HowTo.CRAM-MD5.txt] + * <Rootless installation> [HowTo.Rootless.txt] + * <POP-before-SMTP> [HowTo.PopBSMTPAndDovecot.txt] + * <Pop Relay Compatibility> [HowTo.PopRelay.txt] + * <Refiltering mail> [HowTo.RefilterMail.txt] + * <Triggering getmail on IMAP access> [HowTo.TriggerGetmailOnIMAPAccess.txt] + * <Outlook calendar sharing using Bynari Connector (commercial software) with + dovecot> [HowTo.BynariConnector.txt] + * <Using fail2ban with Dovecot> [HowTo.Fail2Ban.txt] + * <VMailMgr with Dovecot> [HowTo.VMailMgr.txt] + * <NTLM authentication against Active Directory> + [HowTo.ActiveDirectoryNtlm.txt] + * <Postfix and Dovecot LMTP> [HowTo.PostfixDovecotLMTP.txt] + * Apple Discussion Forum: Mail Services in Mac OS X Server v10.6 Snow Leopard + [http://discussions.apple.com/forum.jspa?forumID=1350] + * <Postfix + Dspam + Dovecot via LTMP> + [HowTo.Virtual+Postfix+Dspam+Dovecot.txt] + * Postfix + Dovecot2.0.13 + MySQL virtual_users as proxy to DBMail + [http://content.fens.org/index.php?q=admin-howto/mail/dovecot2dbmail-proxy] + * <Configuration files> [HowTo.WriteConfiguration.txt] + * <Replacing antispam plugin with sieve> [HowTo.AntispamWithSieve.txt] + * Debian + Dovecot + chasquid how-to + [https://blitiri.com.ar/git/r/chasquid/b/master/t/docs/f=howto.md.html] + * The ultimate mail server setup + [https://www.ohreally.nl/2018/11/19/mail-server/]: Dovecot (IMAP + SASL + + LDA + Pigeonhole) + Postfix + Let's Encrypt + Greylisting + SPF + DKIM + + Bogofilter + Clamav + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/IMAPServer.Hibernation.txt b/doc/wiki/IMAPServer.Hibernation.txt new file mode 100644 index 0000000..21d4a3f --- /dev/null +++ b/doc/wiki/IMAPServer.Hibernation.txt @@ -0,0 +1,51 @@ +Hibernation +=========== + +This is not supported on kqueue based systems currently, such as FreeBSD. + +Dovecot supports moving connections that have issued IDLE to a special holding +process, called imap-hibernate. This process is responsible for holding the +idle processes until they need to be thawed. + +Configuration +------------- + +'imap_hibernate_timeout' specifies the delay before moving users to +'imap-hibernate' process. This requires inter-process communication between +'imap' and 'imap-hibernate' process. + +---%<------------------------------------------------------------------------- +imap_hibernate_timeout = 5s + +service imap { + # Note that this change will allow any process running as + # $default_internal_user (dovecot) to access mails as any other user. + # This may be insecure in some installations, which is why this isn't + # done by default. + unix_listener imap-master { + user = $default_internal_user + } +} + +# The following is the default already in v2.3.1+: +service imap { + extra_groups = $default_internal_group +} +service imap-hibernate { + unix_listener imap-hibernate { + mode = 0660 + group = $default_internal_group + } +} +---%<------------------------------------------------------------------------- + +How it works +------------ + +When client issues IDLE, the connection socket is moved to the hibernation +process. This process is responsible for keeping all connections that are +idling, until they issue some command that requires them to be thawed into a +imap process. This way, memory and CPU resources are saved, since there is only +one hibernation process. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/IMAPServer.txt b/doc/wiki/IMAPServer.txt new file mode 100644 index 0000000..8681cfe --- /dev/null +++ b/doc/wiki/IMAPServer.txt @@ -0,0 +1,20 @@ +Dovecot as an IMAP server +========================= + +Dovecot was optimized since the beginning to work as an efficient IMAP server. +Dovecot supports a lot of IMAP extensions [http://imapwiki.org/Specs]. + +Some of the extensions need to be explicitly enabled: + + * <METADATA> [ImapMetadata.txt] + * <COMPRESS> [Plugins.Compress.txt] + * <SEARCH=FUZZY> [Plugins.FTS.txt] + * <SPECIAL-USE> [MailboxSettings.txt] + * NOTIFY: Set mailbox_list_index=yes + * URLAUTH: Set imap_urlauth_host and mail_attribute_dict + +Other features: + + * <Hibernation> [IMAPServer.Hibernation.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/ImapMetadata.txt b/doc/wiki/ImapMetadata.txt new file mode 100644 index 0000000..8583415 --- /dev/null +++ b/doc/wiki/ImapMetadata.txt @@ -0,0 +1,25 @@ +IMAP METADATA +============= + +Dovecot supports the IMAP METADATA extension (RFC 5464) +[https://tools.ietf.org/html/rfc5464], which allows per-mailbox, per-user data +to be stored and accessed via IMAP commands. + +To activate metadata storage, a <dictionary> [Dictionary.txt] needs to be +configured in the Dovecot configuration using the 'mail_attribute_dict' option. + +To activate the IMAP METADATA commands, the 'imap_metadata' option needs to be +activated. + +Example: + +---%<------------------------------------------------------------------------- +# Store METADATA information within user's Maildir directory +mail_attribute_dict = file:%h/Maildir/dovecot-attributes + +protocol imap { + imap_metadata = yes +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/IndexFiles.txt b/doc/wiki/IndexFiles.txt new file mode 100644 index 0000000..88ff012 --- /dev/null +++ b/doc/wiki/IndexFiles.txt @@ -0,0 +1,192 @@ +Dovecot's index files +===================== + +The basic idea behind Dovecot's index files is that it makes reading the +mailboxes a lot faster. The index files consist of the following files: + + * dovecot.index: Main index file + * dovecot.index.cache: Cached mailbox data + * dovecot.index.log: Transaction log file + * dovecot.index.log.2: .log file is rotated to .log.2 file when it grows too + large. + * dovecot.list.index*: Mailbox list index files + +Each mailbox has its own separate index files. If the index files are disabled, +the same structures are still kept in the memory, except cache file is disabled +completely (because the client probably won't fetch the same data twice within +a connection). + +If index files are missing, Dovecot creates them automatically when the mailbox +is opened. If at any point creating a file or growing a file gives "not enough +disk space" error, the indexes are transparently moved to memory for the rest +of the session. This isn't done with mailbox formats that rely on index files +(e.g. dbox). + +See <Design.Indexes.txt> for more technical information how the index files are +handled. + +Main index +---------- + +The main index contains the following information for each message: + + * IMAP UID + * Current flags and keywords + * Pointer to cache file + * mbox-only: mbox file offset + * mbox-only: MD5 sum of some of the message headers, intended to help find the + message when its X-UID: header hasn't yet been written + * Other extensions in Dovecot v1.1+, such as mailbox sorting data + +This is the same information that most other IMAP servers keep in memory while +the mailbox is open, but Dovecot has the advantage of keeping the information +permanently stored so it's easy to get it when opening the mailbox. + +The index file's header also contains some summary information, such as how +many messages exist, how many of them are unseen and how many are marked with +\Deleted flag. Opening mailboxes and answering to STATUS IMAP commands can be +usually done simply by getting the required information from the index file's +header. This is why these operations are extremely fast with Dovecot compared +to other servers that don't use an equivalent index file. + +Mailbox synchronization +----------------------- + +The main index's header also contains mailbox syncing state: + + * Maildir: cur/ and new/ directories' timestamps + * mbox: mbox file's mtime and size + +The index file is synchronized against mailbox only if the syncing information +changes. + +Cache file +---------- + +Cache file may contain the following information for messages: + + * Message headers (some, not all) + * Sent date (parsed Date: header) + * Received date (IMAP's INTERNALDATE field) + * Physical and virtual message sizes + * Message's parsed MIME structure, allowing to quickly read only a specific + MIME part (IMAP's FETCH BODY[1.2.3] command) + * IMAP's BODY and BODYSTRUCTURE fields + * If both are used, only BODYSTRUCTURE is saved, since BODY can be + generated from it + * IMAP's ENVELOPE isn't cached currently. Instead the headers used to build it + are cached directly. + +IMAP clients can work in many different ways. There are basically 2 types: + + 1. Online clients that ask for the same information multiple times (eg. + webmails, Pine) + 2. Offline clients that usually download first some of the interesting message + headers and only after that the message bodies (possibly automatically, or + possibly only when the user opens the mail). Most IMAP clients behave like + this. + +Cache file is extremely helpful with the type 1 clients. The first time that +client requests message headers or some other metadata they're stored into the +cache file. The second time they ask for the same information Dovecot can now +get it quickly from the cache file instead of opening the message and parsing +the headers. + +For type 2 clients the cache file is helpful if they use multiple clients or if +the data was cached while the message was being saved (Dovecot v1.1+ can do +this). Some of the information is helpful in any case, for example it's +required to know the message's virtual size when downloading the message. +Without the virtual size being in cache Dovecot first has to read the whole +message to calculate it. + +Only the mailbox metadata that client(s) have asked for earlier are stored into +cache file. This allows Dovecot to be adaptive to different clients' needs and +still not waste disk space (and cause extra disk I/O!) for fields that client +never needs. + +Dovecot can cache fields either permanently or temporarily. Temporarily cached +fields are dropped from the cache file after about a week. Dovecot uses two +rules to determine when data should be cached permanently instead of +temporarily: + + 1. Client accessed messages in non-sequential order within this session. This + most likely means it doesn't have a local cache. + 2. Client accessed a message older than one week. + +<Design.Indexes.Cache.txt> explains the reasons for these rules. + +Transaction log +--------------- + +All changes to the main index go through transaction log first. This has two +advantages when the mailbox is accessed using multiple simultaneous +connections: + + 1. It allows getting a list of changes quickly so that IMAP clients can be + notified of the changes. An alternative would be to do a comparison of two + index mappings, which is what most other IMAP servers do. + 2. 'mmap_disable=yes' implementation relies on the transaction log. Instead of + re-reading the whole main index file after each change it's necessary to + only read a few bytes from the transaction log. + +In Dovecot v1.1+ the transaction log plays an even more important role. The +main index file is updated only "once in a while" to reduce disk writes, so it +is common to first read the main index and then apply new changes from the +transaction log on top of that. With empty mailboxes (eg. download+delete POP3 +users) it would even be possible to delete the whole main index and keep only +the transaction log (although this isn't done currently). + +List index +---------- + +Mailbox list index file is called dovecot.list.index[.log] and it basically +contains: + + * Header contains ID => name mapping. The name isn't the full mailbox name, + but rather each hierarchy level has its own ID and name. For example a + mailbox name "foo/bar" (with '/' as separator) would have separate IDs for + "foo" and "bar" names. + * The records contain { parent_uid, uid, name_id } field that can be used to + build the whole mailbox tree. parent_uid=0 means root, otherwise it's the + parent node's uid. + * Each record also contains GUID for each selectable mailbox. If a mailbox is + recreated using the same name, its GUID also changes. Note however that the + UID doesn't change, because the UID refers to the mailbox name, not to the + mailbox itself. + * The records may contain also extensions for allowing mailbox_get_status() to + return values directly from the mailbox list index. + * Storage backends may also add their own extensions to figure out if a record + is up to date. + +Settings +-------- + +Since v2.2.34+ you can configure some of the hardcoded optimization-related +settings. It's not recommended to change these settings without fully +understanding the consequences. + + * 'mail_cache_unaccessed_field_drop': Drop fields that haven't been accessed + for n seconds. + * 'mail_cache_record_max_size': If cache record becomes larger than this, + don't add it. + * 'mail_cache_compress_min_size': Never compress the file if it's smaller than + this. + * 'mail_cache_compress_delete_percentage': Compress the file when n% of + records are deleted (by count, not by size). + * 'mail_cache_compress_continued_percentage': Compress the file when n% of + rows contain continued rows. For example 200% means that the record has 2 + continued rows, i.e. it exists in 3 separate segments in the cache file. + * 'mail_cache_compress_header_continue_count': Compress the file when we need + to follow more than n next_offsets to find the latest cache header. + * 'mail_index_rewrite_min_log_bytes', 'mail_index_rewrite_max_log_bytes': + Rewrite the index when the number of bytes that needs to be read from the + .log on refresh is between these min/max values. + * 'mail_index_log_rotate_min_size', 'mail_index_log_rotate_max_size', + 'mail_index_log_rotate_min_age': Rotate transaction log after it's a) + min_size or larger and it was created at least min_age_secs or b) larger + than max_size. + * 'mail_index_log2_max_age': Delete .log.2 when it's older than + log2_stale_secs. Don't be too eager, because older files are useful for + QRESYNC and dsync. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Exim.txt b/doc/wiki/LDA.Exim.txt new file mode 100644 index 0000000..1b2ab21 --- /dev/null +++ b/doc/wiki/LDA.Exim.txt @@ -0,0 +1,163 @@ +Dovecot LDA with Exim +===================== + +System users +------------ + +Change the localuser router to use dovecot_delivery transport: + +---%<------------------------------------------------------------------------- +localuser: + driver = accept + check_local_user +# local_part_suffix = +* : -* +# local_part_suffix_optional + transport = dovecot_delivery +---%<------------------------------------------------------------------------- + +'check_local_user' is required. It makes Exim execute the transport with the +user's UID and GID and it also sets HOME environment. + +Next create a new transport for dovecot-lda: + +---%<------------------------------------------------------------------------- +dovecot_delivery: + driver = pipe + + # Use /usr/lib/dovecot/dovecot-lda if using Debian's package. + # You may or may not want to add -d $local_part@$domain depending on if you +need a userdb lookup done. + command = /usr/local/libexec/dovecot/dovecot-lda -f $sender_address + + message_prefix = + message_suffix = + log_output + delivery_date_add + envelope_to_add + return_path_add + #group = mail + #mode = 0660 + temp_errors = 64 : 69 : 70: 71 : 72 : 73 : 74 : 75 : 78 +---%<------------------------------------------------------------------------- + +LDA is now running using the local user's UID and GID. The mail is delivered to +the location specified by <mail_location> [MailLocation.txt] setting. Note that +the above configuration doesn't do any <userdb> [UserDatabase.txt] lookups, so +you can't have any per-user configuration. If you want that, see the virtual +user setup below. + +Virtual users +------------- + +Make sure that 'check_local_user' isn't set in the router. + +Single UID +---------- + +Configure the transport to run as the user you want, for example vmail: + +---%<------------------------------------------------------------------------- +dovecot_virtual_delivery: + driver = pipe + command = /usr/local/libexec/dovecot/dovecot-lda -d $local_part@$domain -f +$sender_address + # v1.1+: command = /usr/local/libexec/dovecot/dovecot-lda -d +$local_part@$domain -f $sender_address -a +$original_local_part@$original_domain + message_prefix = + message_suffix = + delivery_date_add + envelope_to_add + return_path_add + log_output + user = vmail + temp_errors = 64 : 69 : 70: 71 : 72 : 73 : 74 : 75 : 78 +---%<------------------------------------------------------------------------- + +You'll also need to have a master authentication socket and give vmail user +access to it. See <LDA.txt> for more information. + +List of temp_errors can be found in '/usr/include/sysexits.h'. + +Multiple UIDs +------------- + +If you need multiple uids/gids you'll need to set dovecot-lda setuid root. See +<LDA#multipleuids> [LDA.txt] for how to do this securely. + +You could alternatively set 'user = root', but this requires that you built +Exim without root being in FIXED_NEVER_USERS list. + +Multiple UIDs, without running dovecot-lda as root +-------------------------------------------------- + +In this mode, dovecot-lda won't be querying Dovecot's master socket, instead +trusting Exim to setup its execution environment. This means you must set up +Exim to get the UID, GID, Home directory from LDAP/SQL/whatever. Here, we're +setting them in the router and the transport automatically inherits them. + +Router configuration +-------------------- + +Insert the following router after your external delivery routers and before +your local system delivery routers. + +This assumes you're using macros set elsewhere to handle your external queries, +as they can quickly become unwieldy to manage. Make sure you adjust it to suit +your installation first! + +---%<------------------------------------------------------------------------- +ldap_local_user: + debug_print = "R: ldap_local_user for $local_part@$domain" + driver = accept + domains = +ldap_local_domains + condition = LDAP_VIRT_COND + router_home_directory = LDAP_VIRT_HOME + user = LDAP_VIRT_UID + group = LDAP_VIRT_GID + #local_part_suffix = +* : -* + #local_part_suffix_optional + transport = dovecot_lda +---%<------------------------------------------------------------------------- + +Transport configuration +----------------------- + +This transport has been tested with Exim 4.69-9 and Dovecot 1:1.2.5-2 +(backported) on Debian Lenny. You also have to set + +---%<------------------------------------------------------------------------- +dovecot_lda: + debug_print = "T: dovecot_lda for $local_part@$domain" + driver = pipe + # Uncomment the following line and comment the one after it if you want +dovecot-lda to try + # to deliver subaddresses into INBOX.{subaddress}. If you do this, uncomment +the + # local_part_suffix* lines in the router as well. Make sure you also change +the separator + # to suit your local setup. + #command = /usr/lib/dovecot/dovecot-lda -e -k -m +"INBOX|${substr_1:$local_part_suffix}" \ + command = /usr/lib/dovecot/dovecot-lda -e -k \ + -f "$sender_address" -a "$original_local_part@$original_domain" + environment = USER=$local_part@$domain + home_directory = /var/mail/home/$domain/$local_part + umask = 002 + message_prefix = + message_suffix = + delivery_date_add + envelope_to_add + return_path_add + log_output + log_defer_output + return_fail_output + freeze_exec_fail + #temp_errors = * + temp_errors = 64 : 69 : 70 : 71 : 72 : 73 : 74 : 75 : 78 +---%<------------------------------------------------------------------------- + +You need to have <home directory> [VirtualUsers.Home.txt] set to have duplicate +database enabled, among other reasons. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Indexing.txt b/doc/wiki/LDA.Indexing.txt new file mode 100644 index 0000000..abde8f8 --- /dev/null +++ b/doc/wiki/LDA.Indexing.txt @@ -0,0 +1,47 @@ +LDA Indexing +============ + +LDA's indexing basically does two things while message is being saved: + + 1. It updates the main index file. + * This improves performance with <mbox> [MailboxFormat.mbox.txt] format, + especially if 'mbox_very_dirty_syncs=no'. + * With <Maildir> [MailboxFormat.Maildir.txt] the benefits of this are + almost irrelevant. + 2. It updates the 'dovecot.index.cache' file. + +Cache file +---------- + +The LDA also updates the cache file, which can be very useful with all mailbox +formats. It means that when an IMAP client wants to fetch the message's +metadata (e.g. some header fields), they're can be retrieved from the cache +file and Dovecot doesn't have to open and parse the message file. There are +some tradeoffs though: + + * LDA indexing wastes disk I/O because it has to open and update index files + * LDA indexing saves disk I/O because it already has the message body in + memory, so it doesn't need to read it from disk. + * IMAP indexing wastes disk I/O because it has to open and read message files + * IMAP indexing may save disk I/O because IMAP process always has index files + opened, and many IMAP clients are configured to download all new message + bodies anyway, so the second time message bodies are read they're already in + memory + +So it depends on IMAP client if it's faster to use LDA or IMAP time indexing. +In any case the user experience is typically faster with LDA indexing, because +the message list metadata can be returned faster when it's pre-indexed. + +See <IndexFiles.txt> for more information about what the index files contain. + +Non-indexed mail delivery +------------------------- + +Ignoring the benefits of cache file updates, the only thing left is the main +index updates. As mentioned above, with Maildir format these benefits are very +small. This also means that it's perfectly fine to use a non-Dovecot MDA to +deliver mails that doesn't update indexes. Dovecot can efficiently see and +index such new mails without doing anything expensive like "rebuilding +indexes". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Postfix.txt b/doc/wiki/LDA.Postfix.txt new file mode 100644 index 0000000..bf519af --- /dev/null +++ b/doc/wiki/LDA.Postfix.txt @@ -0,0 +1,255 @@ +Dovecot LDA with Postfix +======================== + +This page contains only information specific to using LDA with Postfix, see +<LDA.txt> for more information about using the LDA itself. + +System users +------------ + +If you wish you use 'dovecot-lda' for all system users on a single domain mail +host you can do it by editing 'mailbox_command' parameter in + +'/etc/postfix/main.cf' (postconf(5) [http://www.postfix.org/postconf.5.html]): + +---%<------------------------------------------------------------------------- +mailbox_command = /usr/local/libexec/dovecot/dovecot-lda -f "$SENDER" -a +"$RECIPIENT" +# or +mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT" +# or +mailbox_command = /usr/lib/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT" +# or wherever it was installed in your system. +---%<------------------------------------------------------------------------- + +Then run 'postfix reload'. + + * This command doesn't do a <userdb> [UserDatabase.txt] lookup. If you want + that (e.g. for per-user quota lookups) you need to add '-d "$USER"' + parameter. + * Postfix runs 'mailbox_command' with both the uid and gid of the destination + user. This may not allow 'dovecot-lda' to write a lock file in '/var/mail'. + When this directory is writable by a privileged group (say 'main', see the + option 'mail_privileged_group'), we can use the setgid permission bit on the + 'dovecot-lda' executable: + + ---%<---------------------------------------------------------------------- + # chgrp mail /usr/lib/dovecot/dovecot-lda + # chmod 2755 /usr/lib/dovecot/dovecot-lda + ---%<---------------------------------------------------------------------- + + Alas these permission will disappear if you update dovecot. A more robust + way to do so is to compile a relay program '/etc/postfix/dovecot-lda-relay' + that has the setgid permission and execs the real 'dovecot-lda'. + + ---%<---------------------------------------------------------------------- + # cd /etc/postfix + # cat >dovecot-lda-relay.c <<EOF + #include <unistd.h> + char *pgm = "/usr/lib/dovecot/dovecot-lda"; /* wherever dovecot-lda is + located */ + int main{int argc, char**argv) { argv[0]=pgm; execv(pgm,argv); return + 10; } + EOF + # gcc -o dovecot-lda-relay dovecot-lda-relay.c + # chown root:mail dovecot-lda-relay + # chmod 2755 dovecot-lda-relay + ---%<---------------------------------------------------------------------- + + Then, simply invoke '/etc/postfix/dovecot-lda-relay' instead of + 'dovecot-lda' in 'mailbox_command'. + * Postfix's 'mailbox_size_limit' setting applies to all files that are written + via dovecot-lda. The default is 50 MB, so dovecot-lda can't write *any* + files larger than that, including mbox files or log files. This shows up + only in Dovecot's logs: + + ---%<---------------------------------------------------------------------- + dovecot-lda(user): write() failed with mbox file /home/user/mail/foo: File + too large (process was started with ulimit -f limit) + ---%<---------------------------------------------------------------------- + + * If you have trouble seeing anything in Dovecot's logs, see <LDA#Logging> + [LDA.txt]. + +Virtual users +------------- + +Dovecot LDA is very easy to use on large scale installations with Postfix +virtual domains support, just add a 'dovecot' service in +'/etc/postfix/master.cf' (master(5) [http://www.postfix.org/master.5.html]): + +---%<------------------------------------------------------------------------- +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f +${sender} -d ${recipient} +---%<------------------------------------------------------------------------- + +An example using address extensions (ie user+extension@domain.com (don't forget +to define the proper recipient_delimiter in Postfix's main.cf)) to deliver to +the folder 'extension' in your maildir (If you wish to preserve the case of +${extension}, remove the 'hu'flags [http://www.postfix.org/pipe.8.html], and be +sure to utilize <Modifiers> [Variables.txt] in your dovecot.conf for mail +locations and other configuration parameters that are expecting lower case): + +---%<------------------------------------------------------------------------- +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f +${sender} -d ${user}@${nexthop} -m ${extension} + +# or if you have a INBOX/ namespace prefix: +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f +${sender} -d ${user}@${nexthop} -m INBOX/${extension} +---%<------------------------------------------------------------------------- + +This example ignores address extensions (ie user+extension@domain.com delivers +just like user@domain.com ), but still shows the original address for Sieve: + +---%<------------------------------------------------------------------------- +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/dovecot-lda -f ${sender} -a +${original_recipient} -d ${user}@${nexthop} +---%<------------------------------------------------------------------------- + +Replace 'vmail' above with your virtual mail user account. + +Then set 'virtual_transport' to 'dovecot' in '/etc/postfix/main.cf': + +---%<------------------------------------------------------------------------- +dovecot_destination_recipient_limit = 1 +virtual_mailbox_domains = your.domain.here +virtual_transport = dovecot +---%<------------------------------------------------------------------------- + +And remember to run + +---%<------------------------------------------------------------------------- +postfix reload +---%<------------------------------------------------------------------------- + +Virtual users with multiple uids/gids +------------------------------------- + +If you need multiple uids/gids you'll need to set dovecot-lda setuid root or +invoke it through sudo. See <LDA#multipleuids> [LDA.txt] for how to do this +securely. + +Postfix with a NFS mail store +----------------------------- + +If you are experiencing problems with dovecot-lda processes hanging when +delivering to an NFS mail store, it's likely that the dovecot-lda process is +hanging while waiting for free locks. The occurrence of this can be greatly +reduced, if not eradicated, by forcing Postfix to only deliver to the same +recipient one at a time. + +---%<------------------------------------------------------------------------- +dovecot_destination_concurrency_limit = 1 +---%<------------------------------------------------------------------------- + +Prevent backscatter +------------------- + +To prevent backscatter you should configure Postfix to reject mail for non +existent recipients. + +This is the default behaviour (smtpd_reject_unlisted_recipient = yes) so +there's no need to set "reject_unlisted_recipient" in any of your restriction. +But: Postfix must know if a recipient exists. Depending on how you've +configured Dovecot and Postfix this can be done several ways. + +System users +------------ + +If you only use local system users this is no problem - all valid recipients +can be found in the local password or alias database. + +Virtual users (static) +---------------------- + +When you use virtual users and domains you should maintain a list of valid +recipients. The relevant settings settings are: + +*virtual_alias_maps, virtual_mailbox_maps* + +For static verification you can maintain the content of the files yourself. For +every recipient or alias you need one entry. Example: + +*virtual_alias_maps* + +---%<------------------------------------------------------------------------- +name_recipient@example.com external@example.net +---%<------------------------------------------------------------------------- + +*virtual_mailbox_maps* + +---%<------------------------------------------------------------------------- +name@example.com OK +recipient@example.com available +---%<------------------------------------------------------------------------- + +Don't forget to run "postmap" afterwards. + +*Info:* if you use the Dovecot LDA or LMTP it doesn't matter what you use +behind the recipient address. Use "OK", the full name of the user or else. + +Virtual users (dynamic) +----------------------- + +Do you already use a database (MySQL, PostgreSQL) for Dovecot? Use the same +source for Postfix. You only have to to define a valid sql query for Postfix. +Example: + +---%<------------------------------------------------------------------------- +virtual_mailbox_maps = proxy:mysql:/etc/postfix/virtual_mailbox_maps.cf +---%<------------------------------------------------------------------------- + +*virtual_mailbox_maps.cf* + +---%<------------------------------------------------------------------------- +user = mysql-user +password = mysql-password +hosts = unix:/var/run/mysql/mysqld.sock +dbname = mailserver +query = SELECT name FROM mailbox WHERE email='%s' +---%<------------------------------------------------------------------------- + +This query will return the value of the filed "name" from table "mailbox" if +the email address of the recipient matches the email from the field "email". +This is enough for Postfix because Postfix must only know if the recipient +exists. The value doesn't matter. When you use a database (or LDAP) there's no +need to manually maintain a file with valid recipients. + +*Info:* If you use "relay_domains" instead of "virtual_mailbox_domains" you +have to use "relay_recipient_maps" instead of "virtual_mailbox_maps". + +Dynamic address verification with LMTP +-------------------------------------- + +With Dovecot 2.0 you can also use LMTP and the Postfix setting +"reject_unverified_recipient" for dynamic address verification. It's really +nice because Postfix doesn't need to query an external datasource (MySQL, +LDAP...). Postfix maintain a local database with existing/non existing +addresses (you can configure how long positive/negative results should be +cached). + +To use LMTP and dynamic address verification you must first get Dovecot +working. Then you can configure Postfix to use LMTP and set +"reject_unverified_recipient" in the smtpd_recipient_restrictions. + +On every incoming email Postfix will probe if the recipient address exists. You +will see similar entries in your logfile: + +---%<------------------------------------------------------------------------- +Recipient address rejected: undeliverable address: host +tux.example.com[private/dovecot-lmtp] said: 550 5.1.1 < tzknvtr@example.com > +User doesn't exist: tzknvtr@example.com (in reply to RCPT TO command); from=< +cnrilrgfclra@spammer.org > to=< tzknvtr@example.com > +---%<------------------------------------------------------------------------- + +If the recipient address exists (status=deliverable) Postfix accepts the mail. + +*Info:* you can not use "reject_unverified_recipient" with "pipe" so this +doesn't work with the Dovecot LDA "deliver". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Qmail.txt b/doc/wiki/LDA.Qmail.txt new file mode 100644 index 0000000..d13756d --- /dev/null +++ b/doc/wiki/LDA.Qmail.txt @@ -0,0 +1,31 @@ +Dovecot LDA with Qmail +====================== + +System users +------------ + +The delivery command you need is + +---%<------------------------------------------------------------------------- +|/var/qmail/bin/preline -f /usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +(You may need to adjust the paths to match your qmail and dovecot +installations.) The 'preline' command will add the 'Return-Path:' and +'Delivered-To:' lines, because 'dovecot-lda' doesn't recognize qmail's +environment variables. + +For site-wide usage, put that in '/var/qmail/control/defaultdelivery' (assuming +you installed qmail according to LWQ [http://www.lifewithqmail.org/lwq.html]). +Or, save it as '.qmail' in selected users' home directories. + +Virtual users +------------- + +Add the '-d' parameter to specify the destination username: + +---%<------------------------------------------------------------------------- +|/var/qmail/bin/preline -f /usr/local/libexec/dovecot/dovecot-lda -d $EXT@$USER +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Sendmail.txt b/doc/wiki/LDA.Sendmail.txt new file mode 100644 index 0000000..b14e4f0 --- /dev/null +++ b/doc/wiki/LDA.Sendmail.txt @@ -0,0 +1,105 @@ +Dovecot LDA with Sendmail +========================= + +The following describes how to configure Sendmail to use 'dovecot-lda' where +'root' permission is not granted and Dovecot runs under a single user ID. It +may need some adjustment for more typical setups. Other assumptions are that +Sendmail is configured for virtual hosting and that local-system mail delivery +is not handled by 'dovecot-lda'. + +Allowing that 'sendmail.mc' has 'MAILER(procmail)dnl' included, edit +'sendmail.cf' adding these lines after the 'Mprocmail' definition: + +---%<------------------------------------------------------------------------- +######################*****############## +### DOVECOT Mailer specification ### +##################*****################## +Mdovecot, P=/usr/local/libexec/dovecot/dovecot-lda, F=DFMPhnu9, + S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP/HdrFromSMTP, + T=DNS/RFC822/X-Unix, + A=/usr/local/libexec/dovecot/dovecot-lda -d $u +---%<------------------------------------------------------------------------- + +If you're using 'sendmail.mc' then put the lines above into a new file +'/usr/share/sendmail-cf/mailer/dovecot.m4' and put 'MAILER(dovecot)' into your +'sendmail.mc' + +=================================== + +Another method of doing the above is by editing your 'hostname.mc' with the +following three lines: + +---%<------------------------------------------------------------------------- +FEATURE(`local_procmail', +`/usr/local/libexec/dovecot/dovecot-lda',`/usr/local/libexec/dovecot/dovecot-lda +-d $u') +MODIFY_MAILER_FLAGS(`LOCAL', `-f') +MAILER(procmail) +---%<------------------------------------------------------------------------- + +After editing 'hostname.mc' with the above, be sure to remake your +'hostname.cf' file. This is confirmed to work with: + + * dovecot-1.0.7 + * FreeBSD 6.3-RELEASE-p3 i386 + * sendmail Version 8.14.2 + * Compiled with: DNSMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7 + NAMED_BIND NETINET NETINET6 NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF + STARTTLS TCPWRAPPERS USERDB XDEBUG + +=================================== + +If 'sendmail' runs under a different non-'root' UID via + + * 'define(`confRUN_AS_USER', `sendmail')dnl' + +in 'sendmail.mc', then the /env_put(t_strconcat("RESTRICT_/ lines in +'deliver.c' must be commented-out. + +Now add a + +---%<------------------------------------------------------------------------- +virtualdomain.example.com vmail:vmail +---%<------------------------------------------------------------------------- + +line for each virtual domain to 'mailertable.cf' and run 'makemap hash +mailertable.db < mailertable.cf'. The 'dovecot' (or some other random text) +after the colon character is required, else 'sendmail' will fail to pass +command arguments to 'dovecot-lda' correctly. Make sure all the virtual +domains are in the 'virtuserdomains' file. + +=========================================== + +(Fedora 14: dovecot 2.0.8 & sendmail 8.14.4) + +Summing up all previous experience, one may keep all virtual user accounts +under one system account. + +The sendmail's "U=" mailer option with changing the owner of lda (to "keeper" +here for instance): + +---%<------------------------------------------------------------------------- +-rwxr-xr-x. 1 keeper mail 14536 Dec 7 16:43 /usr/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +allows to run virtual users under one system account without applying SUID. + +Sendmail can pass a user account to LDA with or without the domain. Passing a +user name without the domain can be achievedwith S=/R= rewriting rules of the +local mailer. Finally, into '/usr/share/sendmail-cf/mailer/dovecot.m4' goes the +block of lines: + +---%<------------------------------------------------------------------------- +Mdovecot, P=/usr/libexec/dovecot/dovecot-lda, + F=l59DFMPhnu, + S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, + M=51200000, + U=keeper:mail, + T=DNS/RFC822/X-Unix, + A=/usr/libexec/dovecot/dovecot-lda -d $u +---%<------------------------------------------------------------------------- + +dovecot.m4 [http://sites.google.com/site/mclroy/dovecot/dovecot-m4] can be a +bit more complex. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.txt b/doc/wiki/LDA.txt new file mode 100644 index 0000000..80e2ee6 --- /dev/null +++ b/doc/wiki/LDA.txt @@ -0,0 +1,298 @@ +Dovecot LDA +=========== + +The Dovecot LDA is a <local delivery agent> [MDA.txt], which takes mail from an +<MTA.txt> and delivers it to a user's mailbox, while keeping Dovecot index +files up to date. Nowadays you should probably use the <LMTP server> [LMTP.txt] +instead, because it's somewhat easier to configure (especially related to +permissions) and gives better performance. + +This page describes the common settings required to make LDA work. You should +read it first, and then the MTA specific pages: + + * <LDA.Postfix.txt> + * <LDA.Exim.txt> + * <LDA.Sendmail.txt> + * <LDA.Qmail.txt> + * <LDA.ZMailer.txt> + +Main features of Dovecot LDA +---------------------------- + + * <Mailbox indexing during mail delivery> [LDA.Indexing.txt], providing faster + mailbox access later + * <Quota enforcing by a plugin> [Quota.txt] + * <Sieve language support by a plugin> [Pigeonhole.Sieve.txt] + * Mail filtering + * Mail forwarding + * Vacation auto-reply + +Common configuration +-------------------- + +The settings are listed in the example 'conf.d/15-lda.conf' file. The important +settings are: + + * 'postmaster_address' is used as the From: header address in bounce mails + * 'hostname' is used in generated Message-IDs and in Reporting-UA: header in + bounce mails + * 'sendmail_path' is used to send mails. Note that the default is + '/usr/sbin/sendmail', which doesn't necessarily work the same as + '/usr/lib/sendmail'. + * Alternatively you can use 'submission_host' to send mails via the + specified SMTP server. + * 'auth_socket_path' specifies the UNIX socket to auth-userdb where LDA can + lookup userdb information when '-d' parameter is used. See below how to + configure Dovecot to configure the socket. + +Note that the config files must be world readable to enable dovecot-lda process +read them, while running with user privileges. You can put password related +settings to a separate file, which you include with '!include_try' and +dovecot-lda skips them. + +Parameters +---------- + +Parameters accepted by dovecot-lda: + + * '-d <username>': Destination username. If given, the user information is + looked up from dovecot-auth. Typically used with virtual users, but not + necessarily with system users. + * '-a <address>': Original envelope recipient address (e.g. user+ext@domain), + typically same as SMTP's RCPT TO: value. If not specified, it's taken from + header specified by 'lda_original_recipient_header' setting (v2.0.3+). If + the header doesn't exist either, defaults to same as username. + * '-r <address>': Final envelope recipient address. Defaults to -a address, + but may differ if e.g. aliases are used or when dropping the +ext part. + (v2.0.3+) + * '-f <address>': Envelope sender address. If not specified and message data + begins with a valid mbox-style "From " line, the address is taken from it. + * '-c <path>': Alternative configuration file path. + * '-m <mailbox>': Destination mailbox (default is INBOX). If the mailbox + doesn't exist, it will not be created (unless the lda_mailbox_autocreate + setting is set to yes). If message couldn't be saved to the mailbox for any + reason, it's delivered to INBOX instead. + * If Sieve plugin is used, this mailbox is used as the "keep" action's + mailbox. It's also used if there is no Sieve script or if the script + fails for some reason. + * Deliveries to namespace prefix will result in saving the mail to INBOX + instead. For example if you have "Mail/" namespace, this allows you to + specify 'dovecot-lda -m Mail/$mailbox' where mail is stored to + Mail/$mailbox or to INBOX if $mailbox is empty. + * The mailbox name is specified the same as it's visible in IMAP client. + For example if you've a Maildir with '.box.sub/' directory and your + namespace configuration is 'prefix=INBOX/', 'separator=/', the correct + way to deliver mail there is to use '-m INBOX/box/sub' + * '-e': If mail gets rejected, write the rejection reason to stderr and exit + with EX_NOPERM. The default is to send a rejection mail ourself. + * '-k': Don't clear all environment at startup. + * '-p <path>': Path to the mail to be delivered instead of reading from stdin. + If using maildir the file is hard linked to the destination if possible. + This allows a single mail to be delivered to multiple users using hard + links, but currently it also prevents dovecot-lda from updating cache file + so it shouldn't be used unless really necessary. + * '-o name=value': Override a setting from dovecot.conf. You can give this + parameter multiple times. + +Return values +------------- + +dovecot-lda will exit with one of the following values: + + * 0 (EX_OK): Delivery was successful. + * 64 (EX_USAGE): Invalid parameter given. + * 67 (EX_NOUSER): The destination username was not found. + * 77 (EX_NOPERM): -e parameter was used and mail was rejected. Typically this + happens when user is over quota and 'quota_full_tempfail=no'. + * 75 (EX_TEMPFAIL): A temporary failure. This is returned for almost all + failures. See the log file for details. + +System users +------------ + +You can use LDA with a few selected system users (ie. user is found from +'/etc/passwd' / NSS) by calling dovecot-lda in the user's '~/.forward' file: + +---%<------------------------------------------------------------------------- +| "/usr/local/libexec/dovecot/dovecot-lda" +---%<------------------------------------------------------------------------- + +This should work with any MTA which supports per-user '.forward' files. For +qmail's per-user setup, see <LDA.Qmail.txt>. + +This method doesn't require the authentication socket explained below since +it's executed as the user itself. + +Virtual users +------------- + +With a lookup +------------- + +Give the destination username to dovecot-lda with '-d' parameter, for example: + +---%<------------------------------------------------------------------------- +dovecot-lda -f $FROM_ENVELOPE -d $DEST_USERNAME +---%<------------------------------------------------------------------------- + +You'll need to set up a auth-userdb socket for dovecot-lda so it knows where to +find mailboxes for the users: + +---%<------------------------------------------------------------------------- +service auth { + unix_listener auth-userdb { + mode = 0600 + user = vmail # User running dovecot-lda + #group = vmail # Or alternatively mode 0660 + dovecot-lda user in this +group + } +} +---%<------------------------------------------------------------------------- + +The auth-userdb socket can be used to do <userdb> [UserDatabase.txt] lookups +for given usernames or get a list of all users. Typically the result will +contain the user's UID, GID and home directory, but depending on your +configuration it may return other information as well. So the information is +similar to what can be found from eg.'/etc/passwd' for system users. This means +that it's probably not a problem to use mode=0666 for the socket, but you +should try to restrict it more just to be safe. + +Without a lookup +---------------- + +If you have already looked up the user's home directory and you don't need a +userdb lookup for any other reason either (such as overriding settings for +specific users), you can run dovecot-lda similar to how it's run for system +users: + +---%<------------------------------------------------------------------------- +HOME=/path/to/user/homedir dovecot-lda -f $FROM_ENVELOPE +---%<------------------------------------------------------------------------- + +This way you don't need to have a master listener socket. Note that you should +verify the user's existence prior to running dovecot-lda, otherwise you'll end +up having mail delivered to nonexistent users as well. + +You must have set the proper UID (and GID) before running dovecot-lda. It's not +possible to run dovecot-lda as root without '-d' parameter. + +Multiple UIDs +------------- + +If you're using more than one UID for users, you're going to have problems +running dovecot-lda, as most MTAs won't let you run dovecot-lda as root. Best +solution is to use <LMTP.txt> instead, but if you can't do that, there are two +ways to work around this problem: + + 1. Make dovecot-lda setuid-root. + 2. Use sudo to wrap the invocation of dovecot-lda. + +Making dovecot-lda setuid-root: +------------------------------- + +Beware: *it's insecure to make dovecot-lda setuid-root*, especially if you have +untrusted users in your system.*Setuid-root dovecot-lda can be used to gain +root privileges*. You should take extra steps to make sure that untrusted users +can't run it and potentially gain root privileges. You can do this by making +sure only your MTA has execution access to it. For example: + +---%<------------------------------------------------------------------------- +# chgrp secmail /usr/local/libexec/dovecot/dovecot-lda +# chmod 04750 /usr/local/libexec/dovecot/dovecot-lda +# ls -l /usr/local/libexec/dovecot/dovecot-lda +-rwsr-x--- 1 root secmail 4023932 2010-06-15 16:23 dovecot-lda +---%<------------------------------------------------------------------------- + +Then start dovecot-lda as a user that belongs to secmail group. Note that you +have to recreate these rights after each update of dovecot. + +Using sudo: +----------- + +Alternatively, you can use sudo to wrap the invocation of dovecot-lda. This has +the advantage that updates will not clobber the setuid bit, but note that *it +is just as insecure being able to run dovecot-lda via sudo as setuid-root*. +Make sure you only give your MTA the ability to invoke dovecot-lda via sudo. + +First configure sudo to allow 'dovelda' user to invoke dovecot-lda by adding +the following to your '/etc/sudoers': + +---%<------------------------------------------------------------------------- +Defaults:dovelda !syslog +dovelda ALL=NOPASSWD:/usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +Then configure your MTA to invoke dovecot-lda as user 'dovelda' and via sudo: + +---%<------------------------------------------------------------------------- +/usr/bin/sudo /usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +instead of just plain '/usr/local/libexec/dovecot/dovecot-lda'. + +Problems with dovecot-lda +------------------------- + + * If you are using <prefetch userdb> [UserDatabase.Prefetch.txt], keep in mind + that 'dovecot-lda' does not make a password query and thus will not work if + '-d' parameter is used. The <UserDatabase.Prefetch.txt> page explains how to + fix this. + * See <Checkpassword> [PasswordDatabase.CheckPassword.txt] for how to make + dovecot-lda work with checkpassword. + +Logging +------- + + * Normally Dovecot logs everything through its log process, which is running + as root. dovecot-lda doesn't, which means that you might need some special + configuration for it to log anything at all. + * If dovecot-lda fails to write to log files it exits with temporary failure. + * If you have trouble finding where Dovecot logs by default, see + <Logging.txt>. + * Note that Postfix's 'mailbox_size_limit' setting applies to all files that + are written to. So if you have a limit of 50 MB, dovecot-lda can't write to + log files larger than 50 MB and you'll start getting temporary failures. + +If you want dovecot-lda to keep using Dovecot's the default log files: + + * If you're logging to syslog, make sure the syslog socket (usually + '/dev/log') has enough write permissions for dovecot-lda. For example set it + world-read/writable:'chmod a+rw /dev/log'. + * If you're logging to Dovecot's default log files again you'll need to give + enough write permissions to the log files for dovecot-lda. + +You can also specify different log files for dovecot-lda. This way you don't +have to give any extra write permissions to other log files or the syslog +socket. You can do this by overriding the 'log_path' and 'info_log_path' +settings: + +---%<------------------------------------------------------------------------- +protocol lda { + .. + # remember to give proper permissions for these files as well + log_path = /var/log/dovecot-lda-errors.log + info_log_path = /var/log/dovecot-lda.log +} +---%<------------------------------------------------------------------------- + +For using syslog with dovecot-lda, set the paths empty: + +---%<------------------------------------------------------------------------- +protocol lda { + .. + log_path = + info_log_path = + # You can also override the default syslog_facility: + #syslog_facility = mail +} +---%<------------------------------------------------------------------------- + +Plugins +------- + + * Most of the <Dovecot plugins> [Plugins.txt] work with dovecot-lda. + * Virtual quota can be enforced using <Quota plugin> [Quota.txt]. + * Sieve language support can be added with the <Pigeonhole Sieve plugin> + [Pigeonhole.Sieve.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LMTP.Exim.txt b/doc/wiki/LMTP.Exim.txt new file mode 100644 index 0000000..d1150b2 --- /dev/null +++ b/doc/wiki/LMTP.Exim.txt @@ -0,0 +1,153 @@ +Contents + + + 1. Using LMTP over UNIX Socket + + 2. Using LMTP over TCP Socket + + 3. Striping domain to avoid user unknown / doesn't exist error + + 4. Verifying recipients using LMTP + + 5. Delivering mails case insensitively + +Exim provides support for LMTP over UNIX sockets using the LMTP transport +[http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_lmtp_transport.html], +your distribution may/not provide this, run exim -bV and check for 'lmtp' in +'Transports:'. Support for LMTP over TCP sockets is provided by the SMTP +transport +[http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_smtp_transport.html]. + +Using LMTP over UNIX Socket +--------------------------- + +Use this configuration if dovecot runs on the same host as exim. + +Example router: + +---%<------------------------------------------------------------------------- +local_user: + debug_print = "R: local_user for $local_part@$domain" + driver = accept + domains = +local_domains + check_local_user + transport = dovecot_lmtp + cannot_route_message = Unknown user +---%<------------------------------------------------------------------------- + +Example transport: + +---%<------------------------------------------------------------------------- +dovecot_lmtp: + driver = lmtp + socket = /var/run/dovecot/lmtp + #maximum number of deliveries per batch, default 1 + batch_max = 200 + #allow suffixes/prefixes (default unset) + rcpt_include_affixes +---%<------------------------------------------------------------------------- + +Using LMTP over TCP Socket +-------------------------- + +Example router: + +---%<------------------------------------------------------------------------- +local_user: + transport = dovecot_lmtp + domains = +local_domains + driver = manualroute + route_list = "* 192.168.1.0 byname" + #if destination server is the local host enable this + #self = send +---%<------------------------------------------------------------------------- + +Set IP and port as appropriate to your setup. + +Example transport: + +---%<------------------------------------------------------------------------- +dovecot_lmtp: + driver = smtp + #allow suffixes/prefixes (default unset) + rcpt_include_affixes + protocol = lmtp + port = 2525 +---%<------------------------------------------------------------------------- + +Striping domain to avoid user unknown / doesn't exist error +----------------------------------------------------------- + +If you are using a userdb which does not have domain names, you may need to add +a setting to 20-lmtp.conf + +---%<------------------------------------------------------------------------- +protocol lmtp { + ... + # use %n to strip away the domain part + auth_username_format = %n +} +---%<------------------------------------------------------------------------- + +Symptoms: + + * Exim says something like "LMTP error after RCPT ... 550 ... User doesn't + exist someuser@somedomain" + * Dovecot verbose log says something like "auth-worker(9048): + passwd(someuser@somedomain): unknown user" + +Verifying recipients using LMTP +------------------------------- + +You can use callout verification to avoid accepting mail for addresses which do +not exist in Dovecot. Below is a config snippet which could be used in +acl_smtp_rcpt to achieve this: + +---%<------------------------------------------------------------------------- +deny + message = invalid recipient + domains = +local_domains + !verify = recipient/callout=no_cache +---%<------------------------------------------------------------------------- + +For more information on address verification see the related section of the +Exim specification +[http://www.exim.org/exim-html-current/doc/html/spec_html/ch-access_control_lists.html#SECTaddressverification]. + +Delivering mails case insensitively +----------------------------------- + +*Warning: *Just use this setup if all your login names contain only lower case +characters! (On Linux see /etc/adduser.conf under NAME_REGEX variable). + +Exim retains the case of the local part. Dovecot's LMTP /may/ fail looking up +an incorrect cased local part in your userdb. You can solve this problem by +extending the /protocol lmtp/ section: + +---%<------------------------------------------------------------------------- +protocol lmtp { + ... + # use %Ln to strip away the domain part + auth_username_format = %Lu +} +---%<------------------------------------------------------------------------- + +(If you don't mind allowing case insensitive logins for dovecoth +authentication, you may set /auth_username_format/ in the global configuration +accordingly and renounce the above change). + +In case you prefer to configure exim to lower case the local part instead, add +a router just before your local delivery router: + +---%<------------------------------------------------------------------------- +lowercase_local: + debug_print = "R: lower case local_part for local delivery" + driver = redirect + redirect_router = local_user + data = ${lc:${local_part}} +---%<------------------------------------------------------------------------- + +Make sure to reference the name you have chosen for your local delivery router +within /redirect_router/. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LMTP.txt b/doc/wiki/LMTP.txt new file mode 100644 index 0000000..ead650a --- /dev/null +++ b/doc/wiki/LMTP.txt @@ -0,0 +1,131 @@ +LMTP Server +=========== + +LMTP uses the same settings as <LDA.txt>, as specified in 'conf.d/15-lda.conf' +in example configuration. There is also a bit of extra configuration in +'conf.d/20-lmtp.conf'. The main difference is that the LDA is a short-running +process, started as a binary from command line, while LMTP is a long-running +process started by Dovecot's master process. + +Envelope Addresses +------------------ + +Compared to dovecot-lda parameters, the addresses are taken from: + + * -f / Envelope sender address: This is the MAIL FROM: value from LMTP + session. + * -r / Final envelope recipient address: This is the RCPT TO: value from LMTP + session. + * -a / Original envelope recipient address: This defaults to same as RCPT TO: + value, but may be overridden by reading it from a header specified by + 'lda_original_recipient_header' setting. + * -d / Destination username: This is the same as RCPT TO: value, but with the + "+extension" part removed when 'recipient_delimiter' setting is set. If + usernames differ from recipient email addresses, the userdb must handle the + translation. + +Listeners +--------- + +You can configure LMTP to be listening on TCP or UNIX sockets: + +---%<------------------------------------------------------------------------- +# add lmtp to protocols, otherwise its listeners are ignored +protocols = imap pop3 lmtp + +service lmtp { + inet_listener lmtp { + address = 192.168.0.24 127.0.0.1 ::1 + port = 24 + } + + unix_listener lmtp { + #mode = 0666 + } +} +---%<------------------------------------------------------------------------- + +The UNIX listener on $base_dir/lmtp is enabled by default when protocols +setting contains lmtp. + +Security +-------- + +Unfortunately LMTP process currently needs to run as root, and only temporarily +drop privileges to users. Otherwise it couldn't handle mail deliveries to more +than a single user with different UID. If you're using only a single global +UID/GID, you can improve security by running lmtp processes as that user: + +---%<------------------------------------------------------------------------- +service lmtp { + user = vmail +} +---%<------------------------------------------------------------------------- + +LMTP Proxying +------------- + +It's possible to use Dovecot LMTP server as a proxy to remote LMTP or SMTP +servers. The configuration is similar to <IMAP/POP3 proxying> +[PasswordDatabase.ExtraFields.Proxy.txt], but you'll need to tell Dovecot LMTP +to issue passdb lookups: + +---%<------------------------------------------------------------------------- +lmtp_proxy = yes +---%<------------------------------------------------------------------------- + +Performance +----------- + +For higher volume sites, it may be desirable to increase the number of active +listener processes. A range of 5 to 20 is probably good for most sites: + +---%<------------------------------------------------------------------------- +service lmtp { + process_min_avail = 5 +} +---%<------------------------------------------------------------------------- + +Logging +------- + +If you want to store LMTP delivery logs to a different file, you can do it +with: + +---%<------------------------------------------------------------------------- +service lmtp { + executable = lmtp -L +} +protocol lmtp { + info_log_path = /var/log/dovecot-lmtp.log +} +---%<------------------------------------------------------------------------- + +For rawlogs, please see <Debugging.Rawlog.txt> + +Plugins +------- + + * Most of the <Dovecot plugins> [Plugins.txt] work with LMTP. + * Virtual quota can be enforced using <Quota plugin> [Quota.txt]. + * 'lmtp_rcpt_check_quota = yes' enables quota checking already at RCPT TO + stage. This check isn't done for proxied connections. + * Sieve language support can be added with the <Pigeonhole Sieve plugin> + [Pigeonhole.Sieve.txt]. + +Address extension delivery +-------------------------- + +To make address extension work with LMTP you must check these variables are set + + * lmtp_save_to_detail_mailbox = yes + * recipient_delimiter = + + +Using LMTP with different MTAs +------------------------------ + + * <Postfix> [HowTo.PostfixDovecotLMTP.txt] + * <Exim> [LMTP.Exim.txt] + * HALON [http://wiki.halon.se/LMTP] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Logging.txt b/doc/wiki/Logging.txt new file mode 100644 index 0000000..ef5dc04 --- /dev/null +++ b/doc/wiki/Logging.txt @@ -0,0 +1,181 @@ +Contents + + + 1. Dovecot Logging + + 1. Internal Errors + + 2. Changing Log File Paths + + 3. Syslog Example + + 4. Rotating Logs + + 5. Logging verbosity + +Dovecot Logging +=============== + +*Dovecot always logs a detailed error message* if something goes wrong. If it +doesn't, it's considered a bug and will be fixed. However, almost always the +problem is that *you're looking at the wrong log file*; error messages may be +logged to a different file than informational messages. + +You can find the log file locations by running: + +---%<------------------------------------------------------------------------- +doveadm log find +---%<------------------------------------------------------------------------- + +Dovecot log configuration is found in the conf.d/10-logging.conf file in the +dovecot configuration folder (usually */etc/dovecot* but may also be +*/usr/local/etc/dovecot*). + +By default Dovecot logs to syslog using *mail* facility. You can change the +facility from 'syslog_facility' setting. The syslog configuration is often in +'/etc/syslog.conf' or '/etc/rsyslog*' files. You can also configure Dovecot to +write to log files directly, see below. + +When using syslog, Dovecot uses 5 different logging levels: + + * *debug*: Debug-level message. + * *info*: Informational messages. + * *warning*: Warnings that don't cause an actual error, but are useful to know + about. + * *err*: Non-fatal errors. + * *crit*: Fatal errors that cause the process to die. + +Where exactly these messages are logged depends entirely on your syslog +configuration. Often everything is logged to '/var/log/mail.log' or +'/var/log/maillog', and *err* and *crit* are logged to '/var/log/mail.err'. +This is not necessarily true for your configuration though. + +In an ideal configuration the errors would be logged to a separate file than +non-errors. For example you could set 'syslog_facility = local5' and set: + +---%<------------------------------------------------------------------------- +local5.* -/var/log/dovecot.log +local5.warning;local5.error;local5.crit -/var/log/dovecot-errors.log +---%<------------------------------------------------------------------------- + +Here all the Dovecot messages get logged into 'dovecot.log', while all the +important error/warning messages get logged into 'dovecot-errors.log'. + +Internal Errors +--------------- + +If IMAP or POP3 processes encounter some error, they don't show the exact +reason for clients. Instead they show: + +---%<------------------------------------------------------------------------- +Internal error occurred. Refer to server log for more information. [2006-01-07 +22:35:11] +---%<------------------------------------------------------------------------- + +The point is that whenever anything unexpected happens, Dovecot doesn't leak +any extra information about it to clients. They don't need it and they might +try to exploit it in some ways, so the less they know the better. + +The real error message is written to the error log file. The timestamp is meant +for you to help you find it. + +Changing Log File Paths +----------------------- + +If you don't want to use syslog, or if you just can't find the Dovecot's error +logs, you can make Dovecot log elsewhere as well: + +---%<------------------------------------------------------------------------- +log_path = /var/log/dovecot.log +# If you want everything in one file, just don't specify info_log_path and +debug_log_path +info_log_path = /var/log/dovecot-info.log +# Leave empty in order to send debug-level messages to info_log_path +debug_log_path = /var/log/dovecot-debug.log +---%<------------------------------------------------------------------------- + +The warning and error messages will go to file specified by 'log_path', while +informative messages goes to 'info_log_path' and debug messages goes to +'debug_log_path'. If you do this, make sure you're really looking at the +'log_path' file for error messages, since the "Starting up" message is written +to 'info_log_path' file. + +Syslog Example +-------------- + +Dovecot logging asynchronously via 'syslog_facility = local5' with basic rules: + +---%<------------------------------------------------------------------------- +local5.* -/var/log/dovecot.log +local5.info -/var/log/dovecot.info +local5.warn -/var/log/dovecot.warn +local5.err -/var/log/dovecot.err +:msg,contains,"stored mail into mailbox"\ + -/var/log/dovecot.lmtp +---%<------------------------------------------------------------------------- + +Rotating Logs +------------- + +If you change from syslog to an external log file, you can use logrotate +(available on most recent linux distros) to maintain the Dovecot logfile so it +doesn't grow beyond a manageable size. Save the below scriptlet as +'/etc/logrotate.d/dovecot': + +---%<------------------------------------------------------------------------- +/var/log/dovecot*.log { + weekly + rotate 4 + missingok + notifempty + compress + delaycompress + sharedscripts + postrotate + doveadm log reopen + endscript +} +---%<------------------------------------------------------------------------- + +*Note:* doveadm is not working properly with SELinux (e.g. doveadm cannot read +config file when called from logrotate context). SELinux safe postrotate +alternative scriplet: + +---%<------------------------------------------------------------------------- +postrotate + kill -s 0 `cat /var/run/dovecot/master.pid` || kill -s USR1 `cat +/var/run/dovecot/master.pid` +endscript +---%<------------------------------------------------------------------------- + +*Note 2:* When 'syslog_facility = local5' is used for logging (example above), +the line "/var/log/dovecot.log" should be added to the +'/etc/logrotate.d/syslog' file to enable rotation (no +'/etc/logrotate.d/dovecot' in this case!). + +Logging verbosity +----------------- + +There are several settings that control logging verbosity. By default they're +all disabled, but they may be useful for debugging. + + * 'auth_verbose=yes' enables logging all failed authentication attempts. + * 'auth_debug=yes' enables all authentication debug logging (also enables + 'auth_verbose'). Passwords are logged as '<hidden>'. + * 'auth_debug_passwords=yes' does everything that 'auth_debug=yes' does, but + it also removes password hiding (but only if you are not using PAM, since + PAM errors aren't written to Dovecot's own logs). + * 'mail_debug=yes' enables all kinds of mail related debug logging, such as + showing where Dovecot is looking for mails. + * 'verbose_ssl=yes' enables logging SSL errors and warnings. Even without this + setting if connection is closed because of an SSL error, the error is logged + as the disconnection reason. + * 'auth_verbose_passwords=no|plain|sha1' If authentication fails, this setting + logs the used password. If you don't really need to know what the password + itself was, but are more interested in knowing if the user is simply trying + to use the wrong password every single time or if it's a brute force attack, + you can set this to "sha1" and only the SHA1 of the password is logged. + That's enough to know if the password is same or different between login + attempts. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LoginProcess.txt b/doc/wiki/LoginProcess.txt new file mode 100644 index 0000000..0918e1b --- /dev/null +++ b/doc/wiki/LoginProcess.txt @@ -0,0 +1,126 @@ +Login processes +=============== + +The main purpose of login processes is to handle the IMAP, POP3, +<Submission.txt> (v2.3), and <ManageSieve> [Pigeonhole.ManageSieve.txt] +connections before the user has logged in. The login processes don't need to be +able to do anything else than let the user log in, so they can run in highly +restricted environment. By default they are run as a non-privileged "dovenull" +user chrooted into a non-writable directory containing only authentication UNIX +sockets. + +Login processes also handle proxying the SSL and TLS connections even after the +user has logged in. This way all the SSL code runs in the same restricted +environment, which means that a security hole in the SSL library gives the +attacker access only to the restricted chroot, rather than possibly all the +users' mails. + +The default login settings should be good enough for small sites. There are two +ways to run the login processes: the high-security mode and the +high-performance mode. Both are discussed separately below. + +For explanation on the various settings for services, see <Services.txt> + +High-security mode +------------------ + +You can enable high-security mode with: + +---%<------------------------------------------------------------------------- +service imap-login { + service_count = 1 + #process_min_avail = 0 + #process_limit = $default_process_limit + #vsz_limit = 64M +} +service pop3-login { + service_count = 1 +} +---%<------------------------------------------------------------------------- + +This is the default. It works by using a new imap-login or pop3-login process +for each incoming connection. Since the processes run in a highly restricted +chroot, running each connection in a separate process means that in case there +is a security hole in Dovecot's pre-authentication code or in the SSL library, +the attacker can't see other users' connections and can't really do anything +destructive. The only way out of it is to find and exploit a kernel security +hole. + +Since one login process can handle only one connection, the service's +'process_limit' setting limits the number of users that can be logging in at +the same time (defaults to 'default_process_limit=100'). SSL/TLS proxying +processes are also counted here, so if you're using SSL/TLS you'll need to make +sure this count is higher than the maximum number of users that can be logged +in simultaneously. With TLS/SSL connections, the login process will not +terminate, and remains to perform proxying between imap backend process and the +client. + + * If the maximum login process count is reached, the oldest process in + logging-in state (ie. non-proxying) is destroyed. + * To avoid startup latency for new client connections, set 'process_min_avail' + to higher than zero. That many idling processes are always kept around + waiting for new connections. + * 'vsz_limit' should be fine at its default 64MB value. + +High-performance mode +--------------------- + +You can enable high-performance mode with: + +---%<------------------------------------------------------------------------- +service imap-login { + service_count = 0 + #client_limit = $default_client_limit + process_min_avail = 4 # number of CPU cores + vsz_limit = 1G +} +service pop3-login { + service_count = 0 +} +---%<------------------------------------------------------------------------- + +It works by using a number of long running login processes, each handling a +number of connections. This loses much of the security benefits of the login +process design, because in case of a security hole (in Dovecot or SSL library) +the attacker is now able to see other users logging in and steal their +passwords, read their mails, etc. + + * 'process_min_avail' should be set to be at least the number of CPU cores in + the system, so that all of them will be used. + * Otherwise new processes are created only once an existing one's connection + count reaches client_limit + * Default client_limit * process_limit = 1000*100 = 100k connections + * vsz_limit should be increased to avoid out of memory errors, especially if + you're using SSL/TLS. + +Login access check sockets +-------------------------- + +Dovecot login processes can check via UNIX socket if the incoming connection +should be allowed to log in. This is most importantly implemented to enable TCP +wrappers support for Dovecot. + +TCP wrappers support +-------------------- + +You must have built Dovecot with support for TCP wrappers. You can do this by +giving '--with-libwrap' parameter to 'configure'. + +Add to dovecot.conf: + +---%<------------------------------------------------------------------------- +login_access_sockets = tcpwrap + +service tcpwrap { + unix_listener login/tcpwrap { + group = $default_login_user + mode = 0600 + user = $default_login_user + } +} +---%<------------------------------------------------------------------------- + +Remember to configure your rules! The format is described in hosts.allow(5) and +hosts.deny(5). Files used are usually /etc/hosts.allow and /etc/hosts.deny. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MDA.txt b/doc/wiki/MDA.txt new file mode 100644 index 0000000..a4a0af3 --- /dev/null +++ b/doc/wiki/MDA.txt @@ -0,0 +1,17 @@ +MDA (LDA) +========= + +An MDA is a _M_ail _D_elivery _A_gent. + An LDA is a _L_ocal _D_elivery _A_gent. + These two terms are synonyms. + +An MDA is being passed messages from an <MTA.txt> and delivers it to a real or +virtual mailbox. + +Common choices include: + + * <maildrop.txt> + * procmail [http://www.procmail.org/] (appears to be unmaintained) + * <Dovecot's `deliver` LDA> [LDA.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MTA.txt b/doc/wiki/MTA.txt new file mode 100644 index 0000000..74722e4 --- /dev/null +++ b/doc/wiki/MTA.txt @@ -0,0 +1,54 @@ +MTA +=== + +MTA is an acronym for _M_ail _T_ransport _A_gent. It is the software that works +behind the scenes to transport E-Mail messages from one computer to another. +MUAs (such as mutt, thunderbird, sylpheed, evolution, kmail) hand off newly +sent messages to an MTA. MTAs talk to other MTAs, and either deliver mail +locally or hand it off for delivery to an <MDA/LDA> [MTA.txt] if it was +destined to the local system. + +MTA is a generic term and usually refers to one of these popular software +packages: + + * Postfix [http://www.postfix.org/] is Wietse Venema's secure, fast and + flexible mailer. Default on SUSE Linux and NetBSD. + * Exim [http://www.exim.org/] is Philip Hazel's flexible mailer. Default on + Debian GNU/Linux. + * Sendmail [http://www.sendmail.org/] the original BSD mailer. Default on + FreeBSD; a Sun spinoff is used on Solaris. + * Courier [http://www.courier-mta.org/] was inspired by qmail, but intends to + do things right. + * qmail [http://cr.yp.to/qmail.html] is an obsolete and unmaintained server. + Its POP3 part can be taken over by Dovecot. Qmail started off boasting about + speed and security in the mid-1990s, but has lots of unfixed bugs (this + document includes patches where known), + [http://home.pages.de/~mandree/qmail-bugs.html] among them security bugs + that remain unfixed, and the security guarantee (500 USD) denied. If you + really intend to continue using it, read Dave Sill's Life with qmail + [http://www.lifewithqmail.org/] which contains instructions to work around + some of qmail's security issues. + * HALON [https://halon.io/] is a commercial MTA, which supports Dovecot Auth + and LMTP. It's based on FreeBSD with anti-spam/virus, DKIM, DMARC, DANE, + etc. + +Some people also subsume mail fetching utilities under the MTA category, among +them: + + * fetchmail [http://www.fetchmail.info/] a fast mail retriever for the POP2, + POP3/KPOP/SDPS, IMAP2/IMAP4, ODMR and ETRN protocols, SSL and Kerberos + capable. It forwards the retrieved messages to SMTP/ESMTP, LMTP servers or + into an <LDA.txt>. Developed out of Carl Harris's popclient by Eric S. + Raymond. Fetchmail is now maintained by <MatthiasAndree.txt> and Rob F. + Funk. + * getmail [http://pyropus.ca/software/getmail/] a POP3/SDPS and IMAP4 (with + SSL) enabled mail retrieval utility written in Python. Developed by Chales + Cazabon. + +These mail fetching utilities can be used to store mail for later retrieval by +dovecot. + + Contrast this to the <message delivery agent (MDA)> [MDA.txt], sometimes +called local delivery agent (LDA). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.LocalDisk.txt b/doc/wiki/MailLocation.LocalDisk.txt new file mode 100644 index 0000000..4bd99fe --- /dev/null +++ b/doc/wiki/MailLocation.LocalDisk.txt @@ -0,0 +1,50 @@ +Mail storage in local disk +========================== + +Filesystems +----------- + + * See <MailboxFormat.Maildir.txt> for Maildir-specific filesystem + optimizations + * Dovecot doesn't rely on atime updates, so you can mount the filesystem with + noatime + +Index files +----------- + +Keeping index files on a different disk than the mail spool gives you better +performance. The indexes have a lot of write activity so it is recommended to +use RAID-10 instead of RAID-5 for them. + +Fsyncing +-------- + +By default Dovecot calls fsync() and fdatasync() whenever it's useful to +prevent potential data loss. The main reason for this is so that Dovecot won't +lie that the message was saved to the disk, if in fact a power failure a second +later would lose the message. With IMAP clients this is perhaps a less serious +problem, because the lost message was most likely either a mail in Draft +mailbox or a message in "Sent Messages" mailbox. In any case a message that the +user had already seen. However if <LDA.txt> loses a message, the user never +even knew that the message existed, unless the sender decides to resend it. + +Since power failures and kernel panics are quite rare, many people are tempted +to disable fsyncing because it may increase the performance quite a lot. +Dovecot allows this by setting 'mail_fsync=never'. However you should consider +leaving it at "optimized" for LDA and LMTP. You can do this with: + +---%<------------------------------------------------------------------------- +# Default to no fsyncing +mail_fsync = never + +protocol lda { + # Enable fsyncing for LDA + mail_fsync = optimized +} +protocol lmtp { + # Enable fsyncing for LMTP + mail_fsync = optimized +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.Maildir.txt b/doc/wiki/MailLocation.Maildir.txt new file mode 100644 index 0000000..4552cf8 --- /dev/null +++ b/doc/wiki/MailLocation.Maildir.txt @@ -0,0 +1,142 @@ +Maildir configuration +===================== + +See <MailboxFormat.Maildir.txt> for a complete description of how Dovecot has +implemented Maildir support. + +Mail location +------------- + +Maildir exists almost always in '~/Maildir' directory. The mail location is +specified with: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +---%<------------------------------------------------------------------------- + +Directory layout +---------------- + +By default Dovecot uses Maildir++ directory layout. This means that all +mailboxes are stored in a single directory and prefixed with a dot. For +example: + + * Maildir/.folder/ + * Maildir/.folder.subfolder/ + +If you want maildirs to use hierarchical directories, such as: + + * Maildir/folder/ + * Maildir/folder/subfolder/ + +you'll need to enable fs layout: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:LAYOUT=fs +---%<------------------------------------------------------------------------- + +Control files +------------- + +Dovecot stores some Maildir metadata into two control files: + + * 'dovecot-uidlist' file contains IMAP UID <-> Maildir filename mapping + * 'dovecot-keywords' file contains Maildir filename flag (a..z = 0..25) <-> + keyword name mapping + +Both of these files are described fully in <MailboxFormat.Maildir.txt>. The +important thing to remember about them however is that they shouldn't be +treated the same way as index files. Index files can be deleted and rebuilt +without any side effects, but if you delete control files you'll cause messages +to get new UIDs and possibly lose keyword names. + +If the messages get new UIDs, the IMAP clients will invalidate their local +cache and download the messages all over again. If you do this for all the +users, you could cause huge disk I/O bursts to your server. + +Dovecot can't currently handle not being able to write the control files, so it +will cause problems with <filesystem quota> [Quota.FS.txt]. To avoid problems +with this, you should place control files into a partition where quota isn't +checked. You can specify this by adding ':CONTROL=<path>' to 'mail_location', +for example: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:CONTROL=/var/no-quota/%u +---%<------------------------------------------------------------------------- + +Index files +----------- + +See <MailLocation#indexfiles> [MailLocation.txt] for full explanation of how to +change the index path. For example: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:INDEX=/var/indexes/%u +---%<------------------------------------------------------------------------- + +Optimizations +------------- + + * 'maildir_very_dirty_syncs=yes': Assume that only Dovecot accesses 'cur/' + directory. If another process (or even Dovecot which doesn't update index + files) does changes to 'cur/' while the mailbox is simultaneously being + modified by Dovecot, Dovecot may not notice those external changes. It's + still safe to deliver new mails to 'new/' using non-Dovecot software (except + with 'mailbox_list_index=yes' changes aren't noticed outside INBOX). + * 'maildir_copy_with_hardlinks=yes' (default): When copying a message, do it + with hard links whenever possible. This makes the performance much better, + and it's unlikely to have any side effects. Only reason to disable this is + if you're using a filesystem where hard links are slow (e.g. HFS+). + * 'maildir_stat_dirs=no' (default): Assume that all the files beginning with a + dot in the maildir are maildirs. You shouldn't have any non-directory files + beginning with a dot in the maildirs, but if you do you may need to set this + to "yes", in which case Dovecot needs to stat() each directory entry, which + degrades the performance. Some filesystems provide the + directory/non-directory status for free without having to stat(). In those + filesystems this setting is ignored. + +Filesystem optimizations +------------------------ + +See <MailboxFormat/Maildir#Maildir_and_filesystems> +[MailboxFormat.Maildir.txt]. + +Mailbox directory name +---------------------- + +When using 'LAYOUT=fs', there is a potential for naming collisions between +Maildir's 'new/', 'cur/' and 'tmp/' subdirectories, and mail folders of the +same names. For example, consider a mail folder "foo/bar". Under 'LAYOUT=fs', +data for this mail folder will be stored at under Maildir's usual three +directories '~/Maildir/foo/bar/{new,cur,tmp}/'. + +If the user then tries to create a mail folder "foo/bar/new", this would then +imply that data should be stored in Maildir's three directories +'~/Maildir/foo/bar/new/{new,cur,tmp}/'. But this would overlap Maildir's 'new/' +subdirectory of mail folder "foo/bar". + +This may not be a problem in many installations, but if a risk of collisions +with Maildir's three subdirectory names is perceived, then the 'DIRNAME' +parameter can be used. For example, if we specify mail location as: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:LAYOUT=fs:DIRNAME=mAildir +---%<------------------------------------------------------------------------- + +then this will push Maildir's 'new/', 'cur/' and 'tmp/' subdirectories down +into a subdirectory 'mAildir/', so a mail folder "foo/bar" would be stored at +'~/Maildir/foo/bar/mAildir/{new,cur,tmp}/'. A mail folder "foo/bar/new" would +be stored at '~/Maildir/foo/bar/new/mAildir/{new,cur,tmp}/', which would then +have no overlap with the mail folder "foo/bar". + +'DIRNAME' affects INBOX slightly differently. Without 'DIRNAME', INBOX will be +stored at '~/Maildir/{new,cur,tmp}/', but when 'DIRNAME' is specified, we get +an extra path component 'INBOX/' immediately prior to the 'DIRNAME' value, so +in the example above INBOX would be stored at +'~/Maildir/INBOX/mAildir/{new,cur,tmp}/'. + +The value for 'DIRNAME' should be chosen carefully so as to minimise the +chances of clashing with mail folder names. In the example here, unusual +upper/lower casing has been used. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.SharedDisk.txt b/doc/wiki/MailLocation.SharedDisk.txt new file mode 100644 index 0000000..528a663 --- /dev/null +++ b/doc/wiki/MailLocation.SharedDisk.txt @@ -0,0 +1,75 @@ +Mail storage on shared disks +============================ + +Dovecot supports keeping mails and index files in clustered filesystems, such +as Red Hat GFS [http://www.redhat.com/gfs/], Oracle OCFS2 +[http://oss.oracle.com/projects/ocfs2/] and HP Polyserve Matrix +[http://h18006.www1.hp.com/storage/software/clusteredfs/index.html]. + +Dovecot also supports keeping mails and index files on NFS. Everything +described in this page applies to NFS as well, but see <NFS.txt> for more +NFS-specific problems and optimizations. + +Memory mapping +-------------- + +By default Dovecot mmap()s the index files. This may not work with all +clustered filesystems, and it most certainly won't work with NFS. Setting +'mmap_disable = yes' disables mmap() and Dovecot does its own internal caching. +If mmap() is supported by your filesystem, it's still not certain that it gives +better performance. Try benchmarking to make sure. + +Locking +------- + +Dovecot supports locking index files with fcntl (default), flock or dotlocks. +Some clustered filesystems may not support fcntl, so you can change it to use +flock instead. Fcntl locks may also cause problems with some NFS +configurations, in which case you can try if switching to dotlocks helps. Note +that dotlocks are the slowest locking method. You can change the locking method +from 'lock_method' setting. Regardless of the 'lock_method' setting, Dovecot +always uses dotlocks for some locks. + +Clock synchronization +--------------------- + +Run ntpd in each computer to make sure their clocks are synchronized. If the +clocks are more than one second apart from each others and multiple computers +access the same mailbox simultaneously, you may get errors. + +Caching +------- + +Your cluster will probably perform better if users are usually redirected to +the same server. This is because the mailbox may already be cached in the +memory and it may also reduce the traffic between the clusterfs nodes. You can +use <director> [Director.txt] service to do this easily automatically. Or at +the very least make sure that your load balancer redirects connections from the +same IP address to the same server. + +FUSE / GlusterFS +---------------- + +FUSE caches dentries and file attributes internally. If you're using multiple +GlusterFS clients to access the same mailboxes, you're going to have problems. +Worst of these problems can be avoided by using NFS cache flushes, which just +happen to work with FUSE as well: + +---%<------------------------------------------------------------------------- +mail_nfs_index = yes +mail_nfs_storage = yes +---%<------------------------------------------------------------------------- + +These probably don't work perfectly. + +Samba / CIFS +------------ + +Dovecot's temporary files may include a colon character ':' in their filename, +which is not a permitted character when using cifs.Dovecot also renames the +temporary files whilst holding a lock in them, which generates the error 'Text +file is busy'. + +Cifs/smbfs is unlikely to work as a remote filesystem. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.dbox.txt b/doc/wiki/MailLocation.dbox.txt new file mode 100644 index 0000000..6ced4e3 --- /dev/null +++ b/doc/wiki/MailLocation.dbox.txt @@ -0,0 +1,88 @@ +dbox configuration +================== + +See <MailboxFormat.dbox.txt> for a description of the dbox mailbox format. +NOTE:*You must not lose the dbox index files, they can't be regenerated without +data loss*. + +Mail location +------------- + +dbox can be used in two ways: + + 1. One message per file (*single-dbox*), similar to <Maildir> + [MailboxFormat.Maildir.txt]. + 2. Multiple messages per file (*multi-dbox*), but unlike <mbox> + [MailboxFormat.mbox.txt] multiple files per mailbox. + +To use *single-dbox*, use the tag 'sdbox' in the <mail location> +[MailLocation.txt], for example: + +---%<------------------------------------------------------------------------- +# single-dbox +mail_location = sdbox:~/dbox +---%<------------------------------------------------------------------------- + +For backwards compatibility, 'dbox' is an alias to 'sdbox' in the mail +location. + +To use *multi-dbox*, use the tag 'mdbox' in the <mail location> +[MailLocation.txt], for example: + +---%<------------------------------------------------------------------------- +# multi-dbox +mail_location = mdbox:~/mdbox +---%<------------------------------------------------------------------------- + +Alternate storage +----------------- + +dbox has a feature for transparently moving message data to an <Alternate +storage> [MailboxFormat.dbox.txt] area. + +To specify an alternate storage area, use the 'ALT' parameter in the mail +location. For example, specifying the mail location as: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:/var/vmail/%d/%n:ALT=/altstorage/vmail/%d/%n +---%<------------------------------------------------------------------------- + +will make Dovecot look for message data first under '/var/vmail/%d/%n', and if +it is not found there it will look under '/altstorage/vmail/%d/%n' instead. + +Keep the unmounted '/altstorage' directory permissions such that Dovecot mail +processes can't create directories under it (e.g. root:root 0755). This way if +the alt storage isn't mounted for some reason, Dovecot won't think that all the +messages in alt storage were deleted and lose their flags. + +Mailbox directory name +---------------------- + +When using the default hierarchical layout, there is a potential for naming +collisions between dbox's 'dbox-Mails/' subdirectory and mail folders of the +same name. For example, consider a mail folder "foo/bar". Under the default +hierarchical layout, data about this mail folder would be stored at +'~/mdbox/mailboxes/foo/bar/dbox-Mails/'. + +If the user then tried to create a mail folder "foo/bar/dbox-Mails", this would +then imply that data would be stored at +'~/mdbox/mailboxes/foo/bar/dbox-Mails/dbox-Mails/'. But this would overlap the +'dbox-Mails/' subdirectory of mail folder "foo/bar". + +This may not be a problem in many installations, but if a risk of collisions +with the default name "dbox-Mails" is perceived, then the 'DIRNAME' parameter +can be used. For example, if we specify mail location as: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:~/mdbox:DIRNAME=DbOx-mAiLs +---%<------------------------------------------------------------------------- + +then this will make data for mail folders be stored in a subdirectory +'DbOx-mAiLs/' instead of the default 'dbox-Mails/', so a mail folder "foo/bar" +would have data stored at '~/mdbox/mailboxes/foo/bar/DbOx-mAiLs/'. + +The value for 'DIRNAME' should be chosen carefully so as to minimise the +chances of clashing with mail folder names. In the example here, unusual +upper/lower casing has been used. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.mbox.txt b/doc/wiki/MailLocation.mbox.txt new file mode 100644 index 0000000..98d6f73 --- /dev/null +++ b/doc/wiki/MailLocation.mbox.txt @@ -0,0 +1,194 @@ +mbox configuration +================== + +See <MailboxFormat.mbox.txt> for a complete description of how Dovecot has +implemented mbox support. + +Mail location +------------- + +In many systems the user's mails are by default stored in '/var/mail/username' +file. This file is called INBOX in IMAP world. Since IMAP supports multiple +mailboxes, you'll need to have a directory for them as well. Usually '~/mail' +is a good choice for this. For installation such as this, the mail location is +specified with: + +---%<------------------------------------------------------------------------- +# %u is replaced with the username that logs in +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +It's in no way a requirement to have the INBOX in '/var/mail/' directory. In +fact this often just brings problems because Dovecot might not be able to write +dotlock files to the directory (see below). You can avoid this completely by +just keeping everything in '~/mail/': + +---%<------------------------------------------------------------------------- +# INBOX exists in ~/mail/inbox +mail_location = mbox:~/mail +---%<------------------------------------------------------------------------- + +Index files +----------- + +See <MailLocation#indexfiles> [MailLocation.txt] for full explanation of how to +change the index path. For example: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=/var/indexes/%u +---%<------------------------------------------------------------------------- + +Locking +------- + +Make sure that all software accessing the mboxes are using the same locking +methods in the same order. The order is important to prevent deadlocking. From +Dovecot's side you can change these from 'mbox_read_locks' and +'mbox_write_locks' settings. See <MboxLocking.txt> for more information. + +/var/mail/ dotlocks +------------------- + +Often mbox write locks include dotlock, which means that Dovecot needs to +create a new "<mbox>.lock" file to the directory where the mbox file exists. If +your INBOXes are in '/var/mail/' directory you may have to give Dovecot write +access to the directory. There are two ways the '/var/mail/' directory's +permissions have traditionally been set up: + + * World-writable with sticky bit set, allowing anyone to create new files but + not overwrite or delete existing files owned by someone else (ie. same as + /tmp). You can do this with 'chmod a+rwxt /var/mail' + * Directory owned by a mail group and the directory set to group-writable + (mode=0770, group=mail) + +You can give Dovecot access to mail group by setting: + +---%<------------------------------------------------------------------------- +mail_privileged_group = mail +---%<------------------------------------------------------------------------- + +NOTE: With <LDA.txt> the 'mail_privileged_group' setting unfortunately doesn't +work, so you'll have to use the sticky bit, disable dotlocking completely or +use LMTP server instead. + +/var/mail/* permissions +----------------------- + +In some systems the '/var/mail/$USER' files have 0660 mode permissions. This +causes Dovecot to try to preserve the file's group, and if it doesn't have +permissions to do so, it'll fail with an error: + +---%<------------------------------------------------------------------------- +imap(user): Error: chown(/home/user/mail/.imap/INBOX, -1, 12(mail)) failed: +Operation not permitted (egid=1000(user), group based on /var/mail/user) +---%<------------------------------------------------------------------------- + +There is rarely any real need for the files to have 0660 mode, so the best +solution for this problem is to just change the mode to 0600: + +---%<------------------------------------------------------------------------- +chmod 0600 /var/mail/* +---%<------------------------------------------------------------------------- + +Optimizations +------------- + +The settings below are related to mbox performance. See +<MailboxFormat.mbox.txt> for more complete description of what they do. + + * 'mbox_lazy_writes=yes' (default): Metadata updates, such as writing X-UID + headers or flag changes, aren't written to mbox file until the mailbox is + closed or CHECK or EXPUNGE IMAP commands are sent by the client. The mbox + rewrites can be costly, so this may avoid a lot of disk writes. + * 'mbox_dirty_syncs=yes' (default): Dovecot assumes that external mbox file + changes only mean that new messages were appended to it. Without this + setting Dovecot re-reads the whole mbox file whenever it changes. There are + various safeguards in place to make this setting safe even when other + changes than appends were done to the mbox. The only downside to this + setting is that external message flag modifications may not be visible + immediately. + * 'mbox_very_dirty_syncs=yes' (not default): When opening mbox file that has + been changed externally, don't re-read it. Otherwise similar to + 'mbox_dirty_syncs=yes'. + * 'mbox_min_index_size=n': If mbox file is smaller than n kilobytes, don't + update its index files. If an index file exists for it, it's still read + however. + +Only /var/mail/ mboxes +---------------------- + +With POP3 it's been traditional that users have their mails only in the +'/var/mail/' directory. IMAP however supports having multiple mailboxes, so +each user has to have a private directory where the mailboxes are stored. +Dovecot also needs a directory for its index files unless you disable them +completely. + +If you *really* want to use Dovecot as a plain POP3 server without index files, +you can work around the problem of not having the per-user directory: + + * Set users' home directory in userdb to some empty non-writable directory, + for example '/var/empty' + * Modify 'mail_location' setting so that the mail root directory is also the + empty directory and append ':INDEX=MEMORY' to it. For example: + 'mail_location = mbox:/var/empty:INBOX=/var/mail/%u:INDEX=MEMORY' + * Note that if you have IMAP users, they'll see the '/var/empty' as the + directory containing other mailboxes than INBOX. If the directory is + writable, all the users will have their mailboxes shared. + +Directory layout +---------------- + +By default Dovecot uses filesystem layout under mbox. This means that mail is +stored in mbox files under hierarchical directories, for example: + + * '~/mail/inbox' - mbox file containing mail for INBOX + * '~/mail/foo' - mbox file containing mail for mailbox "foo" + * '~/mail/bar/baz' - mbox file containing mail for mailbox "bar/baz" + +One upshot of this is that it is not normally possible to have mailboxes which +are subfolders of mailboxes containing messages. + +As an alternative, it is possible to configure Dovecot to store all mailboxes +in a single directory with hierarchical levels separated by a dot. This can be +configured by adding ':LAYOUT=maildir++' to the mail location. There are, +however, some further considerations when doing this; see +<MboxChildFolders.txt> for some examples. + +Control files +------------- + +Under mbox format, Dovecot maintains the subscribed mailboxes list in a file +'.subscriptions' which by default is stored in the mail location root. So in +the example configuration this would be at '~/mail/.subscriptions'. + +If you want to put this somewhere else, you can change the directory in which +the '.subscriptions' file is kept by using the 'CONTROL' parameter. So for +example, if we configured the mail location using: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:CONTROL=~/mail-control +---%<------------------------------------------------------------------------- + +then the subscribed mailboxes list would be maintained at +'~/mail-control/.subscriptions'. + +One practical application of the 'CONTROL' parameter is described at +<MboxChildFolders.txt>. + +Message file name +----------------- + +By default, Dovecot stores messages for INBOX in an mbox file called "inbox", +and messages for all other mailboxes in an mbox file whose relative path is +equivalent to the name of the mailbox. Under this scheme, it is not possible to +have mailboxes which contain both messages and child mailboxes. + +However, the behaviour (for mailboxes other than INBOX) can be changed using +the 'DIRNAME' parameter. If the 'DIRNAME' parameter is specified with a +particular value, then Dovecot will store messages in a file with a name of +that value, in a directory with a name equivalent to the mailbox name. + +There are, however, some further considerations when doing this; see +<MboxChildFolders.txt> for an example. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.txt b/doc/wiki/MailLocation.txt new file mode 100644 index 0000000..eb0942b --- /dev/null +++ b/doc/wiki/MailLocation.txt @@ -0,0 +1,343 @@ +Mail location +============= + + * For mbox-specific settings, see <MailLocation.mbox.txt> + * For Maildir-specific settings, see <MailLocation.Maildir.txt> + * For dbox-specific settings, see <MailLocation.dbox.txt> + +There are three different places where the mail location is looked up from: + + 1. 'mail_location' setting in 'dovecot.conf' is used if nothing else overrides + it. + 2. 'mail' <userdb field> [UserDatabase.txt] overrides 'mail_location' setting. + + 3. 'location' setting inside namespaces overrides everything. Usually this + should be used only for public and shared namespaces. + +Autodetection +------------- + +By default the 'mail_location' setting is empty, which means that Dovecot +attempts to locate automatically where your mails are. This is done by looking, +in order, at: + + * '~/mdbox/' + * '~/sdbox/' + * '~/Maildir/' + * '~/mail/.imap/' + * '~/mail/inbox' + * '~/mail/mbox' + * '~/Mail/.imap/' + * '~/Mail/inbox' + * '~/Mail/mbox' + +For autodetection to work, one of the above locations has to be populated; when +autodetection is active, Dovecot will not attempt to create a mail folder. Note +that '.imap' is a directory, and 'inbox' and 'mbox' are files. + +It's usually a good idea to explicitly specify where the mails are, even if the +autodetection happens to work, in particular to benefit from auto-creation of +the folder for new users. + +Mailbox autocreation +-------------------- + +Dovecot in the 1.x era created mailboxes automatically regardless of whether +mail_location was set. In 2.x autocreation only gets triggered if mail_location +is correctly set. You'll see something like this if you enable debug logging: + +---%<------------------------------------------------------------------------- +Debug: Namespace : /home/user/Mail doesn't exist yet, using default permissions +Debug: Namespace : Using permissions from /home/user/Mail: mode=0700 +gid=default +---%<------------------------------------------------------------------------- + +and a 'Mail/.imap' directory will be present once that process has concluded. +This is the easiest way to ensure a freshly created user is correctly set up +for access via Dovecot. + +Format +------ + +The format of the mailbox location specification is as follows: + + * / <mailbox-format> [MailboxFormat.txt]/ : /path/ [ : /key/ = /value/ … ] + +where: + + * /mailbox-format/ is a tag identifying one of the formats described at + <Mailbox Formats> [MailboxFormat.txt]. + * /path/ is the path to a directory where the mail is stored. This must be an + absolute path, not a relative path. Even if relative paths appear to work, + this usage is deprecated and will likely stop working at some point. Do not + use the home directory, for reasons see <Home vs. mail directory> + [VirtualUsers.Home.txt] + * /key/ = /value/ can appear zero or more times to set various optional + parameters. Possible values for /key/ are: + * 'INDEX' : specifies the location of <index files> [MailLocation.txt]. + * 'ITERINDEX' : Perform mailbox listing using the INDEX directories + instead of the mail root directories. Mainly useful when the INDEX + storage is on a faster storage. It takes no value. (v2.2.32+) + * 'INBOX' : specifies the location of the <INBOX> [MailLocation.txt]. + * 'LAYOUT' : specifies the directory layout to use: + * Maildir++: The default used by Maildir format + * fs: The default used by mbox and dbox formats + * index: Uses mailbox GUIDs as the directory names. The mapping between + mailbox names and GUIDs exists in dovecot.list.index* files. + * 'NO-NOSELECT' : Automatically delete any \ <NoSelect.txt> mailboxes that + have no children. These mailboxes are sometimes confusing to users. Also + if a \ <NoSelect.txt> mailbox is attempted to be created with "CREATE + box/", it's created as selectable mailbox instead. (LAYOUT=Maildir++ + always behaves this same way.) (v2.2.32+) + * 'UTF-8' : Store mailbox names on disk using UTF-8 instead of modified + UTF-7. + * 'BROKENCHAR' : Specifies an escape character that is used for broken + mailbox names. If mailbox name can't be changed reversibly to UTF-8 and + back, encode the problematic parts using<broken_char><hex> in the + user-visible UTF-8 name. The broken_char itself also has to be encoded + the same way. This can be useful with imapc to access mailbox names that + aren't valid mUTF-7 charset from remote servers. (v2.2.32+) + * 'CONTROL' : Specifies the location of control files under the <mbox> + [MailLocation.mbox.txt] or <Maildir> [MailLocation.Maildir.txt] formats. + * 'VOLATILEDIR' : Specifies the location of volatile files. This includes + lock files and potentially other files that don't need to exist + permanently. This is especially useful to avoid creating lock files to + NFS or other remote filesystems. (v2.2.32+) + * 'SUBSCRIPTIONS' : specifies the file used for storing subscriptions. The + default is "subscriptions". If you're trying to avoid name collisions + with a mailbox named "subscriptions", then also consider setting + 'MAILBOXDIR'. + * 'MAILBOXDIR' : specifies directory name under which all mailbox + directories are stored. With <dbox formats> [MailboxFormat.dbox.txt] the + default is "mailboxes/" while with other mailbox formats the default is + empty. Typically this should be changed only for <lazy_expunge namespace> + [Plugins.Lazyexpunge.txt] with mdbox. + * 'DIRNAME' : specifies the directory name used for mailbox directories, or + in the case of mbox specifies the mailbox message file name. With <dbox + formats> [MailboxFormat.dbox.txt] the default is "dbox-Mails/" while with + other mailbox formats the default is empty. Can be used under either + <mbox> [MailLocation.mbox.txt], <Maildir> [MailLocation.Maildir.txt] or + <dbox> [MailLocation.dbox.txt] formats. Note that this directory is used + only for the mail directory and the alt directory, not for index/control + directories (but see below). + * 'FULLDIRNAME' : Same as 'DIRNAME', but use the directory name also for + index and control directory paths. This should be used instead of + 'DIRNAME' for new installations. (v2.2.8+) + * 'ALT' : specifies the <Alternate storage> [MailLocation.dbox.txt] path + for dbox formats. + * The colons and equals signs are literal and there are no spaces in an actual + mailbox location specification. + +Variables +--------- + +You can use several variables in the 'mail_location' setting. See +<Variables.txt> for a full list, but the most commonly used ones are: + + * '%u': Full username. + * '%n': User part in user@domain, same as %u if there's no domain. + * '%d': Domain part in user@domain, empty if there's no domain. + +Typical settings +---------------- + +Typically with Maildir it would be set to: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +---%<------------------------------------------------------------------------- + +with mbox: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +or if you'd like to use the <dbox> [MailboxFormat.dbox.txt] format: + +---%<------------------------------------------------------------------------- +# single-dbox +mail_location = sdbox:~/dbox +---%<------------------------------------------------------------------------- + +or: + +---%<------------------------------------------------------------------------- +# multi-dbox +mail_location = mdbox:~/mdbox +---%<------------------------------------------------------------------------- + +Use only absolute paths. Even if relative paths would appear to work, they +might just as well break some day. + +Directory hashing +----------------- + +You can use two different kinds of hashes in <variables> [Variables.txt]: + + * %H modifiers returns a 32bit hash of the given string as hex. For example + '%2.256H' would return max. 256 different hashes in range 00 .. ff. + * %M returns a MD5 hash of the string as hex. This can be used for two level + hashing by getting substrings of the MD5 hash. For example '%1Mu/%2.1Mu/%u' + returns directories from '0/0/user' to 'f/f/user'. + +Index files +----------- + +Index files are by default stored under the same directory as mails. With +maildir they are stored in the actual maildirs, with mbox they are stored under +'.imap/' directory. You may want to change the index file location if you're +using <NFS.txt> or if you're setting up <shared mailboxes> +[SharedMailboxes.txt]. + +You can change the index file location by adding ':INDEX=<path>' to +mail_location. For example: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:INDEX=/var/indexes/%u +---%<------------------------------------------------------------------------- + +The index directories are created automatically, but note that it requires that +Dovecot has actually access to create the directories. Either make sure that +the index root directory ('/var/indexes' in the above example) is writable to +the logged in user, or create the user's directory with proper permissions +before the user logs in. + +If you really want to, you can also disable the index files completely by +appending ':INDEX=MEMORY'. + +Private index files (v2.2+) +--------------------------- + +Since v2.2 the recommended way to enable private flags for shared mailboxes is +to create private indexes with :INDEXPVT=<path>. See +<SharedMailboxes.Public.txt> for more information. + +INBOX path +---------- + +INBOX path can be specified to exist elsewhere than the rest of the mailboxes, +for example: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +mail_location = maildir:~/Maildir:INBOX=~/Maildir/.INBOX +---%<------------------------------------------------------------------------- + +Note that it's still not possible to mix maildir and mbox formats this way. You +need to use <namespaces> [Namespaces.txt] for that. + +Homeless users +-------------- + +Having a home directory for users is highly recommended. The <Pigeonhole.txt> +<Sieve plugin> [Pigeonhole.Sieve.txt] already requires a home directory to +work, and it probably won't be the last feature to require a home. See +<VirtualUsers#homedirs> [VirtualUsers.txt] for more reasons why it's a good +idea, and how to give Dovecot a home directory even if you don't have a "real +home directory". + +If you really don't want to set any home directory, you can use something like: + +---%<------------------------------------------------------------------------- +mail_location = maildir:/home/%u/Maildir +---%<------------------------------------------------------------------------- + +Per-user mail locations +----------------------- + +It's possible to override the default 'mail_location' for specific users by +making the <user database> [UserDatabase.txt] return 'mail' <extra field> +[UserDatabase.ExtraFields.txt]. See the <user database> [UserDatabase.txt] page +for the specific userdb you're using for more information how to do this. +Below are however a couple of examples. + +Note that %h doesn't work in the userdb queries or templates. ~/ gets expanded +later, so use it instead. + +Note also that since 'location' specified within a <namespace> [Namespaces.txt] +overrides mail_location setting, in case you specified that parameter, you'll +have to override in in the user database, specifying 'namespace/inbox/location' +extra field instead of 'mail'. + +SQL +--- + +---%<------------------------------------------------------------------------- +user_query = SELECT home, uid, gid, mail FROM users WHERE user = '%u' +---%<------------------------------------------------------------------------- + +LDAP +---- + +---%<------------------------------------------------------------------------- +user_attrs = homeDirectory=home, uidNumber=uid, gidNumber=gid, +mailLocation=mail +---%<------------------------------------------------------------------------- + +Passwd-file +----------- + +---%<------------------------------------------------------------------------- +user:{PLAIN}password:1000:1000::/home/user::userdb_mail=mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +Mixing mbox and maildir +----------------------- + +It's possible to use both mboxes and maildirs for the same user by configuring +multiple namespaces. See <Namespaces.txt>. + +Having both mboxes and maildirs mixed within the same namespace isn't currently +supported. + +Custom mailbox location detection +--------------------------------- + +Dovecot by default detects the mailboxes in this order: + + 1. maildir: ~/Maildir + 2. mbox: ~/mail, and /var/mail/%u if it exists + 3. mbox: ~/Mail, and /var/mail/%u if it exists + +If you need something else, you can override the 'mail_executable' setting to +run a script, which sets the MAIL environment properly. For example: + +---%<------------------------------------------------------------------------- +#!/bin/sh + +if [ -d $HOME/.maildir ]; then + export MAIL=maildir:$HOME/.maildir +else + export MAIL=mbox:$HOME/mail:INBOX=/var/mail/$USER +fi +export USERDB_KEYS="$USERDB_KEYS mail" + +exec "$@" +---%<------------------------------------------------------------------------- + +Custom namespace location +------------------------- + +If you need to override namespace's location, first give it a name ("inbox" +below): + +---%<------------------------------------------------------------------------- +namespace inbox { + .. +} +---%<------------------------------------------------------------------------- + +Then in the script use: + +---%<------------------------------------------------------------------------- +#!/bin/sh + +# do the lookup here +location=mbox:$HOME/mail + +export USERDB_KEYS="$USERDB_KEYS namespace/inbox/location" +exec env "NAMESPACE/INBOX/LOCATION=$location" "$@" +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.Cydir.txt b/doc/wiki/MailboxFormat.Cydir.txt new file mode 100644 index 0000000..9882bbf --- /dev/null +++ b/doc/wiki/MailboxFormat.Cydir.txt @@ -0,0 +1,20 @@ +Cydir +===== + +This mailbox format is very similar to Cyrus IMAP's internal mail store: + + * Messages are stored in "<uid>." named files + * Cyrus's 'cyrus.index' is equivalent to Dovecot's 'dovecot.index'. Dovecot + however also requires 'dovecot.index.log' for its indexing to work. + * Cyrus's 'cyrus.cache' is equivalent to Dovecot's 'dovecot.index.cache'. + +Cydir is a very simple format internally. It relies on Dovecot's <index files> +[IndexFiles.txt] completely for its functionality. If the index files are lost, +all the message flags are lost. Currently the code can't even rebuild index +files if they're lost. + +Cydir is mostly meant to be used for benchmarking and stress testing index +handling code. Its code is small and simple, so it can also act as an example +for writing new mail storage backends. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.MH.txt b/doc/wiki/MailboxFormat.MH.txt new file mode 100644 index 0000000..cbe7190 --- /dev/null +++ b/doc/wiki/MailboxFormat.MH.txt @@ -0,0 +1,52 @@ +MH Mailbox Format +================= + +The MH mailbox format originated with a system developed by the RAND +corporation and the University of California. Each email message is stored in +a single file, with directories indicating folders and subfolders. The index +or order of the messages in the folder determine what each message is named +(which may not correspond to the inode index). The "safe" way to guarantee a +message gets written to a mail folder is to first write the message out to a +randomly chosen temporary file name, then link the file to the number LAST+1, +where LAST is the last sequential message in the folder. If the link fails, +increment the counter and try again. + +MH folders also maintains a meta-file called '~/Mail/.mh_context' that contains +information about the most current folder and message chosen. Each sub-folder +also contains a meta-file called '.mh_sequences' or '.xmhcache', which +maintains keyword association lists for stored queries. New messages are +stored in the "unseen" sequence for a folder. Procmail itself does not bother +making changes to this file, rather simply delivers the message to the folder +and leaves determining new messages as an exercise for the MUA. For example: + +---%<------------------------------------------------------------------------- +unseen: 1-3 8 15 +projectB: 2-8 10 +---%<------------------------------------------------------------------------- + +shows two stored sequences of messages. Command-line utilities can then use +these sequences as shortcuts.'show unseen', for example, is short-hand for +'show 1-3 8 15'. + +Deleted emails are indicated by prepending a "," to the name. One of the +largest problems that IMAP servers have with MH format is the volatility of the +email message name itself. The command-line utility 'sortm' is used to sort +mail folders by date or string matching. To do this, messages are actually +renamed to reflect the new sort order. IMAP servers are required to maintain +an index of the folder contents, so when the names of the file entries cannot +be guaranteed to be stable, IMAP servers have to throw out previous index +caches and re-index. + +When operating with a shell account on a machine that also provides IMAP access +to folders, users are encouraged not to re-sort email locally while accessing +the IMAP server remotely. + +Links + + * NMH [http://www.nongnu.org/nmh/]: New MH Client + * Original RAND MH [http://rand-mh.sourceforge.net] Code + * MH-Book [http://rand-mh.sourceforge.net/book/] + * Mutt Manual [http://www.mutt.org/doc/manual/manual-4.html#ss4.6]: Describing + how it handles MH folders + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.Maildir.txt b/doc/wiki/MailboxFormat.Maildir.txt new file mode 100644 index 0000000..5629b7d --- /dev/null +++ b/doc/wiki/MailboxFormat.Maildir.txt @@ -0,0 +1,400 @@ +Maildir +======= + +This format debuted with the qmail server in the mid-1990s. Each mailbox folder +is a directory and each message a file. This improves efficiency because +individual emails can be modified, deleted and added without affecting the +mailbox or other emails, and makes it safer to use on networked file systems +such as NFS. + +Dovecot extensions +------------------ + +Since the standard maildir specification [https://cr.yp.to/proto/maildir.html] +doesn't provide everything needed to fully support the IMAP protocol, Dovecot +had to create some of its own non-standard extensions. The extensions still +keep the maildir standards compliant, so MUAs not supporting the extensions can +still safely use it as a normal maildir. + +IMAP UID mapping +---------------- + +IMAP requires each message to have a permanent unique ID number. Dovecot uses +'dovecot-uidlist' file to keep UID <-> filename mapping. The file is basically +in the same format as Courier IMAP's courierimapuiddb file, except for one +difference (see below). + +The file begins with a header: + +---%<------------------------------------------------------------------------- +3 V1275660208 N25022 G3085f01b7f11094c501100008c4a11c1 +---%<------------------------------------------------------------------------- + + * 3 is the file format version number used by Dovecot v1.1+ + * 1275660208 is the IMAP UIDVALIDITY + * 25022 is the UID that will be given to the next added message + * 3085f01b7f11094c501100008c4a11c1 is the 128 bit mailbox global UID in hex + * There may be other fields, and the order of these fields isn't important + +Version 1 file format is compatible with Courier. Version 2 was used by a few +non-release versions. + +After the header comes the list of UID <-> filename mappings: + +---%<------------------------------------------------------------------------- +25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2, +25017 W2481 :1276533073.M242911P3632.kurkku:2,F +---%<------------------------------------------------------------------------- + + * 25006, 25017 are message UIDs + * 2481 is the second message's virtual size. First message contains it in the + filename itself, so it's not duplicated. + * There may be more fields before ':' character + * Rest of the line after ':' is the last known filename. This filename doesn't + necessarily exist currently, because filename changes every time message's + flags change. Dovecot doesn't waste disk I/O by rewriting uidlist file every + time flags change, but whenever it is rewritten the latest filenames are + used. This allows Dovecot to try to guess what the message's current + filename is and if successful, avoid having to scan the directory's + contents. + +The dovecot-uidlist file doesn't need to be locked for reading. When writing +dovecot-uidlist.lock file needs to be created. New lines can be appended to the +end of file, but existing data must never be directly modified, it can only be +replaced with rename() call. + +dovecot-uidlist is updated lazily to optimize for disk I/O. If a message is +expunged, it may not be removed from dovecot-uidlist until sometimes later. +This means that if you create a new file using the same file name as what +already exists in dovecot-uidlist, Dovecot thinks you "unexpunged" message by +restoring a message from backup. This causes a warning to be logged and the +file to be renamed. + +Note that messages must not be modified once they've been delivered. IMAP (and +Dovecot) requires that messages are immutable. If you wish to modify them in +any way, create a new message instead and expunge the old one. + +IMAP keywords +------------- + +All the non-standard message flags are called keywords in IMAP. Some clients +use these automatically for marking spam (eg. $Junk, $NonJunk, $Spam, $NonSpam +keywords). Thunderbird uses labels which map to keywords $Label1, $Label2, etc. + +Dovecot stores keywords in the maildir filename's flags field using letters +a..z. This means that only 26 keywords are possible to store in the maildir. If +more are used, they're still stored in Dovecot's index files. The mapping from +single letters to keyword names is stored in dovecot-keywords file. The file is +in format: + +---%<------------------------------------------------------------------------- +0 $Junk +1 $NonJunk +---%<------------------------------------------------------------------------- + +0 means letter 'a' in the maildir filename, 1 means 'b' and so on. The file +doesn't need to be locked for reading, but when writing dovecot-uidlist file +must be locked. The file must not be directly modified, it can only be replaced +with rename() call. + +For example, a file named + +---%<------------------------------------------------------------------------- +1234567890.M20046P2137.mailserver,S=4542,W=4642:2,Sb +---%<------------------------------------------------------------------------- + +would be flagged as '$NonJunk' with the above keywords. + +Maildir filename extensions +--------------------------- + +The standard filename definition is: "<base filename>:2,<flags>". Dovecot has +extended the<flags> field to be "<flags>[,<non-standard fields>]". This means +that if Dovecot sees a comma in the<flags> field while updating flags in the +filename, it doesn't touch anything after the comma. However other maildir MUAs +may mess them up, so it's still not such a good idea to do that. Basic<flags> +are described here [https://cr.yp.to/proto/maildir.html]. The <non-standard +fields> isn't used by Dovecot for anything currently. + +Dovecot supports reading a few fields from the <base filename>: + + * ',S=<size>': <size> contains the file size. Getting the size from the + filename avoids doing a stat(), which may improve the performance. This is + especially useful with <Maildir++ quota> [Quota.Maildir.txt]. + * ',W=<vsize>': <vsize> contains the file's RFC822.SIZE, ie. the file size + with linefeeds being CR+LF characters. If the message was stored with CR+LF + linefeeds,<size> and <vsize> are the same. Setting this may give a small + speedup because now Dovecot doesn't need to calculate the size itself. + +A maildir filename with those fields would look something like: +'1035478339.27041_118.foo.org,S=1000,W=1030:2,S' + +Usage of timestamps +------------------- + +Timestamps of message files: + + * 'mtime' is used as IMAP INTERNALDATE, RFC 3501 sec 2.3.3. + [https://www.faqs.org/rfcs/rfc3501.html], must never change, see sec. + 2.3.1.1. 4) + * 'ctime' used as Dovecot's internal "save/copy date", unless the correct + value is found from 'dovecot.index.cache'. This is used only by external + commands, e.g. "doveadm expunge savedbefore". + * 'atime' not used + +Timestamps of 'cur' and 'new' directories: + + * 'mtime' is used to detect changes of the mailbox and may force regeneration + of <index files> [IndexFiles.txt] + * 'atime' and 'ctime' not used + +Filename examples +----------------- + ++---------------------------------------------------------------------------------------------+----------------------------+ +| Filename | Explanation | ++---------------------------------------------------------------------------------------------+----------------------------+ +| *1491941793*.M41850P8566V0000000000000015I0000000004F3030E_0.mx1.example.com,S=10956:2,STln | UNIX timestamp of arrival | ++---------------------------------------------------------------------------------------------+----------------------------+ +| 1491941793.M41850P8566V0000000000000015I0000000004F3030E_0.mx1.example.com,*S=10956*:2,STln | Size of e-mail | ++---------------------------------------------------------------------------------------------+----------------------------+ +| 1491941793.M41850P8566V0000000000000015I0000000004F3030E_0.mx1.example.com,S=10956:2,*STln* | *S*=seen (marked as read) | +| | *T*=trashed | +| | *l*=IMAP tag #12 (0=a, 1=b,| +| | 2=c, etc) as defined in | +| | that folder's | +| | dovecot-keywords. | +| | *n*=IMAP tag #14 (0=a, 1=b,| +| | 2=c, etc) as defined in | +| | that folder's | +| | dovecot-keywords. | ++---------------------------------------------------------------------------------------------+----------------------------+ + +Maildir and filesystems +----------------------- + +General comparisons of Maildir on different filesystems +------------------------------------------------------- + + * https://www.thesmbexchange.com/eng/qmail_fs_benchmark.html + * https://www.htiweb.inf.br/benchmark/fsbench.htm (including some graphs) + +Linux ext2 / ext3 +----------------- + +The main disadvantage is that searching can be slightly slower, and access to +very large mailboxes (thousands of messages) can get slow with filesystems +which don't have directory indexes. + +Old versions of ext2 and ext3 on Linux don't support directory indexing (to +speed up access), but newer versions of ext3 do, although you may have to +manually enable it. You can check if the indexing is already enabled with +tune2fs: + +---%<------------------------------------------------------------------------- +tune2fs -l /dev/hda3 | grep features +---%<------------------------------------------------------------------------- + +If you see dir_index, you're all set. If dir_index is missing, add it using: + +---%<------------------------------------------------------------------------- +umount /dev/hda3 +tune2fs -O dir_index /dev/hda3 +e2fsck -fD /dev/hda3 +mount /dev/hda3 +---%<------------------------------------------------------------------------- + +ReiserFS +-------- + +ReiserFS was built to be fast with lots of small files, so it works well with +maildir. + +XFS +--- + +XFS performance seems to depend on a lot of factors, also on the system and the +file system parameters. + + * There are early reports on the dovecot mailing list which suggest that XFS + seems quite a lot slower than ext3 or + ReiserFS:https://dovecot.org/list/dovecot/2007-January/018994.html + * But then again others recommend XFS for the use with Maildir and dovecot: + https://dovecot.org/list/dovecot/2006-May/013216.html + * This 2007 Linux.conf.au talk about "Choosing and Tuning Linux File Systems" + (Slides as PDF) + [https://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks/348.pdf] + also recommends XFS for Maildir (alternatively ext3 with small blocks and + high inodetofile ratio) + * Someone else wrote here in the wiki: XFS on TSL 3.0.5 works almost twice as + fast as our prior EXT3 installation of which is significant in size. + ReiserFS is also a good option. + * Comparisons which suggest XFS as being best choice: + * https://www.thesmbexchange.com/eng/qmail_fs_benchmark.html + * https://www.htiweb.inf.br/benchmark/fsbench.htm + +Various tips +------------ + + * Mounting XFS with 'logbufs=8' option might increase the speed. + * Create the XFS with options '-b size=1024 -d su=16k,sw=3 -l + logdev=<some_other_device>' (Source: + https://www.thesmbexchange.com/eng/qmail_fs_benchmark.html) + * Use 'mkfs.xfs -f -l size=32768b,version=2' and 'mount.xfs -o + noatime,logbufs=8,logbsize=131072' (Source: + https://www.htiweb.inf.br/benchmark/fsbench.htm) + +NFS +--- + +NFS v3 performance can be adversely affected by readdirplus, which causes the +NFS server to stat() every file in a directory. The solution under Linux is to +make sure the NFS filesystem is mounted with the "nordirplus" option. + + * https://dovecot.org/list/dovecot/2012-July/066939.html + +Directory Structure +------------------- + +By default Dovecot uses Maildir++ +[https://www.courier-mta.org/imap/README.maildirquota.html] directory layout +for organizing mailbox directories. This means that all the folders are +directly inside '~/Maildir' directory: + + * '~/Maildir/new', '~/Maildir/cur' and '~/Maildir/tmp' directories contain the + messages for INBOX. The 'tmp' directory is used during delivery, new + messages arrive in 'new' and read shall be moved to 'cur' by the clients. + * '~/Maildir/.folder/' is a mailbox folder + * '~/Maildir/.folder.subfolder/' is a subfolder of a folder (ie. + "folder/subfolder") + +You can also optionally use the "fs" layout by appending ':LAYOUT=fs' to +<mail_location> [MailLocation.txt]. This makes the folder structure look like: + + * '~/Maildir/new', '~/Maildir/cur' and '~/Maildir/tmp' directories contain the + messages for INBOX, just like with Maildir++. + * '~/Maildir/folder/' is a mailbox folder + * '~/Maildir/folder/subfolder/' is a subfolder of a folder + +Filesystem permissions +---------------------- + +See <SharedMailboxes.Permissions.txt> for how permissions are set for newly +created files and directories. + +Since Dovecot v2.0 "Permissions for newly created mail files are no longer +copied from dovecot-shared file", see <Upgrading.2.0.txt>. + +Issues with the specification +----------------------------- + +Locking +------- + +Although maildir was designed to be lockless, Dovecot locks the maildir while +doing modifications to it or while looking for new messages in it. This is +required because otherwise Dovecot might temporarily see mails incorrectly +deleted, which would cause trouble. Basically the problem is that if one +process modifies the maildir (eg. a rename() to change a message's flag), +another process in the middle of listing files at the same time could skip a +file. The skipping happens because readdir() system call doesn't guarantee that +all the files are returned if the directory is modified between the calls to +it. This problem exists with all the commonly used filesystems. + +Because Dovecot uses its own non-standard locking ('dovecot-uidlist.lock' +dotlock file), other MUAs accessing the maildir don't support it. This means +that if another MUA is updating messages' flags or expunging messages, Dovecot +might temporarily lose some message. After the next sync when it finds it +again, an error message may be written to log and the message will receive a +new UID. + +Delivering mails to new/ directory doesn't have any problems, so there's no +need for LDAs to support any type of locking. + +Mail delivery +------------- + +Qmail's how a message is delivered page +[https://www.qmail.org/man/man5/maildir.html] suggests to deliver the mail like +this: + + 1. Create a unique filename (only "time.pid.host" here, later Maildir spec has + been updated to allow more uniqueness identifiers) + 2. Do 'stat(tmp/<filename>)'. If the 'stat()' found a file, wait 2 seconds and + go back to step 1. + 3. Create and write the message to the 'tmp/<filename>'. + 4. link() it into new/ directory. Although not mentioned here, the link() + could again fail if the mail existed in new/ dir. In that case you should + probably go back to step 1. + +All this trouble is rather pointless. Only the first step is what really +guarantees that the mails won't get overwritten, the rest just sounds nice. +Even though they might catch a problem once in a while, they give no guaranteed +protection and will just as easily pass duplicate filenames through and +overwrite existing mails. + +Step 2 is pointless because there's a race condition between steps 2 and 3. +PID/host combination by itself should already guarantee that it never finds +such a file. If it does, something's broken and the stat() check won't help +since another process might be doing the same thing at the same time, and you +end up writing to the same file in tmp/, causing the mail to get corrupted. + +In step 4 the link() would fail if an identical file already existed in the +maildir, right? Wrong. The file may already have been moved to cur/ directory, +and since it may contain any number of flags by then you can't check with a +simple stat() anymore if it exists or not. + +Step 2 was pointed out to be useful if clock had moved backwards. However again +this doesn't give any actual safety guarantees, because an identical base +filename could already exist in cur/. Besides if the system was just rebooted, +the file in tmp/ could probably be even overwritten safely (assuming it wasn't +already link()ed to new/). + +So really, all that's important in not getting mails overwritten in your +maildir is the step 1: Always create filenames that are guaranteed to be +unique. Forget about the 2 second waits and such that the Qmail's man page +talks about. + +Maildir and mail header metadata +-------------------------------- + +Unlike when using <mbox> [MailboxFormat.mbox.txt] as <mailbox format> +[MailboxFormat.txt], where mail headers (for example 'Status', 'X-UID', etc.) +are <used to determine and store meta-data> [MailboxFormat.mbox.txt], the mail +headers within maildir files are (usually)*not* used for this purpose by +dovecot; neither when mails are created/moved/etc. via IMAP nor when maildirs +are placed (e.g. copied or moved in the filesystem) in a mail location (and +then "imported" by dovecot).Therefore, it is (usually) *not* necessary, to +strip any such mail headers at the MTA, MDA or LDA (as it is recommended with +<mbox> [MailboxFormat.mbox.txt]). + +There is one exception, though, namely when 'pop3_reuse_xuidl=yes' (which is +however rather deprecated):In this case 'X-UIDL' is used for the POP3 UIDLs. +Therefore,*in this case, is recommended to strip the 'X-UIDL' mail headers +_case-insensitively_ at the MTA, MDA or LDA*. + +Procmail Problems +----------------- + +Maildir format is somewhat compatible with MH format. This is sometimes a +problem when people configure their procmail to deliver mails to 'Maildir/new'. +This makes procmail create the messages in MH format, which basically means +that the file is called 'msg.inode_number'. While this appears to work first, +after expunging messages from the maildir the inodes are freed and will be +reused later. This means that another file with the same name may come to the +maildir, which makes Dovecot think that an expunged file reappeared into the +mailbox and an error is logged. + +The proper way to configure procmail to deliver to a Maildir is to use +'Maildir/' as the destination. + +References +---------- + + * Official Maildir format page [https://cr.yp.to/proto/maildir.html] + * Qmail's how to deliver to Maildir man page + [https://www.qmail.org/man/man5/maildir.html] + * Maildir++ [https://www.courier-mta.org/imap/README.maildirquota.html] + * Wikipedia [https://en.wikipedia.org/wiki/Maildir] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.dbox.txt b/doc/wiki/MailboxFormat.dbox.txt new file mode 100644 index 0000000..8df69bf --- /dev/null +++ b/doc/wiki/MailboxFormat.dbox.txt @@ -0,0 +1,251 @@ +dbox +==== + +dbox is Dovecot's own high-performance mailbox format. The original version was +introduced in v1.0 alpha4, but since then it has been completely redesigned in +v1.1 series and improved even further in v2.0. + +dbox can be used in two ways: + + 1. *single-dbox* ('sdbox' in <mail location> [MailLocation.txt]): One message + per file, similar to <Maildir> [MailboxFormat.Maildir.txt]. For backwards + compatibility,'dbox' is an alias to 'sdbox' in <mail_location> + [MailLocation.txt]. + 2. *multi-dbox* ('mdbox' in <mail location> [MailLocation.txt]): Multiple + messages per file, but unlike <mbox> [MailboxFormat.mbox.txt] multiple + files per mailbox. + +One of the main reasons for dbox's high performance is that it uses Dovecot's +index files as the only storage for message flags and keywords, so the indexes +don't have to be "synchronized". Dovecot trusts that they're always up-to-date +(unless it sees that something is clearly broken). This also means that *you +must not lose the dbox index files, they can't be regenerated without data +loss*. + +dbox has a feature for transparently moving message data to an alternate +storage area. See <Alternate storage> [MailboxFormat.dbox.txt] below. + +dbox storage is extensible. Single instance attachment storage was already +implemented as such extension. + +Layout +------ + +By default, the dbox filesystem layout will be as follows. Data which isn't the +actual message content is stored in a layout common to both *single-dbox* and +*multi-dbox*: + + * <mail location root>'/mailboxes/INBOX/dbox-Mails/dovecot.index*' - Index + files for INBOX + * <mail location root>'/mailboxes/foo/dbox-Mails/dovecot.index*' - Index files + for mailbox "foo" + * <mail location root>'/mailboxes/foo/bar/dbox-Mails/dovecot.index*' - Index + files for mailbox "foo/bar" + * <mail location root>'/dovecot.mailbox.log*' - Mailbox changelog + * <mail location root>'/subscriptions' - subscribed mailboxes list + * <mail location root>'/dovecot-uidvalidity*' - IMAP UID validity + +Note that with dbox the Index files actually contain significant data which is +held nowhere else. Index files for both *single-dbox* and *multi-dbox* contain +message flags and keywords. For *multi-dbox*, the index file also contains the +map_uids which link (via the "map index") to the actual message data. This data +cannot be automatically recreated, so it is important that Index files are +treated with the same care as message data files. + +Index files can be stored in a different location by using the INDEX parameter +in the mail location specification. If the INDEX parameter is specified, it +will make Dovecot look for the Index files as follows: + + * <INDEX location>'/mailboxes/INBOX/dbox-Mails/dovecot.index*' - Index files + for INBOX + * <INDEX location>'/mailboxes/foo/dbox-Mails/dovecot.index*' - Index files for + mailbox "foo" + * <INDEX location>'/mailboxes/foo/bar/dbox-Mails/dovecot.index*' - Index files + for mailbox "foo/bar" + +Actual message content is stored differently depending on whether it is +*single-dbox* or *multi-dbox*. + +Under *single-dbox* we have: + + * <mail location root>'/mailboxes/INBOX/dbox-Mails/u.*' - Numbered files + ('u.1','u.2', ...) each containing one message of INBOX + * <mail location root>'/mailboxes/foo/dbox-Mails/u.*' - Files each containing + one message for mailbox "foo" + * <mail location root>'/mailboxes/foo/bar/dbox-Mails/u.*' - Files each + containing one message for for mailbox "foo/bar" + +Under *multi-dbox* we have: + + * <mail location root>'/storage/dovecot.map.index*' - "Map index" containing a + record for each message stored + * <mail location root>'/storage/m.*' - Numbered files ('m.1', 'm.2', ...) each + containing one or multiple messages + +With Dovecot versions 2.0.4 and later, setting the INDEX parameter sets the +location of the "map index" as well as the location of the mailbox indexes. So +this would make the "map index" be stored as follows: + + * <INDEX location>'/storage/dovecot.map.index*' - "Map index" containing a + record for each message stored + +Multi-dbox +---------- + +You can enable multi-dbox with: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:~/mdbox +---%<------------------------------------------------------------------------- + +The directory layout (under '~/mdbox/') is: + + * '~/mdbox/storage/' contains the actual mail data for all mailboxes + * '~/mdbox/mailboxes/' contains directories for mailboxes and their index + files + +The storage directory has files: + + * 'dovecot.map.index*' files contain the "map index" + * 'm.*' files contain the mail data + +Each m.* file contains one or more messages. 'mdbox_rotate_size' setting can be +used to configure how large the files can grow. + +The map index contains a record for each message: + + * map_uid: Unique growing 32 bit number for the message. + * refcount: 16 bit reference counter for this message. Each time the message + is copied the refcount is increased. + * file_id: File number containing the message. For example if file_id=5, the + message is in file 'm.5'. + * offset: Offset to message within the file. + * size: Space used by the message in the file, including all metadata. + +Mailbox indexes refer to messages only using map_uids. This allows messages to +be moved to different files by updating only the map index. Copying is done +simply by appending a new record to mailbox index containing the existing +map_uid and increasing its refcount. If refcount grows over 32768, currently +Dovecot gives an error message. It's unlikely anyone really wants to copy the +same message that many times. + +Expunging a message only decreases the message's refcount. The space is later +freed in "purge" step. This is typically done in a nightly cronjob when there's +less disk I/O activity. The purging first finds all files that have refcount=0 +mails. Then it goes through each file and copies the refcount>0 mails to other +mdbox files (to the same files as where newly saved messages would also go), +updates the map index and finally deletes the original file. So there is never +any overwriting or file truncation. + +The purging can be invoked explicitly running <doveadm purge> +[Tools.Doveadm.Purge.txt]. + +There are several safety features built into dbox to avoid losing messages or +their state if map index or mailbox index gets corrupted: + + * Each message has a 128 bit globally unique identifier (GUID). The GUID is + saved to message metadata in m.* files and also to mailbox indexes. This + allows Dovecot to find messages even if map index gets corrupted. + * Whenever index file is rewritten, the old index is renamed to + 'dovecot.index.backup'. If the main index becomes corrupted, this backup + index is used to restore flags and figure out what messages belong to the + mailbox. + * Initial mailbox where message was saved to is stored in the message metadata + in m.* files. So if all indexes get lost, the messages are put to their + initial mailboxes. This is better than placing everything into a single + mailbox. + +Alternate storage +----------------- + +Unlike Maildir, with dbox the message file names don't change. This makes it +possible to support storing files in multiple directories or mount points. dbox +supports looking up files from "altpath" if they're not found from the primary +path. This means that it's possible to move older mails that are rarely +accessed to cheaper (slower) storage. + +To enable this functionality, use the 'ALT' parameter in the mail location. For +example, specifying the mail location as: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:/var/vmail/%d/%n:ALT=/altstorage/vmail/%d/%n +---%<------------------------------------------------------------------------- + +will make Dovecot look for message data first under '/var/vmail/%d/%n' +("primary storage"), and if it is not found there it will look under +'/altstorage/vmail/%d/%n' ("alternate storage") instead. There's no problem +having the same (identical) file in both storages. + +Keep the unmounted '/altstorage' directory permissions such that Dovecot mail +processes can't create directories under it (e.g. root:root 0755). This way if +the alt storage isn't mounted for some reason, Dovecot won't think that all the +messages in alt storage were deleted and lose their flags. + +When messages are moved from primary storage to alternate storage, only the +actual message data (stored in files 'u.*' under *single-dbox* and 'm.*' under +*multi-dbox*) is moved to alternate storage; everything else remains in the +primary storage. + +Message data can be moved from primary storage to alternate storage using +<doveadm altmove> [Tools.Doveadm.Altmove.txt]. (In theory you could also do +this with some combination of cp/mv, but better not to go there unless you +really need to. The updates must be atomic in any case, so direct cp won't +work.) + +The granularity at which data is moved to alternate storage is individual +messages. This is true even for *multi-dbox* when multiple messages are stored +in a single 'm.*' storage file. If individual messages from an 'm.*' storage +file need to be moved to alternate storage, the message data is written out to +a different 'm.*' storage file (either new or existing) in the alternate +storage area and the "map index" updated accordingly. + +Alternate storage is completely transparent at the IMAP/POP level. Users +accessing mail through IMAP or POP cannot normally tell if any given message is +stored in primary storage or alternate storage. Conceivably users might be able +to measure a performance difference; the point is that there is no IMAP/POP +command which could be used to expose this information. It is entirely possible +to have a mail folder which contains a mix of messages stored in primary +storage and alternate storage. + +dbox and mail header metadata +----------------------------- + +Unlike when using <mbox> [MailboxFormat.mbox.txt] as <mailbox format> +[MailboxFormat.txt], where mail headers (for example 'Status', 'X-UID', etc.) +are <used to determine and store meta-data> [MailboxFormat.mbox.txt], the mail +headers within dbox files are (usually)*not* used for this purpose by dovecot; +neither when mails are created/moved/etc. via IMAP nor when dboxes are placed +(e.g. copied or moved in the filesystem) in a mail location (and then +"imported" by dovecot).Therefore, it is (usually) *not* necessary, to strip any +such mail headers at the MTA, MDA or LDA (as it is recommended with <mbox> +[MailboxFormat.mbox.txt]). + +There is one exception, though, namely when 'pop3_reuse_xuidl=yes' (which is +however rather deprecated):In this case 'X-UIDL' is used for the POP3 UIDLs. +Therefore,*in this case, is recommended to strip the 'X-UIDL' mail headers +_case-insensitively_ at the MTA, MDA or LDA*. + +Accessing expunged mails with mdbox +----------------------------------- + +"mdbox_deleted" storage can be used to access mdbox's all mails that are +completely deleted (reference count=0). The mdbox_deleted parameters should +otherwise be exactly the same as mdbox's. Then you can use e.g. doveadm fetch +or doveadm import commands to access the mails. For example: + +---%<------------------------------------------------------------------------- +# If you have mail_location=mdbox:~/mdbox:INDEX=/var/index/%u +doveadm import mdbox_deleted:~/mdbox:INDEX=/var/index/%u "" subject oops +---%<------------------------------------------------------------------------- + +This finds a deleted mail with subject "oops" and imports it into INBOX. + +Mail delivery +============= + +Some MTA configurations have the MTA directly dropping mail into Maildirs or +mboxes. Since most MTAs don't understand the dbox format, this option is not +available. Instead, the MTA could be set up to use <LMTP#mta> [LMTP.txt] or +<the Dovecot Local Delivery Agent> [LDA.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.imapc.txt b/doc/wiki/MailboxFormat.imapc.txt new file mode 100644 index 0000000..0fee740 --- /dev/null +++ b/doc/wiki/MailboxFormat.imapc.txt @@ -0,0 +1,178 @@ +imapc storage +============= + +The imapc storage accesses a remote IMAP server as if it were a regular Dovecot +mailbox format. Dovecot can treat it as a very dummy storage or optionally a +more capable storage. + +---%<------------------------------------------------------------------------- +# In-memory index files: +mail_location = imapc: + +# Store index files locally: +mail_location = imapc:~/imapc +---%<------------------------------------------------------------------------- + +Connection settings +------------------- + +Do a regular IMAP LOGIN to imap.example.com using 'a LOGIN user@example.com +secret': + +---%<------------------------------------------------------------------------- +imapc_host = imap.example.com +imapc_password = secret +imapc_port = 143 +imapc_user = user@example.com +---%<------------------------------------------------------------------------- + +If you want to use a master user login, set: + +---%<------------------------------------------------------------------------- +imapc_master_user = masteruser +---%<------------------------------------------------------------------------- + +For SSL, use either: + + * ---%<---------------------------------------------------------------------- + imapc_ssl = imaps + imapc_port = 993 + ---%<---------------------------------------------------------------------- + + * ---%<---------------------------------------------------------------------- + imapc_ssl = starttls + imapc_port = 143 + ---%<---------------------------------------------------------------------- + +For testing you may want to disable all SSL certificate checks: + +---%<------------------------------------------------------------------------- +imapc_ssl_verify = no +---%<------------------------------------------------------------------------- + +You can use other SASL mechanisms besides PLAIN by specifying (the first one +advertised by IMAP server is used): + +---%<------------------------------------------------------------------------- +imapc_sasl_mechanisms = external plain login +---%<------------------------------------------------------------------------- + +The SASL client mechanisms are implemented in Dovecot's lib-sasl/ code. It's +possible to add more with plugins. + +Other settings +-------------- + +imapc_cmd_timeout: + How long to wait for an IMAP command to reply before disconnecting and + retrying (default: 5 mins). + +imapc_list_prefix: + Access only mailboxes under this prefix. For example + 'imapc_list_prefix=INBOX/' + +imapc_max_idle_time: + Send something (NOOP/DONE) to server after not sending anything for this + amount of time (default: 29 mins). + +imapc_rawlog_dir: + Log all IMAP traffic input/output to this directory. + +Features and workarounds +------------------------ + +'imapc_features' setting is a space-separated list of features and workarounds +that can be enabled. + +Optimizations +------------- + +rfc822.size: + Allow passing through message sizes using 'FETCH RFC822.SIZE' + +fetch-headers: + Allow fetching specific message headers using 'FETCH BODY.PEEK[HEADER.FIELDS + (..)]' + +fetch-bodystructure: + Allow fetching BODY and BODYSTRUCTURE from remote server. (v2.2.30+) + +search: + Allow using 'SEARCH' command. + +Features +-------- + +gmail-migration: + Enable GMail-specific migration: Use IMAP X-GM-MSGID as POP3 UIDL. Add + $GMailHaveLabels keyword to mails that have X-GM-LABELS except for \Muted (to + be used for migrating only archived emails in "All Mails"). Add + pop3_deleted_flag to mails that don't exist in POP3 server. + +proxyauth: + Use Sun/Oracle IMAP-server specific PROXYAUTH command to do master user + authentication. Normally this would be done using the SASL PLAIN + authentication. + +throttle:INIT:MAX:SHRINK: + When receiving [THROTTLED] response (from GMail), throttling is applied. INIT + = initial throttling msecs (default: 50 ms), afterwards each subsequent + [THROTTLED] doubles the throttling until MAX is reached (default: 16000 ms). + When [THROTTLED] is not received for a while, it's shrank again. The initial + shrinking is done after SHRINK (default: 500 ms). If [THROTTLED] is received + again within this timeout, it's doubled, otherwise both throttling and the + next shrinking timeout is shrank to 3/4. + +modseq: + Access MODSEQ and HIGHESTMODSEQ (v2.2.24+) + +delay-login: + Don't connect to the remote server until some command requires it. By default + the server is connected to immediately on login. (v2.2.29+) + +Quota +----- + +Using the "imapc" quota backend allows asking for the quota from remote IMAP +server (v2.2.30+). By default it uses GETQUOTAROOT INBOX for getting the quota. +There are also two parameters that can be used to control how the quota is +looked up: + + * box=<mailbox>: Use GETQUOTAROOT <mailbox> + * root=<name>: Use GETQUOTA <name> + +For example: + +---%<------------------------------------------------------------------------- +plugin { + quota = imapc:root=User Quota +} +---%<------------------------------------------------------------------------- + +Workarounds +----------- + +zimbra-workarounds: + Fetch full message using 'BODY.PEEK[HEADER] BODY.PEEK[TEXT]' instead of just + 'BODY.PEEK[]' because the header differs between these two when there are + illegal control chars or 8bit chars. This mainly caused problems with dsync, + but as of v2.2.22+ this should no longer be a problem and there's probably no + need to enable this workaround. + +no-examine: + Use SELECT instead of EXAMINE even when we don't want to modify anything in + the mailbox. This is a Courier-workaround where it didn't permanently assign + UIDVALIDITY to an EXAMINEd mailbox, but assigned it for SELECTed mailbox. + +fetch-msn-workarounds: + Try to ignore wrong message sequence numbers in FETCH replies whenever + possible, preferring to use the returned UID number instead. + +fetch-fix-broken-mails: + If a FETCH returns 'NO' (but not 'NO [LIMIT]' or 'NO [SERVERBUG]'), assume + the mail is broken in server and just treat it as if it were an empty email. + NOTE: This is often a dangerous option! It's not safe to assume that NO means + a permanent error rather than a temporary error. This feature should be + enabled only for specific users who have been determined to be broken. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.mailstore.txt b/doc/wiki/MailboxFormat.mailstore.txt new file mode 100644 index 0000000..3993602 --- /dev/null +++ b/doc/wiki/MailboxFormat.mailstore.txt @@ -0,0 +1,17 @@ +Mailstore +========= + +This format originated with exim. Each message is written as two unique +filenames ending in .env and .msg. The .env contains the message envelope, and +the .msg file contains the actual message. When an email is delivered, the +email header is created with the suffix .tmp, and the message is created with +.msg. When the .msg file is completed, the .tmp file is renamed to end in .env. +Programs wishing to access an email must wait for both files to be present, or +for the absence of the .tmp file. + +References +---------- + + * http://www.exim.org/exim-html-4.60/doc/html/spec.html/ch26.html#id2639666 + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.mbox.txt b/doc/wiki/MailboxFormat.mbox.txt new file mode 100644 index 0000000..438f9d8 --- /dev/null +++ b/doc/wiki/MailboxFormat.mbox.txt @@ -0,0 +1,290 @@ +Mbox Mailbox Format +=================== + +Contents + + + 1. Mbox Mailbox Format + + 1. Locking + + 1. Dotlock + + 2. Deadlocks + + 2. Directory Structure + + 3. Dovecot's Metadata + + 4. Dovecot's Speed Optimizations + + 5. From Escaping + + 6. Mbox Variants + + 7. References + +Usually UNIX systems are configured by default to deliver mails to +'/var/mail/username' or '/var/spool/mail/username' mboxes. In IMAP world these +files are called INBOX mailboxes. IMAP protocol supports multiple mailboxes +however, so there needs to be a place for them as well. Typically they're +stored in '~/mail/' or '~/Mail/' directories. + +The mbox file contains all the messages of a single mailbox. Because of this, +the mbox format is typically thought of as a slow format. However with +Dovecot's indexing this isn't true. Only expunging messages from the beginning +of a large mbox file is slow with Dovecot, most other operations should be +fast. Also because all the mails are in a single file, searching is much faster +than with maildir. + +Modifications to mbox may require moving data around within the file, so +interruptions (eg. power failures) can cause the mbox to break more or less +badly. Although Dovecot tries to minimize the damage by moving the data in a +way that data should never get lost (only duplicated), mboxes still aren't +recommended to be used for important data. + +Locking +------- + +Locking is a mess with mboxes. There are multiple different ways to lock a +mbox, and software often uses incompatible locking. See <MboxLocking.txt> for +how to check what locking methods some commonly used programs use. + +There are at least four different ways to lock a mbox: + + * *dotlock*: 'mailboxname.lock' file created by almost all software when + writing to mboxes. This grants the writer an exclusive lock over the mbox, + so it's usually not used while reading the mbox so that other processes can + also read it at the same time. So while using a dotlock typically prevents + actual mailbox corruption, it doesn't protect against read errors if mailbox + is modified while a process is reading. + * *flock*: 'flock()' system call is quite commonly used for both read and + write locking. The read lock allows multiple processes to obtain a read lock + for the mbox, so it works well for reading as well. The one downside to it + is that it doesn't work if mailboxes are stored in NFS. + * *fcntl*: Very similar to *flock*, also commonly used by software. In some + systems this 'fcntl()' system call is compatible with 'flock()', but in + other systems it's not, so you shouldn't rely on it.*fcntl* works with NFS + if you're using lockd daemon in both NFS server and client. + * *lockf*: POSIX 'lockf()' locking. Because it allows creating only exclusive + locks, it's somewhat useless so Dovecot doesn't support it. With Linux + 'lockf()' is internally compatible with 'fcntl()' locks, but again you + shouldn't rely on this. + +Dotlock +------- + +Another problem with dotlocks is that if the mailboxes exist in '/var/mail/', +the user may not have write access to the directory, so the dotlock file can't +be created. There are a couple of ways to work around this: + + * Give a mail group write access to the directory and then make sure that all + software requiring access to the directory runs with the group's privileges. + This may mean making the binary itself setgid-mail, or using a separate + dotlock helper program which is setgid-mail. With Dovecot this can be done + by setting 'mail_privileged_group = mail'. + * Set sticky bit to the directory ('chmod +t /var/mail'). This makes it + somewhat safe to use, because users can't delete each others mailboxes, but + they can still create new files (the dotlock files). The downside to this is + that users can create whatever files they wish in there, such as a mbox for + newly created user who hadn't yet received mail. + +Deadlocks +--------- + +If multiple lock methods are used, which is usually the case since dotlocks +aren't typically used for read locking, the order in which the locking is done +is important. Consider if two programs were running at the same time, both use +dotlock and fcntl locking but in different order: + + * Program A: fcntl locks the mbox + * Program B at the same time: dotlocks the mbox + * Program A continues: tries to dotlock the mbox, but since it's already + dotlocked by B, it starts waiting + * Program B continues: tries to fcntl lock the mbox, but since it's already + fcntl locked by A, it starts waiting + +Now both of them are waiting for each others locks. Finally after a couple of +minutes they time out and fail the operation. + +Directory Structure +------------------- + +By default, when listing mailboxes, Dovecot simply assumes that all files it +sees are mboxes and all directories mean that they contain sub-mailboxes. There +are two special cases however which aren't listed: + + * '.subscriptions' file contains IMAP's mailbox subscriptions. + * '.imap/' directory contains Dovecot's index files. + +Because it's not possible to have a file which is also a directory, it's not +normally possible to create a mailbox and child mailboxes under it. + +However if you really want to be able to have mailboxes containing both +messages and child mailboxes under mbox, then Dovecot can be configured to do +this, subject to certain provisos; see <MboxChildFolders.txt>. + +Dovecot's Metadata +------------------ + +Dovecot uses C-Client (ie. UW-IMAP, Pine) compatible headers in mbox messages +to store metadata. These headers are: + + * X-IMAPbase: Contains UIDVALIDITY, last used UID and list of used keywords + * X-IMAP: Same as X-IMAPbase but also specifies that the message is a "pseudo + message" + * X-UID: Message's allocated UID + * Status: R (\Seen) and O (non-\Recent) flags + * X-Status: A (\Answered), F (\Flagged), T (\Draft) and D (\Deleted) flags + * X-Keywords: Message's keywords + * Content-Length: Length of the message body in bytes + +Whenever any of these headers exist, Dovecot treats them as its own private +metadata. It does sanity checks for them, so the headers may also be modified +or removed completely. None of these headers are sent to IMAP/POP3 clients when +they read the mail. + +*The MTA, MDA or LDA should strip all these headers _case-insensitively_ before +writing the mail to the mbox.* + +Only the first message contains the X-IMAP or X-IMAPbase header. The difference +is that when all the messages are deleted from mbox file, a "pseudo message" is +written to the mbox which contains X-IMAP header. This is the "DON'T DELETE +THIS MESSAGE -- FOLDER INTERNAL DATA" message which you hate seeing when using +non-C-client and non-Dovecot software. This is however important to prevent +abuse, otherwise the first mail which is received could contain faked +X-IMAPbase header which could cause trouble. + +If message contains X-Keywords header, it contains a space-separated list of +keywords for the mail. Since the same header can come from the mail's sender, +only the keywords are listed in X-IMAP header are used. + +The UID for a new message is calculated from "last used UID" in X-IMAP header + +1. This is done always, so fake X-UID headers don't really matter. This is also +why the pseudo message is important. Otherwise the UIDs could easily grow over +2^31 which some clients start treating as negative numbers, which then cause +all kinds of problems. Also when 2^32 is exceeded, Dovecot will also start +having some problems. + +Content-Length is used as long as another valid mail starts after that many +bytes. Because the byte count must be exact, it's quite unlikely that abusing +it can cause messages to be skipped (or rather appended to the previous +message's body). + +Status and X-Status headers are trusted completely, so it's pretty good idea to +filter them in LDA if possible. + +Dovecot's Speed Optimizations +----------------------------- + +Updating messages' flags and keywords can be a slow operation since you may +have to insert a new header (Status, X-Status, X-Keywords) or at least insert +data in the header's value. Some mbox MUAs do this simply by rewriting all of +the mbox after the inserted data. If the mbox is large, this can be very slow. +Dovecot optimizes this by always leaving some space characters after some of +its internal headers. It can use this space to move only minimal amount of data +necessary to get the necessary data inserted. Also if data is removed, it just +grows these spaces areas. + +'mbox_lazy_writes' setting works by adding and/or updating Dovecot's metadata +headers only after closing the mailbox or when messages are expunged from the +mailbox. C-Client works the same way. The upside of this is that it reduces +writes because multiple flag updates to same message can be grouped, and +sometimes the writes don't have to be done at all if the whole message is +expunged. The downside is that other processes don't notice the changes +immediately (but other Dovecot processes do notice because the changes are in +index files). + +'mbox_dirty_syncs' setting tries to avoid re-reading the mbox every time +something changes. Whenever the mbox changes (ie. timestamp or size), it first +checks if the mailbox's size changed. If it didn't, it most likely meant that +only message flags were changed so it does a full mbox read to find it. If the +mailbox shrunk, it means that mails were expunged and again Dovecot does a full +sync. Usually however the only thing besides Dovecot that modifies the mbox is +the LDA which appends new mails to the mbox. So if the mbox size was grown, +Dovecot first checks if the last known message is still where it was last time. +If it is, Dovecot reads only the newly added messages and goes into a "dirty +mode". As long as Dovecot is in dirty mode, it can't be certain that mails are +where it expects them to be, so whenever accessing some mail, it first verifies +that it really is the correct mail by finding its X-UID header. If the X-UID +header is different, it fallbacks to a full sync to find the mail's correct +position. The dirty mode goes away after a full sync. If 'mbox_lazy_writes' was +enabled and the mail didn't yet have X-UID header, Dovecot uses MD5 sum of a +couple of headers to compare the mails. + +'mbox_very_dirty_syncs' does the same as 'mbox_dirty_syncs', but the dirty +state is kept also when opening the mailbox. Normally opening the mailbox does +a full sync if it had been changed outside Dovecot. + +From Escaping +------------- + +In mboxes a new mail always begins with a "From " line, commonly referred to as +From_-line. To avoid confusion, lines beginning with "From " in message bodies +are usually prefixed with '>' character while the message is being written to +in mbox. + +Dovecot doesn't currently do this escaping however. Instead it prevents this +confusion by adding Content-Length headers so it knows later where the next +message begins. Dovecot doesn't either remove the '>' characters before sending +the data to clients. Both of these will probably be implemented later. + +Mbox Variants +------------- + +There are a few minor variants of this format: + +*mboxo* is the name of original mbox format originated with Unix System V. +Messages are stored in a single file, with each message beginning with a line +containing "From SENDER DATE". If "From " (case-sensitive, with the space) +occurs at the beginning of a line anywhere in the email, it is escaped with a +greater-than sign (to ">From "). Lines already quoted as such, for example +">From " or ">>>From " are *not* quoted again, which leads to irrecoverable +corruption of the message content. + +*mboxrd* was named for Raul Dhesi in June 1995, though several people came up +with the same idea around the same time. An issue with the mboxo format was +that if the text ">From " appeared in the body of an email (such as from a +reply quote), it was not possible to distinguish this from the mailbox format's +quoted ">From ". mboxrd fixes this by always quoting already quoted "From " +lines (e.g. ">From ", ">>From ", ">>>From ", etc.) as well, so readers can just +remove the first ">" character. This format is used by qmail and getmail +(>=4.35.0). + +*mboxcl* format was originated with Unix System V Release 4 mail tools. It adds +a Content-Length field which indicates the number of bytes in the message. This +is used to determine message boundaries. It still quotes "From " as the +original mboxo format does (and *not* as mboxrd does it). + +*mboxcl2* is like mboxcl but does away with the "From " quoting. + +*MMDF* (Multi-channel Memorandum Distribution Facility mailbox format) was +originated with the MMDF daemon. The format surrounds each message with lines +containing four control-A's. This eliminates the need to escape From: lines. + +Dovecot currently uses mboxcl2 format internally, but it's planned to move to +combination of mboxrd and mboxcl. + +*How a message is read stored in mbox extension ?* + + * An email client reader scans throughout mbox file looking for From_ lines. + * Any From_ line marks the beginning of a message. + * Once the reader finds a message, it extracts a (possibly corrupted) envelope + sender and delivery date out of the From_ line. + * It then reads until the next From_ line or scans till the end of file, + whenever From_ comes first. + * It removes the last blank line and deletes the quoting of >From_ lines and + >>From_ lines and so on. + +References +---------- + + * Wikipedia [http://en.wikipedia.org/wiki/Mbox] + * Qmail mbox [http://www.qmail.org/man/man5/mbox.html] + * Mbox family + [http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html] + * CommuniGatePro mbox + [http://www.communigate.com/CommuniGatePro/Mailboxes.html#mbox] + * MBOX File Viewer [http://www.freeviewer.org/mbox/] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.mbx.txt b/doc/wiki/MailboxFormat.mbx.txt new file mode 100644 index 0000000..c5d76a2 --- /dev/null +++ b/doc/wiki/MailboxFormat.mbx.txt @@ -0,0 +1,24 @@ +mbx +=== + +A high performance mbox-replacement created by Mark Crispin while at University +of Washington for UW-IMAP. Each message in the file is preceded by a record +carrying all the metadata that IMAP-protocol needs. This allows changing the +metadata easily by modifying the fixed-size header, rather than moving data +around in a file like with mbox. + +File locking is handled more intelligently (only appends may need to wait for +locks), making this format a good choice for shared mailboxes. + +Downsides contain: + + * Messages don't get deleted from disk until there's only one client accessing + the mailbox, so you may need to add some forced downtime to get it done. + * Works only in local filesystem. + * Expunging is still as costly and as fragile as with mbox. + * At least on 32 bit systems, mailboxes are limited to 2GB size. (A mailbox + can't be accessed anymore without manual fixing if it reached 2GB (minus 1B) + of size on disk.) This is probably UW-IMAP -specific implementation problem + rather than actual mailbox format issue. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.txt b/doc/wiki/MailboxFormat.txt new file mode 100644 index 0000000..0f70184 --- /dev/null +++ b/doc/wiki/MailboxFormat.txt @@ -0,0 +1,147 @@ +Mailbox Formats +=============== + +Mailbox formats supported by Dovecot: ++-----------------------------+---------------+-------------------+-------------+ +| *Name* | *Tag* | *Description* | ++-----------------------------+---------------+-------------------+-------------+ +| <mbox> | 'mbox' | Traditional UNIX | +| [MailboxFormat.mbox.txt] | | mailbox format. | +| | | Users' INBOX | +| | | mboxes are | +| | | commonly stored in| +| | | '/var/spool/mail' | +| | | or '/var/mail' | +| | | directory. Single | +| | | file contains | +| | | multiple messages.| ++-----------------------------+---------------+-------------------+-------------+ +| <Maildir> | 'maildir' | One file contains | +| [MailboxFormat.Maildir.txt] | | one message. A | +| | | reliable choice | +| | | since files are | +| | | never modified and| +| | | all operations are| +| | | atomic. The | +| | | top-level | +| | | 'Maildir' | +| | | directory contains| +| | | the 'Maildir/cur',| +| | | 'Maildir/new' and | +| | | 'Maildir/tmp' | +| | | subdirectories. | ++-----------------------------+---------------+-------------------+-------------+ +| <dbox> | 'sdbox' | *single-dbox*, one| Dovecot's | +| [MailboxFormat.dbox.txt] | | message per file | own high | +| | | | performance | +| | | | mailbox | +| | | | format. | +| | | | Messages are| +| | | | stored in | +| | | | one or more | +| | | | files, each | +| | | | containing | +| | | | one or more | +| | | | messages. | ++-----------------------------+---------------+-------------------+-------------+ +| 'mdbox' | *multi-dbox*, | +| | multiple | +| | messages per | +| | file | ++-----------------------------+---------------+-------------------+-------------+ +| 'dbox' | deprecated | +| | alias for | +| | 'sdbox' | ++-----------------------------+---------------+-------------------+-------------+ +| <Cydir> | 'cydir' | Dovecot's own | +| [MailboxFormat.Cydir.txt] | | simple and high | +| | | performance | +| | | Cyrus-like mailbox| +| | | format. It should | +| | | be mostly used for| +| | | testing and | +| | | benchmarking only.| ++-----------------------------+---------------+-------------------+-------------+ +| <imapc> | 'imapc' | Use remote IMAP | +| [MailboxFormat.imapc.txt] | | server as mail | +| | | storage. | ++-----------------------------+---------------+-------------------+-------------+ +| <pop3c> | 'pop3c' | Use remote POP3 | +| [MailboxFormat.pop3c.txt] | | server as mail | +| | | storage. | ++-----------------------------+---------------+-------------------+-------------+ + +The *Tag* column indicates the tag which is used at the beginning of a <mailbox +location specification> [MailLocation.txt]. + +Mailbox formats *not* supported by Dovecot: ++-------------------------------+---------------------------------------------+ +| *Name* | *Description* | ++-------------------------------+---------------------------------------------+ +| <mbx> [MailboxFormat.mbx.txt] | UW-IMAP's old high performance mailbox | +| | format. One file contains all the mailboxes,| +| | so expunges may still be slow. | ++-------------------------------+---------------------------------------------+ +| <mix> [MailboxFormat.mix.txt] | UW-IMAP's new (2006) high performance | +| | mailbox format. Similar to multi-dbox. | ++-------------------------------+---------------------------------------------+ +| <mailstore> | A format created by Exim. | +| [MailboxFormat.mailstore.txt] | | ++-------------------------------+---------------------------------------------+ +| <MH> [MailboxFormat.MH.txt] | One file contains one message. Sort order of| +| | the folder determines the message ID and | +| | name. Actively used by projects such as | +| | MH-E, NMH, exmh. Experimentally supported | +| | by UW-IMAP | +| | [https://www.washington.edu/imap/]. | ++-------------------------------+---------------------------------------------+ +| <MMDF> [MailboxFormat.MMDF.txt]| Similar to mbox, but instead of From-line | +| | separators it uses four '^A' characters | ++-------------------------------+---------------------------------------------+ +| <Cyrus> | One file contains one message, plus there | +| [MailboxFormat.Cyrus.txt] | are a couple of index/cache files. Commonly | +| | referred to as being maildir-like, although | +| | they have only a single thing in common. | ++-------------------------------+---------------------------------------------+ + +Adding support for new formats for Dovecot isn't very difficult, although it +can be time consuming. Dovecot exposes a nice and simple API which needs to be +implemented. Use Cydir format as an example. + +Software Support +---------------- + ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| *Format/Software* | *Dovecot | *UW-IMAP | *Courier-IMAP | *Exim | *Postfix | *PINE | *mutt | *procmail | *maildrop | +| | [https://www.dovecot.org/]* | [https://www.washington.edu/imap/]* | [https://www.courier-mta.org/imap/]* | [https://www.exim.org/]* | [https://www.postfix.org/]* | [https://www.washington.edu/pine/]* | [https://www.mutt.org/]* | [https://www.procmail.org/]* | [https://www.courier-mta.org/maildrop/]* | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| <mbox> [MailboxFormat.mbox.txt]| Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| <mbx> [MailboxFormat.mbx.txt] | No | Yes | No | Yes | No | Yes | No | No | No | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| <maildir> | Yes | No | Yes | Yes | Yes | No | Yes | Yes (3.22) | Yes | +| [MailboxFormat.Maildir.txt] | | | | | | | | | | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| <mailstore> | No | No | No | Yes | No | No | No | No | No | +| [MailboxFormat.mailstore.txt] | | | | | | | | | | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| <dbox> [MailboxFormat.dbox.txt]| Yes | No | No | No | No | No | No | No | No | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| <MH> [MailboxFormat.MH.txt] | No | Yes | No | No | No | Yes | Yes | Yes | No | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ + +Conversion Between Mailbox Formats +---------------------------------- + +See <Migration.MailFormat.txt>. + +References +---------- + + * Mutt mailbox formats: https://rucus.ru.ac.za/docs/mutt/manual58.html + * Article on mailbox formats: + https://www.livejournal.com/users/rfunk/1571.html + * Mbox and maildir comparison: + https://www.linuxmail.info/mbox-maildir-mail-storage-formats/ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxSettings.txt b/doc/wiki/MailboxSettings.txt new file mode 100644 index 0000000..3e94581 --- /dev/null +++ b/doc/wiki/MailboxSettings.txt @@ -0,0 +1,81 @@ +Mailbox settings +================ + +Since Dovecot v2.1 one can assign SPECIAL-USE RFC 6154 +[http://www.faqs.org/rfcs/rfc6154.html] tags and specify, which mailboxes to +create and/or subscribe to automatically. + +The autocreated mailboxes are created lazily to disk only when accessed for the +first time. The autosubscribed mailboxes aren't written to subscriptions file, +unless SUBSCRIBE command is explicitly used for them. + +The mailbox section name specifies the mailbox name. If it has spaces, you can +put it in "quotes". The mailbox settings are: + + * auto: Autocreate/subscribe mailbox? + * no: Neither + * create: Autocreate, but don't autosubscribe + * subscribe: Autocreate and autosubscribe + * special_use: Space-separated list of SPECIAL-USE flags to use for the + mailbox. There are no validity checks, so you could specify anything you + want in here, but it's not a good idea to use other than the standard ones + specified in the RFC. + * NOTE: Due to a bug in Dovecot v2.2.30+ if special-use flags are used, + SPECIAL-USE needs to be added to post-login CAPABILITY response as RFC + 6154 mandates. You can do this with 'imap_capability = +SPECIAL-USE' + * autoexpunge=<time>: (v2.2.20+) Automatically at user deinitialization + expunge all mails in this mailbox whose saved-timestamp is older than<time> + (e.g. autoexpunge=30d). This removes the need for <expire plugin> + [Plugins.Expire.txt] if you don't care that the expunging may not always + happen in time. + * For IMAP and POP3 this happens after the client is already disconnected. + * For LMTP this happens when the user's mail delivery is finished. Note + that if there are multiple recipients this may delay delivering the mails + to the other recipients. + * Also doveadm and other processes verify this, which may be unnecessary. + So it may be better to explicitly enable this only inside protocol imap, + pop3 and maybe lmtp. You can do this with e.g.:'protocol imap { namespace + inbox { mailbox Spam { autoexpunge = 10d } } }' + * mailbox_list_index=yes is highly recommended when using this setting, as + it avoids actually opening the mailbox to see if anything needs to be + expunged. + * autoexpunge_max_mails=<number>: (v2.2.25+) Mails are expunged until mail + count is at autoexpunge_max_mails or below. After these messages are + removed, autoexpunge will then try to expunge mails based on the + 'autoexpunge' setting. + +---%<------------------------------------------------------------------------- +namespace inbox { + #prefix = INBOX. # the namespace prefix isn't added again to the mailbox +names. + inbox = yes + # ... + + mailbox Trash { + auto = no + special_use = \Trash + } + mailbox Drafts { + auto = no + special_use = \Drafts + } + mailbox Sent { + auto = subscribe # autocreate and autosubscribe the Sent mailbox + special_use = \Sent + } + mailbox "Sent Messages" { + auto = no + special_use = \Sent + } + mailbox Spam { + auto = create # autocreate Spam, but don't autosubscribe + special_use = \Junk + } + mailbox virtual/All { # if you have a virtual "All messages" mailbox + auto = no + special_use = \All + } +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Makefile.am b/doc/wiki/Makefile.am new file mode 100644 index 0000000..13b0939 --- /dev/null +++ b/doc/wiki/Makefile.am @@ -0,0 +1,282 @@ +if BUILD_DOCS +wikidir = $(docdir)/wiki +wiki_DATA = $(wikifiles) +endif + +EXTRA_DIST = \ + $(wikifiles) + +wikifiles = ACL.txt \ + AixPluginsSupport.txt \ + AttachmentIndicator.txt \ + AuthDatabase.CheckPassword.txt \ + AuthDatabase.Dict.txt \ + AuthDatabase.LDAP.AuthBinds.txt \ + AuthDatabase.LDAP.PasswordLookups.txt \ + AuthDatabase.LDAP.Userdb.txt \ + AuthDatabase.LDAP.txt \ + AuthDatabase.Lua.txt \ + AuthDatabase.Passwd.txt \ + AuthDatabase.PasswdFile.txt \ + AuthDatabase.SQL.txt \ + AuthDatabase.VPopMail.txt \ + AuthDatabase.txt \ + Authentication.Caching.txt \ + Authentication.Kerberos.txt \ + Authentication.MasterUsers.txt \ + Authentication.Mechanisms.DigestMD5.txt \ + Authentication.Mechanisms.NTLM.txt \ + Authentication.Mechanisms.Winbind.txt \ + Authentication.Mechanisms.txt \ + Authentication.MultipleDatabases.txt \ + Authentication.PasswordSchemes.txt \ + Authentication.Penalty.txt \ + Authentication.Policy.txt \ + Authentication.RestrictAccess.txt \ + Authentication.txt \ + BasicConfiguration.txt \ + Chrooting.txt \ + Clients.NegativeUIDs.txt \ + Clients.txt \ + CompilingSource.txt \ + ConfigFile.txt \ + Debugging.Authentication.txt \ + Debugging.ProcessTracing.txt \ + Debugging.Rawlog.txt \ + Debugging.Thunderbird.txt \ + Design.Arrays.txt \ + Design.AuthProcess.txt \ + Design.AuthProtocol.txt \ + Design.Buffers.txt \ + Design.Code.txt \ + Design.Dcrypt.txt \ + Design.DoveadmProtocol.HTTP.txt \ + Design.DoveadmProtocol.txt \ + Design.Dsync.txt \ + Design.Events.txt \ + Design.Indexes.Cache.txt \ + Design.Indexes.MailIndexApi.txt \ + Design.Indexes.MainIndex.txt \ + Design.Indexes.TransactionLog.txt \ + Design.Indexes.txt \ + Design.InputStreams.txt \ + Design.Lua.txt \ + Design.MailProcess.txt \ + Design.Memory.txt \ + Design.OutputStreams.txt \ + Design.ParameterForwarding.txt \ + Design.Plugins.txt \ + Design.Processes.txt \ + Design.Storage.ErrorHandling.txt \ + Design.Storage.Mail.txt \ + Design.Storage.MailNamespace.txt \ + Design.Storage.MailStorage.txt \ + Design.Storage.MailUser.txt \ + Design.Storage.Mailbox.Save.txt \ + Design.Storage.Mailbox.Search.txt \ + Design.Storage.Mailbox.Sync.txt \ + Design.Storage.Mailbox.Transaction.txt \ + Design.Storage.Mailbox.txt \ + Design.Storage.MailboxList.txt \ + Design.Storage.Plugins.txt \ + Design.Strings.txt \ + Design.txt \ + Dict.txt \ + Dictionary.txt \ + Director.txt \ + DomainLost.txt \ + Errors.ChgrpNoPerm.txt \ + Events.txt \ + FindMailLocation.txt \ + FinishBasicConfiguration.txt \ + HAProxy.txt \ + HowTo.AntispamWithSieve.txt \ + HowTo.EximAndDovecotSASL.txt \ + HowTo.ImapcProxy.txt \ + HowTo.PopBSMTPAndDovecot.txt \ + HowTo.PopRelay.txt \ + HowTo.PostfixAndDovecotSASL.txt \ + HowTo.Rootless.txt \ + HowTo.SimpleVirtualInstall.txt \ + HowTo.WriteConfiguration.txt \ + HowTo.txt \ + IMAPServer.Hibernation.txt \ + IMAPServer.txt \ + ImapMetadata.txt \ + IndexFiles.txt \ + LDA.Exim.txt \ + LDA.Indexing.txt \ + LDA.Postfix.txt \ + LDA.Qmail.txt \ + LDA.Sendmail.txt \ + LDA.txt \ + LMTP.Exim.txt \ + LMTP.txt \ + Logging.txt \ + LoginProcess.txt \ + MDA.txt \ + MTA.txt \ + MailLocation.LocalDisk.txt \ + MailLocation.Maildir.txt \ + MailLocation.SharedDisk.txt \ + MailLocation.dbox.txt \ + MailLocation.mbox.txt \ + MailLocation.txt \ + MailboxFormat.Cydir.txt \ + MailboxFormat.MH.txt \ + MailboxFormat.Maildir.txt \ + MailboxFormat.dbox.txt \ + MailboxFormat.imapc.txt \ + MailboxFormat.mailstore.txt \ + MailboxFormat.mbox.txt \ + MailboxFormat.mbx.txt \ + MailboxFormat.txt \ + MailboxSettings.txt \ + MboxChildFolders.txt \ + MboxLocking.txt \ + MboxProblems.txt \ + Migration.BincIMAP.txt \ + Migration.Courier.txt \ + Migration.Cyrus.txt \ + Migration.Dsync.txt \ + Migration.Gmail.txt \ + Migration.Linuxconf.txt \ + Migration.MailFormat.txt \ + Migration.Online.txt \ + Migration.Teapop.txt \ + Migration.UW.txt \ + Migration.Vm-pop3d.txt \ + Migration.txt \ + MissingMailboxes.txt \ + Mountpoints.txt \ + NFS.txt \ + Namespaces.txt \ + OSCompatibility.txt \ + POP3Server.txt \ + PasswordDatabase.BSDAuth.txt \ + PasswordDatabase.ExtraFields.AllowNets.txt \ + PasswordDatabase.ExtraFields.Host.txt \ + PasswordDatabase.ExtraFields.NoDelay.txt \ + PasswordDatabase.ExtraFields.NoLogin.txt \ + PasswordDatabase.ExtraFields.Proxy.txt \ + PasswordDatabase.ExtraFields.User.txt \ + PasswordDatabase.ExtraFields.txt \ + PasswordDatabase.IMAP.txt \ + PasswordDatabase.PAM.txt \ + PasswordDatabase.Shadow.txt \ + PasswordDatabase.Static.txt \ + PasswordDatabase.oauth2.txt \ + PasswordDatabase.txt \ + PerformanceTuning.txt \ + Pigeonhole.Installation.txt \ + Pigeonhole.ManageSieve.Clients.txt \ + Pigeonhole.ManageSieve.Configuration.txt \ + Pigeonhole.ManageSieve.Install.txt \ + Pigeonhole.ManageSieve.Troubleshooting.txt \ + Pigeonhole.ManageSieve.txt \ + Pigeonhole.Sieve.Configuration.Dict.txt \ + Pigeonhole.Sieve.Configuration.File.txt \ + Pigeonhole.Sieve.Configuration.LDAP.txt \ + Pigeonhole.Sieve.Configuration.txt \ + Pigeonhole.Sieve.Examples.txt \ + Pigeonhole.Sieve.Extensions.Duplicate.txt \ + Pigeonhole.Sieve.Extensions.Editheader.txt \ + Pigeonhole.Sieve.Extensions.Include.txt \ + Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt \ + Pigeonhole.Sieve.Extensions.Vacation.txt \ + Pigeonhole.Sieve.Extensions.Variables.txt \ + Pigeonhole.Sieve.Extensions.txt \ + Pigeonhole.Sieve.Plugins.Extdata.txt \ + Pigeonhole.Sieve.Plugins.Extprograms.txt \ + Pigeonhole.Sieve.Plugins.IMAPFilterSieve.txt \ + Pigeonhole.Sieve.Plugins.IMAPSieve.txt \ + Pigeonhole.Sieve.Plugins.Pipe.txt \ + Pigeonhole.Sieve.Plugins.txt \ + Pigeonhole.Sieve.Troubleshooting.txt \ + Pigeonhole.Sieve.Usage.txt \ + Pigeonhole.Sieve.txt \ + Pigeonhole.txt \ + Plugins.Apparmor.txt \ + Plugins.Autocreate.txt \ + Plugins.CharsetAlias.txt \ + Plugins.Compress.txt \ + Plugins.Expire.txt \ + Plugins.FTS.Lucene.txt \ + Plugins.FTS.Solr.txt \ + Plugins.FTS.Squat.txt \ + Plugins.FTS.txt \ + Plugins.LastLogin.txt \ + Plugins.Lazyexpunge.txt \ + Plugins.Listescape.txt \ + Plugins.MailCrypt.txt \ + Plugins.MailFilter.txt \ + Plugins.MailLog.txt \ + Plugins.MailboxAlias.txt \ + Plugins.Notify.txt \ + Plugins.NotifyStatus.txt \ + Plugins.PushNotification.txt \ + Plugins.QuotaClone.txt \ + Plugins.Snarf.txt \ + Plugins.Stats.txt \ + Plugins.Trash.txt \ + Plugins.VarExpandCrypt.txt \ + Plugins.Virtual.txt \ + Plugins.Welcome.txt \ + Plugins.Zlib.txt \ + Plugins.txt \ + PostLoginScripting.txt \ + PreAuth.txt \ + QuickConfiguration.txt \ + Quota.Configuration.txt \ + Quota.Count.txt \ + Quota.Dict.txt \ + Quota.Dirsize.txt \ + Quota.FS.txt \ + Quota.Maildir.txt \ + Quota.txt \ + Replication.txt \ + RunningDovecot.txt \ + SSL.CertificateClientImporting.txt \ + SSL.CertificateCreation.txt \ + SSL.DovecotConfiguration.txt \ + SSL.SNIClientSupport.txt \ + SSL.txt \ + Sasl.txt \ + SecurityTuning.txt \ + Services.txt \ + SharedMailboxes.ClusterSetup.txt \ + SharedMailboxes.Permissions.txt \ + SharedMailboxes.Public.txt \ + SharedMailboxes.Shared.txt \ + SharedMailboxes.Symlinks.txt \ + SharedMailboxes.txt \ + SocketUnavailable.txt \ + Statistics.Old.txt \ + Statistics.txt \ + Submission.txt \ + SystemUsers.txt \ + TestInstallation.txt \ + TestPop3Installation.txt \ + TimeMovedBackwards.txt \ + Timeouts.txt \ + Upgrading.1.0.txt \ + Upgrading.1.1.txt \ + Upgrading.1.2.txt \ + Upgrading.2.0.txt \ + Upgrading.2.1.txt \ + Upgrading.2.2.txt \ + Upgrading.2.3.txt \ + Upgrading.txt \ + UserDatabase.ExtraFields.txt \ + UserDatabase.NSS.txt \ + UserDatabase.Prefetch.txt \ + UserDatabase.Static.txt \ + UserDatabase.txt \ + UserIds.txt \ + Variables.txt \ + VirtualUsers.Home.txt \ + VirtualUsers.txt \ + WhyDoesItNotWork.txt \ + maildrop.txt \ + mutt.txt \ + uw2dovecot.sh.txt diff --git a/doc/wiki/Makefile.in b/doc/wiki/Makefile.in new file mode 100644 index 0000000..6aa234e --- /dev/null +++ b/doc/wiki/Makefile.in @@ -0,0 +1,920 @@ +# Makefile.in generated by automake 1.16.3 from Makefile.am. +# @configure_input@ + +# Copyright (C) 1994-2020 Free Software Foundation, Inc. + +# This Makefile.in is free software; the Free Software Foundation +# gives unlimited permission to copy and/or distribute it, +# with or without modifications, as long as this notice is preserved. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY, to the extent permitted by law; without +# even the implied warranty of MERCHANTABILITY or FITNESS FOR A +# PARTICULAR PURPOSE. + +@SET_MAKE@ + +VPATH = @srcdir@ +am__is_gnu_make = { \ + if test -z '$(MAKELEVEL)'; then \ + false; \ + elif test -n '$(MAKE_HOST)'; then \ + true; \ + elif test -n '$(MAKE_VERSION)' && test -n '$(CURDIR)'; then \ + true; \ + else \ + false; \ + fi; \ +} +am__make_running_with_option = \ + case $${target_option-} in \ + ?) ;; \ + *) echo "am__make_running_with_option: internal error: invalid" \ + "target option '$${target_option-}' specified" >&2; \ + exit 1;; \ + esac; \ + has_opt=no; \ + sane_makeflags=$$MAKEFLAGS; \ + if $(am__is_gnu_make); then \ + sane_makeflags=$$MFLAGS; \ + else \ + case $$MAKEFLAGS in \ + *\\[\ \ ]*) \ + bs=\\; \ + sane_makeflags=`printf '%s\n' "$$MAKEFLAGS" \ + | sed "s/$$bs$$bs[$$bs $$bs ]*//g"`;; \ + esac; \ + fi; \ + skip_next=no; \ + strip_trailopt () \ + { \ + flg=`printf '%s\n' "$$flg" | sed "s/$$1.*$$//"`; \ + }; \ + for flg in $$sane_makeflags; do \ + test $$skip_next = yes && { skip_next=no; continue; }; \ + case $$flg in \ + *=*|--*) continue;; \ + -*I) strip_trailopt 'I'; skip_next=yes;; \ + -*I?*) strip_trailopt 'I';; \ + -*O) strip_trailopt 'O'; skip_next=yes;; \ + -*O?*) strip_trailopt 'O';; \ + -*l) strip_trailopt 'l'; skip_next=yes;; \ + -*l?*) strip_trailopt 'l';; \ + -[dEDm]) skip_next=yes;; \ + -[JT]) skip_next=yes;; \ + esac; \ + case $$flg in \ + *$$target_option*) has_opt=yes; break;; \ + esac; \ + done; \ + test $$has_opt = yes +am__make_dryrun = (target_option=n; $(am__make_running_with_option)) +am__make_keepgoing = (target_option=k; $(am__make_running_with_option)) +pkgdatadir = $(datadir)/@PACKAGE@ +pkgincludedir = $(includedir)/@PACKAGE@ +pkglibdir = $(libdir)/@PACKAGE@ +pkglibexecdir = $(libexecdir)/@PACKAGE@ +am__cd = CDPATH="$${ZSH_VERSION+.}$(PATH_SEPARATOR)" && cd +install_sh_DATA = $(install_sh) -c -m 644 +install_sh_PROGRAM = $(install_sh) -c +install_sh_SCRIPT = $(install_sh) -c +INSTALL_HEADER = $(INSTALL_DATA) +transform = $(program_transform_name) +NORMAL_INSTALL = : +PRE_INSTALL = : +POST_INSTALL = : +NORMAL_UNINSTALL = : +PRE_UNINSTALL = : +POST_UNINSTALL = : +build_triplet = @build@ +host_triplet = @host@ +subdir = doc/wiki +ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 +am__aclocal_m4_deps = $(top_srcdir)/m4/ac_checktype2.m4 \ + $(top_srcdir)/m4/ac_typeof.m4 $(top_srcdir)/m4/arc4random.m4 \ + $(top_srcdir)/m4/blockdev.m4 $(top_srcdir)/m4/c99_vsnprintf.m4 \ + $(top_srcdir)/m4/clock_gettime.m4 $(top_srcdir)/m4/crypt.m4 \ + $(top_srcdir)/m4/crypt_xpg6.m4 $(top_srcdir)/m4/dbqlk.m4 \ + $(top_srcdir)/m4/dirent_dtype.m4 $(top_srcdir)/m4/dovecot.m4 \ + $(top_srcdir)/m4/fd_passing.m4 $(top_srcdir)/m4/fdatasync.m4 \ + $(top_srcdir)/m4/flexible_array_member.m4 \ + $(top_srcdir)/m4/glibc.m4 $(top_srcdir)/m4/gmtime_max.m4 \ + $(top_srcdir)/m4/gmtime_tm_gmtoff.m4 \ + $(top_srcdir)/m4/ioloop.m4 $(top_srcdir)/m4/iovec.m4 \ + $(top_srcdir)/m4/ipv6.m4 $(top_srcdir)/m4/libcap.m4 \ + $(top_srcdir)/m4/libtool.m4 $(top_srcdir)/m4/libwrap.m4 \ + $(top_srcdir)/m4/linux_mremap.m4 $(top_srcdir)/m4/ltoptions.m4 \ + $(top_srcdir)/m4/ltsugar.m4 $(top_srcdir)/m4/ltversion.m4 \ + $(top_srcdir)/m4/lt~obsolete.m4 $(top_srcdir)/m4/mmap_write.m4 \ + $(top_srcdir)/m4/mntctl.m4 $(top_srcdir)/m4/modules.m4 \ + $(top_srcdir)/m4/notify.m4 $(top_srcdir)/m4/nsl.m4 \ + $(top_srcdir)/m4/off_t_max.m4 $(top_srcdir)/m4/pkg.m4 \ + $(top_srcdir)/m4/pr_set_dumpable.m4 \ + $(top_srcdir)/m4/q_quotactl.m4 $(top_srcdir)/m4/quota.m4 \ + $(top_srcdir)/m4/random.m4 $(top_srcdir)/m4/rlimit.m4 \ + $(top_srcdir)/m4/sendfile.m4 $(top_srcdir)/m4/size_t_signed.m4 \ + $(top_srcdir)/m4/sockpeercred.m4 $(top_srcdir)/m4/sql.m4 \ + $(top_srcdir)/m4/ssl.m4 $(top_srcdir)/m4/st_tim.m4 \ + $(top_srcdir)/m4/static_array.m4 $(top_srcdir)/m4/test_with.m4 \ + $(top_srcdir)/m4/time_t.m4 $(top_srcdir)/m4/typeof.m4 \ + $(top_srcdir)/m4/typeof_dev_t.m4 \ + $(top_srcdir)/m4/uoff_t_max.m4 $(top_srcdir)/m4/vararg.m4 \ + $(top_srcdir)/m4/want_apparmor.m4 \ + $(top_srcdir)/m4/want_bsdauth.m4 \ + $(top_srcdir)/m4/want_bzlib.m4 \ + $(top_srcdir)/m4/want_cassandra.m4 \ + $(top_srcdir)/m4/want_cdb.m4 \ + $(top_srcdir)/m4/want_checkpassword.m4 \ + $(top_srcdir)/m4/want_clucene.m4 $(top_srcdir)/m4/want_db.m4 \ + $(top_srcdir)/m4/want_gssapi.m4 $(top_srcdir)/m4/want_icu.m4 \ + $(top_srcdir)/m4/want_ldap.m4 $(top_srcdir)/m4/want_lua.m4 \ + $(top_srcdir)/m4/want_lz4.m4 $(top_srcdir)/m4/want_lzma.m4 \ + $(top_srcdir)/m4/want_mysql.m4 $(top_srcdir)/m4/want_pam.m4 \ + $(top_srcdir)/m4/want_passwd.m4 $(top_srcdir)/m4/want_pgsql.m4 \ + $(top_srcdir)/m4/want_prefetch.m4 \ + $(top_srcdir)/m4/want_shadow.m4 \ + $(top_srcdir)/m4/want_sodium.m4 $(top_srcdir)/m4/want_solr.m4 \ + $(top_srcdir)/m4/want_sqlite.m4 \ + $(top_srcdir)/m4/want_stemmer.m4 \ + $(top_srcdir)/m4/want_systemd.m4 \ + $(top_srcdir)/m4/want_textcat.m4 \ + $(top_srcdir)/m4/want_unwind.m4 $(top_srcdir)/m4/want_zlib.m4 \ + $(top_srcdir)/m4/want_zstd.m4 $(top_srcdir)/configure.ac +am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \ + $(ACLOCAL_M4) +DIST_COMMON = $(srcdir)/Makefile.am $(am__DIST_COMMON) +mkinstalldirs = $(install_sh) -d +CONFIG_HEADER = $(top_builddir)/config.h +CONFIG_CLEAN_FILES = +CONFIG_CLEAN_VPATH_FILES = +AM_V_P = $(am__v_P_@AM_V@) +am__v_P_ = $(am__v_P_@AM_DEFAULT_V@) +am__v_P_0 = false +am__v_P_1 = : +AM_V_GEN = $(am__v_GEN_@AM_V@) +am__v_GEN_ = $(am__v_GEN_@AM_DEFAULT_V@) +am__v_GEN_0 = @echo " GEN " $@; +am__v_GEN_1 = +AM_V_at = $(am__v_at_@AM_V@) +am__v_at_ = $(am__v_at_@AM_DEFAULT_V@) +am__v_at_0 = @ +am__v_at_1 = +SOURCES = +DIST_SOURCES = +am__can_run_installinfo = \ + case $$AM_UPDATE_INFO_DIR in \ + n|no|NO) false;; \ + *) (install-info --version) >/dev/null 2>&1;; \ + esac +am__vpath_adj_setup = srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; +am__vpath_adj = case $$p in \ + $(srcdir)/*) f=`echo "$$p" | sed "s|^$$srcdirstrip/||"`;; \ + *) f=$$p;; \ + esac; +am__strip_dir = f=`echo $$p | sed -e 's|^.*/||'`; +am__install_max = 40 +am__nobase_strip_setup = \ + srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*|]/\\\\&/g'` +am__nobase_strip = \ + for p in $$list; do echo "$$p"; done | sed -e "s|$$srcdirstrip/||" +am__nobase_list = $(am__nobase_strip_setup); \ + for p in $$list; do echo "$$p $$p"; done | \ + sed "s| $$srcdirstrip/| |;"' / .*\//!s/ .*/ ./; s,\( .*\)/[^/]*$$,\1,' | \ + $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \ + if (++n[$$2] == $(am__install_max)) \ + { print $$2, files[$$2]; n[$$2] = 0; files[$$2] = "" } } \ + END { for (dir in files) print dir, files[dir] }' +am__base_list = \ + sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \ + sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g' +am__uninstall_files_from_dir = { \ + test -z "$$files" \ + || { test ! -d "$$dir" && test ! -f "$$dir" && test ! -r "$$dir"; } \ + || { echo " ( cd '$$dir' && rm -f" $$files ")"; \ + $(am__cd) "$$dir" && rm -f $$files; }; \ + } +am__installdirs = "$(DESTDIR)$(wikidir)" +DATA = $(wiki_DATA) +am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP) +am__DIST_COMMON = $(srcdir)/Makefile.in +DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) +ACLOCAL = @ACLOCAL@ +ACLOCAL_AMFLAGS = @ACLOCAL_AMFLAGS@ +AMTAR = @AMTAR@ +AM_DEFAULT_VERBOSITY = @AM_DEFAULT_VERBOSITY@ +APPARMOR_LIBS = @APPARMOR_LIBS@ +AR = @AR@ +AUTH_CFLAGS = @AUTH_CFLAGS@ +AUTH_LIBS = @AUTH_LIBS@ +AUTOCONF = @AUTOCONF@ +AUTOHEADER = @AUTOHEADER@ +AUTOMAKE = @AUTOMAKE@ +AWK = @AWK@ +BINARY_CFLAGS = @BINARY_CFLAGS@ +BINARY_LDFLAGS = @BINARY_LDFLAGS@ +BISON = @BISON@ +CASSANDRA_CFLAGS = @CASSANDRA_CFLAGS@ +CASSANDRA_LIBS = @CASSANDRA_LIBS@ +CC = @CC@ +CCDEPMODE = @CCDEPMODE@ +CDB_LIBS = @CDB_LIBS@ +CFLAGS = @CFLAGS@ +CLUCENE_CFLAGS = @CLUCENE_CFLAGS@ +CLUCENE_LIBS = @CLUCENE_LIBS@ +COMPRESS_LIBS = @COMPRESS_LIBS@ +CPP = @CPP@ +CPPFLAGS = @CPPFLAGS@ +CRYPT_LIBS = @CRYPT_LIBS@ +CXX = @CXX@ +CXXCPP = @CXXCPP@ +CXXDEPMODE = @CXXDEPMODE@ +CXXFLAGS = @CXXFLAGS@ +CYGPATH_W = @CYGPATH_W@ +DEFS = @DEFS@ +DEPDIR = @DEPDIR@ +DICT_LIBS = @DICT_LIBS@ +DLLIB = @DLLIB@ +DLLTOOL = @DLLTOOL@ +DSYMUTIL = @DSYMUTIL@ +DUMPBIN = @DUMPBIN@ +ECHO_C = @ECHO_C@ +ECHO_N = @ECHO_N@ +ECHO_T = @ECHO_T@ +EGREP = @EGREP@ +EXEEXT = @EXEEXT@ +FGREP = @FGREP@ +FLEX = @FLEX@ +FUZZER_CPPFLAGS = @FUZZER_CPPFLAGS@ +FUZZER_LDFLAGS = @FUZZER_LDFLAGS@ +GREP = @GREP@ +INSTALL = @INSTALL@ +INSTALL_DATA = @INSTALL_DATA@ +INSTALL_PROGRAM = @INSTALL_PROGRAM@ +INSTALL_SCRIPT = @INSTALL_SCRIPT@ +INSTALL_STRIP_PROGRAM = @INSTALL_STRIP_PROGRAM@ +KRB5CONFIG = @KRB5CONFIG@ +KRB5_CFLAGS = @KRB5_CFLAGS@ +KRB5_LIBS = @KRB5_LIBS@ +LD = @LD@ +LDAP_LIBS = @LDAP_LIBS@ +LDFLAGS = @LDFLAGS@ +LD_NO_WHOLE_ARCHIVE = @LD_NO_WHOLE_ARCHIVE@ +LD_WHOLE_ARCHIVE = @LD_WHOLE_ARCHIVE@ +LIBCAP = @LIBCAP@ +LIBDOVECOT = @LIBDOVECOT@ +LIBDOVECOT_COMPRESS = @LIBDOVECOT_COMPRESS@ +LIBDOVECOT_DEPS = @LIBDOVECOT_DEPS@ +LIBDOVECOT_DSYNC = @LIBDOVECOT_DSYNC@ +LIBDOVECOT_LA_LIBS = @LIBDOVECOT_LA_LIBS@ +LIBDOVECOT_LDA = @LIBDOVECOT_LDA@ +LIBDOVECOT_LDAP = @LIBDOVECOT_LDAP@ +LIBDOVECOT_LIBFTS = @LIBDOVECOT_LIBFTS@ +LIBDOVECOT_LIBFTS_DEPS = @LIBDOVECOT_LIBFTS_DEPS@ +LIBDOVECOT_LOGIN = @LIBDOVECOT_LOGIN@ +LIBDOVECOT_LUA = @LIBDOVECOT_LUA@ +LIBDOVECOT_LUA_DEPS = @LIBDOVECOT_LUA_DEPS@ +LIBDOVECOT_SQL = @LIBDOVECOT_SQL@ +LIBDOVECOT_STORAGE = @LIBDOVECOT_STORAGE@ +LIBDOVECOT_STORAGE_DEPS = @LIBDOVECOT_STORAGE_DEPS@ +LIBEXTTEXTCAT_CFLAGS = @LIBEXTTEXTCAT_CFLAGS@ +LIBEXTTEXTCAT_LIBS = @LIBEXTTEXTCAT_LIBS@ +LIBICONV = @LIBICONV@ +LIBICU_CFLAGS = @LIBICU_CFLAGS@ +LIBICU_LIBS = @LIBICU_LIBS@ +LIBOBJS = @LIBOBJS@ +LIBS = @LIBS@ +LIBSODIUM_CFLAGS = @LIBSODIUM_CFLAGS@ +LIBSODIUM_LIBS = @LIBSODIUM_LIBS@ +LIBTIRPC_CFLAGS = @LIBTIRPC_CFLAGS@ +LIBTIRPC_LIBS = @LIBTIRPC_LIBS@ +LIBTOOL = @LIBTOOL@ +LIBUNWIND_CFLAGS = @LIBUNWIND_CFLAGS@ +LIBUNWIND_LIBS = @LIBUNWIND_LIBS@ +LIBWRAP_LIBS = @LIBWRAP_LIBS@ +LINKED_STORAGE_LDADD = @LINKED_STORAGE_LDADD@ +LIPO = @LIPO@ +LN_S = @LN_S@ +LTLIBICONV = @LTLIBICONV@ +LTLIBOBJS = @LTLIBOBJS@ +LT_SYS_LIBRARY_PATH = @LT_SYS_LIBRARY_PATH@ +LUA_CFLAGS = @LUA_CFLAGS@ +LUA_LIBS = @LUA_LIBS@ +MAINT = @MAINT@ +MAKEINFO = @MAKEINFO@ +MANIFEST_TOOL = @MANIFEST_TOOL@ +MKDIR_P = @MKDIR_P@ +MODULE_LIBS = @MODULE_LIBS@ +MODULE_SUFFIX = @MODULE_SUFFIX@ +MYSQL_CFLAGS = @MYSQL_CFLAGS@ +MYSQL_CONFIG = @MYSQL_CONFIG@ +MYSQL_LIBS = @MYSQL_LIBS@ +NM = @NM@ +NMEDIT = @NMEDIT@ +NOPLUGIN_LDFLAGS = @NOPLUGIN_LDFLAGS@ +OBJDUMP = @OBJDUMP@ +OBJEXT = @OBJEXT@ +OTOOL = @OTOOL@ +OTOOL64 = @OTOOL64@ +PACKAGE = @PACKAGE@ +PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@ +PACKAGE_NAME = @PACKAGE_NAME@ +PACKAGE_STRING = @PACKAGE_STRING@ +PACKAGE_TARNAME = @PACKAGE_TARNAME@ +PACKAGE_URL = @PACKAGE_URL@ +PACKAGE_VERSION = @PACKAGE_VERSION@ +PANDOC = @PANDOC@ +PATH_SEPARATOR = @PATH_SEPARATOR@ +PGSQL_CFLAGS = @PGSQL_CFLAGS@ +PGSQL_LIBS = @PGSQL_LIBS@ +PG_CONFIG = @PG_CONFIG@ +PIE_CFLAGS = @PIE_CFLAGS@ +PIE_LDFLAGS = @PIE_LDFLAGS@ +PKG_CONFIG = @PKG_CONFIG@ +PKG_CONFIG_LIBDIR = @PKG_CONFIG_LIBDIR@ +PKG_CONFIG_PATH = @PKG_CONFIG_PATH@ +QUOTA_LIBS = @QUOTA_LIBS@ +RANLIB = @RANLIB@ +RELRO_LDFLAGS = @RELRO_LDFLAGS@ +RPCGEN = @RPCGEN@ +RUN_TEST = @RUN_TEST@ +SED = @SED@ +SETTING_FILES = @SETTING_FILES@ +SET_MAKE = @SET_MAKE@ +SHELL = @SHELL@ +SQLITE_CFLAGS = @SQLITE_CFLAGS@ +SQLITE_LIBS = @SQLITE_LIBS@ +SQL_CFLAGS = @SQL_CFLAGS@ +SQL_LIBS = @SQL_LIBS@ +SSL_CFLAGS = @SSL_CFLAGS@ +SSL_LIBS = @SSL_LIBS@ +STRIP = @STRIP@ +SYSTEMD_CFLAGS = @SYSTEMD_CFLAGS@ +SYSTEMD_LIBS = @SYSTEMD_LIBS@ +VALGRIND = @VALGRIND@ +VERSION = @VERSION@ +ZSTD_CFLAGS = @ZSTD_CFLAGS@ +ZSTD_LIBS = @ZSTD_LIBS@ +abs_builddir = @abs_builddir@ +abs_srcdir = @abs_srcdir@ +abs_top_builddir = @abs_top_builddir@ +abs_top_srcdir = @abs_top_srcdir@ +ac_ct_AR = @ac_ct_AR@ +ac_ct_CC = @ac_ct_CC@ +ac_ct_CXX = @ac_ct_CXX@ +ac_ct_DUMPBIN = @ac_ct_DUMPBIN@ +am__include = @am__include@ +am__leading_dot = @am__leading_dot@ +am__quote = @am__quote@ +am__tar = @am__tar@ +am__untar = @am__untar@ +bindir = @bindir@ +build = @build@ +build_alias = @build_alias@ +build_cpu = @build_cpu@ +build_os = @build_os@ +build_vendor = @build_vendor@ +builddir = @builddir@ +datadir = @datadir@ +datarootdir = @datarootdir@ +dict_drivers = @dict_drivers@ +docdir = @docdir@ +dvidir = @dvidir@ +exec_prefix = @exec_prefix@ +host = @host@ +host_alias = @host_alias@ +host_cpu = @host_cpu@ +host_os = @host_os@ +host_vendor = @host_vendor@ +htmldir = @htmldir@ +includedir = @includedir@ +infodir = @infodir@ +install_sh = @install_sh@ +libdir = @libdir@ +libexecdir = @libexecdir@ +localedir = @localedir@ +localstatedir = @localstatedir@ +mandir = @mandir@ +mkdir_p = @mkdir_p@ +moduledir = @moduledir@ +oldincludedir = @oldincludedir@ +pdfdir = @pdfdir@ +prefix = @prefix@ +program_transform_name = @program_transform_name@ +psdir = @psdir@ +rundir = @rundir@ +runstatedir = @runstatedir@ +sbindir = @sbindir@ +sharedstatedir = @sharedstatedir@ +sql_drivers = @sql_drivers@ +srcdir = @srcdir@ +ssldir = @ssldir@ +statedir = @statedir@ +sysconfdir = @sysconfdir@ +systemdservicetype = @systemdservicetype@ +systemdsystemunitdir = @systemdsystemunitdir@ +target_alias = @target_alias@ +top_build_prefix = @top_build_prefix@ +top_builddir = @top_builddir@ +top_srcdir = @top_srcdir@ +@BUILD_DOCS_TRUE@wikidir = $(docdir)/wiki +@BUILD_DOCS_TRUE@wiki_DATA = $(wikifiles) +EXTRA_DIST = \ + $(wikifiles) + +wikifiles = ACL.txt \ + AixPluginsSupport.txt \ + AttachmentIndicator.txt \ + AuthDatabase.CheckPassword.txt \ + AuthDatabase.Dict.txt \ + AuthDatabase.LDAP.AuthBinds.txt \ + AuthDatabase.LDAP.PasswordLookups.txt \ + AuthDatabase.LDAP.Userdb.txt \ + AuthDatabase.LDAP.txt \ + AuthDatabase.Lua.txt \ + AuthDatabase.Passwd.txt \ + AuthDatabase.PasswdFile.txt \ + AuthDatabase.SQL.txt \ + AuthDatabase.VPopMail.txt \ + AuthDatabase.txt \ + Authentication.Caching.txt \ + Authentication.Kerberos.txt \ + Authentication.MasterUsers.txt \ + Authentication.Mechanisms.DigestMD5.txt \ + Authentication.Mechanisms.NTLM.txt \ + Authentication.Mechanisms.Winbind.txt \ + Authentication.Mechanisms.txt \ + Authentication.MultipleDatabases.txt \ + Authentication.PasswordSchemes.txt \ + Authentication.Penalty.txt \ + Authentication.Policy.txt \ + Authentication.RestrictAccess.txt \ + Authentication.txt \ + BasicConfiguration.txt \ + Chrooting.txt \ + Clients.NegativeUIDs.txt \ + Clients.txt \ + CompilingSource.txt \ + ConfigFile.txt \ + Debugging.Authentication.txt \ + Debugging.ProcessTracing.txt \ + Debugging.Rawlog.txt \ + Debugging.Thunderbird.txt \ + Design.Arrays.txt \ + Design.AuthProcess.txt \ + Design.AuthProtocol.txt \ + Design.Buffers.txt \ + Design.Code.txt \ + Design.Dcrypt.txt \ + Design.DoveadmProtocol.HTTP.txt \ + Design.DoveadmProtocol.txt \ + Design.Dsync.txt \ + Design.Events.txt \ + Design.Indexes.Cache.txt \ + Design.Indexes.MailIndexApi.txt \ + Design.Indexes.MainIndex.txt \ + Design.Indexes.TransactionLog.txt \ + Design.Indexes.txt \ + Design.InputStreams.txt \ + Design.Lua.txt \ + Design.MailProcess.txt \ + Design.Memory.txt \ + Design.OutputStreams.txt \ + Design.ParameterForwarding.txt \ + Design.Plugins.txt \ + Design.Processes.txt \ + Design.Storage.ErrorHandling.txt \ + Design.Storage.Mail.txt \ + Design.Storage.MailNamespace.txt \ + Design.Storage.MailStorage.txt \ + Design.Storage.MailUser.txt \ + Design.Storage.Mailbox.Save.txt \ + Design.Storage.Mailbox.Search.txt \ + Design.Storage.Mailbox.Sync.txt \ + Design.Storage.Mailbox.Transaction.txt \ + Design.Storage.Mailbox.txt \ + Design.Storage.MailboxList.txt \ + Design.Storage.Plugins.txt \ + Design.Strings.txt \ + Design.txt \ + Dict.txt \ + Dictionary.txt \ + Director.txt \ + DomainLost.txt \ + Errors.ChgrpNoPerm.txt \ + Events.txt \ + FindMailLocation.txt \ + FinishBasicConfiguration.txt \ + HAProxy.txt \ + HowTo.AntispamWithSieve.txt \ + HowTo.EximAndDovecotSASL.txt \ + HowTo.ImapcProxy.txt \ + HowTo.PopBSMTPAndDovecot.txt \ + HowTo.PopRelay.txt \ + HowTo.PostfixAndDovecotSASL.txt \ + HowTo.Rootless.txt \ + HowTo.SimpleVirtualInstall.txt \ + HowTo.WriteConfiguration.txt \ + HowTo.txt \ + IMAPServer.Hibernation.txt \ + IMAPServer.txt \ + ImapMetadata.txt \ + IndexFiles.txt \ + LDA.Exim.txt \ + LDA.Indexing.txt \ + LDA.Postfix.txt \ + LDA.Qmail.txt \ + LDA.Sendmail.txt \ + LDA.txt \ + LMTP.Exim.txt \ + LMTP.txt \ + Logging.txt \ + LoginProcess.txt \ + MDA.txt \ + MTA.txt \ + MailLocation.LocalDisk.txt \ + MailLocation.Maildir.txt \ + MailLocation.SharedDisk.txt \ + MailLocation.dbox.txt \ + MailLocation.mbox.txt \ + MailLocation.txt \ + MailboxFormat.Cydir.txt \ + MailboxFormat.MH.txt \ + MailboxFormat.Maildir.txt \ + MailboxFormat.dbox.txt \ + MailboxFormat.imapc.txt \ + MailboxFormat.mailstore.txt \ + MailboxFormat.mbox.txt \ + MailboxFormat.mbx.txt \ + MailboxFormat.txt \ + MailboxSettings.txt \ + MboxChildFolders.txt \ + MboxLocking.txt \ + MboxProblems.txt \ + Migration.BincIMAP.txt \ + Migration.Courier.txt \ + Migration.Cyrus.txt \ + Migration.Dsync.txt \ + Migration.Gmail.txt \ + Migration.Linuxconf.txt \ + Migration.MailFormat.txt \ + Migration.Online.txt \ + Migration.Teapop.txt \ + Migration.UW.txt \ + Migration.Vm-pop3d.txt \ + Migration.txt \ + MissingMailboxes.txt \ + Mountpoints.txt \ + NFS.txt \ + Namespaces.txt \ + OSCompatibility.txt \ + POP3Server.txt \ + PasswordDatabase.BSDAuth.txt \ + PasswordDatabase.ExtraFields.AllowNets.txt \ + PasswordDatabase.ExtraFields.Host.txt \ + PasswordDatabase.ExtraFields.NoDelay.txt \ + PasswordDatabase.ExtraFields.NoLogin.txt \ + PasswordDatabase.ExtraFields.Proxy.txt \ + PasswordDatabase.ExtraFields.User.txt \ + PasswordDatabase.ExtraFields.txt \ + PasswordDatabase.IMAP.txt \ + PasswordDatabase.PAM.txt \ + PasswordDatabase.Shadow.txt \ + PasswordDatabase.Static.txt \ + PasswordDatabase.oauth2.txt \ + PasswordDatabase.txt \ + PerformanceTuning.txt \ + Pigeonhole.Installation.txt \ + Pigeonhole.ManageSieve.Clients.txt \ + Pigeonhole.ManageSieve.Configuration.txt \ + Pigeonhole.ManageSieve.Install.txt \ + Pigeonhole.ManageSieve.Troubleshooting.txt \ + Pigeonhole.ManageSieve.txt \ + Pigeonhole.Sieve.Configuration.Dict.txt \ + Pigeonhole.Sieve.Configuration.File.txt \ + Pigeonhole.Sieve.Configuration.LDAP.txt \ + Pigeonhole.Sieve.Configuration.txt \ + Pigeonhole.Sieve.Examples.txt \ + Pigeonhole.Sieve.Extensions.Duplicate.txt \ + Pigeonhole.Sieve.Extensions.Editheader.txt \ + Pigeonhole.Sieve.Extensions.Include.txt \ + Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt \ + Pigeonhole.Sieve.Extensions.Vacation.txt \ + Pigeonhole.Sieve.Extensions.Variables.txt \ + Pigeonhole.Sieve.Extensions.txt \ + Pigeonhole.Sieve.Plugins.Extdata.txt \ + Pigeonhole.Sieve.Plugins.Extprograms.txt \ + Pigeonhole.Sieve.Plugins.IMAPFilterSieve.txt \ + Pigeonhole.Sieve.Plugins.IMAPSieve.txt \ + Pigeonhole.Sieve.Plugins.Pipe.txt \ + Pigeonhole.Sieve.Plugins.txt \ + Pigeonhole.Sieve.Troubleshooting.txt \ + Pigeonhole.Sieve.Usage.txt \ + Pigeonhole.Sieve.txt \ + Pigeonhole.txt \ + Plugins.Apparmor.txt \ + Plugins.Autocreate.txt \ + Plugins.CharsetAlias.txt \ + Plugins.Compress.txt \ + Plugins.Expire.txt \ + Plugins.FTS.Lucene.txt \ + Plugins.FTS.Solr.txt \ + Plugins.FTS.Squat.txt \ + Plugins.FTS.txt \ + Plugins.LastLogin.txt \ + Plugins.Lazyexpunge.txt \ + Plugins.Listescape.txt \ + Plugins.MailCrypt.txt \ + Plugins.MailFilter.txt \ + Plugins.MailLog.txt \ + Plugins.MailboxAlias.txt \ + Plugins.Notify.txt \ + Plugins.NotifyStatus.txt \ + Plugins.PushNotification.txt \ + Plugins.QuotaClone.txt \ + Plugins.Snarf.txt \ + Plugins.Stats.txt \ + Plugins.Trash.txt \ + Plugins.VarExpandCrypt.txt \ + Plugins.Virtual.txt \ + Plugins.Welcome.txt \ + Plugins.Zlib.txt \ + Plugins.txt \ + PostLoginScripting.txt \ + PreAuth.txt \ + QuickConfiguration.txt \ + Quota.Configuration.txt \ + Quota.Count.txt \ + Quota.Dict.txt \ + Quota.Dirsize.txt \ + Quota.FS.txt \ + Quota.Maildir.txt \ + Quota.txt \ + Replication.txt \ + RunningDovecot.txt \ + SSL.CertificateClientImporting.txt \ + SSL.CertificateCreation.txt \ + SSL.DovecotConfiguration.txt \ + SSL.SNIClientSupport.txt \ + SSL.txt \ + Sasl.txt \ + SecurityTuning.txt \ + Services.txt \ + SharedMailboxes.ClusterSetup.txt \ + SharedMailboxes.Permissions.txt \ + SharedMailboxes.Public.txt \ + SharedMailboxes.Shared.txt \ + SharedMailboxes.Symlinks.txt \ + SharedMailboxes.txt \ + SocketUnavailable.txt \ + Statistics.Old.txt \ + Statistics.txt \ + Submission.txt \ + SystemUsers.txt \ + TestInstallation.txt \ + TestPop3Installation.txt \ + TimeMovedBackwards.txt \ + Timeouts.txt \ + Upgrading.1.0.txt \ + Upgrading.1.1.txt \ + Upgrading.1.2.txt \ + Upgrading.2.0.txt \ + Upgrading.2.1.txt \ + Upgrading.2.2.txt \ + Upgrading.2.3.txt \ + Upgrading.txt \ + UserDatabase.ExtraFields.txt \ + UserDatabase.NSS.txt \ + UserDatabase.Prefetch.txt \ + UserDatabase.Static.txt \ + UserDatabase.txt \ + UserIds.txt \ + Variables.txt \ + VirtualUsers.Home.txt \ + VirtualUsers.txt \ + WhyDoesItNotWork.txt \ + maildrop.txt \ + mutt.txt \ + uw2dovecot.sh.txt + +all: all-am + +.SUFFIXES: +$(srcdir)/Makefile.in: @MAINTAINER_MODE_TRUE@ $(srcdir)/Makefile.am $(am__configure_deps) + @for dep in $?; do \ + case '$(am__configure_deps)' in \ + *$$dep*) \ + ( cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh ) \ + && { if test -f $@; then exit 0; else break; fi; }; \ + exit 1;; \ + esac; \ + done; \ + echo ' cd $(top_srcdir) && $(AUTOMAKE) --foreign doc/wiki/Makefile'; \ + $(am__cd) $(top_srcdir) && \ + $(AUTOMAKE) --foreign doc/wiki/Makefile +Makefile: $(srcdir)/Makefile.in $(top_builddir)/config.status + @case '$?' in \ + *config.status*) \ + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh;; \ + *) \ + echo ' cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__maybe_remake_depfiles)'; \ + cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__maybe_remake_depfiles);; \ + esac; + +$(top_builddir)/config.status: $(top_srcdir)/configure $(CONFIG_STATUS_DEPENDENCIES) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh + +$(top_srcdir)/configure: @MAINTAINER_MODE_TRUE@ $(am__configure_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(ACLOCAL_M4): @MAINTAINER_MODE_TRUE@ $(am__aclocal_m4_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(am__aclocal_m4_deps): + +mostlyclean-libtool: + -rm -f *.lo + +clean-libtool: + -rm -rf .libs _libs +install-wikiDATA: $(wiki_DATA) + @$(NORMAL_INSTALL) + @list='$(wiki_DATA)'; test -n "$(wikidir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(wikidir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(wikidir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(wikidir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(wikidir)" || exit $$?; \ + done + +uninstall-wikiDATA: + @$(NORMAL_UNINSTALL) + @list='$(wiki_DATA)'; test -n "$(wikidir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(wikidir)'; $(am__uninstall_files_from_dir) +tags TAGS: + +ctags CTAGS: + +cscope cscopelist: + + +distdir: $(BUILT_SOURCES) + $(MAKE) $(AM_MAKEFLAGS) distdir-am + +distdir-am: $(DISTFILES) + @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + list='$(DISTFILES)'; \ + dist_files=`for file in $$list; do echo $$file; done | \ + sed -e "s|^$$srcdirstrip/||;t" \ + -e "s|^$$topsrcdirstrip/|$(top_builddir)/|;t"`; \ + case $$dist_files in \ + */*) $(MKDIR_P) `echo "$$dist_files" | \ + sed '/\//!d;s|^|$(distdir)/|;s,/[^/]*$$,,' | \ + sort -u` ;; \ + esac; \ + for file in $$dist_files; do \ + if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \ + if test -d $$d/$$file; then \ + dir=`echo "/$$file" | sed -e 's,/[^/]*$$,,'`; \ + if test -d "$(distdir)/$$file"; then \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \ + cp -fpR $(srcdir)/$$file "$(distdir)$$dir" || exit 1; \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + cp -fpR $$d/$$file "$(distdir)$$dir" || exit 1; \ + else \ + test -f "$(distdir)/$$file" \ + || cp -p $$d/$$file "$(distdir)/$$file" \ + || exit 1; \ + fi; \ + done +check-am: all-am +check: check-am +all-am: Makefile $(DATA) +installdirs: + for dir in "$(DESTDIR)$(wikidir)"; do \ + test -z "$$dir" || $(MKDIR_P) "$$dir"; \ + done +install: install-am +install-exec: install-exec-am +install-data: install-data-am +uninstall: uninstall-am + +install-am: all-am + @$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am + +installcheck: installcheck-am +install-strip: + if test -z '$(STRIP)'; then \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + install; \ + else \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + "INSTALL_PROGRAM_ENV=STRIPPROG='$(STRIP)'" install; \ + fi +mostlyclean-generic: + +clean-generic: + +distclean-generic: + -test -z "$(CONFIG_CLEAN_FILES)" || rm -f $(CONFIG_CLEAN_FILES) + -test . = "$(srcdir)" || test -z "$(CONFIG_CLEAN_VPATH_FILES)" || rm -f $(CONFIG_CLEAN_VPATH_FILES) + +maintainer-clean-generic: + @echo "This command is intended for maintainers to use" + @echo "it deletes files that may require special tools to rebuild." +clean: clean-am + +clean-am: clean-generic clean-libtool mostlyclean-am + +distclean: distclean-am + -rm -f Makefile +distclean-am: clean-am distclean-generic + +dvi: dvi-am + +dvi-am: + +html: html-am + +html-am: + +info: info-am + +info-am: + +install-data-am: install-wikiDATA + +install-dvi: install-dvi-am + +install-dvi-am: + +install-exec-am: + +install-html: install-html-am + +install-html-am: + +install-info: install-info-am + +install-info-am: + +install-man: + +install-pdf: install-pdf-am + +install-pdf-am: + +install-ps: install-ps-am + +install-ps-am: + +installcheck-am: + +maintainer-clean: maintainer-clean-am + -rm -f Makefile +maintainer-clean-am: distclean-am maintainer-clean-generic + +mostlyclean: mostlyclean-am + +mostlyclean-am: mostlyclean-generic mostlyclean-libtool + +pdf: pdf-am + +pdf-am: + +ps: ps-am + +ps-am: + +uninstall-am: uninstall-wikiDATA + +.MAKE: install-am install-strip + +.PHONY: all all-am check check-am clean clean-generic clean-libtool \ + cscopelist-am ctags-am distclean distclean-generic \ + distclean-libtool distdir dvi dvi-am html html-am info info-am \ + install install-am install-data install-data-am install-dvi \ + install-dvi-am install-exec install-exec-am install-html \ + install-html-am install-info install-info-am install-man \ + install-pdf install-pdf-am install-ps install-ps-am \ + install-strip install-wikiDATA installcheck installcheck-am \ + installdirs maintainer-clean maintainer-clean-generic \ + mostlyclean mostlyclean-generic mostlyclean-libtool pdf pdf-am \ + ps ps-am tags-am uninstall uninstall-am uninstall-wikiDATA + +.PRECIOUS: Makefile + + +# Tell versions [3.59,3.63) of GNU make to not export all variables. +# Otherwise a system limit (for SysV at least) may be exceeded. +.NOEXPORT: diff --git a/doc/wiki/MboxChildFolders.txt b/doc/wiki/MboxChildFolders.txt new file mode 100644 index 0000000..4c3bde4 --- /dev/null +++ b/doc/wiki/MboxChildFolders.txt @@ -0,0 +1,187 @@ +Mail folders containing both messages and sub-folders +----------------------------------------------------- + +Under mbox, it is not normally possible to have a mail folder which contains +both messages and sub-folders. This is because there would be a filesystem name +collision between the name of the mbox file containing the messages and the +name of the directory containing the sub-folders, for example: + + * Mail folder "foo" containing messages would be stored in a file at + '~/mail/foo'. + * Mail folder "foo/bar" containing messages would be stored in a file at + '~/mail/foo/bar', but this cannot happen because this relies on the + existence of a directory '~/mail/foo/' which can't exist because there is + already a file with that name. + +If however there is a requirement to be able to have a mail folder which +contains both messages and sub-folders, then there are a couple of ways to do +it: + + 1. Maildir++ layout + 2. Messages in named file + +These approaches are described in more detail below. + +Maildir++ layout +---------------- + +Under mbox, Dovecot normally stores mail folders in "filesystem" layout. In +this layout, mail folders are stored in mbox files (potentially under +subdirectories) with the same relative path as the mail folder path, for +example: + + * '~/mail/foo' - mbox file containing mail for mail folder "foo"; can not + create any mail sub-folders of "foo" + * '~/mail/bar/baz' - mbox file containing mail for mail folder "bar/baz"; can + not create any mail sub-folders of "bar/baz" + * ('~/mail/inbox' - mbox file containing mail for INBOX) + +However, Dovecot can be configured to keep mbox mail in a Maildir++-like +layout. This makes Dovecot keep mail in mbox files where all the mailbox folder +naming levels are separated with dots (with a leading dot), for example: + + * '~/mail/.foo' - mbox file containing mail for mail folder "foo" + * '~/mail/.foo.bar' - mbox file containing mail for mail folder "foo/bar". We + can now do this. + * '~/mail/.bar.baz' - mbox file containing mail for mail folder "bar/baz" + * ('~/mail/inbox' - mbox file containing mail for INBOX) + +This can be enabled by adding ':LAYOUT=maildir++' to the mail location, for +example: + +---%<------------------------------------------------------------------------- +# Incomplete example. Do not use! +mail_location = mbox:~/mail:LAYOUT=maildir++ +---%<------------------------------------------------------------------------- + +However there is a problem. Under mbox, setting 'LAYOUT=maildir++' alone leaves +Dovecot unable to place index files, which would likely result in performance +issues. So when using 'LAYOUT=maildir++' with mbox, it is advisable to also +configure 'INDEX'. Now, mail files (other than 'inbox') all have names +beginning with a dot, so if we like we can store other things in the '~/mail' +directory by using names which do not begin with a dot. So we could think to +use 'INDEX' to indexes at '~/mail/index/', for example: + +---%<------------------------------------------------------------------------- +# Incomplete example. Do not use! +mail_location = mbox:~/mail:LAYOUT=maildir++:INDEX=~/mail/index +---%<------------------------------------------------------------------------- + +If we do this, then indexes will be kept at '~/mail/index/' and this will not +clash with any names used for mail folders. There is one more thing we may want +to consider though. By default Dovecot will maintain a list of subscribed +folders in a file '.subscriptions' under the mail location root. In this case +that means it would end up at '~/mail/.subscriptions'. This would then mean +that it would be impossible to create a mail folder called "subscriptions". We +can get around this by using the 'CONTROL' parameter to move the +'.subscriptions' file somewhere else, for example into the directory +'~/mail/control' (again choosing a name which doesn't begin with a dot so we +don't collide with the names of mbox files storing mail folders). That gives +us: + +---%<------------------------------------------------------------------------- +# Trick mbox configuration which allows a mail folder which contains both +# messages and sub-folders +mail_location = +mbox:~/mail:LAYOUT=maildir++:INDEX=~/mail/index:CONTROL=~/mail/control +---%<------------------------------------------------------------------------- + +This then allows mail folders which contains both messages and sub-folders +without possibility of naming collisions between mail folders and other data. + +There is one further wrinkle. Specifying ':LAYOUT=maildir++' for mbox changes +the default hierarchy separator from a slash to a dot. This should not be a +problem for IMAP clients as the hierarchy separator is exposed through IMAP. +However anything which expects to just "know" that the hierarchy separator is a +slash may get confused. This can be worked around by using a <namespace> +[Namespaces.txt] to set the folder separator back to a slash again. + +Messages in named file +---------------------- + +Under mbox, Dovecot normally stores mail folders in "filesystem" layout. In +this layout, mail folders are stored in mbox files (potentially under +subdirectories) with the same relative path as the mail folder path, for +example: + + * '~/mail/foo' - mbox file containing mail for mail folder "foo"; can not + create any mail sub-folders of "foo" + * '~/mail/bar/baz' - mbox file containing mail for mail folder "bar/baz"; can + not create any mail sub-folders of "bar/baz" + * ('~/mail/inbox' - mbox file containing mail for INBOX) + +In the example above, we can't create any sub-folders of "foo" because there is +a file 'foo' in the way. So we could think to get rid of that file and put a +directory there instead. But if we do that then we need somewhere to put the +messages for folder "foo". We could think to put them in a specially-named file +in the directory 'foo/'. Then if we wanted to create a sub-folder of "foo" we +would be fine because we could then do that. The rule would then be that +messages go into the specially-named file in the directory corresponding to the +mail folder name. We want want to choose a special name which would be unlikely +to collide with a folder name. We could think to use something like +'mBoX-MeSsAgEs'. Now, it turns out that you can configure Dovecot to do this +using the 'DIRNAME' parameter. For example, using a configuration of: + +---%<------------------------------------------------------------------------- +# Incomplete example. Do not use! +mail_location = mbox:~/mail:DIRNAME=mBoX-MeSsAgEs +---%<------------------------------------------------------------------------- + +we would get a layout like this: + + * '~/mail/inbox' - mbox file containing mail for INBOX + * '~/mail/foo/mBoX-MeSsAgEs' - mbox file containing mail for mail folder "foo" + + * '~/mail/foo/bar/mBoX-MeSsAgEs' - mbox file containing mail for mail folder + "foo/bar" + +However there is a problem. Under mbox, setting 'DIRNAME' alone leaves Dovecot +unable to place index files, which would likely result in performance issues, +or worse, if the index directory gets created first, this will obstruct the +creation of the mbox file. So when using 'DIRNAME' with mbox, it is also +necessary to configure 'INDEX'. The question then arises where to put index +files. + +Any directory under the '~/mail' directory could be considered as a mail +folder. We could think to use a name beginning with a dot, for example +'~/mail/.index' but that would then mean that it would not be possible to +create a mail folder called ".index"; unlikely, but it would be nice to have as +few implementation-specific restrictions as possible. + +In addition, by default, Dovecot will create a file '.subscriptions' at the +mail location root to hold a list of mailbox subscriptions. This would make it +impossible to create a mail folder called ".subscriptions". But we can move the +'.subscriptions' file to another directory by using the 'CONTROL' parameter. To +get round these issues, we can add another directory layer which separates +these purposes. For example, using the configuration: + +---%<------------------------------------------------------------------------- +# Trick mbox configuration which allows a mail folder which contains both +# messages and sub-folders +mail_location = +mbox:~/mail/mailboxes:DIRNAME=mBoX-MeSsAgEs:INDEX=~/mail/index:CONTROL=~/mail/control +---%<------------------------------------------------------------------------- + +would result in the following layout: + + * '~/mail/mailboxes/foo/mBoX-MeSsAgEs' - mbox file containing messages for + mail folder "foo" + * '~/mail/mailboxes/foo/bar/mBoX-MeSsAgEs' - mbox file containing messages for + mail folder "foo/bar" + * '~/mail/mailboxes/inbox' - mbox file containing messages for INBOX + * '~/mail/control/.subscriptions' - file containg list of subscribed mailboxes + + * '~/mail/index/INBOX/dovecot.index.*' - index files for INBOX + * '~/mail/index/foo/dovecot.index.*' - index files for mail folder "foo" + * '~/mail/index/foo/bar/dovecot.index.*' - index files for mail folder + "foo/bar" + * '~/mail/index/dovecot.mailbox.log' - other index files + +Restrictions on mail folder names are then minimised; we can't have mail +folders with the names "mBoX-M ''eSsAgEs", "dovecot.index.*, or +"dovecot.mailbox.log". + +Unlike the Maildir++ layout approach above, because we are still using +"filesystem" layout, the hierarchy separator remains as a slash. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MboxLocking.txt b/doc/wiki/MboxLocking.txt new file mode 100644 index 0000000..47c7c4e --- /dev/null +++ b/doc/wiki/MboxLocking.txt @@ -0,0 +1,70 @@ +Mbox Locking +============ + +The only standard way to lock an mbox is using a method called "dotlock". This +means that a file named '<mailbox-name>.lock' is created in the same directory +as the mailbox being locked. This works pretty well when the mbox is locked for +writing, but for reading it's very inefficient. That's why other locking +methods have been used. + +*It's important that all software that's reading or writing to mboxes use the +same locking settings.* If they use different methods, they might read/write to +an mbox while another process is modifying it, and see corrupted mails. If they +use the same methods but in a different order, they can both end up in a +deadlock. + +If you want to know more details about locking, see <the mbox format page> +[MailboxFormat.mbox.txt]. + +For Dovecot you can configure locking using the 'mbox_read_locks' and +'mbox_write_locks' settings. The defaults are: + +---%<------------------------------------------------------------------------- +mbox_read_locks = fcntl +mbox_write_locks = dotlock fcntl +---%<------------------------------------------------------------------------- + +Here's a list of how to find out the locking settings for other software: + +Procmail +-------- + +---%<------------------------------------------------------------------------- +# procmail -v 2>&1|grep Locking +Locking strategies: dotlocking, fcntl() +---%<------------------------------------------------------------------------- + +Postfix +------- + +Postfix has two different ways to deliver to mboxes. One is the "mailbox" +transport and another one is the "virtual" transport. + +---%<------------------------------------------------------------------------- +# postconf mailbox_delivery_lock +mailbox_delivery_lock = fcntl, dotlock +# postconf virtual_mailbox_lock +virtual_mailbox_lock = fcntl +---%<------------------------------------------------------------------------- + +In the above case, if you used the "mailbox" transport, you'd have to change +Dovecot's configuration to 'mbox_write_locks = fcntl dotlock' or vice versa for +Postfix. + +If you used the "virtual" transport, it doesn't really matter if the "dotlock" +is missing, since the "fcntl" is common with Dovecot and Postfix. + +mutt +---- + +---%<------------------------------------------------------------------------- +mutt -v|grep -i lock +---%<------------------------------------------------------------------------- + +Debian +------ + +Debian's policy specifies that all software should use "fcntl and then dotlock" +locking, but this probably applies only to most commonly used software. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MboxProblems.txt b/doc/wiki/MboxProblems.txt new file mode 100644 index 0000000..3b646b3 --- /dev/null +++ b/doc/wiki/MboxProblems.txt @@ -0,0 +1,134 @@ +mbox problems +============= + +External modifications +---------------------- + +In general Dovecot doesn't mind if you modify the mbox file externally. It's +fine if external software expunges messages or appends new ones. However moving +around existing messages, inserting messages in the middle of the file or +modifying existing messages isn't allowed. + +Especially modifying existing messages (eg. removing attachments) may cause all +kinds of problems. If you do that, at the minimum go and delete +'dovecot.index.cache' file from the mailbox, otherwise weird things may happen. +However IMAP protocol guarantees that messages don't change at all, and +deleting Dovecot's cache file doesn't clear clients' local caches, so it still +may not work right. + +If you insert messages, or if you "undelete" messages (eg. replace mbox from a +backup), you may see errors in Dovecot's logs: + +---%<------------------------------------------------------------------------- +mbox sync: UID inserted in the middle of mailbox /home/tss/mail/inbox (817 > +787, seq=18, idx_msgs=32) +---%<------------------------------------------------------------------------- + +This is normal. Dovecot just assigned new UIDs for the messages. See below for +other reasons why UID insertions could happen. + +Debugging UID insertions +------------------------ + +The above error message can be read as: "18th message in the mbox file +contained X-UID: 787 header, however the index file at that position told the +message was supposed to have UID 817. There are 32 messages currently in the +index file." + +There are four possibilities why the error message could happen: + + 1. Message with a X-UID: 787 header really was inserted in the mbox file. For + example you replaced mbox from a backup. + 2. Something changed the X-UID headers. I don't think any existing software + can cause this. + 3. The message was expunged from the index file, but for some reason it wasn't + expunged from the mbox file. The index file is updated only after a + successful mbox file modification, so this shouldn't really happen either. + 4. If this problem happens constantly, it could mean that you're sharing the + same index file for multiple different mboxes! + * This could happen if you let Dovecot do mailbox autodetection and it + sometimes uses '/var/mail/%u' (when it exists) and other times + '~/mail/inbox'. Use an explicit <mail_location> [MailLocation.txt] + setting to make sure the same INBOX is used. + * Another possibility is that you're sharing index files between multiple + users. Each user must have their own home directory. + +It's possible that broken X-UID headers in mails and 'mbox_lazy_writes=yes' +combination has some bugs. If you're able to reproduce such an error, please +let me know how. Dovecot versions earlier than 1.0.rc27 have some known bugs. + +UIDVALIDITY changes +------------------- + +UIDVALIDITY is stored in X-IMAPbase: or X-IMAP: header of the first message in +mbox file. This is done by both Dovecot and UW-IMAP (and Pine). It's also +stored in 'dovecot.index' file. It shouldn't normally change, because if it +does it means that client has to download all the messages for the mailbox +again. + +If the UIDVALIDITY in mbox file doesn't match the one in 'dovecot.index' file, +Dovecot logs an error: + +---%<------------------------------------------------------------------------- +UIDVALIDITY changed (1100532544 -> 1178155834) in mbox file +/home/user/mail/mailbox +---%<------------------------------------------------------------------------- + +This can happen when the following happens: + + 1. Dovecot accesses the mailbox saving the current UIDVALIDITY to + 'dovecot.index' file. + 2. The UIDVALIDITY gets lost from the mbox file + * X-IMAP: or X-IMAPbase: header gets lost because something else than + Dovecot or UW-IMAP deletes the first message + * The whole file gets truncated + * Something else than Dovecot deletes or renames the mbox + 3. The mailbox is accessed (or created if necessary) by UW-IMAP or Pine. It + notices that the mailbox is missing UIDVALIDITY, so it assigns a new + UIDVALIDITY and writes the X-IMAPbase: or X-IMAP: header. + * Also Dovecot that's configured to not use index files behaves the same. + 4. Dovecot accesses again the mailbox. UIDVALIDITY in the mbox file's header + doesn't match the one in 'dovecot.index' file. It logs an error and updates + the UIDVALIDITY in the index file to the new one. + +Crashes +------- + +Dovecot's mbox code is a bit fragile because of the way it works. However +instead of just corrupting the mbox file, it usually assert-crashes whenever it +notices an inconsistency. You may see crashes such as: + +---%<------------------------------------------------------------------------- +Panic: mbox /home/user/mail/mailbox: seq=2 uid=45 uid_broken=0 originally +needed 12 bytes, now needs 27 bytes +---%<------------------------------------------------------------------------- + +This is a bit difficult problem to fix. Usually this crash has been related to +Dovecot rewriting some headers that were broken. If you see these crashes, it +would really help if you were able to reproduce the crash. + +If you have such a mailbox which crashes every time when it's tried to be +opened, please put the mbox through mbox anonymizer +[http://dovecot.org/tools/mbox-anonymize.pl] and send it, the mailbox's +'dovecot.index' and 'dovecot.index.log' files to tss@iki.fi. None of those +files contain any actual message contents so it's be safe to send them. + +Avoiding crashes and errors +--------------------------- + +Since the problems usually have been related to broken headers, you should be +able to avoid them by filtering out all the Dovecot's internal metadata +headers. This is a good idea to do in any case. If you use <Dovecot LDA> +[LDA.txt] it does this filtering automatically. Otherwise you could do this in +your SMTP server. The headers that you should filter out are: + + * Content-Length + * Status + * X-IMAP + * X-IMAPbase + * X-Keywords + * X-Status + * X-UID + * X-UIDL (if you're using 'pop3_reuse_xuidl=yes') + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.BincIMAP.txt b/doc/wiki/Migration.BincIMAP.txt new file mode 100644 index 0000000..6a42ba3 --- /dev/null +++ b/doc/wiki/Migration.BincIMAP.txt @@ -0,0 +1,349 @@ +Binc IMAP +========= + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read <Migration.txt> page first carefully.* + +If you're using only Binc IMAP, it's possible to do a transparent Dovecot +migration. + +Binc IMAP v1.2 and later +------------------------ + +binc2dovecot.pl attempts to do as perfect migration as possible. Basically it +reads Binc's uidlist files from the specified maildir and it's (sub-)folders +and generates 'dovecot-uidlist' files out of them. It also converts Binc's +subscription file. This script hasn't been tested with Binc IMAP< 1.2. + +If Binc has been used with the IMAPdir depot format, it need to be converted to +Maildir++ with the script IMAPdir2Maildir++ before running binc2dovecot.pl. + +IMAPdir2Maildir++ + +---%<------------------------------------------------------------------------- +#!/bin/bash +# BINC mailbox definition for the example parameters below +# Mailbox { +# depot = "IMAPdir", +# type = "Maildir", +# path = "Maildir", +# } + +# +# Parameters: set according to your local system settings +# + +# Path to the IMAPdir +IMAPdirName="${1}/Maildir" + +# Path to the new Maildir++ directory +maildirName="${1}/Maildir" + +# Name used for the inbox with IMAPdir +inboxName="INBOX" + +# the character . is invalid for Maildir++ +# What string shall it be replaced with? +dotReplacementString="_cdot_" + +# +# Below here nothing should need to be adjusted +# + +# Initialise some variables and settings +shopt -s dotglob +orgDir=$(pwd) + +# loop through all file names according to the pattern in $FILES +cd $IMAPdirName +FILES="*" +for file in $FILES +do + # Skip over Maildir++ directories + if [[ ${file:0:1} = "." && -e ${file}/maildirfolder ]] + then + continue + fi + # Skip over non-directory file names + if [[ ! -d $file ]] + then + continue + fi + + # Move INBOX contents according to Maildir++ specification + if [[ $file = $inboxName ]] + then + mv ${file}/* $maildirName + rmdir $file + continue + fi + + # create Maildir++ compliant new folder name + newFile=${file//\\\\/\\} + newFile=${newFile//\\./$dotReplacementString} + newFile=${newFile/#./$dotReplacementString} + newFile=.$newFile + + # rename folder name according to Maildir++ specification & add maildirfolder +file + mv "$file" "${maildirName}/$newFile" + owner=$(stat -c %u "${maildirName}/$newFile") + group=$(stat -c %g "${maildirName}/$newFile") + touch "${maildirName}/$newFile/maildirfolder" + chown $owner:$group "${maildirName}/$newFile/maildirfolder" + chmod 600 "${maildirName}/$newFile/maildirfolder" +done + +# Adapt subscriptions file +mv .bincimap-subscribed "${maildirName}/.bincimap-subscribed" +sed -i "s/\./$dotReplacementString/g" "${maildirName}/.bincimap-subscribed" + +# Return to original working directory +cd $orgDir +---%<------------------------------------------------------------------------- + +Usage: ./IMAPdir2Maildir++ /path/to/user + +binc2dovecot.pl + +---%<------------------------------------------------------------------------- +#!/usr/bin/perl + +use IO::File; +use IO::Dir; +use File::stat; +use File::Basename; +use strict; + +### Parameters to adapt to local cofiguration +# Name of the Maildir++ directory relative to the path passed as argument +my $mailbox = $ARGV[0]."/Maildir"; +# Name space used by BINC for private folders, with IMAPdir often = "" +our $namespace = "INBOX."; + +### Nothing should need to be changed below here +our $indent = 0; + +die("Mailbox doesn't exist") + if (!-d $mailbox); +parse_mailbox($mailbox); + +# Sanity check for namespace +my $subscriptionsSize = -s $mailbox.'/subscriptions'; +if ($subscriptionsSize == 0) { + print $/; + print $/; + print "WARNING: Your new subscriptions file is empty. Are you using the +correct namespace? If not re-run script with correct namespace parameter.", $/; +} + +sub parse_mailbox +{ + my ($mailbox) = @_; + print " " x $indent, "Parsing ", $mailbox, " ...", $/; + $indent += 2; + + my $mb = IO::Dir->new($mailbox) + or die("Unable to open mailbox $mailbox"); + while(my $file = $mb->read()) + { + my $absfile = $mailbox."/".$file; + next + if ($file eq "." || $file eq ".."); + if ($file eq ".bincimap-subscribed" && -f $absfile) + { + convert_subscribtions($absfile); + } + elsif ($file eq "bincimap-cache" && -f $absfile) + { + convert_cache($absfile); + } + elsif (substr($file, 0, 1) eq "." && -d $absfile && -e +$absfile."/maildirfolder") + { + parse_mailbox($absfile); + } + } + $mb->close(); + + $indent -= 2; + return 1; +} + +sub convert_cache +{ + my ($infile) = @_; + my $dir = dirname($infile); + my %uids = (); + + print " " x $indent, "Converting cache...", $/; + my $in = IO::File->new("<".$infile) + or die("Unable to open cache file $infile"); + my ($blockopen, $uid) = (0, 0); + my $id = ""; + while(my $line = $in->getline()) + { + if ($line =~ /^\d+\s{$/) + { + $blockopen = 1; + $uid = 0; + $id = ""; + next; + } + elsif ($blockopen && $line =~ /^}$/) + { + $blockopen = 0; + next; + } + elsif ($blockopen && $line =~ /^\t_UID\s=\s(\d+),?$/) + { + $uid = $1; + } + elsif ($blockopen && $line =~ /^\t_ID\s=\s"?(.*?)"?,?$/) + { + $id = $1; + } + if ($uid > 0 && length($id) > 0) + { + $uids{$uid} = $id; + $uid = 0; + $id = ""; + next; + } + } + $in->close(); + + if (scalar(keys(%uids)) <= 0) + { + print " " x $indent, "Empty uidlist. Skipping...", $/; + return 1; + } + + my $uidvalfile = $dir."/bincimap-uidvalidity"; + my ($uidvalidity, $uidnext) = (0, 0); + die("Error: File $uidvalfile doesn't exist") + if (!-f $uidvalfile); + $in = IO::File->new("<".$uidvalfile) + or die("Unable to open file: $uidvalfile"); + while(my $line = $in->getline()) + { + if ($line =~ /^\t_uidvalidity\s=\s(\d+),?$/) + { + $uidvalidity = $1; + } + elsif ($line =~ /^\t_uidnext\s=\s(\d+),?$/) + { + $uidnext = $1; + } + } + $in->close(); + + die("Error: either uidnext ($uidnext) or uidvalidity ($uidvalidity) is +invalid") + if ($uidnext <= 0 || $uidvalidity <= 0); + + my $version = 1; + my $outfile = $dir."/dovecot-uidlist";; + my $out = IO::File->new(">".$outfile) + or die("Unable to create cache file $outfile"); + $out->print($version, " ", $uidvalidity, " ", $uidnext, $/); + foreach my $uid (sort{$a <=> $b} (keys(%uids))) + { + $out->print($uid, " ", $uids{$uid}, $/); + } + $out->close(); + + my $stat = stat($infile); + chown($stat->uid, $stat->gid, $outfile); + chmod(0600, $outfile); + + return 1; +} + +sub convert_subscribtions +{ + my ($infile) = @_; + my $dir = dirname($infile); + my @cache = (); + + print " " x $indent, "Converting subscriptions...", $/; + my $in = IO::File->new("<".$infile) + or die("Unable to open file: $infile"); + while(my $line = $in->getline()) + { + next + if ($line !~ /^$namespace/); + $line =~ s/^$namespace?//; + $line =~ s/\n$//; + $line =~ s/\r$//; + $line =~ s/\//\./g; + next + if (length($line) <= 0); + next + if (!-d $dir."/.".$line); + push(@cache, $line) + if (scalar(grep{$_ eq $line}(@cache)) <= 0); + } + $in->close(); + + my $outfile = $dir."/subscriptions"; + my $out = IO::File->new(">".$outfile) + or die("Unable to create subscriptions file: $outfile"); + foreach my $subscription (@cache) + { + $out->print($subscription, $/); + } + $out->close(); + + my $stat = stat($infile); + chown($stat->uid, $stat->gid, $outfile); + chmod(0600, $outfile); + + return 1; +} +---%<------------------------------------------------------------------------- + +Usage: ./binc2dovecot.pl /path/to/user + +NOTE: /path/to/user/Maildir MUST exist. If "./Maildir" isn't your default +maildir-name, you can edit this at the top of the script. + +---%<------------------------------------------------------------------------- +Example 1: +# find /var/pop -mindepth 1 -maxdepth 1 -type d -exec +/path/to/binc2dove.pl {} \; + +Example 2: +# find /usr/local/vpopmail/domains -mindepth 2 -maxdepth 2 -type d -exec +/path/to/binc2dove.pl {} \; +---%<------------------------------------------------------------------------- + +Dovecot configuration +--------------------- + +Binc IMAP by default uses "INBOX/" as the IMAP namespace for private mailboxes. +If you want a transparent migration, you'll need to configure Dovecot to use a +namespace with "INBOX/" prefix as well. + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +namespace { + separator = / + prefix = INBOX/ + inbox = yes +} +---%<------------------------------------------------------------------------- + +Manual conversion +----------------- + + * Binc's '.bincimap-subscribed' file is compatible with Dovecot's + 'subscriptions' file, but you need to remove the "INBOX/" prefixes from the + mailboxes. + * Binc's 'bincimap-cache + bincimap-uidvalidity' are NOT compatible with + Dovecot's 'dovecot-uidlist' file. See file format documention or above + script for conversion. + * Binc's message flags are compatible with Dovecot (as they are specified by + the Maildir specification) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Courier.txt b/doc/wiki/Migration.Courier.txt new file mode 100644 index 0000000..5e87ab5 --- /dev/null +++ b/doc/wiki/Migration.Courier.txt @@ -0,0 +1,87 @@ +Courier IMAP/POP3 +================= + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read <Migration.txt> page first carefully.* + +Courier v0.43 and later to Dovecot v1.1+ +---------------------------------------- + +courier-dovecot-migrate.pl +[https://dovecot.org/tools/courier-dovecot-migrate.pl] does a perfect migration +from Courier IMAP and POP3, preserving IMAP UIDs and POP3 UIDLs. It reads +Courier's 'courierimapuiddb' and 'courierpop3dsizelist' files and produces +'dovecot-uidlist' file from it. + +Before doing the actual conversion you can run the script and see if it +complains about any errors and such, for example: + +---%<------------------------------------------------------------------------- +# ./courier-dovecot-migrate.pl --to-dovecot --recursive /home +Finding maildirs under /home +/home/user/Maildir/dovecot-uidlist already exists, not overwritten +/home/user/Maildir2: No imap/pop3 uidlist files +Total: 69 mailboxes / 6 users + 0 errors +No actual conversion done, use --convert parameter +---%<------------------------------------------------------------------------- + +The actual conversion can be done for all users at once by running the script +with '--convert --recursive' parameters. Make sure the conversion worked by +checking that 'dovecot-uidlist' files were created to all maildirs (including +to subfolders). + +The '--recursive' option goes through only one level down in directory +hierarchies. This means that if you have some kind of a directory hashing +scheme (or even domain/username/), it won't convert all of the files. + +You can also convert each user as they log in for the first time, using +<PostLoginScripting.txt> with a script something like: + +---%<------------------------------------------------------------------------- +#!/bin/sh +# WARNING: Be sure to use mail_drop_priv_before_exec=yes, +# otherwise the files are created as root! + +courier-dovecot-migrate.pl --quiet --to-dovecot --convert Maildir +# This is for imap, create a similar script for pop3 too +exec /usr/local/libexec/dovecot/imap +---%<------------------------------------------------------------------------- + +FIXME: The script should rename also folder names that aren't valid mUTF-7. +Dovecot can't otherwise access such folders. + +Dovecot configuration +--------------------- + +Courier by default uses "INBOX." as the IMAP namespace for private mailboxes. +If you want a transparent migration, you'll need to configure Dovecot to use a +namespace with "INBOX." prefix as well. + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir + +namespace { + prefix = INBOX. + separator = . + inbox = yes +} +---%<------------------------------------------------------------------------- + +See also <Namespaces#Backwards Compatibility: Courier IMAP> [Namespaces.txt]. + +Manual conversion +----------------- + + * Courier's 'courierimapsubscribed' file is compatible with Dovecot's + 'subscriptions' file, but you need to remove the "INBOX." prefixes from the + mailboxes./This is true even if you set namespace prefix to "INBOX." as + described above./ + * Courier's 'courierimapuiddb' file is compatible with Dovecot's + 'dovecot-uidlist' file, just rename it. + * Courier's message flags are compatible with Dovecot (as they are specified + by the Maildir specification) + * Courier's message keywords implementation isn't Dovecot compatible. There + doesn't exist a simple way to convert the keywords manually. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Cyrus.txt b/doc/wiki/Migration.Cyrus.txt new file mode 100644 index 0000000..874c4bb --- /dev/null +++ b/doc/wiki/Migration.Cyrus.txt @@ -0,0 +1,81 @@ +Cyrus +===== + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read <Migration.txt> page first carefully.* + +For POP3 UIDL compatibility, use one of: + +Cyrus versions up to v2.1.3: + ---%<----------------------------------------------------------------------- + # Cyrus (old format - up to v2.1.3) + pop3_uidl_format = %u + ---%<----------------------------------------------------------------------- + +Cyrus versions v2.1.4 and newer: + ---%<----------------------------------------------------------------------- + # Cyrus (new format - v2.1.4 and above) + pop3_uidl_format = %v.%u + ---%<----------------------------------------------------------------------- + +Mail storage migration +---------------------- + +There exists several scripts which can be used to convert Cyrus mail storages +to Maildir. They all read the Cyrus mail directories directly, so they don't +need a running Cyrus installation. + + * cyrus2dovecot [https://github.com/a-schild/cyrus2dovecot] (originally + [http://cyrus2dovecot.sw.fu-berlin.de/] by Freie Universität Berlin) allows + you to perform a server transition which is fully transparent to both POP + and IMAP users, as virtually all available metadata is preserved during the + conversion. This includes message UIDs, INTERNALDATEs, IMAP folder + subscriptions, the UIDVALIDITY and UIDNEXT values for each folder, as well + as all IMAP flags (including the first 26 user-defined keywords). Messages + marked as "delayed expunge" won't be migrated over. It also has support for + sieve migration and virtual domain support.Cyrus2Dovecot is supposed to work + with all Cyrus releases up to (at least) version 2.3.x. So far, it has been + tested with Cyrus 1.4, 2.1.18, 2.2.12, 2.3.12p2, 2.3.1. + * cyrus2courier [http://madness.at/projects/] is Dovecot-compatible. A + non-official v1.6ts release [http://dovecot.org/tools/] works up to Cyrus + v2.3.9. It should be able to preserve message UIDs, INTERNALDATEs, flags and + the first 26 keywords. It works only with the supported Cyrus versions, so + if Cyrus once again changes its internal formats this tool might break + again. + * cyrus2maildir.py [http://www.majid.info/mylos/weblog/2006/03/08-1.html] (for + Cyrus v2.2) preserves (only) INTERNALDATEs and \Seen flags. + * cyrus2dovecot [http://trukenmueller.de/cyrus2dovecot] (by Trukenmüller) + doesn't preserve timestamps or flags. + +You can also do the <migration via IMAP protocol using dsync> +[Migration.Dsync.txt]. + +Migration of passwords +---------------------- + +Some installations of Cyrus store passwords using /Cyrus/ SASL (not to be +confused with other SASL implementations). Passwords are stored in +'/etc/sasldb2', in Berkeley DB format. On Debian, the command 'db4.2_dump -p +/etc/sasldb2' may allow you access to the passwords. This could be +incorporated into a script to copy the passwords to an LDAP directory for use +with other mail servers (e.g. Dovecot). + +For Fedora Core 3 (and probably other versions) the command is just 'db_dump -p +/etc/sasldb2'. + +Namespaces +---------- + +Cyrus uses one of the following namespace configurations depending on the +altnamespace and unixhierarchysep options: + +---%<------------------------------------------------------------------------- +namespace inbox { + prefix = INBOX. # no altnamespace + #prefix = "" # altnamespace + separator = . # no unixhierarchysep + #separator = / # unixhierarchysep +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Dsync.txt b/doc/wiki/Migration.Dsync.txt new file mode 100644 index 0000000..cd3e9a7 --- /dev/null +++ b/doc/wiki/Migration.Dsync.txt @@ -0,0 +1,165 @@ +Migrating from any IMAP/POP3 server to Dovecot via dsync +======================================================== + +You need Dovecot v2.1.4+ for this. + +This page describes how to migrate mails to Dovecot from a remote IMAP/POP3 +server, preserving the IMAP UIDs, POP3 UIDLs and everything else. + +We'll assume that you can log in with master user "master" that has a password +"masteruser-secret". If you can't use master users, you'll need the users' +plaintext passwords and change the appropriate configuration to use them. + +See also server-specific details: + + * <Gmail> [Migration.Gmail.txt] + +Generic configuration +--------------------- + +dsync can be configured with features and workarounds. This is done by using +dsync_features setting in config file, such as + +---%<------------------------------------------------------------------------- +dsync_features = empty-header-workaround +---%<------------------------------------------------------------------------- + + * empty-header-workaround - enables dsync to cope with broken (Zimbra) servers + that sometimes send FETCH replies that are missing all the headers, even + though the mail actually has headers and another FETCH might return them. In + this situation dsync simply assumes for existing mails that the mails are a + match. (v2.2.25.3+) + +In v2.2.30+ dsync can avoid creating a single huge transaction that can fail +all at once. In case of failures the sync can be then done incrementally. + +---%<------------------------------------------------------------------------- +dsync_commit_msgs_interval = 100 # default in v2.2.30+ +---%<------------------------------------------------------------------------- + +In v2.2.33+ you can specify which email headers are used in incremental syncing +for checking whether the local email matches the remote email. This should only +include headers that can be efficiently downloaded from the remote server. The +default is: + +---%<------------------------------------------------------------------------- +dsync_hashed_headers = Date Message-ID +---%<------------------------------------------------------------------------- + +IMAP migration configuration +---------------------------- + +Set up configuration for the IMAP server you wish to migrate from: + +---%<------------------------------------------------------------------------- +imapc_host = imap.example.com + +# Authenticate as masteruser / masteruser-secret, but use a separate login +user. +# If you don't have a master user, remove the imapc_master_user setting. +imapc_user = %u +imapc_master_user = masteruser +imapc_password = masteruser-secret + +imapc_features = rfc822.size +# If you have Dovecot v2.2.8+ you may get a significant performance improvement +with fetch-headers: +imapc_features = $imapc_features fetch-headers +# Read multiple mails in parallel, improves performance +mail_prefetch_count = 20 + +# If the old IMAP server uses INBOX. namespace prefix, set: +#imapc_list_prefix = INBOX + +# for SSL: +#imapc_port = 993 +#imapc_ssl = imaps +#ssl_client_ca_dir = /etc/ssl +#imapc_ssl_verify = yes +# for <2.2.0: change ssl_client_ca_dir=/etc/ssl to imapc_ssl_ca_dir=/etc/ssl +---%<------------------------------------------------------------------------- + +POP3 migration configuration +---------------------------- + +Set up configuration for the POP3 server you wish to migrate from: + +---%<------------------------------------------------------------------------- +pop3c_host = pop3.example.com + +# Authenticate as masteruser / masteruser-secret, but use a separate login +user. +# If you don't have a master user, remove the pop3c_master_user setting. +pop3c_user = %u +pop3c_master_user = masteruser +pop3c_password = masteruser-secret + +# for SSL: +#pop3c_port = 995 +#pop3c_ssl = pop3s +# for <2.2.0: change ssl_client_ca_dir=/etc/ssl to pop3c_ssl_ca_dir=/etc/ssl +#ssl_client_ca_dir = /etc/ssl +#pop3c_ssl_verify = yes + +# make sure you have also the regular inbox namespace defined, e.g.: +#namespace inbox { +# inbox = yes +#} +namespace { + prefix = POP3-MIGRATION-NS/ + location = pop3c: + list = no + hidden = yes +} +protocol doveadm { + mail_plugins = $mail_plugins pop3_migration +} +plugin { + pop3_migration_mailbox = POP3-MIGRATION-NS/INBOX +} +---%<------------------------------------------------------------------------- + +The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is +handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3 +server and figures out which IMAP messages match which POP3 messages and then +returns the appropriate POP3 UIDL. + +Running +------- + +Make sure destination is exactly as source, deleting/reverting any changes in +destination if necessary: + +---%<------------------------------------------------------------------------- +doveadm -o mail_fsync=never backup -R -u user@domain imapc: +---%<------------------------------------------------------------------------- + +or incremental one-way merge (it's ok to do changes on both sides): + +---%<------------------------------------------------------------------------- +doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc: +---%<------------------------------------------------------------------------- + +(Fsyncing is disabled just for migration efficiency.) + +For per-user user/passwords use: + +---%<------------------------------------------------------------------------- +doveadm -o imapc_user=foo -o pop3c_user=foo -o imapc_password=bar -o +pop3c_password=bar backup -R -u user@domain imapc: +---%<------------------------------------------------------------------------- + +Once the users are migrated, remember to remove the pop3_migration plugin from +settings. + +Problems +-------- + + * POP3 message order (when it's different from IMAP message order) is + currently preserved only when destination is Maildir, sdbox or mdbox. + * If source POP3 server merges multiple IMAP mailboxes into one POP3 INBOX, + the migration won't be transparent. + * If source IMAP and POP3 servers return messages somehow differently, + pop3-migration plugin might not be able to match the messages + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Gmail.txt b/doc/wiki/Migration.Gmail.txt new file mode 100644 index 0000000..eccac1a --- /dev/null +++ b/doc/wiki/Migration.Gmail.txt @@ -0,0 +1,73 @@ +Migration from Gmail to Dovecot +=============================== + +You can use <dsync migration via IMAP> [Migration.Dsync.txt] protocol, but +there are a few things different with Gmail compared to other IMAP servers: + + * With Gmail when you delete a mail from POP3, the mail is only hidden from + future POP3 sessions, but it's still available via IMAP. If you wish to + preserve this functionality, there's a 'pop3_deleted_flag' setting in + Dovecot v2.2.2+. + * Gmail has labels. If a message has multiple labels, it shows up in multiple + IMAP folders, but it's still the same message and uses quota only once for + that message. Dovecot currently doesn't have such support, so the migration + will copy the message to multiple folders and each instance will use up + quota. There's currently no easy fix for this, although there are some + future plans to optionally not count message copies towards quota. + * Even though the quota is duplicated, it doesn't mean that the storage + usage has to be duplicated. Use the dsync -v parameter to avoid this. See + https://github.com/dovecot/core/commit/70df8f39fb3db7c49b18c855178f8172176a037a + * Gmail has virtual folders: "All Mail", "Starred" and "Important". From + migration point of view this means that the migration should skip most of + these folders, since their mails are in other folders anyway. With v2.2.3+ + you can tell dsync to skip these folders:'doveadm sync -x '\Flagged' -x + '\Important'' - by using the \flag parameters dsync finds the folders by + their SPECIAL-USE flag rather than their name (which may be different for + different user depending on their language). + * The "All Mail" folder actually contains also "archived mails" that don't + exist in any other folder. These mails need to be migrated. See below. + * Google requires that SSL/TLS be enabled to connect through IMAP. Make sure + that the following are enabled in your Dovecot configuration and set to + appropriate values based on your distribution (usually either one is + enough): + + ---%<---------------------------------------------------------------------- + ssl_client_ca_dir = (''your distribution's trusted TLS CA store (Fedora / + CentOS / Redhat uses /etc/pki/tls/ ))'' + ssl_client_ca_file = (''your distribution's trusted TLS CA file (Fedora / + CentOS / Redhat uses /etc/pki/tls/cert.pem ))'' + ---%<---------------------------------------------------------------------- + +GMail Migration Feature (v2.2.16+) +---------------------------------- + +There's a new 'imapc_features=gmail-migration' setting that helps with this +migration. It will: + + * Set the pop3_deleted_flag to mails that no longer exist in POP3 + * Return POP3 UIDL in GMail format so dsync can preserve it. + * Add a new $GmailHaveLabels keyword to archived mails in the \All mailbox, + which means those mails are not archived. You probably don't want to migrate + these mails. + * Note that mails in the \Important and \Flagged mailboxes are marked with + "\Important" and "\Starred" labels. If you don't migrate mails that have + $GmailHaveLabels then you must not exclude the \Flagged and \Important + mailboxes or some of the mails won't be migrated. + +For example use a command line: + +v2.2.16: + +---%<------------------------------------------------------------------------- +doveadm backup -a 'virtual/All' -F '-$GmailHaveLabels' -R -u user@domain imapc: +---%<------------------------------------------------------------------------- + +Unfortunately -F parameter isn't working in v2.2.17 & v2.2.18 because of +another parameter collision caused -F to be accidentally used for another +purpose. For v2.2.19+ use: + +---%<------------------------------------------------------------------------- +doveadm backup -a 'virtual/All' -O '-$GmailHaveLabels' -R -u user@domain imapc: +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Linuxconf.txt b/doc/wiki/Migration.Linuxconf.txt new file mode 100644 index 0000000..8c74d62 --- /dev/null +++ b/doc/wiki/Migration.Linuxconf.txt @@ -0,0 +1,64 @@ +Linuxconf Migration +=================== + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read <Migration.txt> page first carefully.* + +For those of you who still remember it and use it, Linuxconf +[http://www.solucorp.qc.ca/linuxconf/] and its patched UW-IMAP companion VIMAP +[http://vimap.sourceforge.net/] had a really simple way of doing virtual +domains. They use standard passwd/shadow files, except each domain has its own +files, such as: + +---%<------------------------------------------------------------------------- +/etc/vmail/passwd.domain1.com +/etc/vmail/shadow.domain1.com +/etc/vmail/passwd.domain2.com +/etc/vmail/shadow.domain2.com +---%<------------------------------------------------------------------------- + +The mail is stored in <mbox format> [MailboxFormat.mbox.txt]. The INBOX exists +in '/var/spool/vmail/domain.com/user' and the other mailboxes in +'/vhome/domain.com/home/user'. + +To make Dovecot Linuxconf compatible, use these settings: + +---%<------------------------------------------------------------------------- +# passwd-like file for use with Linuxconf virtual domains +passdb { + driver = passwd-file + # Path for passwd-file + args = /etc/vmail/shadow.%d +} +userdb { + driver = passwd-file + # Path for passwd-file + args = /etc/vmail/passwd.%d +} + +mail_location = mbox:%h:INBOX=/var/spool/vmail/%d/%n +# Or if you want to place Dovecot's index/cache files to a separate directory +# to avoid adding them to backups: +#mail_location = +mbox:%h:INBOX=/var/spool/vmail/%d/%n:INDEX=/nobackup/imap-indexes/%d/%n +# Note that you may need to modify imap-indexes directory's permissions so +# that Dovecot can create the directories. +---%<------------------------------------------------------------------------- + +You also need to rename all the '.mailboxlist' files to '.subscriptions' to +preserve the mailbox subscriptions. + +The Linuxconf virtual email system is actually pretty good especially if you +are merging several existing single domain servers into one virtual domain +server. All you have to do is copy over your existing passwd/shadow files into +the /etc/vmail folder and rename them. You will need to do some editing on the +passwd file to point to where you wish to store your email. + +Exim works very well with this configuration. The Exim Wiki has a detailed +description of how to configure Exim [http://www.exim.org/eximwiki/Linuxconf] +to work with this virtual user structure. + +More information about Linuxconf can be found at their Home Site +[http://www.solucorp.qc.ca/linuxconf/]. Exim [http://www.exim.org] info here. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.MailFormat.txt b/doc/wiki/Migration.MailFormat.txt new file mode 100644 index 0000000..5fb1921 --- /dev/null +++ b/doc/wiki/Migration.MailFormat.txt @@ -0,0 +1,247 @@ +Converting between mailbox formats +================================== + +If you want a transparent migration, the biggest problem is preserving message +UIDs. See <Migration.txt> for the problems this may cause. If you do the +conversion with dsync, it preserves the UIDs. + +dsync +----- + +With dsync you can convert between any two mailbox formats that Dovecot +supports. As much of the mailbox state is preserved as possible. Typically it's +everything. See <Tools.Dsync.txt> for full documentation, here are only a +couple of examples: + + * mbox -> maildir migration. Set 'mail_location=maildir:~/Maildir' and run + 'dsync -u username mirror mbox:~/mail:INBOX=/var/mail/username' + * maildir -> mdbox migration. Set 'mail_location=mdbox:~/mdbox' and run 'dsync + -u username mirror maildir:~/Maildir' + * maildir -> sdbox migration. Set 'mail_location=sdbox:~/sdbox' and run 'dsync + -u username mirror maildir:~/Maildir' + +If you can successfully use dsync, you can skip the rest of this page. + +Converting from mbox to Maildir +------------------------------- + + * mb2md.pl with Dovecot modifications [http://dovecot.org/tools/mb2md.pl] can + convert mails, preserving UIDs and keywords. + * See also [attachment:migrateuser.sh script to drive the full migration + for a user]. + * This script requires patched 'mailutil' that supports Maildir. One + working 'mailutil' binary is RHEL4 PINE RPM + [http://dag.wieers.com/rpm/packages/pine/pine-4.64-3.el4.rf.i386.rpm] + from the DAG RPM Repository [http://dag.wieers.com/rpm/]. You could + also patch any version of pine/c-client and patch it with the Maildir + patches from + http://staff.washington.edu/chappa/pine/info/maildir.html. + * mb2md.py [http://dovecot.org/list/dovecot/2008-March/029736.html] can + convert also message UIDs. + * Yet another way to fix the UIDL migration problem. If you can generate an + uildlist ("messagenumber uidl" pairs), use my new -U uidllist option to + inject X-UILD: headers in the converted Maildir-file. The modified mb2md + script is avalable here:mb2md.xuidl.pl.gz + [http://www.chaos.dk/~sch/mb2md.xuidl/mb2md.xuidl.pl.gz]. I used this to + convert a cucipop installation to dovecot. pop3_reuse_xuidl=yes will do the + rest./-- <SoerenSchroeder.txt> 2009-02-13/ + +Check also the *User-Contributed Maildir Support* section on the qmail +community site [http://www.qmail.org/top.html#usersoft] for more choices. + +Example (user's mail in '~someuser/mail' and INBOX in '/var/mail/someuser'): + +---%<------------------------------------------------------------------------- +cd ~someuser +mb2md-3.20.pl -s mail -R +mb2md-3.20.pl -m -s /var/mail/someuser +mv mail mail.old +---%<------------------------------------------------------------------------- + +Now the mail will be in '~someuser/Maildir'. Do not forget to migrate the +subscriptions as well, otherwise the new maildir will seem to have only an +inbox when viewed through a mail client that supports them. This can be as +simple as copying the old '~someuser/mail/.subscriptions' file to +'~someuser/Maildir/subscriptions' (warning: I have not tested this extensively, +my subscription list and folder hierarchy was very simplistic). + +Hierarchy separator change +-------------------------- + +The default hierarchy separator with Maildir is '.' instead of '/' which is +common with mboxes. To keep the migration transparent to users, you can keep +the '/' separator by using <namespaces> [Namespaces.txt]. In any case you need +to replace the '/' with '.' in the subscriptions file: + + * ---%<---------------------------------------------------------------------- + sed 's:/:.:g' subscriptions > subscriptions.new + mv subscriptions.new subscriptions + ---%<---------------------------------------------------------------------- + +UW-IMAP's subscriptions file is in '~/.mailboxlist'. Dovecot's mbox +subscriptions is in '<mbox root dir>/.subscriptions'. Dovecot's Maildir +subscriptions is in '<maildir root>/subscriptions'. + +Also if you're migrating from UW-IMAP, you probably had "mail/" prefixes in the +mailbox names. You can again use <namespaces> [Namespaces.txt] to let clients +use the prefix, or you can tell your users to remove the namespace prefix from +their clients and change the subscriptions file: + +---%<------------------------------------------------------------------------- +sed 's/^mail\.//' subscriptions > subscriptions.new +mv subscriptions.new subscriptions +---%<------------------------------------------------------------------------- + +Note that because Maildir uses '.' as the hierarchy separator in filesystem, +it's not possible to have mailbox names containing '.' characters, even if you +changed the separator in namespaces. If you really want to have dots, the only +way to do this is by modifying the filesystem separator in 'MAILDIR_FS_SEP' and +'MAILDIR_FS_SEP_S' defines in 'src/lib-storage/index/maildir/maildir-storage.h' +file in the sources. Do not be tempted to change 'MAILDIR_FS_SEP' et al to +'/'; it won't work. + +Converting from Maildir to mbox +------------------------------- + +This is especially helpful if you want to archive your mail to a single file +for storage on a CD, a PC, etc. But it can also be helpful if you want to use +<mbox> [MailboxFormat.mbox.txt] with Dovecot. + +Use the reformail program that comes with maildrop +[http://www.courier-mta.org/maildrop/]. You can also use the formail program +that comes with procmail [http://www.procmail.org/]. Here is a simple script +showing how this works. + +To use it, adjust the script to invoke the right command according to your +system. + +Then 'cd' to the user's home directory (one level above 'Maildir') and run the +script with two arguments: the mailbox name (You can use "." for the top-level +folder), and the output mbox filename, for example: + +---%<------------------------------------------------------------------------- +cd ~hans +perl dw-maildirtombox.pl . >/tmp/hans-inbox +perl dw-maildirtombox.pl Sent >/tmp/hans-sent +---%<------------------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +#!/usr/bin/env perl +# dw-maildirtombox.pl +# dw = Dovecot Wiki :-) +# NOTE! The output file must not contain single quotes (')! +# figure out which program to run +$cmd="reformail -f1"; +system("$cmd </dev/null >/dev/null 2>/dev/null") == 0 or $cmd="formail"; +system("$cmd </dev/null >/dev/null 2>/dev/null") == 0 +or die "cannot find reformail or formail on your \$PATH!\nAborting"; +$dir=$ARGV[0]; +$outputfile=$ARGV[1]; +if (($outputfile eq '') || ($dir eq '')) +{ die "Usage: ./archivemail.pl mailbox outputfile\nAborting"; } +if (!stat("Maildir/$dir/cur") || !stat("Maildir/$dir/new")) +{ die "Maildir/$dir is not a maildir.\nAborting"; } +@files = (<Maildir/$dir/cur/*>,<Maildir/$dir/new/*>); +foreach $file (@files) { + next unless -f $file; # skip non-regular files + next unless -s $file; # skip empty files + next unless -r $file; # skip unreadable files + $file =~ s/'/'"'"'/; # escape ' (single quote) + $run = "cat '$file' | $cmd >>'$outputfile'"; + system($run) == 0 or warn "cannot run \"$run\"."; +} +---%<------------------------------------------------------------------------- + +Converting from MBX to Maildir +------------------------------ + +See the uw2dovecot.pl +[http://wiki.dovecot.org/Migration/UW?action=AttachFile&do=view&target=uw2dovecot.pl] +as mentioned on the <Migration.UW.txt> page. + +Converting from MBX to mbox +--------------------------- + +If you are using UW-IMAP and using the <MBX format> [MailboxFormat.mbx.txt], +you will need to convert it to <mbox> [MailboxFormat.mbox.txt] format. The +conversion process isn't pretty, but here is a script that works. You will need +to get and compile the mailutil program from the UW-IMAP web site +[http://www.washington.edu/imap/]. + +---%<------------------------------------------------------------------------- +#! /bin/sh +# Written by Marc Perkel - public domain +# overhauled by Matthias Andree, 2006 +# Usage: mbx-convert <filename> +# This code assumes there a user named "marc" with the primary group "marc". +# Change to any real user on your system. +# Yes - it look bizzare - but it gets the job done +# abort on error +set -e +user=marc +group=marc +homedir=/home/$user +if [ $# -ne 1 ] ; then + echo >&2 "Usage: $0 <filename>" + exit 1 +fi +# set up automatic cleanup +trap 'rm -f "${homedir}"/in.$$ "${homedir}"/out.$$' 0 +# First copy to users home dir and make the user the owner +cp "$1" "${homedir}/in.$$" +chown "$user":"$group" "${homedir}/in.$$" +# Run mailutil to convert as the user other than root +# mailutil requires this +su "$user" -c "mailutil copy in.$$ \#driver.unix/out.$$" +# create new file with same permissions/owner as old +cp -p "$1" "${1}.new" +# cat instead of copy leaves the original owner and permissions alone +if cat "${homedir}/out.$$" >"${1}.new" ; then + # cat succeeded, rename file into place + mv "${1}.new" "$1" +else + # cat failed, remove temp file + rm -f "${1}.new" + exit 1 +fi +---%<------------------------------------------------------------------------- + +Make a copy of some folders and test it first. Once you are satisfied that it +works then: + + * Write a script to convert all your files. + * Shut down your email system so files can't be modified + * Copy all your email to a backup location in case you have to revert + * Run the script + * Turn on Dovecot and test to verify it is working + * Important - make sure that you changed your SMTP configuration to write mbox + and not MBX. + * Turn on SMTP and verify it is all working + +*User comments:* + + * Is this hassle actually necessary? I have run mailutil as root like this + (mailutil as provided by the PINE 4.61 package for SUSE Linux + 10.0):'mailutil copy /tmp/foo.mbx.orig '#driver.unix//tmp/foo.test'' and did + not encounter any problems./-- <MatthiasAndree.txt>, 2006-05-18/ + * I did the same (using 'mailutil') but it doesn't maintain UIDs or + UIDVALIDITY. So I hacked[attachment:mbx2mbox.tgz this] together to do the + migration./-- <JulianFitzell.txt>, 2008-08-02/ + * Tried to do 'mailutil -v copy /tmp/foo '#driver.unix'/tmp/foo.unix', but + mailutil argues 'Can't open mailbox /tmp/foo: no such mailbox'. (mailutil as + from the Debian package uw-mailutils in Debian 5.0 Lenny) strace shows that + it searches for /tmp/foo/cur, i.e. a Maildir format mailbox. (WTF?) No idea + yet how to get this working. And I'm really glad when I got rid of MBX. -- + <AxelBeckert.txt>, 2009-06-19 + * As Mark Crispin always says: Don't use UW-IMAP with buggy (Maildir) + patches, compile your own. + * Same problem here - just reinstalled uw-imapd, used thunderbird to create + a "normal" mailfolder and copied the old folder contents - F.Fernandez + * At least with ubuntu mailutil from uw-mailutils it starts search from + root *ALWAYS* and if you start with slash it tries to convert from + maildir format 'mailutil -v copy /tmp/foo '#driver.unix'/tmp/foo.unix' = + convert maildir formated folder /tmp/foo/cur to mbox and 'mailutil -v + copy tmp/foo '#driver.unix'/tmp/foo.unix' = convert /tmp/foo file to mbox + - Manwe, 2010-03-09 + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Online.txt b/doc/wiki/Migration.Online.txt new file mode 100644 index 0000000..83d8bb8 --- /dev/null +++ b/doc/wiki/Migration.Online.txt @@ -0,0 +1,86 @@ +Online Mailbox Format Conversion +-------------------------------- + +This page aims to help with the tasks around planned mailbox migrations to e.g. +mdbox. To perform this exercise with minimum service interruption a Postfix +check can be implemented to put mail for the accounts to be migrated on HOLD: + +Postfix main.cf +--------------- + +---%<------------------------------------------------------------------------- +smtpd_recipient_restrictions = + [...] + check_recipient_access hash:/etc/postfix/recipient_maintenance, + permit_mynetworks, + reject_unauth_destination +---%<------------------------------------------------------------------------- + +/etc/postfix/recipient_maintenance +---------------------------------- + +---%<------------------------------------------------------------------------- +user@domain.tld HOLD Planned maintenance +user1@domain.tld HOLD Planned maintenance +user2@domain.tld HOLD Planned maintenance +[...] +---%<------------------------------------------------------------------------- + +Virtual Domain Aliases in Postfix can be easily used to populate the *List of +Recipients* to be converted (and to be queued on HOLD). The following basic +script helps populating the database and needs to be adapted to the actual +alias location: + +---%<------------------------------------------------------------------------- +#!/bin/bash +set -e +ALIASES="/etc/postfix/virtual" +MAINTALIASES="/etc/postfix/recipient_maintenance" + +doveadm user $1 || { echo 'user query failed.'; exit 1; } + +sed -e "s/\(.*\)$1/\1/;s/[ \t]*$//; s/$/ HOLD Planned maintenance: account=$1/" +$ALIASES >>$MAINTALIASES +grep $1 $MAINTALIASES +postmap $MAINTALIASES +echo '[ Complete ]' +---%<------------------------------------------------------------------------- + +Usage: + +---%<------------------------------------------------------------------------- +$ maintalias.sh user@domain.tld +---%<------------------------------------------------------------------------- + +Finally running *postmap recipient_maintenance* will create the database (in +case this part is not scripted). + +Actual Migration +---------------- + +Disable new logins for the users by adding 'allow_nets=' to their userdb +records, effectively disallowing ALL nets. Close open sessions if any: + +---%<------------------------------------------------------------------------- +$ doveadm who <user> +$ doveadm kick <user> +---%<------------------------------------------------------------------------- + +Convert the mailbox using the backup option: + +---%<------------------------------------------------------------------------- +$ dsync -u user@domain.tld backup mdbox:~/mdbox +---%<------------------------------------------------------------------------- + +Adjust userdb entries (if required) to reflect new 'mail_location'. Clear the +'recipient_maintenance' DB and undo 'allow_nets=' entries for the migrated +users. + +Finally release and requeue all mail previously held: + +---%<------------------------------------------------------------------------- +$ postsuper -r [ALL] +$ postqueue -f +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Teapop.txt b/doc/wiki/Migration.Teapop.txt new file mode 100644 index 0000000..c6498bb --- /dev/null +++ b/doc/wiki/Migration.Teapop.txt @@ -0,0 +1,73 @@ +Teapop 0.3.8 +============ + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read <Migration.txt> page first carefully.* + +First at all sorry for my bad English. At work I had to migrate our existing +teapop/mbox setup to Dovecot/Maildir without change the UIDL. At first I think +I must set the option pop3_uidl_format to %Mf. But this doesn't work because +Teapop use different algorithm. So the only way I found was to set the X-UIDL +in the mbox and then use the mb2md script. My Co-worker Robert (many thanks for +programming the script) and I use the following script: + +---%<------------------------------------------------------------------------- +#!/usr/bin/perl +use Digest::MD5; +$context = Digest::MD5->new; +$gotmail = 0; +$counter = 1; +@mail = (); +while (<>) +{ + $line = $_; + if(/^From /) + { + if ($gotmail) + { + processMail (); + $counter++; + $context->reset(); + @mail = (); + } + $gotmail = 1; + } + push (@mail, $line); + next if ($line =~ /^(Status|X-Status|Lines|Content-Length): /); + $context->add($line); +} +if ($gotmail) +{ + processMail (); +} +else +{ + print STDERR "Mailbox is empty!\n"; +} +sub processMail () +{ + if ($#mail > 2) + { + print shift(@mail); + print shift(@mail); + print "X-UIDL: " . $context->hexdigest() . "\n"; + foreach $l (@mail) + { + print $l; + } + } + else + { + print STDERR "Email has less then 3 lines!\n"; + } +} +---%<------------------------------------------------------------------------- + +Usage: scriptname $mboxfile > $newbox + +The script read the mbox file and generate the MD5 sum, if the line don't start +with Status,X-Status,Lines and Content-Length, for each mail and insert the +X-UIDL: after the Return-Path line. After that you can use the mb2md +script.Important: You must set the 'pop3_reuse_xuidl=yes'. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.UW.txt b/doc/wiki/Migration.UW.txt new file mode 100644 index 0000000..5ba5e88 --- /dev/null +++ b/doc/wiki/Migration.UW.txt @@ -0,0 +1,137 @@ +UW-IMAP +======= + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read <Migration.txt> page first carefully.* + +Namespaces +---------- + +By default UW-IMAP allows access to whole home directory. Since the home +directory may contain many other files as well, many people have chosen to +store their mails in the 'mail/' directory. This usually means that IMAP +clients have been configured to use 'mail/' as their "IMAP namespace prefix" +(the clients use different names for this). This doesn't work with Dovecot, +because Dovecot shows clients only the 'mail/' directory instead of the whole +home directory. So if the IMAP namespace was kept as 'mail/', Dovecot would try +to access the '~/mail/mail/' directory. + +There are three ways to fix this: + + 1. Remove the IMAP namespace prefix from the clients. + 2. Use namespaces to allow users to keep using the prefix. See "Backwards + Compatibility" in <Namespaces.txt> for an example configuration. + 3. Configure Dovecot to use home directory ('mail_location = + mbox:~/:INBOX=/var/mail/%u' and set 'mail_full_filesystem_access=yes'). The + latter is needed to make '~/mail' and '~user/mail' prefixes work. + +A typical mailbox location setting is: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +~/mbox file +----------- + +If a '~/mbox' file exists, UW-IMAP moves all the mails from '/var/mail/user' +into the '~/mbox' file. Currently Dovecot doesn't support this feature. There +are two possibilities to handle this: + + * Move everyone's mails to '~/mbox' and reconfigure your <MDA.txt> to deliver + new mails there by default. + * Move the existing mails from '~/mbox' back to '/var/mail/user'. + +Subscriptions +------------- + +UW-IMAP keeps the list of subscribed mailboxes in '~/.mailboxlist' file, while +Dovecot keeps them in '~/mail/.subscriptions' file. UW-IMAP's subscriptions +also contain the mailboxes with their prefixes, for example: + +---%<------------------------------------------------------------------------- +mail/box +~/mail/box2 +~user/mail/box3 +---%<------------------------------------------------------------------------- + + * If you removed the prefix from the IMAP clients, you'll also have to remove + these prefixes. + * You can use <uw2dovecot.sh.txt> script to copy all the users' '.mailboxlist' + files to '.subscriptions' files (without any prefix removal). + * It's possible to keep using the '.mailboxlist' filename (as long as it's in + the same directory) by modifying 'SUBSCRIPTION_FILE_NAME' define in + 'src/lib-storage/index/mbox/mbox-storage.h' + +UIDs, flags and keywords +------------------------ + +Dovecot uses UW-IMAP compatible metadata headers in mboxes, so it's possible to +migrate back and forth without losing any flags, keywords or cause IMAP UIDs to +change. + +Locking +------- + +UW-IMAP uses dotlock + flock() as the default locking combination, while +Dovecot uses dotlock + fcntl() by default. See <MboxLocking.txt> to determine +what are the correct lock settings for you. To use UW-IMAP compatible locking, +use: + +---%<------------------------------------------------------------------------- +mbox_read_locks = flock +mbox_write_locks = dotlock flock +---%<------------------------------------------------------------------------- + +DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA +------------------------------------------------- + +The first message in a mbox file may contain the subject "DON'T DELETE THIS +MESSAGE -- FOLDER INTERNAL DATA". This system message has information about the +mailbox and its state. + +Dovecot v1.0 uses these system messages. Keep them. + +Maildir conversion +------------------ + +If you're planning on migrating to Dovecot, you might also want to switch to +Maildir format as well. However it might be easier to first migrate from +UW-IMAP + mbox to Dovecot + mbox, and only then migrate the users to Maildir +format. + + * http://people.redhat.com/rkeech/maildir-migration.txt describes how to + migrate from UW-IMAP+mbox to Dovecot v0.99 + Maildir. Note that Dovecot + v0.99 has slightly different configuration file settings. + * <Migration.MailFormat.txt> has some migration scripts + * Here is a tool ([attachment:uw2dovecot.pl]) that will convert mbox, mbx, and + mix formatted UW-IMAP folders to Maildir/dovecot format. + +UW-POP3 (UW-IMAP's POP3 wrapper, ipop3d) +======================================== + +By default Dovecot generates POP3 UIDLs differently than UW-POP3, which causes +POP3 clients to redownload them as new messages. You can avoid this by setting: + +---%<------------------------------------------------------------------------- +pop3_uidl_format = %08Xv%08Xu +---%<------------------------------------------------------------------------- + +To utilize the UW login conversion to lowecase, we recommend that you use the +following option: + +---%<------------------------------------------------------------------------- +auth_username_format = %Lu +---%<------------------------------------------------------------------------- + +You can confirm that the old and the new UIDLs match: + +---%<------------------------------------------------------------------------- +telnet localhost 110 +user test +pass test +uidl +quit +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.Vm-pop3d.txt b/doc/wiki/Migration.Vm-pop3d.txt new file mode 100644 index 0000000..550eafc --- /dev/null +++ b/doc/wiki/Migration.Vm-pop3d.txt @@ -0,0 +1,42 @@ +Vm-pop3d +======== + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read <Migration.txt> page first carefully.* + +Vm-pop3d uses the Message-ID: header data for UIDL, Dovecot does not support +this as it is not unique enough. The following Perl script will take the +Message-ID: data from all mails in a mbox and put the data into the X-UIDL: +header which Dovecot can use with the 'pop3_reuse_xuidl' setting: + +---%<------------------------------------------------------------------------- +#!/usr/bin/env perl +use Email::Simple; +my @totalmail=<STDIN>; +my $mail = join("",@totalmail); +my $email = Email::Simple->new($mail); +my $msg_id = $email->header("Message-Id"); +$msg_id =~ s#<##g; +$msg_id =~ s#>##g; +$email->header_set("X-UIDL", $msg_id); +print $email->as_string; +---%<------------------------------------------------------------------------- + +Requires email::simple, though the default setting in email::simple is to wrap +headers at 77 characters, which then causes problems when Outlook clients issue +UIDL, the workaround for this is to edit the Perl module, on Debian Etch this +involves editing '/usr/share/perl5/Email/Simple.pm', find "sub _fold" and +change {0,77} to a suitably higher value. On more recent versions of +email::simple, you may need to edit 'Headers.pm' instead, with the line to look +for being "_default_fold_at" + +You then run the script like so: + +---%<------------------------------------------------------------------------- +formail -q- -s perl script.pl < inbox > newinbox +---%<------------------------------------------------------------------------- + +When Dovecot now looks at newinbox, it will use the X-UIDL: header and clients +will not redownload mail. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Migration.txt b/doc/wiki/Migration.txt new file mode 100644 index 0000000..a1a5945 --- /dev/null +++ b/doc/wiki/Migration.txt @@ -0,0 +1,166 @@ +Migration to Dovecot +==================== + +*WARNING: Badly done migration will cause your IMAP and/or POP3 clients to +re-download all mails. Read this page carefully.* + +This page contains generic information related to migrating from another IMAP +or POP3 server to Dovecot. You should read this page, and then look at the +server-specific instructions: + + * <Migrating from any IMAP/POP3 server to Dovecot via dsync> + [Migration.Dsync.txt] - this is the recommended way to do all migrations + nowadays + * <UW-IMAP / UW-POP3> [Migration.UW.txt] + * <Linuxconf / VIMAP> [Migration.Linuxconf.txt] + * <Courier IMAP and POP3> [Migration.Courier.txt] + * <Cyrus IMAP and POP3> [Migration.Cyrus.txt] + * <vm-pop3d> [Migration.Vm-pop3d.txt] + * <teapop> [Migration.Teapop.txt] + * <Binc IMAP> [Migration.BincIMAP.txt] + * <Gmail> [Migration.Gmail.txt] + +Dovecot is one of the easiest IMAP servers to migrate to because of its +powerful configuration options. Dovecot can store email in both <mbox> +[MailboxFormat.mbox.txt] and <Maildir> [MailboxFormat.Maildir.txt] formats, +making it compatible with many existing servers. Dovecot is also very flexible +as to where it stores the email. It supports many different databases for +storing <passwords> [PasswordDatabase.txt] <user information> +[UserDatabase.txt]. + +Migration involves several separate tasks. You either need to convert your data +or make Dovecot read your existing data. Dovecot is very good at being +compatible and configurable, so it is likely to read your existing mailboxes +and user and password configurations. Tasks for conversion include: + + * <Where and how to store the email> [MailLocation.txt] + * <User authentication> [PasswordDatabase.txt] + * <User database configuration> [UserDatabase.txt] (home directory structure, + UID and GID) + +IMAP migration +-------------- + +When migrating mails from another IMAP server, you should make sure that these +are preserved: + + 1. Message flags + * Lost flags can be really annoying, you most likely want to avoid it. + 2. Message UIDs and UIDVALIDITY value + * If UIDs are lost, at the minimum clients' message cache gets cleaned and + messages are re-downloaded as new. + * Some IMAP clients store metadata by assigning it to specific UID, if + UIDs are changed these will be lost. + 3. Mailbox subscription list + * Users would be able to manually subscribe them again if you don't want + to mess with it. + +POP3 migration +-------------- + +When migrating mails from another POP3 server, you should try to preserve the +old UIDLs. If POP3 client is configured to keep mails in the server and the +messages' UIDLs change, all the messages are downloaded again as new messages. + +*Don't trust the migration scripts or anything you see in this wiki. Verify +manually that the UIDLs are correct before exposing real clients to Dovecot.* +You can do this by logging in using your old POP3 server, issuing UIDL command +and saving the output. Then log in using Dovecot and save its UIDL output as +well. Use e.g.'diff' command to verify that the lists are identical. Note that: + + * *If a client already saw changed UIDLs and decided to start re-downloading + mails, it's unlikely there is anything you can do to stop it. Even going + back to your old server is unlikely to help at that point.* + * Some (many?) POP3 clients also require that the message ordering is + preserved. + * Some clients re-download all mails if you change the hostname in the client + configuration. Be aware of this when testing. + +Some servers (UW, Cyrus) implementing both IMAP and POP3 protocols use the IMAP +UID and UIDVALIDITY values for generating the POP3 UIDL values. To preserve the +POP3 UIDL from such servers you'll need to preserve the IMAP UIDs and set +'pop3_uidl_format' properly. + +If the server doesn't use IMAP UIDs for the POP3 UIDL, you'll need to figure +out another way to do it. One way is to put the UIDL value into X-UIDL: header +in the mails and set 'pop3_reuse_xuidl=yes'. Some POP3 servers (QPopper) write +the X-UIDL: header themselves, making the migration easy. + +Some POP3 servers using Maildir uses the maildir base filename as the UIDL. You +can use 'pop3_uidl_format = %f' to do this. + +Here is a list of POP3 servers and how they generate their UIDs. Please update +if you know more: + + * UW-POP3: 'pop3_uidl_format = %08Xv%08Xu' + * qmail-pop3d: 'pop3_uidl_format = %f' + * Cyrus <= 2.1.3: 'pop3_uidl_format = %u' + * Cyrus >= 2.1.4: 'pop3_uidl_format = %v.%u' + * Citadel [http://www.citadel.org] (all versions): 'pop3_uidl_format = %u' + * Dovecot 0.99: 'pop3_uidl_format = %v.%u' + * tpop3d [http://www.ex-parrot.com/~chris/tpop3d/]: + * Maildir: 'pop3_uidl_format = %Mf' (MD5 sum of the maildir base filename + in hex) + * mbox: MD5 sum in hex of first 512 bytes of the message (or of the full + message if it's less than 512 bytes). + * popa3d [http://www.openwall.com/popa3d/] Generates MD5 sum from a couple of + headers. Dovecot uses compatible MD5 sums internally, but converts them into + UIDL strings in a bit different way. + * teapop [http://www.toontown.org/teapop/] 0.3.8: + * Maildir: 'pop3_uidl_format = %Mf' (MD5 sum of the maildir base filename + in hex) + * mbox: MD5 sum of the message without the following lines: + Status,X-Status,Lines and Content-Length. + * Cucipop mbox: v1.31 uses its own homebrew checksum based on headers and + body. Injection of X-UIDL: headers and pop3_reuse_xuidl=yes is the way to + go. + * qpopper: 'pop3_reuse_xuidl=yes' + * Courier: 'pop3_uidl_format = %f' + +IMAP <-> IMAP copying +--------------------- + +Note that you can do IMAP -> Dovecot migration using <dsync> +[Migration.Dsync.txt], which preserves IMAP UIDs and other metadata. This is +the recommended way of doing migrations. But there are also other options: + +If you don't care about preserving messages' UIDs, you can always migrate from +another IMAP server to Dovecot by downloading the messages via IMAP from the +old server and then uploading them to Dovecot via IMAP. + +There are several different tools for this, for example UW-IMAP +[http://www.washington.edu/imap/]'s mailutil, imapsync +[http://freshmeat.net/projects/imapsync], YippieMove +[http://www.yippiemove.com] and Larch [https://github.com/rgrove/larch]. + +imapsync +-------- + +Here's an example of how to run imapsync for a single user: + +---%<------------------------------------------------------------------------- +imapsync --syncinternaldates \ + --host1 192.168.1.57 --authmech1 LOGIN --user1 leah@example.com --password1 +secret \ + --host2 127.0.0.1 --authmech2 LOGIN --user2 leah@example.com --password2 +secret +---%<------------------------------------------------------------------------- + +It is quite easy to script this for a number of users, assuming you have their +passwords. Even if you do not, imapsync also supports logging in as an admin +user that has the ability to copy message for sub users, and a variety of other +authentication options. + +Larch +----- + +Here's an example of how to run Larch: + +---%<------------------------------------------------------------------------- +larch --from imap://mail1.example.com --to imap://mail2.example.com +---%<------------------------------------------------------------------------- + +When run it will ask you for usernames and passwords that will be used for +logging into servers, but you can also specify them on the command line. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MissingMailboxes.txt b/doc/wiki/MissingMailboxes.txt new file mode 100644 index 0000000..752d5d4 --- /dev/null +++ b/doc/wiki/MissingMailboxes.txt @@ -0,0 +1,54 @@ +Missing mailboxes +================= + +Namespaces +---------- + +Dovecot by default doesn't use any "personal IMAP namespace prefix", which +clients often call either "IMAP namespace" or "IMAP prefix". With Courier you +probably had this set to "INBOX.", with UW-IMAP you might have set it to +"mail/". So, the solution is simply to set this field empty and restart your +IMAP client. If this helps, but you don't want to modify the clients' +configuration, see <Namespaces.txt>. + +Mail location +------------- + +If it didn't help, you might have <mail_location> [MailLocation.txt] setting +wrong. If it's unset, Dovecot tries to detect where your mail is stored by +looking at '~/Maildir', '~/mail', '/var/spool/mail/' and '/var/mail/' +directories. Depending on what you want, Dovecot might have guessed wrong. See +<TestInstallation.txt> for how to figure out what exactly is the problem. + +Missing INBOX (mbox) +-------------------- + +See if the mails are stored in '~/mbox' file. If '~/mbox' file exists, UW-IMAP +moves mails there from '/var/mail/user'. Dovecot supports this with <snarf +plugin> [Plugins.Snarf.txt]. + +Subscriptions +------------- + +Dovecot uses different filenames for list of mailbox subscriptions. You'll need +to rename these to ones that Dovecot wants (currently '.subscriptions' for mbox +and 'subscriptions' for Maildir). See <Migration.txt> for more information. + +Troubleshooting +--------------- + +If it's still not working, check first if the problem is with IMAP client or +server configuration. Easiest way to do this is to talk IMAP directly. (include +the A, B, C): + +---%<------------------------------------------------------------------------- +telnet imap.example.org 143 +A login username password +B list "" * +C logout +---%<------------------------------------------------------------------------- + +If you see a list of expected mailboxes, the problem is with your IMAP client. +If not, set 'mail_debug=yes' and look at the logs. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Mountpoints.txt b/doc/wiki/Mountpoints.txt new file mode 100644 index 0000000..1f920af --- /dev/null +++ b/doc/wiki/Mountpoints.txt @@ -0,0 +1,55 @@ +Mountpoints +=========== + +---%<------------------------------------------------------------------------- +master: Warning: /mnt/foo is no longer mounted. See +http://wiki2.dovecot.org/Mountpoints +---%<------------------------------------------------------------------------- + +Dovecot wants to keep track of mountpoints that might contain emails. If such a +mountpoint is suddenly not mounted, it's handled as an error condition rather +than as "this is a new user". This prevents other confusion that follows such a +situation. + +The list of mount points is maintained in the file /var/run/dovecot/mounts. + +Removing mountpoints +-------------------- + +If you permanently removed a mountpoint, you can tell Dovecot to forget about +it with: + +---%<------------------------------------------------------------------------- +doveadm mount remove /mnt/foo +---%<------------------------------------------------------------------------- + +Ignoring mountpoints +-------------------- + +Dovecot already internally filters out many mountpoints and filesystems that +are pretty much guaranteed not to contain any emails, but of course it can't +know about everything. This is especially problematic with filesystems that are +sometimes mounted and sometimes not. You can permanently tell Dovecot to ignore +a mountpoint with: + +---%<------------------------------------------------------------------------- +doveadm mount add /mnt/foo ignore +---%<------------------------------------------------------------------------- + +or ignore all mountpoints under '/mnt': + +---%<------------------------------------------------------------------------- +doveadm mount add '/mnt/*' ignore +---%<------------------------------------------------------------------------- + +Adding mountpoints +------------------ + +Mountpoints are added automatically when Dovecot starts up. If you mount a new +filesystem with mails while Dovecot is running you can also add it manually: + +---%<------------------------------------------------------------------------- +doveadm mount add /mnt/foo +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/NFS.txt b/doc/wiki/NFS.txt new file mode 100644 index 0000000..edd2ff6 --- /dev/null +++ b/doc/wiki/NFS.txt @@ -0,0 +1,161 @@ +NFS +=== + +NFS is commonly used in one of these ways: + + 1. Dovecot is run in a single computer. + 2. Dovecot is run in multiple computers, users are redirected more or less + randomly to different computers. + 3. Dovecot is run in multiple computers, each user is assigned a specific + computer which is used whenever possible. + +*The only way to reliably implement the 2nd setup is with the <director> +[Director.txt] service.* + +Dovecot configuration +--------------------- + +Single Dovecot server setup or <Dovecot director> [Director.txt] cluster setup: + +---%<------------------------------------------------------------------------- +mmap_disable = yes +#dotlock_use_excl = no # only needed with NFSv2, NFSv3+ supports O_EXCL and +it's faster +mail_fsync = always +mail_nfs_storage = no +mail_nfs_index = no +---%<------------------------------------------------------------------------- + +Multi-server setup that *tries* to flush NFS caches (increases NFS operations, +and *isn't fully reliable*), try not to use this: + +---%<------------------------------------------------------------------------- +mmap_disable = yes +#dotlock_use_excl = no # only needed with NFSv2, NFSv3+ supports O_EXCL and +it's faster +mail_fsync = always +# These settings slow things down and don't fully work, use director proxy +instead: +mail_nfs_storage = yes +mail_nfs_index = yes +---%<------------------------------------------------------------------------- + +Common issues +------------- + +Clock synchronization +--------------------- + +Run ntpd in the NFS server and all the NFS clients to make sure their clocks +are synchronized. If the clocks are more than one second apart from each others +and multiple computers access the same mailbox simultaneously, you may get +errors. + +NFS caching problems +-------------------- + +NFS caching is a big problem when multiple computers are accessing the same +mailbox simultaneously. The best fix for this is to prevent it from happening. +Configure your setup so that a user always gets redirected to the same server +(unless it's down). This also means that mail deliveries must be done by the +same server, or alternatively it shouldn't update index files. + +Dovecot flushes NFS caches when needed if you set 'mail_nfs_storage=yes', but +unfortunately this doesn't work 100%, so you can get random errors. + +Disabling NFS attribute cache helps a lot in getting rid of caching related +errors, but this makes the performance MUCH worse and increases the load on NFS +server. This can usually be done by giving 'actimeo=0' or 'noac' mount option. + +Index files +----------- + +If you keep the index files stored on NFS, you'll need to set +'mmap_disable=yes'. If you're not running lockd you'll have to set +'lock_method=dotlock', but this degrades performance. Note that some NFS +installations have problems with lockd. If you're beginning to get all kinds of +locking related errors, try if the problems go away with dotlocking. + +With mbox/Maildir formats (but not dbox!) it's also possible to store index +files on local disk instead of on NFS. If the user gets redirected to different +servers, the local indexes are automatically created/updated. If the user is +(nearly) always redirected to the same server this should be fine and you would +likely get higher performance than indexes stored on NFS, but if the server +changes it can be slow to recreate the index/cache files. + +Single computer setup +--------------------- + +This doesn't really differ from keeping mails stored locally. For better +performance you should keep index files stored in a local disk. + +Random redirects to multiple servers +------------------------------------ + +You should avoid this setup whenever possible. Besides the NFS cache problems +described above, mailbox contents can't be cached as well in the memory either. +This is more problematic with mbox than with maildir, but in both cases if a +client is redirected to a different server when reconnecting, the new server +will have to read some data via the NFS into memory, while the original server +might have had the data already cached. + +If you choose to use this setup, at the very least try to make connections from +a single IP redirected into the same server. This avoids the biggest problems +with clients that use multiple connections. + +Per-user redirects to multiple servers +-------------------------------------- + +This method performs a lot better than random redirects. It maximizes the +caching possibilities and prevents the problems caused by simultaneous mailbox +access. + +New mail deliveries are often still handled by different computers. This isn't +a problem with maildir as long as you're not using <LDA.txt> (i.e. +dovecot-uidlist file or index files shouldn't get updated). It shouldn't be a +problem with mboxes either as long as you're using fcntl locking. This problem +can be fully solved by using LMTP protocol to deliver the mails to the correct +server (possibly using Dovecot's LMTP proxy). + +NFS clients +=========== + +Here's a list of kernels that have been tried as NFS clients: + + * FreeBSD has a caching bug + [http://www.freebsd.org/cgi/query-pr.cgi?pr=123755] which causes problems + when mailbox is being accessed from different computers at the same time + * Linux 2.6.16: 'utime()' is buggy, fix in here + [http://client.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-007-fix_setattr_clobber.dif]. + With the fix applied, utime() seems to work perfectly. High-volume systems + may experience VFS lock sync issues and for these the complete patchset at + http://www.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-NFS_ALL.dif is + suggested and appears to work well in production. + * Linux 2.6.18: Seems to have intermittent caching issues. The same .config + with 2.6.20.1 has been tested and appears to work well. + * Linux 2.4.8: Has caching problems, don't know if they can be solved + * Solaris: If it's completely broken, see + http://dovecot.org/list/dovecot/2006-December/018145.html + * The Connectathon test suite is very useful to verify a healthy NFS setup, + see http://www.connectathon.org/nfstests.html + +Misc notes +========== + + * readdirplus isn't really needed by Dovecot and it can slow down some NFS + servers. Use "nordirplus" mount option to disable it. + * Dovecot doesn't care about root_squash setting, all the root-owned files are + in /var/run typically which is not in NFS + * In an environment using Debian (2.6.18) clients with Isilon NFS cluster + nodes - the following mount options were found to be the most + successful:'rsize=32768,wsize=32768,hard,fg,lock,nfsvers=3,tcp,retrans=0,nordirplus + 0 0' + * To learn more about NFS caching and other issues, mostly from a programmer's + point of view, see NFS Coding HOWTO + [http://iki.fi/tss/nfs-coding-howto.html] + * Use such permissions for the unmounted mount point root directory that + Dovecot can't create files under it. Otherwise if the NFS server isn't + mounted for any reason and user access mails, a new empty user mail + directory is created, which breaks things. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Namespaces.txt b/doc/wiki/Namespaces.txt new file mode 100644 index 0000000..3d05dac --- /dev/null +++ b/doc/wiki/Namespaces.txt @@ -0,0 +1,393 @@ +Namespaces +========== + +Contents + + + 1. Namespaces + + 1. Configuration + + 1. Namespace types + + 2. Hierarchy separators + + 3. Namespace settings + + 4. From userdb + + 2. Shared Mailboxes + + 3. Examples + + 1. Mixed mbox and Maildir + + 2. Backwards Compatibility: UW-IMAP + + 3. Backwards Compatibility: Courier IMAP + + 4. Per-user Namespace Location From SQL + + 5. Hidden subscription namespace + +Dovecot supports fully configurable namespaces. Their original and primary +purpose is to provide Namespace IMAP extension (RFC 2342 +[http://www.faqs.org/rfcs/rfc2342.html]) support, which allows giving IMAP +clients hints about where to locate mailboxes and whether they're private, +shared or public. Unfortunately most IMAP clients don't support this extension. + +Dovecot namespaces can be used for several other purposes too: + + * Changing the hierarchy separator + * Providing backwards compatibility when switching from another IMAP server + * Provides support for <public> [SharedMailboxes.Public.txt] and <shared> + [SharedMailboxes.Shared.txt] mailboxes + * Allows having mails in multiple different locations with possibly different + formats + +Configuration +------------- + +In v2.1+ there's a default inbox namespace added in '10-mail.conf'. If the +configuration doesn't explicitly specify a namespace (as was in v2.0 and older) +a default namespace is created automatically. + +The section name in namespaces (e.g. 'namespace sectionname { .. } ' is used +only internally within configuration. It's not required at all, but it allows +you to update an existing namespace (like how '15-mailboxes.conf' does) or have +userdb override namespace settings for specific users +('namespace/sectionname/prefix=foo/'). + +Namespace types +--------------- + +There are 3 types of namespaces: + + * private: Typically contains only user's own private mailboxes. + * shared: Contains other users' <shared mailboxes> + [SharedMailboxes.Shared.txt]. + * public: Contains <public mailboxes> [SharedMailboxes.Public.txt]. + +Hierarchy separators +-------------------- + +Hierarchy separator specifies the character that is used to separate a parent +mailbox from its child mailbox. For example if you have a mailbox "foo" with a +child mailbox "bar", the full path to the child mailbox would be "foo/bar" if +the separator was '/'. With a separator '.' it would be "foo.bar". + +IMAP clients, Sieve scripts and many parts of Dovecot configuration use the +configured separator when referring to mailboxes. This means that if you change +the separator, you may break things. + +However, changing the separator doesn't change the on-disk "layout separator". +For example: ++-----------------------------+--------+-----+----------+---------------------+ +| mail_location | Layout | NS | Mailbox | Directory | +| | sep | sep | name | | ++-----------------------------+--------+-----+----------+---------------------+ +| maildir:~/Maildir | . | . | foo.bar | ~/Maildir/.foo.bar/ | ++-----------------------------+--------+-----+----------+---------------------+ +| maildir:~/Maildir | . | / | foo/bar | ~/Maildir/.foo.bar/ | ++-----------------------------+--------+-----+----------+---------------------+ +| maildir:~/Maildir:LAYOUT=fs | / | . | foo.bar | ~/Maildir/foo/bar/ | ++-----------------------------+--------+-----+----------+---------------------+ +| maildir:~/Maildir:LAYOUT=fs | / | / | foo/bar | ~/Maildir/foo/bar/ | ++-----------------------------+--------+-----+----------+---------------------+ + +Note how the "namespace separator" changes only the "Mailbox name", but doesn't +change the directory where the mails are stored. The "layout separator" can +only be changed by changing the LAYOUT, which also affects the entire directory +structure. + +The layout separator also restricts the mailbox names. For example if the +layout separator is '.', you can't just set separator to '/' and create a +mailbox named "foo.bar". If you need to do this, you can use <listescape> +[Plugins.Listescape.txt] plugin to add escape the mailbox names as necessary. + +A commonly used separator is '/'. It probably causes the least amount of +trouble with different IMAP clients.'^' separator is troublesome with +Thunderbird.When '\' should be used it must be quoted, so one sets separator = +"\\" + +You should use the same hierarchy separator for all namespaces. All list=yes +namespaces must use the same separator, but if you find it necessary (e.g. for +backwards compatibility namespaces) you may use different separators for +list=no namespaces. + +Namespace settings +------------------ + + * type: See the "Namespace types" section above + * separator: See the "Hierarchy separators" section above + * prefix: The namespace prefix how it's visible in the NAMESPACE reply (if + hidden=no) and mailbox list (if list=yes). + * location: <Mailbox location> [MailLocation.txt]. The default is to use + 'mail_location' setting. + * inbox: "yes", if this namespace contains the user's INBOX. There is only one + INBOX, so only one namespace can have inbox=yes. + * hidden: "yes", if this namespace shouldn't be listed in NAMESPACE reply. + * list: "yes" (default), if this namespace and its mailboxes should be listed + by LIST command when the namespace prefix isn't explicitly specified as a + parameter. "children" means the namespace prefix list listed only if it has + child mailboxes. + * subscriptions: "yes" (default) if this namespace should handle its own + subscriptions. If "no", then the first parent namespace with + subscriptions=yes will handle it. For example if it's "no" for a namespace + with prefix=foo/bar/, Dovecot first sees if there's a prefix=foo/ namespace + with subscriptions=yes and then a namespace with an empty prefix. If neither + is found, an error is given. + * ignore_on_failure: Normally Dovecot fails if it can't successfully create a + namespace. Set this to "yes" to continue even if the namespace creation + fails (e.g. public namespace points to inaccessible location). + * disabled: Set to "yes" to quickly disable this namespace. Especially useful + when returned by a userdb lookup to give per-user namespaces. + * alias_for: If multiple namespaces point to the same location, they should be + marked as aliases against one primary namespace. This avoids duplicating + work for some commands (listing the same mailbox multiple times). The value + for alias_for is the primary namespace's prefix. For example if the primary + namespace has empty prefix, set 'alias_for=' for the alias namespace. Or if + primary has 'prefix=INBOX/', use 'alias_for=INBOX/'. + * mailbox { .. } settings can be used to autocreate/autosubscribe mailboxes + and set their SPECIAL-USE flags. + +From userdb +----------- + +To change namespace settings from userdb, you need to return +"namespace/<name>/setting=value". To create a namespace, make sure you first +return "namespace=<name>[,<name>,...]" and settings after this. Note that the +"namespace" setting must list all the namespaces that are used - there's +currently no way to simply "add" a namespace. + +---%<------------------------------------------------------------------------- +userdb { + driver = static + args = namespace=inbox,special +namespace/special/location=sdbox:/var/special/%u +namespace/special/prefix=special/ +} +---%<------------------------------------------------------------------------- + +Shared Mailboxes +---------------- + +See <SharedMailboxes.txt>. + +Examples +-------- + +Mixed mbox and Maildir +---------------------- + +If you have your INBOX as mbox in '/var/mail/username' and the rest of the +mailboxes in Maildir format under '~/Maildir', you can do this by creating two +namespaces: + +---%<------------------------------------------------------------------------- +namespace { + separator = / + prefix = "#mbox/" + location = mbox:~/mail:INBOX=/var/mail/%u + inbox = yes + hidden = yes + list = no +} +namespace { + separator = / + prefix = + location = maildir:~/Maildir +} +---%<------------------------------------------------------------------------- + +Without the 'list = no' setting in the first namespace, clients would see the +"#mbox" namespace as a non-selectable mailbox named "#mbox" but with child +mailboxes (the mbox files in the '~/mail' directory), ie. like a directory. So +specifically with 'inbox = yes', having 'list = no' is often desirable. + +Backwards Compatibility: UW-IMAP +-------------------------------- + +When switching from UW-IMAP and you don't want to give users full access to +filesystem, you can create hidden namespaces which allow users to access their +mails using their existing namespace settings in clients. + +---%<------------------------------------------------------------------------- +# default namespace +namespace inbox { + separator = / + prefix = + inbox = yes +} +# for backwards compatibility: +namespace compat1 { + separator = / + prefix = mail/ + hidden = yes + list = no + alias_for = +} +namespace compat2 { + separator = / + prefix = ~/mail/ + hidden = yes + list = no + alias_for = +} +namespace compat3 { + separator = / + prefix = ~%u/mail/ + hidden = yes + list = no + alias_for = +} +---%<------------------------------------------------------------------------- + +Backwards Compatibility: Courier IMAP +------------------------------------- + +*Recommended:* You can continue using the same INBOX. namespace as Courier: + +---%<------------------------------------------------------------------------- +namespace inbox { + separator = . + prefix = INBOX. + inbox = yes +} +---%<------------------------------------------------------------------------- + +*Alternatively:* Create the INBOX. as a compatibility name, so old clients can +continue using it while new clients will use the empty prefix namespace: + +---%<------------------------------------------------------------------------- +namespace inbox { + separator = / + prefix = + inbox = yes +} + +namespace compat { + separator = . + prefix = INBOX. + inbox = no + hidden = yes + list = no + alias_for = +} +---%<------------------------------------------------------------------------- + +The "separator=/" allows the INBOX to have child mailboxes. Otherwise with +"separator=." it wouldn't be possible to know if "INBOX.foo" means INBOX's +"foo" child or the root "foo" mailbox in "INBOX." compatibility namespace. With +"separator=/" the difference is clear with "INBOX/foo" vs. "INBOX.foo". + +The alternative configuration is not recommended, as it may introduce there +problems: + + * Although clients may do LIST INBOX.*, they may still do LSUB *, resulting in + mixed results. + * If clients used empty namespace with Courier, they now see the mailboxes + with different names, resulting in redownloading of all mails (except + INBOX). + * Some clients may have random errors auto-detecting the proper default + folders (Sent, Drafts etc) if the client settings refer to old paths while + the server lists new paths. + +See also <Migration.Courier.txt>. + +Per-user Namespace Location From SQL +------------------------------------ + +You need to give the namespace a name, for example "docs" below: + +---%<------------------------------------------------------------------------- +namespace docs { + type = public + separator = / + prefix = Public/ +} +---%<------------------------------------------------------------------------- + +Then you have an SQL table like: + +---%<------------------------------------------------------------------------- +CREATE TABLE Namespaces ( +.. + Location varchar(255) NOT NULL, +.. +) +---%<------------------------------------------------------------------------- + +Now if you want to set the namespace location from the Namespaces table, use +something like: + +---%<------------------------------------------------------------------------- +user_query = SELECT Location as 'namespace/docs/location' FROM Namespaces WHERE +.. +---%<------------------------------------------------------------------------- + +Hidden subscription namespace +----------------------------- + +If you follow some advice to separate your INBOX, shared/ and public/ +namespaces by choosing INBOX/ as your prefix for the inboxes you will see, that +you run into troubles with subscriptions.Thats, because there is no parent +namespace for shared/ and public/ if you set 'subscriptions = no' for those +namespaces.If you set 'subscriptions = yes' for shared/ and public/ you will +see yourself in the situation, that all users share the same subscription files +under the location of those mailboxes.One good solution is, to create a so +called "hidden subscription namespace" with subscriptions turned on and setting +'subscriptions = no' for the other namespaces: + +---%<------------------------------------------------------------------------- +namespace subscriptions { + subscriptions = yes + prefix = "" + list = no + hidden = yes +} + +namespace inbox { + inbox = yes + location = + subscriptions = no + mailbox Drafts { + auto = subscribe + special_use = \Drafts + } + mailbox Sent { + auto = subscribe + special_use = \Sent + } + mailbox "Sent Messages" { + special_use = \Sent + } + mailbox Spam { + auto = subscribe + special_use = \Junk + } + mailbox Trash { + auto = subscribe + special_use = \Trash + } + prefix = INBOX/ + separator = / +} +namespace { + type = shared + prefix = shared/%%u/ + location = mdbox:%%h/mdbox:INDEXPVT=%h/mdbox/shared + list = children + subscriptions = no +} +namespace { + type = public + separator = / + prefix = public/ + location = mdbox:/usr/local/mail/public/mdbox:INDEXPVT=%h + subscriptions = no + list = children +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/OSCompatibility.txt b/doc/wiki/OSCompatibility.txt new file mode 100644 index 0000000..57c21b5 --- /dev/null +++ b/doc/wiki/OSCompatibility.txt @@ -0,0 +1,51 @@ +Operating System Compatibility +============================== + +Dovecot is commonly used with Linux, Solaris, FreeBSD, OpenBSD, NetBSD and Mac +OS X. The following operating systems have also worked at least at some point +in Dovecot's existence: + + * BSD/OS 4.2 + * AIX 4.3 + * HP-UX 11i (Dovecot v1.1+) + * Tru64 ?.? + * UnixWare 7.1.4 + * IRIX 6.5 compiles, but SCM_RIGHTS seems to be broken. + * Cygwin apparently doesn't work nowadays. + * MidnightBSD (ports) + +If there are compiling problems with any OS, please send a bug report that +includes the error messages. + +SCM_RIGHTS +---------- + +There is one slightly problematic feature that Dovecot requires for +implementing privilege separation:*SCM_RIGHTS*. If it doesn't work correctly, +you'll get errors on 'fd_send()' or 'fd_read()' such as: + +---%<------------------------------------------------------------------------- +imap-login: fd_send(X) failed: Bad file number +---%<------------------------------------------------------------------------- + +If this happens, you can still use inetd and mailfront +[http://untroubled.org/mailfront/], which executes Dovecot's post-login IMAP or +POP3 binary. You can also try defining 'BUGGY_CMSG_MACROS' in +'src/lib/fdpass.c' to see if it helps. + +Compilers +--------- + +Dovecot should compile with any ANSI-C99 compiler. Dovecot has been known to +compile (at least once in its lifetime) with the following compilers: + + * GCC + * Clang + * Intel CC [http://www.intel.com/software/products/compilers/clin/] + * Tiny CC [http://www.tinycc.org] + * Sun Studio 11 + * Sun Studio 12 + * AIX xlC + * HP-UX cc + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/POP3Server.txt b/doc/wiki/POP3Server.txt new file mode 100644 index 0000000..1c389a7 --- /dev/null +++ b/doc/wiki/POP3Server.txt @@ -0,0 +1,166 @@ +Dovecot as a POP3 server +======================== + +Dovecot was primarily designed to be an IMAP server, so although it works fine +as a POP3 server, it's not really optimized for that. + +Maildir Performance +------------------- + +The main problem with Dovecot's POP3 implementation with Maildir is how to get +messages' size fast enough. The POP3 specification requires that the sizes are +reported exactly, not just approximately. This means that linefeeds must be +counted as CR+LF characters. Normally with Maildir the linefeeds are stored as +plain LF characters, which means that simply getting the file size would +produce the wrong POP3 message size. Some Maildir POP3 servers do this anyway +and violate the POP3 specification. + +Dovecot returns correct message sizes by reading the entire message and +counting the linefeeds correctly. After this is done, the "virtual size" is +stored into dovecot-uidlist file and future calculations can be avoided by +simply looking up the cached value. + +You can also avoid the initial message size calculation by storing the size +directly into the filename. You can do this by appending ,W=<size> at the end +of the base filename. For example '1199932653.M583975P6568.host,W=2211:2,' is a +file whose virtual size is 2211 bytes (and real size somewhat smaller). Note +that this must not be done for existing files, only to newly delivered mails. + +If Dovecot's LDA is used, dovecot-uidlist and the index files are updated upon +message arrival, therefore there will be no message-size performance issues. + +'pop3_fast_size_lookups=yes' setting in v2.0.5+ uses the virtual message sizes +when they're already available, but fallbacks to using the physical message +sizes (violating POP3 specifications, but then again a lot of POP3 servers do +that). + +mbox Performance +---------------- + +Index files are quite useless if your users don't keep mails in the server. +They get first updated when the POP3 session starts to include all the +messages, and after the user has deleted all the mails, they again get updated +to contain zero mails. With this kind of a session the index reads and writes +could have been avoided if the index files had just been completely disabled. + +You may want to try how performance changes if you disable indexes for POP3 +users. You can also try preserving indexes but try different values for +'mbox_min_index_size' setting. + +Do not disable indexing if there are users that do not delete messages after +downloading them. Also, if you use Dovecot LDA, indexes may be helpful to have +fast access to the message sizes. + +Session locking +--------------- + +By default Dovecot allows multiple POP3 connections to the same mailbox. This +is (was?) especially useful for dialup connections which die in the middle of +the download, because the half-dead connections won't keep the mailbox locked. + +Setting 'pop3_lock_session=yes' makes Dovecot lock the mailbox for the whole +session. This is also what the POP3 RFC [http://www.ietf.org/rfc/rfc1939.txt] +specifies that should be done. If another connection comes while the mailbox is +locked, Dovecot waits until the locking times out (2 minutes with +Maildir,'mbox_lock_timeout' with mbox). In future there will be a separate +'pop3_lock_timeout' setting which allows timing out sooner. + +Flag changes +------------ + +By default when a message is RETRed, \Seen flag is added to it. POP3 itself +doesn't support flags, but if the mailbox is opened with IMAP (eg. from +webmail) it's shown as seen. You can disable this (to get better performance) +with 'pop3_no_flag_updates=yes'. + +POP3 client workarounds +----------------------- + +'pop3_client_workarounds' setting allows you to set some workarounds to avoid +POP3 clients breaking with some broken mails. + +Following are supported + + * outlook-no-nuls - Converts 0x0 in data to 0x80 + * oe-ns-eoh - Add missing end of header line + +UIDL format +----------- + +UIDLs are used by POP3 clients to keep track of what messages they've +downloaded, typically only if you've enabled "keep messages in server" option. +If the UIDL changes, the existing messages are re-downloaded as new messages, +which the users don't really appreciate. + +Dovecot supports multiple different ways to set the UIDL format, mostly to make +migrations from other POP3 servers transparent by preserving the old UIDL +values. See <Migration.txt> for how to set the UIDLs to be compatible with your +previous POP3 server. + +For new POP3 servers, the easiest way to set up UIDLs is to use IMAP's +UIDVALIDITY and UID values. The default in Dovecot v1.1+ is: + +---%<------------------------------------------------------------------------- +pop3_uidl_format = %08Xu%08Xv +---%<------------------------------------------------------------------------- + +Another good default is to use the message's global UID: + +---%<------------------------------------------------------------------------- +pop3_uidl_format = %g +---%<------------------------------------------------------------------------- + +However, note that GUIDs may not be unique, as the GUID does not change when a +message is copied. (While copying is not possible using only POP3, it can be +done using IMAP, Sieve, or doveadm.) + +Some formats, such as the previous default '%v.%u', seem to have problems with +Outlook 2003. + +MD5 UIDL format (mbox-only) +--------------------------- + +---%<------------------------------------------------------------------------- +pop3_uidl_format = %m +---%<------------------------------------------------------------------------- + +This works by getting the MD5 sum of a couple of message headers that uniquely +identify the message. The one good thing about MD5 format is that it doesn't +rely on the IMAP UID or UIDVALIDITY value. This allows you to modify the mbox +files in ways that Dovecot doesn't like, without causing the UIDLs to change. +For example: + + * Inserting messages in the middle of mbox files (eg. restoring mbox files + from backups can cause "Expunged message reappeared" errors) + * Reordering messages inside mbox + * <Other random problems> [MboxProblems.txt] causing UID renumbering (although + you should figure out why they're happening) + +The MD5 summing method however doesn't work well if you receive two identical +messages. Usually the MD5 sum is taken from these headers: + + * The first Received: header + * Delivered-To: header + +Normally there won't be a problem, because the MTA adds a unique identifier to +the first Received: header. If the same message is sent to multiple users in +one delivery, the Delivered-To: header is still different, making the MD5 sum +different. + +Except the MTA can be configured to support aliases, so for example sending the +mail to both root@ and webmail@ aliases causes the message to be delivered to +the same user, with identical Received: and Delivered-To: headers. The messages +really are identical, so their MD5 sums are also identical, and that can cause +some POP3 clients to keep downloading the messages over and over again, never +deleting them. + +To avoid this, there's also a 3rd header that is included in the MD5 sum +calculation: + + * X-Delivery-ID: header + +If you use <Dovecot's deliver> [LDA.txt] or IMAP APPEND and 'pop3_uidl_format = +%m', it always appends the X-Delivery-ID: header to saved mailbox. Any existing +X-Delivery-ID: headers in the saved mails are dropped. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.BSDAuth.txt b/doc/wiki/PasswordDatabase.BSDAuth.txt new file mode 100644 index 0000000..c13b800 --- /dev/null +++ b/doc/wiki/PasswordDatabase.BSDAuth.txt @@ -0,0 +1,7 @@ +BSDAuth +======= + +This is similar to <PAM> [PasswordDatabase.PAM.txt], but used by OpenBSD. It +supports 'cache_key' parameter the same way as PAM. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.ExtraFields.AllowNets.txt b/doc/wiki/PasswordDatabase.ExtraFields.AllowNets.txt new file mode 100644 index 0000000..5c80fa4 --- /dev/null +++ b/doc/wiki/PasswordDatabase.ExtraFields.AllowNets.txt @@ -0,0 +1,43 @@ +Allow_nets extra field +---------------------- + +The allow_nets field is a comma separated list of IP addresses and/or networks +where the user is allowed to log in from. If the user tries to log in from +elsewhere, the authentication will fail the same way as if a wrong password was +given. + +Example: 'allow_nets=127.0.0.0/8,192.168.0.0/16,1.2.3.4,4.5.6.7'. + +IPv6 addresses are also allowed. IPv6 mapped IPv4 addresses (eg. +'::ffff:1.2.3.4') are converted to standard IPv4 addresses before matching. +Example:'allow_nets=::1,2001:abcd:abcd::0:0/80,1.2.3.4' + +passwd-file example +------------------- + +---%<------------------------------------------------------------------------- +user:{plain}password::::::allow_nets=192.168.0.0/24 +---%<------------------------------------------------------------------------- + +Keyword 'local' +--------------- + +The keyword 'local' is accepted for Non-IP connections like Unix socket. For +example, with a Postfix/LMTP delivery setup, you must include 'local' for +Postfix to verify the email account: + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = password=test allow_nets=local,127.0.0.1/32 +} +---%<------------------------------------------------------------------------- + +Otherwise, you will see this error in the log: + +---%<------------------------------------------------------------------------- +[/var/run/dovecot/lmtp] said: 550 5.1.1 <test2@example.com> User doesn't exist: +test2@example.com (in reply to RCPT TO command)) +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.ExtraFields.Host.txt b/doc/wiki/PasswordDatabase.ExtraFields.Host.txt new file mode 100644 index 0000000..3a822c2 --- /dev/null +++ b/doc/wiki/PasswordDatabase.ExtraFields.Host.txt @@ -0,0 +1,75 @@ +Login referrals +=============== + +Login referrals are an IMAP extension specified by RFC 2221 +[https://tools.ietf.org/html/rfc2221]. Their purpose is to redirect clients to +an different IMAP4 server in case of hardware failures or organizational +changes. No client action is needed to invoke the LOGIN-REFERRALS capability: +the redirection is triggered by the server and occurs transparently. + +A security consideration is in order. As also stated by RFC 2221, a man in the +middle attack may use a rogue 'password catching' server to collectlogin data +and redirect your clients to their own rogue IMAP4 server. Although this would +be avoided by enforcing SSL/TLS. Login referrals are not supported by many +clients, so you probably don't want to use them anyway. + +Dovecot does not use login referrals by default. + +Configuration +------------- + +Note that the "host" field is also used by <proxying> +[PasswordDatabase.ExtraFields.Proxy.txt]. Login referrals are used only if the +proxy field isn't set. + +Login referrals can be used in two ways: + + 1. Tell the client to log into another server without allowing to log in + locally. + 2. Suggest the client to log into another server, but log it in anyway. + +The following fields can be used to configure login referrals: + + * 'host=s': The destination server's hostname. This field is required for + login referrals to be used. + * 'port=s': The destination server's port. The default is 143. + * 'destuser=s': Tell client to use a different username when logging in. + * 'reason=s': Optional reason to use as the reply to the login command. The + default is "Logged in, but you should use this server instead." + +Using the above settings you can suggest client to log in elsewhere. To require +it, you'll also have to return: + + * 'nologin': User is not allowed to log in. + * 'reason=s': Optional reason. The default is "Try this server instead.". + +Client support +-------------- + +The following clients are known to support login referrals: + + * Pine + * Outlook (but not Outlook Express) + +Examples +-------- + +Forward user to another server after successful authentication: + +---%<------------------------------------------------------------------------- +password_query = SELECT password, host, 'Y' as nologin FROM users WHERE userid += '%u' +---%<------------------------------------------------------------------------- + +Forward all users to another server without authentication: + +---%<------------------------------------------------------------------------- +password_query = \ + SELECT NULL AS password, 'Y' AS nopassword \ + 'imap2.example.com' AS host, \ + 'This server is down, try another one.' AS reason, \ + 'Y' AS nologin, \ + 'Y' AS nodelay +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.ExtraFields.NoDelay.txt b/doc/wiki/PasswordDatabase.ExtraFields.NoDelay.txt new file mode 100644 index 0000000..96b0948 --- /dev/null +++ b/doc/wiki/PasswordDatabase.ExtraFields.NoDelay.txt @@ -0,0 +1,13 @@ +Nodelay extra field +=================== + +If the authentication fails, Dovecot typically waits 0-2 seconds before sending +back the "authentication failed" reply. If this field is set, no such delay is +done. This is commonly used with <proxying> +[PasswordDatabase.ExtraFields.Proxy.txt] and <login referrals> +[PasswordDatabase.ExtraFields.Host.txt]. + +Note that if PAM is used as the passdb, it adds an extra delay which can't be +removed. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.ExtraFields.NoLogin.txt b/doc/wiki/PasswordDatabase.ExtraFields.NoLogin.txt new file mode 100644 index 0000000..fa7ea03 --- /dev/null +++ b/doc/wiki/PasswordDatabase.ExtraFields.NoLogin.txt @@ -0,0 +1,22 @@ +Nologin extra field +=================== + +User isn't allowed to log in even if the password matches. Commonly used with +<proxying> [PasswordDatabase.ExtraFields.Proxy.txt] and <login referrals> +[PasswordDatabase.ExtraFields.Host.txt], but may also be used standalone. One +way to use this would be perhaps: + + * 'nologin=y' + * 'reason=System is being upgraded, please try again later.' + +Unfortunately many clients don't show the reason to the user at all and just +assume that the password was given wrong, so it might not be a good idea to use +this unless the system will be down for days and you don't have a better way to +notify the users. + +Note: if you want to entirely block the user from logging in (i.e. account is +suspended), with no IMAP referral information provided, you must ensure that +neither 'proxy' nor 'host' are defined as one of the passdb extra fields. The +order of preference is:'proxy', 'host', then 'nologin'. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.ExtraFields.Proxy.txt b/doc/wiki/PasswordDatabase.ExtraFields.Proxy.txt new file mode 100644 index 0000000..e86d27d --- /dev/null +++ b/doc/wiki/PasswordDatabase.ExtraFields.Proxy.txt @@ -0,0 +1,307 @@ +Proxying +======== + +Dovecot supports proxying IMAP, POP3, <Submission.txt> (v2.3+), <LMTP.txt>, and +<ManageSieve> [Pigeonhole.ManageSieve.txt] connections to other hosts. The +proxying can be done for all users, or only for some specific users. There are +two ways to do the authentication: + + 1. Forward the password to the remote server. The proxy may or may not perform + authentication itself. This requires that the client uses only plaintext + authentication, or alternatively the proxy has access to users' passwords + in plaintext. + 2. Let Dovecot proxy perform the authentication and login to remote server + using the proxy's <master password> [MasterPassword.txt]. This allows + client to use also non-plaintext authentication. + +The proxy is configured pretty much the same way as <login referrals> +[PasswordDatabase.ExtraFields.Host.txt], with the addition of 'proxy' field. +The common fields to use for both proxying ways are: + + * 'proxy' and 'proxy_maybe': Enables the proxying. Either one of these fields + is required. + * 'proxy_maybe' can be used to implement "automatic proxying". If the proxy + destination matches the current connection, the user gets logged in + normally instead of being proxied. If the same happens with 'proxy', the + login fails with "Proxying loops" error. + * 'proxy_maybe' with LMTP requires v2.1.0+ + * 'proxy_maybe' with 'host=<dns name>' requires v2.1.2+. + * 'auth_proxy_self' setting in dovecot.conf can be used to specify extra + IPs that are also considered to be the proxy's own IPs. (v2.1.2+) + * 'proxy_always' can be used with 'proxy_maybe' to conditionally do + proxying to specified remote host (host isn't self) or to let director + assign a backend host (host is self). So basically this setting just + always sends the 'proxy' extra field to login process, but not + necessarily the 'host'. Useful when dividing users across multiple + director clusters. + * 'host=s': The destination server's *IP address*. This field is required. + * 'source_ip=s': The source IP address to use for outgoing connections. + (v2.2.14+) + * 'port=s': The destination server's port. The default is 143 with IMAP and + 110 with POP3. + * 'protocol=s': The protocol to use for the connection to the destination + server. This field is currently only relevant for LMTP: it can be used to + select either "lmtp" or "smtp". + * 'destuser=s': Tell client to use a different username when logging in. + * 'proxy_mech=s': Tell client to use this SASL authentication mechanism when + logging in. + * 'proxy_timeout': Abort connection after this many seconds. + * 'proxy_nopipelining': Don't pipeline IMAP commands. This is a workaround for + broken IMAP servers that hang otherwise. (v2.2.11+) + * 'proxy_not_trusted': IMAP/POP3 proxying never sends the ID/XCLIENT command + to remote. (v2.2.27+) + +You can use SSL/TLS connection to destination server by returning: + + * 'ssl=yes': Use SSL and require a valid verified remote certificate. + *WARNING: Unless used carefully, this is an insecure setting!* Before + v2.0.16/v2.1.beta1 the host name isn't checked in any way against the + certificate's CN. The only way to use this securely is to only use and allow + your own private CA's certs, anything else is exploitable by a + man-in-the-middle attack. + * Note that 'ssl_client_ca_dir' or 'ssl_client_ca_file' aren't currently + used for verifying the remote certificate, although ideally they will be + in a future Dovecot version. For now you need to add the trusted remote + certificates to 'ssl_ca'. + * *NOTE:* LMTP proxying supports SSL/TLS only since v2.3.1 - for older + versions any ssl/starttls extra field is ignored. + * *NOTE:* doveadm proxying doesn't support SSL/TLS currently - any + ssl/starttls extra field is ignored. + * 'ssl=any-cert': Use SSL, but don't require a valid remote certificate. + * 'starttls=yes': Use STARTTLS command instead of doing SSL handshake + immediately after connected. + * 'starttls=any-cert': Combine starttls and ssl=any-cert. + * Additionally you can also tell Dovecot to send SSL client certificate to the + remote server using 'ssl_client_cert' and 'ssl_client_key' settings in + 'dovecot.conf' (v2.0.17+). + +Set 'login_trusted_networks' to point to the proxies in the backends. This way +you'll get the clients' actual IP addresses logged instead of the proxy's. + +The destination servers don't need to be running Dovecot, but you should make +sure that the Dovecot proxy doesn't advertise more capabilities than the +destination server can handle. For IMAP you can do this by changing +'imap_capability' setting. For POP3 you'll have to modify Dovecot's sources for +now ('src/pop3/capability.h'). Dovecot also automatically sends updated +untagged CAPABILITY reply if it detects that the remote server has different +capabilities than what it already advertised to the client, but some clients +simply ignore the updated CAPABILITY reply. + +v2.2.14+: If your proxy handles a lot of connections (~64k) to the same +destination IP you may run out of TCP ports. The only way to work around this +is to use either multiple destination IPs or ports, or multiple source IPs. +Multiple source IPs can be easily used by adding them to the 'login_source_ips' +setting in 'dovecot.conf'. You can also use hostnames which expand to multiple +IPs. By prefixing the setting with "?" (e.g.'login_source_ips = +?proxy-sources.example.com') Dovecot will use only those IPs that actually +exist in the server, allowing you to share the same config file with multiple +servers. It's probably better not to include the server's default outgoing IP +address in the setting, as explained here +[https://idea.popcount.org/2014-04-03-bind-before-connect/]. + +v2.2.19+: To avoid reconnection load spikes when a backend server dies, you can +tell proxy to spread the client disconnections over a longer time period (after +the server side of the connection is already +disconnected).'login_proxy_max_disconnect_delay' setting in 'dovecot.conf' +controls this (disabled by default). + +v2.2.29+: You can forward arbitratry variables by returning them prefixed with +'forward_'. Dovecot will use protocol dependant way to forward these variables +forward and they will appear on the other side as 'forward_variable' Currently +IMAP/POP3 only feature. This feature requires that the sending host is in +login_trusted_networks. For IMAP the feature works by providing the variables +as part of ID command, such as 'i ID ( ... "x-forward-var" "value")'. For POP3 +the forwarding mecahism uses 'XCLIENT' with 'FORWARD=<base64 encoded blob of +forwarded variables>' + +See <Design.ParameterForwarding.txt> for more details on forwarding. + +Moving users between backends/clusters (v2.2.25+) +------------------------------------------------- + +A safe way to move users from one cluster to another is to do it like: + + * Set delay_until=<timestamp> <passdb extra field> + [PasswordDatabase.ExtraFields.txt] where <timestamp> is the current + timestamp plus some seconds into future (e.g. 31s). You may also want to + append e.g. "+5" for some load balancing if a lot of users are moved at + once. + * Set host=<new host> passdb extra field. This update should be atomic + together with the delay_until field. + * Use "doveadm proxy kick" or "doveadm director kick" to kick the user's + existing connections. + * The processes may still continue running in the backend for a longer + time. If you want to be absolutely sure, you could also run a script to + kill -9 all processes for the user in the backend. This of course has its + own problems. + +The idea here is that while the user's connections are being kicked and the +backend processes are finishing up and shutting down, new connections are being +delayed in the proxy. This delay should be long enough that the user's existing +processes are expected to die, but not so large that clients get connection +timeouts. A bit over 30 seconds is likely a good value. Once the delay_until +timestamp is reached, the connections continue to the new host. + +If you have a lot of users, it helps to group some of them together and do the +host/delay_until updates on a per-group basis rather than per-user basis. + +ID command forwarding (v2.2.29+) +-------------------------------- + +If you want to forward, for some reason, the IMAP ID command provided by the +client, set + +---%<------------------------------------------------------------------------- +imap_id_retain=yes +---%<------------------------------------------------------------------------- + +This will also enable client_id variable in variable expansions for auth +requests, which will contain the ID command as IMAP arglist. + +Password forwarding +------------------- + +If you don't want proxy itself to do authentication, you can configure it to +succeed with any given password. You can do this by returning an empty password +and 'nopassword' field. + +Master password +--------------- + +This way of forwarding requires the destination server to support master user +feature. The users will be normally authenticated in the proxy and the common +proxy fields are returned, but you'll need to return two fields specially: + + * 'master=s': This contains the master username (e.g. "proxy"). It's used as + SASL auhentication ID. + * Alternatively you could return 'destuser=user*master' and set + 'auth_master_user_separator = *'. + * 'pass=s': This field contains the master user's password. + +See <Authentication.MasterUsers.txt> for more information how to configure +this. + +OAuth2 forwarding +----------------- + +If you want to forward <OAuth2> [PasswordDatabase.oauth2.txt] tokens, return +field proxy_mech=%m as extra field. + +Example password forwarding static DB configuration +--------------------------------------------------- + +See <PasswordDatabase.Static.txt> + +Example password forwarding SQL configuration +--------------------------------------------- + +Create the SQL table: + +---%<------------------------------------------------------------------------- +CREATE TABLE proxy ( + user varchar(255) NOT NULL, + host varchar(16) default NULL, + destuser varchar(255) NOT NULL default '', + PRIMARY KEY (user) +); +---%<------------------------------------------------------------------------- + +Insert data to SQL corresponding your users. + +Working data could look like this: ++------+-------------+-----------------+ +| user | host | destuser | ++------+-------------+-----------------+ +| john | 192.168.0.1 | | ++------+-------------+-----------------+ +| joe | 192.168.0.2 | joe@example.com | ++------+-------------+-----------------+ + +The important parts of 'dovecot.conf': + +---%<------------------------------------------------------------------------- +# If you want to trade a bit of security for higher performance, change these +settings: +service imap-login { + service_count = 0 +} +service pop3-login { + service_count = 0 +} + +# If you are not moving mailboxes between hosts on a daily basis you can +# use authentication cache pretty safely. +auth_cache_size = 4096 + +auth_mechanisms = plain +passdb { + driver = sql + args = /usr/local/etc/dovecot/dovecot-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +The important parts of 'dovecot-sql.conf.ext': + +---%<------------------------------------------------------------------------- +driver = mysql +connect = host=sqlhost1 host=sqlhost2 dbname=mail user=dovecot password=secret +password_query = SELECT NULL AS password, 'Y' as nopassword, host, destuser, +'Y' AS proxy FROM proxy WHERE user = '%u' +---%<------------------------------------------------------------------------- + +Example proxy_maybe SQL configuration +------------------------------------- + +Create the SQL table: + +---%<------------------------------------------------------------------------- +CREATE TABLE users ( + user varchar(255) NOT NULL, + domain varchar(255) NOT NULL, + password varchar(100) NOT NULL, + host varchar(16) NOT NULL, + home varchar(100) NOT NULL, + PRIMARY KEY (user) +); +---%<------------------------------------------------------------------------- + +The important parts of 'dovecot.conf': + +---%<------------------------------------------------------------------------- +# user/group who owns the message files: +mail_uid = vmail +mail_gid = vmail + +auth_mechanisms = plain + +passdb { + driver = sql + args = /usr/local/etc/dovecot/dovecot-sql.conf.ext +} +userdb sql { + driver = sql + args = /usr/local/etc/dovecot/dovecot-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +The important parts of 'dovecot-sql.conf.ext': + +---%<------------------------------------------------------------------------- +driver = mysql + +password_query = \ + SELECT concat(user, '@', domain) AS user, password, host, 'Y' AS proxy_maybe +\ + FROM users WHERE user = '%n' AND domain = '%d' + +user_query = SELECT user AS username, domain, home \ + FROM users WHERE user = '%n' AND domain = '%d' +---%<------------------------------------------------------------------------- + +Example proxy LDAP configuration +-------------------------------- + +see: <PasswordDatabase.ExtraFields.txt> for more information, and a worked out +example + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.ExtraFields.User.txt b/doc/wiki/PasswordDatabase.ExtraFields.User.txt new file mode 100644 index 0000000..5c191e6 --- /dev/null +++ b/doc/wiki/PasswordDatabase.ExtraFields.User.txt @@ -0,0 +1,27 @@ +User extra field +================ + +This is mostly useful in case-insensitive username lookups to get the username +returned back using the same casing always. Otherwise depending on your +configuration it may cause problems, such as '/var/mail/user' and +'/var/mail/User' mailboxes created for the same user. + +An example 'password_query' in 'dovecot-sql.conf.ext' would be: + +---%<------------------------------------------------------------------------- +password_query = \ + SELECT concat(user, '@', domain) AS user, password \ + FROM users \ + WHERE user = '%n' and domain = '%d' +---%<------------------------------------------------------------------------- + +You can also update "username" and "domain" fields separately: + +---%<------------------------------------------------------------------------- +password_query = \ + SELECT user AS username, domain, password \ + FROM users \ + WHERE user = '%n' and domain = '%d' +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.ExtraFields.txt b/doc/wiki/PasswordDatabase.ExtraFields.txt new file mode 100644 index 0000000..bffebfb --- /dev/null +++ b/doc/wiki/PasswordDatabase.ExtraFields.txt @@ -0,0 +1,105 @@ +Password database extra fields +============================== + +The primary purpose of a password database lookup is to return the password for +a given user. It may however also return other fields which are treated +specially: + + * * <user> [PasswordDatabase.ExtraFields.User.txt]*: Change the username (eg. + lowercase it). + * *login_user*: Master passdb can use this to change the username. (v2.2.13+) + * * <allow_nets> [PasswordDatabase.ExtraFields.AllowNets.txt]*: Allow user to + log in from only specified IPs (checks against remote client IP). + * *allow_real_nets*: Allow user's network connection to log in from only + specified IPs (checks against /real/ remote IP, e.g. a Dovecot proxy). + * * <proxy and proxy_maybe> [PasswordDatabase.ExtraFields.Proxy.txt]*: Proxy + the connection to another IMAP/POP3 server. + * * <host> [PasswordDatabase.ExtraFields.Host.txt]*: Send login referral to + client (if proxy=y field isn't set). + * * <nologin> [PasswordDatabase.ExtraFields.NoLogin.txt]*: User isn't actually + allowed to log in even if the password matches, with optionally a different + reason given as the authentication failure message. + * * <nodelay> [PasswordDatabase.ExtraFields.NoDelay.txt]*: Don't delay reply + to client in case of an authentication failure. + * *nopassword*: If you want to allow all passwords, use an empty password and + this field. + * *fail*: If set, explicitly fails the passdb lookup. (v2.2.22+) + * *k5principals*: if using "auth_mechanisms = gssapi", may contain Kerberos v5 + principals allowed to map to the current user, bypassing the internal call + to krb5_kuserok(). The database must support credentials lookup. (v2.2+) + * *delay_until*=<UNIX timestamp>[+<max random secs>]: Delay login until this + time. The timestamp must be less than 5 minutes into future or the login + will fail with internal error. The extra random seconds can be used to avoid + a load spike of everybody getting logged in at exactly the same time. + (v2.2.25+) + * *noauthenticate*: Do not perform any authentication, just store extra fields + if user is found. (v2.2.26+/v2.3) + * *forward_<anything>*: In proxy/director, pass the variable to next hop as + forward_<anything>. (v2.2.29+/v2.3) + +How to return these extra fields depends on the password database you use. See +the <password database> [PasswordDatabase.txt] pages on how to do it. Some +passdbs however don't support returning them at all, such as <PAM> +[PasswordDatabase.PAM.txt]. + +The password database may also return fields prefixed with 'userdb_'. These +fields are only saved and used later as if they came from the <user database> +[UserDatabase.txt]'s extra fields. Typically this is done only when using +<prefetch userdb> [UserDatabase.Prefetch.txt]. + +Note that boolean fields are true always if the field exists. So 'nodelay', +'nodelay=yes', 'nodelay=no' and 'nodelay=0' all mean that the nodelay field is +true. With SQL the field is considered to be non-existent if its value is NULL. + +The following suffixes added to a field name are handled specially: + + * *:protected*: Set this field only if it hasn't been set before. + * *:remove*: Remove this field entirely. + +Examples +-------- + +SQL +--- + +*dovecot-sql.conf.ext*: + +---%<------------------------------------------------------------------------- +password_query = SELECT userid as user, password, 'Y' as proxy, host \ + FROM users WHERE userid = '%u' +---%<------------------------------------------------------------------------- + +LDAP +---- + +*dovecot-ldap.conf*: + +---%<------------------------------------------------------------------------- +pass_attrs = \ + =user=%{ldap:user}, \ + =password=%{ldap:userPassword}, + =proxy=%{ldap:proxyEnabled}, \ + =host=%{ldap:hostName} +---%<------------------------------------------------------------------------- + +Note about the "proxy", "proxy_maybe" and any other boolean type fields: these +represent an existence test. Currently this translates to "will proxy (or +proxy_maybe) if this attribute exists". This allows the proxy behaviour to be +selectable per user. To have it "always" on, use a template, e.g.: + +---%<------------------------------------------------------------------------- +pass_attrs = \ + =user=%{ldap:user}, \ + =password=%{ldap:userPassword}, + =proxy=y, \ + =host=%{ldap:hostName} +---%<------------------------------------------------------------------------- + +passwd-file +----------- + +---%<------------------------------------------------------------------------- +user:{plain}pass::::::proxy=y host=127.0.0.1 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.IMAP.txt b/doc/wiki/PasswordDatabase.IMAP.txt new file mode 100644 index 0000000..02f3db8 --- /dev/null +++ b/doc/wiki/PasswordDatabase.IMAP.txt @@ -0,0 +1,32 @@ +Authentication via remote IMAP server +===================================== + +Available driver settings: + + * host=<template> : IP address or hostname. Allows <%variables> + [Variables.txt] (e.g.'host=imap.%d') + * port=<port> + * username=<template> : The default is %u, but this could be changed to for + example '%n@example.com' + * ssl=imaps / ssl=starttls + * ssl_ca_dir=<path> + * ssl_ca_file=<path> (v2.2.30+) + * allow_invalid_cert=yes|no : Whether to allow authentication even if the + certificate isn't trusted. For v2.2 it defaults to "yes". (v2.2.30+) + * rawlog_dir=<path> + +See also <HowTo.ImapcProxy.txt> for how to combine this with imapc storage. + +Example +------- + +Authenticates users against remote IMAP server in IP address 192.168.1.123: + +---%<------------------------------------------------------------------------- +passdb { + driver = imap + args = host=192.168.1.123 +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.PAM.txt b/doc/wiki/PasswordDatabase.PAM.txt new file mode 100644 index 0000000..29646cf --- /dev/null +++ b/doc/wiki/PasswordDatabase.PAM.txt @@ -0,0 +1,234 @@ +PAM - Pluggable Authentication Modules +====================================== + +This is the most common way to authenticate system users nowadays. PAM is not +itself a password database, but rather its configuration tells the system how +exactly to do the authentication. Usually this means using the 'pam_unix.so' +module, which authenticates user from the system's shadow password file. + +Because PAM is not an actual database, only plaintext authentication mechanisms +can be used with PAM. PAM cannot be used as a user database either (although +static user templates could be used to provide the same effect). Usually PAM is +used with <passwd> [AuthDatabase.Passwd.txt] (NSS) or <static> +[UserDatabase.Static.txt] user databases. + +Dovecot should work with Linux PAM, Solaris PAM, OpenPAM (FreeBSD) and ApplePAM +(Mac OS X). + +Service name +------------ + +The PAM configuration is usually in the '/etc/pam.d/' directory, but some +systems may use a single file,'/etc/pam.conf'. By default Dovecot uses +'dovecot' as the PAM service name, so the configuration is read from +'/etc/pam.d/dovecot'. You can change this by giving the wanted service name in +the 'args' parameter. You can also set the service to '%s' in which case +Dovecot automatically uses either 'imap' or 'pop3' as the service, depending on +the actual service the user is logging in to. Here are a few examples: + + * Use '/etc/pam.d/imap' and '/etc/pam.d/pop3': + + ---%<---------------------------------------------------------------------- + passdb { + driver = pam + args = %s + } + ---%<---------------------------------------------------------------------- + + * Use '/etc/pam.d/mail': + + ---%<---------------------------------------------------------------------- + passdb { + driver = pam + args = mail + } + ---%<---------------------------------------------------------------------- + +PAM sessions +------------ + +By giving a 'session=yes' parameter, you can make Dovecot open a PAM session +and close it immediately. Some PAM plugins need this, for instance +'pam_mkhomedir'. With this parameter, 'dovecot.conf' might look something like +this: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = session=yes dovecot +} +---%<------------------------------------------------------------------------- + +PAM credentials +--------------- + +By giving a 'setcred=yes' parameter, you can make Dovecot create PAM +credentials. Some PAM plugins need this. The credentials are never deleted +however, so using this might cause problems with other PAM plugins. + +Limiting the number of PAM lookups +---------------------------------- + +Usually in other software PAM is used to do only a single lookup in a process, +so PAM plugin writers haven't done much testing on what happens when multiple +lookups are done. Because of this, many PAM plugins leak memory and possibly +have some other problems when doing multiple lookups. If you notice that PAM +authentication stops working after some time, you can limit the number of +lookups done by the auth worker process before it dies: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = max_requests=100 +} +---%<------------------------------------------------------------------------- + +The default max_requests value is 100. + +Username changing +----------------- + +A PAM module can change the username. + +Making PAM plugin failure messages visible +------------------------------------------ + +You can replace the default "Authentication failed" reply with PAM's failure +reply by setting: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = failure_show_msg=yes +} +---%<------------------------------------------------------------------------- + +This can be useful with e.g. pam_opie to find out which one time password +you're supposed to give: + +---%<------------------------------------------------------------------------- +1 LOGIN username otp +1 NO otp-md5 324 0x1578 ext, Response: +---%<------------------------------------------------------------------------- + +Restrict IP-Addresses allowed to connect via PAM +------------------------------------------------ + +You can restrict the IP-Addresses allowed to connect via PAM: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + override_fields = allow_nets=10.1.100.0/23,2001:db8:a0b:12f0::/64 +} +---%<------------------------------------------------------------------------- + +Caching +------- + +Dovecot supports caching password lookups by setting 'auth_cache_size' to +non-zero value. For this to work with PAM, you'll also have to give 'cache_key' +parameter. Usually the user is authenticated only based on the username and +password, but PAM plugins may do all kinds of other checks as well, so this +can't be relied on. For this reason the 'cache_key' must contain all the +<variables> [Variables.txt] that may affect authentication. The commonly used +variables are: + + * '%u' - Username. You'll most likely want to use this. + * '%s' - Service. If you use '*' as the service name you'll most likely want + to use this. + * '%r' - Remote IP address. Use this if you do any IP related checks. + * '%l' - Local IP address. Use this if you do any checks based on the local IP + address that was connected to. + +Examples: + +---%<------------------------------------------------------------------------- +# 1MB auth cache size +auth_cache_size = 1024 +passdb { + driver = pam + # username and service + args = cache_key=%u%s * +} +---%<------------------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +# 1MB auth cache size +auth_cache_size = 1024 +passdb { + driver = pam + # username, remote IP and local IP + args = cache_key=%u%r%l dovecot +} +---%<------------------------------------------------------------------------- + +Examples +-------- + +Linux +----- + +Here is an example '/etc/pam.d/dovecot' configuration file which uses standard +UNIX authentication: + +---%<------------------------------------------------------------------------- +auth required pam_unix.so nullok +account required pam_unix.so +---%<------------------------------------------------------------------------- + +Solaris +------- + +For Solaris you will have to edit '/etc/pam.conf'. Here is a working Solaris +example (using 'args = *' instead of the default 'dovecot' service): + +---%<------------------------------------------------------------------------- +imap auth requisite pam_authtok_get.so.1 +imap auth required pam_unix_auth.so.1 +imap account requisite pam_roles.so.1 +imap account required pam_unix_account.so.1 +imap session required pam_unix_session.so.1 +pop3 auth requisite pam_authtok_get.so.1 +pop3 auth required pam_unix_auth.so.1 +pop3 account requisite pam_roles.so.1 +pop3 account required pam_unix_account.so.1 +pop3 session required pam_unix_session.so.1 +---%<------------------------------------------------------------------------- + +Mac OS X +-------- + +On Mac OS X, the '/etc/pam.d/dovecot' file might look like this: + +---%<------------------------------------------------------------------------- +auth required pam_opendirectory.so try_first_pass +account required pam_nologin.so +account required pam_opendirectory.so +password required pam_opendirectory.so +---%<------------------------------------------------------------------------- + +...which, as the equivalent of '/etc/pam.d/login' on OS X 10.9. For very old +versions of OS X (e.g. 10.4), can be represented (where?) as the following in +the on that OS: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = login +} +---%<------------------------------------------------------------------------- + +On older versions of Mac OS X, "passwd" can be used as a userdb to fill in UID, +GID, and homedir information after PAM was used as a passdb, even though +Directory Services prevents "passdb passwd" from working as a username/password +authenticator. This will provide full system user authentication with true +homedir mail storage, without resorting to a single virtual mail user or LDAP: + +---%<------------------------------------------------------------------------- +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.Shadow.txt b/doc/wiki/PasswordDatabase.Shadow.txt new file mode 100644 index 0000000..f103260 --- /dev/null +++ b/doc/wiki/PasswordDatabase.Shadow.txt @@ -0,0 +1,38 @@ +Shadow +====== + +Works at least with Linux and Solaris, but nowadays <PAM> +[PasswordDatabase.PAM.txt] is usually preferred to this. + +This uses auth-worker processes: + +---%<------------------------------------------------------------------------- +passdb { + driver = shadow +} +---%<------------------------------------------------------------------------- + +By default the auth-worker processes are run as "dovecot" user though, which +normally doesn't have access to '/etc/shadow'. If this is a problem, you can +fix it with: + +---%<------------------------------------------------------------------------- +service auth-worker { + # This should be enough: + group = shadow + # If not, just give full root permissions: + #user = root +} +---%<------------------------------------------------------------------------- + +If there are only a few users and you're using '/etc/shadow' file, there's +really no need to use auth-workers. You can disable them with: + +---%<------------------------------------------------------------------------- +passdb { + driver = shadow + args = blocking=no +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.Static.txt b/doc/wiki/PasswordDatabase.Static.txt new file mode 100644 index 0000000..21852e2 --- /dev/null +++ b/doc/wiki/PasswordDatabase.Static.txt @@ -0,0 +1,27 @@ +Static Password Database +======================== + +Static password database is typically used only for testing, proxying setups +and perhaps some other special kind of setups. Static passdb allows all users +to log in with any username. For password you return either: + + * password=secret: All users have "secret" as password. + * nopassword: Users can log in with any password. + +You can return any other <extra fields> [PasswordDatabase.ExtraFields.txt]. You +can use the standard <variables> [Variables.txt] everywhere. + +Example +------- + +Using <Director.txt>: + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = nopassword=y + default_fields = proxy=y host=127.0.0.1 +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.oauth2.txt b/doc/wiki/PasswordDatabase.oauth2.txt new file mode 100644 index 0000000..5928cf5 --- /dev/null +++ b/doc/wiki/PasswordDatabase.oauth2.txt @@ -0,0 +1,192 @@ +Open Authentication v2.0 database +================================= + +Since v2.2.28. This database works with a oauth2 provider such as google or +facebook. You are recommended to use xoauth2 or oauthbearer <authentication +mechanisms> [Authentication.Mechanisms.txt] with this. The responses from +endpoints must be JSON objects. + +Configuration +------------- + +Common +------ + +In dovecot.conf put + +---%<------------------------------------------------------------------------- +auth_mechanisms = $auth_mechanisms oauthbearer xoauth2 + +passdb { + driver = oauth2 + mechanisms = xoauth2 oauthbearer + args = /etc/dovecot/dovecot-oauth2.conf.ext +} +---%<------------------------------------------------------------------------- + +Backend +------- + +Configuration file example for Google +[https://developers.google.com/identity/protocols/OAuth2] + +---%<------------------------------------------------------------------------- +tokeninfo_url = https://www.googleapis.com/oauth2/v3/tokeninfo?access_token= +introspection_url = https://www.googleapis.com/oauth2/v2/userinfo +#force_introspection = yes +username_attribute = email +tls_ca_cert_file = /etc/ssl/certs/ca-certificates.crt +---%<------------------------------------------------------------------------- + +Configuration file example for WSO2 Identity Server +[http://wso2.com/identity-and-access-management] + +---%<------------------------------------------------------------------------- +introspection_mode = post +introspection_url = +https://adminuser:adminpass@server.name:port/oauth2/introspect +username_attribute = username +tls_ca_cert_file = /etc/ssl/certs/ca-certificates.crt +active_attribute = active +active_value = true +---%<------------------------------------------------------------------------- + +Proxy +----- + +If you want to forward oauth2 authentication to your backend, you can use +various ways + +Without proxy authentication + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = nopasssword=y proxy=y proxy_mech=%m ... +} +---%<------------------------------------------------------------------------- + +or with proxy authentication, put into dovecot-oauth2.conf.ext + +---%<------------------------------------------------------------------------- +pass_attrs = proxy=y proxy_mech=%m +---%<------------------------------------------------------------------------- + +Proxy with password grant (since v2.3.6) +---------------------------------------- + +If you want to configure proxy to get token and pass it to backend + +passdb settings + +---%<------------------------------------------------------------------------- +passdb { + driver = oauth2 + mechanisms = oauthbearer xoauth2 + args = /usr/local/etc/dovecot/dovecot-oauth2.token.conf.ext +} + +passdb { + driver = oauth2 + mechanisms = plain login + args = /usr/local/etc/dovecot/dovecot-oauth2.plain.conf.ext +} +---%<------------------------------------------------------------------------- + +put into dovecot-oauth2.token.conf.ext + +---%<------------------------------------------------------------------------- +grant_url = http://localhost:8000/token +client_id = verySecretClientId +client_secret = verySecretSecret +tokeninfo_url = http://localhost:8000/oauth2?oauth= +introspection_url = http://localhost:8000/introspect +introspection_mode = post +use_grant_password = no +debug = yes +username_attribute = username +pass_attrs = pass=%{oauth2:access_token} +---%<------------------------------------------------------------------------- + +put into dovecot-oauth2.plain.conf.ext + +---%<------------------------------------------------------------------------- +grant_url = http://localhost:8000/token +client_id = verySecretClientId +client_secret = verySecretSecret +introspection_url = http://localhost:8000/introspect +introspection_mode = post +use_grant_password = yes +debug = yes +username_attribute = username +pass_attrs = host=127.0.0.1 proxy=y proxy_mech=xoauth2 +pass=%{oauth2:access_token} +---%<------------------------------------------------------------------------- + +Full config file +---------------- + +---%<------------------------------------------------------------------------- +### OAuth2 password database configuration + +## url for verifying token validity. Token is appended to the URL +# tokeninfo_url = http://endpoint/oauth/tokeninfo?access_token= + +## introspection endpoint, used to gather extra fields and other information. +# introspection_url = http://endpoint/oauth/me + +## How introspection is made, valid values are +## auth = GET request with Bearer authentication +## get = GET request with token appended to URL +## post = POST request with token=bearer_token as content +# introspection_mode = auth + +## Force introspection even if tokeninfo contains wanted fields +## Set this to yes if you are using active_attribute +# force_introspection = no + +## wanted scope of validity (optional) +# scope = something + +## username attribute in response (default: email) +# username_attribute = email + +## username normalization format (default: %Lu) +# username_format = %Lu + +## Attribute name for checking whether account is disabled (optional) +# active_attribute = + +## Expected value in active_attribute (empty = require present, but anything +goes) +# active_value = + +## Extra fields to set in passdb response (in passdb static style) +# pass_attrs = + +## Timeout in milliseconds +# timeout_msecs = 0 + +## Enable debug logging +# debug = no + +## Max parallel connections (how many simultaneous connections to open) +# max_parallel_connections = 1 + +## Max pipelined requests (how many requests to send per connection, requires +server-side support) +# max_pipelined_requests = 1 + +## HTTP request raw log directory +# rawlog_dir = /tmp/oauth2 + +## TLS settings +# tls_ca_cert_file = /path/to/ca-certificates.txt +# tls_ca_cert_dir = /path/to/certs/ +# tls_cert_file = /path/to/client/cert +# tls_key_file = /path/to/client/key +# tls_cipher_suite = HIGH:!SSLv2 +# tls_allow_invalid_cert = FALSE +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PasswordDatabase.txt b/doc/wiki/PasswordDatabase.txt new file mode 100644 index 0000000..f41a93e --- /dev/null +++ b/doc/wiki/PasswordDatabase.txt @@ -0,0 +1,182 @@ +Password Databases +================== + +Dovecot authenticates users against password databases. It can also be used to +configure things like <proxies> [PasswordDatabase.ExtraFields.Proxy.txt]. + +You can use multiple databases, so if the password doesn't match in the first +database, Dovecot checks the next one. This can be useful if you want to easily +support having both virtual users and also local system users (see +<Authentication.MultipleDatabases.txt>). + +Success/failure databases +------------------------- + +These databases simply verify if the given password is correct for the user. +Dovecot doesn't get the correct password from the database, it only gets a +"success" or a "failure" reply. This means that these databases can't be used +with non-plaintext <authentication mechanisms> [Authentication.Mechanisms.txt]. + +Databases that belong to this category are: + + * <PAM> [PasswordDatabase.PAM.txt]: Pluggable Authentication Modules. + * <BSDAuth> [PasswordDatabase.BSDAuth.txt]: BSD authentication. + * <CheckPassword> [AuthDatabase.CheckPassword.txt]: External checkpassword + program without Dovecot extensions. + * <IMAP> [PasswordDatabase.IMAP.txt]: Authenticate against remote IMAP server. + + * <OAuth2> [PasswordDatabase.oauth2.txt]: Authenticate against oauth2 provider + (v2.2.29+) + +Lookup databases +---------------- + +Dovecot does a lookup based on the username and possibly other information +(e.g. IP address) and verifies the password validity itself. Fields that the +lookup can return: + + * * <password> [Authentication.PasswordSchemes.txt]*: User's password. + * *password_noscheme*: Like "password", but if a password begins with "{", + assume it belongs to the password itself instead of treating it as a + <scheme> [Authentication.PasswordSchemes.txt] prefix. This is usually + needed only if you use plaintext passwords. + * * <user> [PasswordDatabase.ExtraFields.User.txt]*: Returning a user field + can be used to change the username. Typically used only for case changes + (e.g. "UseR" -> "user"). + * *username*: Like user, but doesn't drop existing domain name (e.g. + "username=foo" for "user@domain" gives "foo@domain"). + * *domain*: Updates the domain part of the username. + * Other special <extra fields> [PasswordDatabase.ExtraFields.txt]. + +Databases that support looking up only passwords, but no user or extra fields: + + * <Passwd> [AuthDatabase.Passwd.txt]: System users (NSS, '/etc/passwd', or + similiar). + * <Shadow> [PasswordDatabase.Shadow.txt]: Shadow passwords for system users + (NSS,'/etc/shadow' or similiar). + * Dovecot supports reading all <password schemes> + [Authentication.PasswordSchemes.txt] from passwd and shadow databases (if + prefix is specified), but that is of course incompatible with all other + tools using/modifying the passwords. + * <VPopMail> [AuthDatabase.VPopMail.txt]: External software used to handle + virtual domains. + +Databases that support looking up everything: + + * <Passwd-file> [AuthDatabase.PasswdFile.txt]: '/etc/passwd'-like file in + specified location. + * <LDAP> [AuthDatabase.LDAP.txt]: Lightweight Directory Access Protocol. + * <SQL> [AuthDatabase.SQL.txt]: SQL database (PostgreSQL, MySQL, SQLite). + * <Dict> [AuthDatabase.Dict.txt]: Dict key-value database (Redis, memcached, + etc.) + * <CheckPassword> [AuthDatabase.CheckPassword.txt]: External checkpassword + program when used with Dovecot extensions. + * <Static> [PasswordDatabase.Static.txt]: Static passdb for simple + configurations + * <Lua> [AuthDatabase.Lua.txt]: Lua script for authentication (v2.3.0+) + +Passdb settings +--------------- + +An example passdb passwd-file with its default settings: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = scheme=ssha256 /usr/local/etc/passwd.replica + default_fields = + override_fields = + + deny = no + master = no + pass = no + skip = never + mechanisms = + username_filter = + + result_failure = continue + result_internalfail = continue + result_success = return-ok + + # v2.2.24+ + auth_verbose = default +} +---%<------------------------------------------------------------------------- + +First we have the settings that provide content for the passdb lookup: + + * driver: The passdb backend name + * args: Arguments for the passdb backend. The format of this value depends on + the passdb driver. Each one uses different args. + * default_fields: Passdb fields (and <extra fields> + [PasswordDatabase.ExtraFields.txt]) that are used, unless overwritten by the + passdb backend. They are in format 'key=value key2=value2 ...'. The values + can contain <%variables> [Variables.txt]. + * override_fields: Same as default_fields, but instead of providing the + default values, these values override what the passdb backend returned. + * auth_verbose: If this is explicitly set to yes or no, it overrides the + global auth_verbose setting. (However, auth_debug=yes overrides the + auth_verbose setting.) (v2.2.24+) + +Then we have the settings which specify when the passdb is used: + + * deny: If "yes", used to provide "denied users database". If the user is + found from the passdb, the authentication will fail. + * master: If "yes", used to provide <master users database> + [Authentication.MasterUsers.txt]. The users listed in the master passdb can + log in as other users. + * pass: This is an alias for 'result_success = continue' as described + below. This was commonly used together with master passdb to specify that + even after a successful master user authentication, the authentication + should continue to the actual non-master passdb to lookup the user. + * skip: Do we sometimes want to skip over this passdb? + * never + * authenticated: Skip if an earlier passdb already authenticated the user + successfully. + * unauthenticated: Skip if user hasn't yet been successfully authenticated + by the previous passdbs. + * mechanisms: Skip, if non-empty and the current auth mechanism is not listed + here. Space or comma-separated list of auth mechanisms (e.g. "PLAIN LOGIN"). + Also "none" can be used to match for a non-authenticating passdb lookup. + (v2.2.30+) + * username_filter: Skip, if non-empty and the username doesn't match the + filter. This is mainly used to assign specific passdbs to specific domains. + Space or comma-separated list of username filters that can have "*" or "?" + wildcards. If any of the filters matches, the filter succeeds. However, + there can also be negative matches preceded by "!". If any of the negative + filters matches, the filter won't succeed. For example if the filter is + "*@example.com *@example2.com !user@example.com", "any@example.com" or + "user@example2.com" matches but "user@example.com" won't match. (v2.2.30+) + +And finally we can control what happens when we're finished with this passdb: + + * result_success: What to do if the authentication succeeded (default: + return-ok) + * result_failure: What to do if authentication failed (default: continue) + * result_internalfail: What to do if the passdb lookup had an internal failure + (default: continue). If any of the passdbs had an internal failure and the + final passdb also returns "continue", the authentication will fail with + "internal error".*WARNING*: If multiple passdbs are required (results are + merged), it's important to set result_internalfail=return-fail to them, + otherwise the authentication could still succeed but not all the intended + extra fields are set. + +The result values that can be used: + + * return-ok: Return success, don't continue to the next passdb. + * return-fail: Return failure, don't continue to the next passdb. + * return: Return earlier passdb's success or failure, don't continue to the + next passdb. If this was the first passdb, return failure. + * continue-ok: Set the current authentication state to success, and continue + to the next passdb. The following passdbs will skip password verification. + * continue-fail: Set the current authentication state to failure, and continue + to the next passdb. The following passdbs will still verify the password. + * continue: Continue to the next passdb without changing the authentication + state. The initial state is failure. If this was set in result_success, the + following passdbs will skip password verification. + +*NOTICE*: when using "continue*" values on a *master* passdb (master = yes), +execution will jump to the first *non-master* passdb instead of continuing with +the next master passdb (verified at lest up to v2.2.27). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PerformanceTuning.txt b/doc/wiki/PerformanceTuning.txt new file mode 100644 index 0000000..80e7fca --- /dev/null +++ b/doc/wiki/PerformanceTuning.txt @@ -0,0 +1,69 @@ +Dovecot performance tuning +========================== + +Disk I/O optimization +--------------------- + +Usually heavily loaded IMAP and POP3 servers don't use much CPU, but they use +all the disk I/O they can get. So reducing disk I/O is probably the most useful +optimization you can do. + + * See <MailLocation.LocalDisk.txt> for generic disk I/O optimizations. + * See <MailLocation.Mbox.txt> for mbox-specific optimizations. + * See <MailLocation.Maildir.txt> for Maildir-specific optimizations. + * See <dbox> [MailboxFormat.dbox.txt] for Dovecot's own high-performance + mailbox format. It usually gives much better performance than mbox/Maildir. + * See <full text search indexes> [Plugins.FTS.txt] for optimizing IMAP SEARCH + command. + * See <POP3Server.txt> for POP3 optimizations, especially + 'pop3_no_flag_updates=yes' + * 'mailbox_list_index=yes' can help a lot by replying to IMAP STATUS (and + similar) lookups from a single index without having to open each mailbox + index separately. This is the default in v2.3+. + * Also 'mailbox_list_index_very_dirty_syncs=yes' makes Dovecot assume that + the list index is up-to-date. + * 'mail_prefetch_count' setting may be helpful with some mailbox formats + * 'mail_location = ...:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u' moves + e.g. lock files to the volatile directory. This is helpful especially if + mail_location otherwise points to a remote filesystem like NFS. (v2.2.32+) + * If acl plugin is used, but only global ACLs are needed, set + 'acl_globals_only=yes' (v2.2.31+) + +CPU usage optimization +---------------------- + + * See <LoginProcess.txt> for optimizing CPU usage caused by logins + * See 'auth_cache_size' setting for caching passdb and userdb lookups + * To distribute password hash calculations to multiple CPU cores (via + auth-worker processes), set 'auth_cache_verify_password_with_worker=yes' + (v2.2.34+) + * Services having client_limit>1 and process_limit>1, set process_min_avail to + the number of CPU cores + * To reduce forks by reusing existing processes for new requests increase + 'service { service_count }' from 1 to higher (e.g. 100) for imap and pop3 + services. It's better not to set it too high or unlimited (0), because + different users use different amounts of memory, and it's wasteful when a + lot of processes end up having a lot of "free" memory. + +Memory usage optimization +------------------------- + +There aren't many settings which affect Dovecot's memory usage. In general +Dovecot uses as much memory as it needs, which is usually quite little. + + * 'auth_cache_size' controls maximum memory size for caching passdb/userdb + lookups + * <High-performance mode for login processes> [LoginProcess.txt]. + * 'imap_hibernate_timeout' controls when to move IDLEing IMAP connections to + wait for changes in a shared imap-hibernate process. This frees up the imap + process. + +Note that these settings do not directly affect the memory usage: + + * 'service { vsz_limit } ': These are simply safe guards against potential + memory leaks. If the process's virtual size reaches the limit, the process + is killed by the kernel. + * 'service { process_limit, client_limit } ': These are mostly to avoid DoS + attacks using up all your memory. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Installation.txt b/doc/wiki/Pigeonhole.Installation.txt new file mode 100644 index 0000000..78b2c56 --- /dev/null +++ b/doc/wiki/Pigeonhole.Installation.txt @@ -0,0 +1,159 @@ +Pigeonhole Installation +======================= + +Contents + + + 1. Pigeonhole Installation + + 1. Getting the sources + + 2. Compiling + + 3. Prebuilt Binaries + + 1. Alpine Linux + + 2. ArchLinux + + 3. RHEL 6 + clones (CentOS, Scientific Linux, ...) + + 4. Debian + + 5. openSUSE + + 6. FreeBSD + + 7. OpenBSD + +Getting the sources +------------------- + +You can download the latest released sources from the Pigeonhole download page +[http://pigeonhole.dovecot.org/download.html]. + +Alternatively, you can get the sources, including the most recent unreleased +changes, from the the Mercurial repository: + +---%<------------------------------------------------------------------------- +hg clone http://hg.rename-it.nl/dovecot-2.0-pigeonhole +---%<------------------------------------------------------------------------- + +Compiling +--------- + +If you downloaded the sources using Mercurial, you will need to execute +'./autogen.sh' first to build the automake structure in your source tree. This +process requires autotools and libtool to be installed. + +If you installed Dovecot from sources, Pigeonhole's configure script should be +able to find the installed 'dovecot-config' automatically: + +---%<------------------------------------------------------------------------- +./configure +make +sudo make install +---%<------------------------------------------------------------------------- + +If this doesn't work, you can use '--with-dovecot=<path>' configure option, +where the path points to a directory containing 'dovecot-config' file. This can +point to an installed file: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=/usr/local/lib/dovecot +make +sudo make install +---%<------------------------------------------------------------------------- + +or to Dovecot source directory that is already compiled: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=../dovecot-2.0.0/ +make +sudo make install +---%<------------------------------------------------------------------------- + +*IMPORTANT*: You need to recompile Pigeonhole when you upgrade Dovecot to a new +version, because otherwise the Sieve interpreter plugin will fail to load with +a version error. + +Prebuilt Binaries +----------------- + +Alpine Linux +------------ + +Pigeonhole can be installed from packages by running: + +---%<------------------------------------------------------------------------- +apk add dovecot-pigeonhole-plugin +---%<------------------------------------------------------------------------- + +ArchLinux +--------- + +Pidgeonhole is available in the community repositories, and can be installed by +running: + +---%<------------------------------------------------------------------------- +pacman -S pigeonhole +---%<------------------------------------------------------------------------- + +RHEL 6 + clones (CentOS, Scientific Linux, ...) +----------------------------------------------- + +Pidgeonhole is available in the main repository, and can be installed by +running: + +---%<------------------------------------------------------------------------- +yum install dovecot-pigeonhole +---%<------------------------------------------------------------------------- + +Debian +------ + +Starting with Debian Wheezy, Pigeonhole binaries are distributed in separate +packages:'dovecot-sieve' for the <Sieve interpreter> [Pigeonhole.Sieve.txt] and +'dovecot-managesieved' for the <ManageSieve service> +[Pigeonhole.ManageSieve.txt]. You can install these by running: + +---%<------------------------------------------------------------------------- +apt-get install dovecot-sieve dovecot-managesieved +---%<------------------------------------------------------------------------- + +Older Debian releases have Sieve and ManageSieve support included in the main +'dovecot-common' package, meaning that this is always available for those +releases once Dovecot is installed. + +openSUSE +-------- + +It is part of the dovecot (dovecot21) rpm. There is no need to install +additional packages. + +FreeBSD +------- + +Pigeonhole can be installed from ports by running: + +---%<------------------------------------------------------------------------- +cd /usr/ports/mail/dovecot-pigeonhole +make install clean +---%<------------------------------------------------------------------------- + +It can be also be installed from packages by running: + +---%<------------------------------------------------------------------------- +pkg install dovecot-pigeonhole +---%<------------------------------------------------------------------------- + +OpenBSD +------- + +Pigeonhole can be installed from packages by running: + +---%<------------------------------------------------------------------------- +pkg_add dovecot-pigeonhole +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.ManageSieve.Clients.txt b/doc/wiki/Pigeonhole.ManageSieve.Clients.txt new file mode 100644 index 0000000..250f0d5 --- /dev/null +++ b/doc/wiki/Pigeonhole.ManageSieve.Clients.txt @@ -0,0 +1,61 @@ +ManageSieve Client Issues +========================= + +Although this ManageSieve server should comply with the draft specification of +theManageSieve protocol, quite a few clients don't. This page lists the known +client problems. + + * *The TLS problem*: + * The core of the TLS problem is that a ManageSieve server is required to send + an unsolicited CAPABILITY response right after successful TLS negotiation. + Older Cyrus servers did not do this and many clients incorporated this + protocol error as the standard, meaning that these do not expect the + CAPABILITY response and thus fail with subsequent commands. However, now + that Cyrus' Timsieved has changed its behaviour towards protocol compliance, + all those clients will follow eventually. The following clients are known to + have this TLS issue: + + Thunderbird Sieve add-on: + TLS broken for old versions. Starting with version 0.1.5 the Thunderbird + Sieve add-on properly supports TLS. + + KMail + kio_sieve: + TLS broken for old versions. This issue is fixed at least in kmail 1.9.9 / + kde 3.5.9. + + SquirrelMail/AvelSieve: + For some users the !Avelsieve client stores scripts but fails to retrieve + them later. This problem is directly caused byAvelSieve's TLS support. A + quick way to fix this is not to enable TLS forManageSieve. AvelSieve + stable (v1.0.1) does not have TLS support at all, so you will see this + happen only with the development or SVN versions. Another issue is that + (at least with avelsieve-1.9.7) it is impossible to delete the last rule + of a script. For avelsieve-1.9.7 you find a patch that fixes these two + issues here + [http://pigeonhole.dovecot.org/client-patches/avelsieve-1.9.7-dovecot.patch]. + + * *Smartsieve, Websieve*: + * These clients are specifically written for Cyrus timsieved and fail on + multiple stages of the protocol when connected to PigeonholeManageSieve. + * /(Stephan Bosch)/: I intend to look at these in the future, but currently + these are very much unavailable for use with Dovecot. But, feel free to + fix these yourself:) + * /(Steffen "Stefreak" Neubauer)/: I've fixed the problems for smartsieve. + Just replace your lib/Managesieve.php with this + here:https://www.commail.org/sieve/lib/Managesieve.phps - HAVE FUN! + * /(Mark Titorenko)/: Stephan provides an updated link to Steffen's + ManageSieve patch in this Dovecot mailing list post dated 2009-09-01 + [http://www.mail-archive.com/dovecot@dovecot.org/msg21862.html] + * /(Stephan Bosch)/: Links moved here + [http://pigeonhole.dovecot.org/patches/] + * *Ruby/Managesieve*: Ruby command line client and library to managesieve + works fine Ruby/Managesieve [http://managesieve.rubyforge.org/] + * *Ruby/Sieve-Parser*: Ruby library for sieve parsing Sieve-Parser + [http://rubygems.org/gems/sieve-parser/] + +*NOTE*: If you add new issues to this list, notify the author or send an e-mail +to the Dovecot mailing list [http://dovecot.org/mailinglists.html]. In any +case, you must make sure that the issue is properly explained and that the +author can contact you for more information. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.ManageSieve.Configuration.txt b/doc/wiki/Pigeonhole.ManageSieve.Configuration.txt new file mode 100644 index 0000000..bf40848 --- /dev/null +++ b/doc/wiki/Pigeonhole.ManageSieve.Configuration.txt @@ -0,0 +1,258 @@ +ManageSieve Configuration +========================= + +*NOTE*: If you have used the Sieve plugin before and you have '.dovecot.sieve' +files in user directories, you are advised to *make a backup first*. Although +theManageSieve daemon takes care to move these files to the Sieve storage +before it is substituted with a symbolic link, this is not a very well tested +operation, meaning that there is a possibility that existing Sieve scripts get +lost. + +The ManageSieve configuration consists of ManageSieve protocol settings and +<Sieve interpreter> [Pigeonhole.Sieve.txt]-related settings. The Sieve +interpreter settings are shared with settings of the <Sieve plugin> +[Pigeonhole.Sieve.txt] for Dovecot's <Local Delivery Agent (LDA)> [LDA.txt] and +<LMTP.txt>. First, the ManageSieve protocol settings are outlined and then the +relevant Sieve settings are described. + +Protocol Configuration +---------------------- + +Along with all other binaries that Dovecot uses, the managesieve and +managesieve-login binaries are installed during 'make install' of the +<Pigeonhole.txt> package. The only thing you need to do to activate the +ManageSieve protocol support in Dovecot is to add 'sieve' to the 'protocols=' +configuration line in your dovecot.conf. The managesieve daemon will listen on +port 4190 by default. As the implementation of the managesieve daemon is +largely based on the original IMAP implementation, it is very similar in terms +of configuration. In addition to most mail daemon config settings, the +managesieve daemon accepts a few more. The following settings can be configured +in the 'protocol sieve' section: + +managesieve_max_line_length = 65536: + The maximum ManageSieve command line length in bytes. This setting is + directly borrowed from IMAP. But, since long command lines are very unlikely + withManageSieve, changing this will not be very useful. + +managesieve_logout_format = bytes=%i/%o: + Specifies the string pattern used to compose the logout message of an + authenticated session. The following substitutions are available: + +: + ---%<----------------------------------------------------------------------- + %i - total number of bytes read from client + %o - total number of bytes sent to client + ---%<----------------------------------------------------------------------- + +managesieve_implementation_string = Dovecot Pigeonhole: + To fool ManageSieve clients that are focused on CMU's timesieved you can + specify the IMPLEMENTATION capability that the Dovecot reports to clients + (e.g. 'Cyrus timsieved v2.2.13'). + +managesieve_max_compile_errors = 5: + The maximum number of compile errors that are returned to the client upon + script upload or script verification. + +managesieve_sieve_capability =, managesieve_notify_capability = : + Respectively the SIEVE and NOTIFY capabilities reported by the ManageSieve + service before authentication. If left unassigned, these will be assigned + dynamically according to what the Sieve interpreter supports by default + (after login this may differ depending on the authenticated user). + +Sieve Interpreter Configuration +------------------------------- + +The part of the <Sieve interpreter> [Pigeonhole.Sieve.txt] configuration that +is relevant forManageSieve mainly consists of the settings that specify where +the user's scripts are stored and where the active script is located. +TheManageSieve service primarily uses the following Sieve interpreter setting +in the 'plugin' section of the Dovecot configuration: + +sieve = file:~/sieve;active=~/.dovecot.sieve : + This specifies the <location> [Pigeonhole.Sieve.Configuration.txt] where the + scripts that are uploaded throughManageSieve are stored. During delivery, the + LDA Sieve plugin uses this location setting to find the active script for + Sieve filtering. The Sieve include extension uses this location for + retrieving ":personal" scripts. If the location type does not allow uploading + scripts, theManageSieve service cannot be used. Currently, only the ' <file> + [Pigeonhole.Sieve.Configuration.File.txt]' <location type> + [Pigeonhole.Sieve.Configuration.txt] supports ManageSieve. + +: + For the ' <file> [Pigeonhole.Sieve.Configuration.File.txt]' <location type> + [Pigeonhole.Sieve.Configuration.txt]: + + * The location is the path to the storage directory for all the user's + personal Sieve scripts. Scripts are stored as separate files with + extension '.sieve'. All other files are ignored when scripts are listed by + aManageSieve client. The storage directory is always generated + automatically if it does not exist (as far as the system permits the user + to do so; no root privileges are used). This is similar to the behavior of + the mail daemons regarding the 'mail_location' configuration. + * ManageSieve maintains a symbolic link pointing to the currently active + script (the script executed at delivery). The location of this symbolic + link can be configured using the ';active=<path>' option. If a regular + file already exists at the location specified by in the ';active=<path>' + location option, it is moved to the storage directory before the symbolic + link is installed. It is renamed to 'dovecot.orig.sieve' and therefore + listed as 'dovecot.orig' by a ManageSieve client. *Note:* It is not wise + to place this active symbolic link inside your mail store, as it may be + mistaken for a mail folder. Inside a maildir for instance, the default + '.dovecot.sieve' would show up as phantom folder //dovecot/sieve/ in your + IMAP tree. + +: + For Pigeonhole versions before v0.3.1, this setting can only be a filesystem + path pointing to a script file, or - whenManageSieve is used - it is the + location of the symbolic link pointing to the active script in the storage + directory. That storage directory is then configured using the deprecated + 'sieve_dir' setting. + +Quota Support +------------- + +By default, users can manage an unlimited number of Sieve scripts on the server +throughManageSieve. However, ManageSieve can be configured to enforce limits on +the number of personal Sieve scripts per user and/or the amount of disk storage +used by these scripts. The maximum size of individual uploaded scripts is +dictated by the configuration of the <Sieve interpreter> +[Pigeonhole.Sieve.txt]. The limits are configured in the plugin section of the +Dovecot configuration as follows: + +sieve_max_script_size = 1M: + The maximum size of a Sieve script. + +sieve_quota_max_scripts = 0: + The maximum number of personal Sieve scripts a single user can have. + +sieve_quota_max_storage = 0: + The maximum amount of disk storage a single user's scripts may occupy. + +A value of 0 for these settings means that no limit is enforced. + +Examples +-------- + +The following provides example configurations for ManageSieve in dovecot.conf +for the various versions. Only sections relevant toManageSieve and the Sieve +plugin are shown. Refer to 20-managesieve.conf in +doc/dovecot/example-config/conf.d, but don't forget to add 'sieve' to the +'protocols' setting if you use it. + +---%<------------------------------------------------------------------------- +... +service managesieve-login { + #inet_listener sieve { + # port = 4190 + #} + + #inet_listener sieve_deprecated { + # port = 2000 + #} + + # Number of connections to handle before starting a new process. Typically + # the only useful values are 0 (unlimited) or 1. 1 is more secure, but 0 + # is faster. <doc/wiki/LoginProcess.txt> + #service_count = 1 + + # Number of processes to always keep waiting for more connections. + #process_min_avail = 0 + + # If you set service_count=0, you probably need to grow this. + #vsz_limit = 64M +} + +service managesieve { + # Max. number of ManageSieve processes (connections) + #process_limit = 1024 +} + +# Service configuration + +protocol sieve { + # Maximum ManageSieve command line length in bytes. ManageSieve usually does + # not involve overly long command lines, so this setting will not normally +need + # adjustment + #managesieve_max_line_length = 65536 + + # Maximum number of ManageSieve connections allowed for a user from each IP +address. + # NOTE: The username is compared case-sensitively. + #mail_max_userip_connections = 10 + + # Space separated list of plugins to load (none known to be useful so far). +Do NOT + # try to load IMAP plugins here. + #mail_plugins = + + # MANAGESIEVE logout format string: + # %i - total number of bytes read from client + # %o - total number of bytes sent to client + #managesieve_logout_format = bytes=%i/%o + + # To fool ManageSieve clients that are focused on CMU's timesieved you can +specify + # the IMPLEMENTATION capability that the dovecot reports to clients. + # For example: 'Cyrus timsieved v2.2.13' + #managesieve_implementation_string = Dovecot Pigeonhole + + # Explicitly specify the SIEVE and NOTIFY capability reported by the server +before + # login. If left unassigned these will be reported dynamically according to +what + # the Sieve interpreter supports by default (after login this may differ +depending + # on the user). + #managesieve_sieve_capability = + #managesieve_notify_capability = + + # The maximum number of compile errors that are returned to the client upon +script + # upload or script verification. + #managesieve_max_compile_errors = 5 + + # Refer to 90-sieve.conf for script quota configuration and configuration of + # Sieve execution limits. +} + +plugin { + # Used by both the Sieve plugin and the ManageSieve protocol + sieve = file:~/sieve;active=~/.dovecot.sieve +} +---%<------------------------------------------------------------------------- + +Proxy +----- + +Like Dovecot's imapd, the ManageSieve login daemon supports proxying to +multiple backend servers. The <proxy configuration wiki page> +[PasswordDatabase.ExtraFields.Proxy.txt] for POP3 and IMAP applies +automatically toManageSieve as well. + +Migration from Dovecot v1.x ManageSieve +--------------------------------------- + +The following has changed since the ManageSieve releases for Dovecot v1.x: + + * For Dovecot v1.0 and v1.1, the 'sieve_dir' setting used by ManageSieve was + called 'sieve_storage'. Also, the 'sieve' and 'sieve_storage' settings were + located inside the 'protocol managesieve' section of the configuration. As + per Dovecot v1.2 these settings are shared with the Sieve plugin and located + in the 'plugin' section of the configuration. Make sure you have updated the + name of the 'sieve_dir' setting and the location of both these settings if + you are upgrading fromManageSieve for Dovecot v1.0/v1.1. + * Pigeonhole ManageSieve does not use the 'mail_location' configuration as a + fall-back anymore to determine a default location for storing Sieve scripts. + It always uses the 'sieve_dir' setting, with default value '~/sieve'. + * The Pigeonhole ManageSieve service now binds to TCP port 4190 by default due + to the IANA port assignment for theManageSieve service. When upgrading from + v1.x, this should be taken into account. For a smooth transition, the + service can be configured manually to listen on both port 2000 and port + 4190, as demonstrated in the example section. + * The Dovecot configuration now calls the ManageSieve protocol 'sieve' instead + of 'managesieve' because it is registered as such with IANA. The binaries + and the services are still called 'managesieve' and 'managesieve-login'. The + example section demonstrates how this affects the configuration. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.ManageSieve.Install.txt b/doc/wiki/Pigeonhole.ManageSieve.Install.txt new file mode 100644 index 0000000..b9f67ad --- /dev/null +++ b/doc/wiki/Pigeonhole.ManageSieve.Install.txt @@ -0,0 +1,2 @@ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.ManageSieve.Troubleshooting.txt b/doc/wiki/Pigeonhole.ManageSieve.Troubleshooting.txt new file mode 100644 index 0000000..d19bd18 --- /dev/null +++ b/doc/wiki/Pigeonhole.ManageSieve.Troubleshooting.txt @@ -0,0 +1,201 @@ +ManageSieve Troubleshooting +=========================== + +Like Dovecot itself, *the ManageSieve service always logs a detailed error +message* if something goes wrong at the server (refer to <Dovecot Logging> +[Logging.txt] for more details): the logs are the first place to look if you +suspect something is wrong. To get additional debug messages in your log file, +you should set 'mail_debug=yes' in dovecot.conf (inside 'protocol sieve {...} +'if you want to enable this forManageSieve only). + +If the client commits protocol violations or sends invalid scripts, an error +response is provided to the client which is not necessarily logged on the +server. A goodManageSieve client presents such error messages to the user. + +Keep in mind that the the ManageSieve service only provides the Sieve +/protocol/, which may be somewhat confusing. This protocol can only be used to +/upload/ Sieve scripts and /activate/ them for execution. Performing the steps +below therefore only verifies that this functionality is working and *not* +whether Sieve scripts are correctly being executed upon delivery. The execution +of Sieve scripts is performed by the Dovecot <Local Delivery Agent (LDA)> +[LDA.txt] or its <LMTP service> [LMTP.txt] using the <LDA Sieve plugin> +[Pigeonhole.Sieve.txt]. If you have problems with Sieve script execution upon +delivery, you are referred to the <Sieve Troubleshooting page> +[Pigeonhole.Sieve.Troubleshooting.txt]. + +Manual Login and Script Upload +------------------------------ + +If you fail to login or upload scripts to the server, it is not necessarily +caused by Dovecot or your configuration. It is often best to test +yourManageSieve server manually first. This also provides you with the direct +error messages from the server without intermission of your client. If you do +not use TLS, you can connect using a simple 'telnet' or 'netcat' connection to +the configured port (typically 4190 or 2000 for older setups). Otherwise you +must use a TLS-capable text protocol client like 'gnutls-cli' as described +below. Upon connection, the server presents the initial greeting with its +capabilities: + +---%<------------------------------------------------------------------------- +"IMPLEMENTATION" "dovecot" +"SASL" "PLAIN" +"SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify +include envelope body relational regex subaddress copy" +"STARTTLS" +OK "Dovecot ready." +---%<------------------------------------------------------------------------- + +Note that the reported 'STARTTLS' capability means that the server accepts TLS, +but, since you are using telnet/netcat, you cannot use this (refer to Manual +TLS Login below). The 'SASL' capability lists the available SASL authentication +mechanisms. If this list is empty and 'STARTTLS' is available, it probably +means that the server forces you to initiate TLS first (as dictated by +''disable_plaintext_auth=yes'' in dovecot.conf). + +Now you need to log in. Although potentially multiple SASL mechanisms are +available, only 'PLAIN' is described here. Authentication is performed using +theManageSieve 'AUTHENTICATE' command. This command typically looks as follows +when the 'PLAIN' mechanism is used: + +---%<------------------------------------------------------------------------- +AUTHENTICATE "PLAIN" "<base64-encoded credentials>" +---%<------------------------------------------------------------------------- + +The credentials are the base64-encoded version of the string +'"\0<username>\0<password"' (in which '\0' represents the ASCII NUL character). +Generating this is cumbersome and a bit daunting for the novice user, so for +convenience a simple Perl script is provided to generate the 'AUTHENTICATE' +command for you. It is available here +[http://pigeonhole.dovecot.org/utilities/sieve-auth-command.pl] and used as +follows: + +---%<------------------------------------------------------------------------- +sieve-auth-command.pl <username> <password> +---%<------------------------------------------------------------------------- + +The command is written to stdout and you can paste this to your protocol +session, e.g.: + +---%<------------------------------------------------------------------------- +AUTHENTICATE "PLAIN" "AHVzZXJuYW1lAHBhc3N3b3Jk" +OK "Logged in." +---%<------------------------------------------------------------------------- + +Now that you are logged in, you can upload a script. This is done using the +'PUTSCRIPT' command. Its first argument is the name for the script and its +second argument is a string literal. A string literal starts with a length +specification ''{<bytes>+}'' followed by a newline. Thereafter the server +expects '<bytes>' bytes of script data. The following uploads a trivial 6 byte +long sieve script that keeps every message (6th byte is the newline character): + +---%<------------------------------------------------------------------------- +PUTSCRIPT "hutsefluts" {6+} +keep; +OK "Putscript completed." +---%<------------------------------------------------------------------------- + +Upon successful upload, you should find a file called 'hutsefluts.sieve' in +your 'sieve_dir' directory. The script should also be listed by the server as +follows when the 'LISTSCRIPTS' command is issued: + +---%<------------------------------------------------------------------------- +LISTSCRIPTS +"hutsefluts" +OK "Listscripts completed." +---%<------------------------------------------------------------------------- + +You can check whether your script is uploaded correctly by downloading it using +the 'GETSCRIPT' command. This command accepts the name of the downloaded script +as its only parameter: + +---%<------------------------------------------------------------------------- +GETSCRIPT "hutsefluts" +{6} +keep; +OK "Getscript completed." +---%<------------------------------------------------------------------------- + +To let the Sieve plugin use your newly uploaded script, you must activate it +using the 'SETACTIVE' command (only one script can be active at any time). The +active script is indicated 'ACTIVE' in the 'LISTSCRIPTS' output, e.g.: + +---%<------------------------------------------------------------------------- +SETACTIVE "hutsefluts" +OK "Setactive completed." +LISTSCRIPTS +"hutsefluts" ACTIVE +OK "Listscripts completed. +---%<------------------------------------------------------------------------- + +The symbolic link configured with the 'sieve' setting should now point to the +activated script in the 'sieve_dir' directory. If no script is active, this +symbolic link is absent. + +Manual TLS Login +---------------- + +When TLS needs to be used during manual testing, 'gnutls-cli' provides the +means to do so. This command-line utility is part of the GNUTLS distribution +and on most systems this should be easy to install. It is used to connect +toManageSieve as follows: + +---%<------------------------------------------------------------------------- +gnutls-cli --starttls -p <port> <host> +---%<------------------------------------------------------------------------- + +This starts the client in plain text mode first. As shown in the previous +section, the server presents a greeting with all capabilities of the server. If +'STARTTLS' is listed, you can issue the 'STARTTLS' command as follows: + +---%<------------------------------------------------------------------------- +STARTTLS +OK "Begin TLS negotiation now." +---%<------------------------------------------------------------------------- + +If an OK response is given by the server you can press 'Ctrl-D' to make +'gnutls-cli' start the TLS negotiation. Upon pressing 'Ctrl-D', 'gnutls-cli' +will show information on the negotiated TLS session and finally the first +response of the server is shown: + +---%<------------------------------------------------------------------------- +"IMPLEMENTATION" "dovecot" +"SASL" "PLAIN" +"SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify +include envelope body relational regex subaddress copy" +OK "TLS negotiation successful." +---%<------------------------------------------------------------------------- + +Hereafter, you can continue to authenticate and upload a script as described in +the previous section. + +Client Problems +--------------- + +If manual efforts to upload a script are successful, but your client still +fails, you need to obtain a view on what the client communicates with the +server. A common method is to sniff the client protocol session using a tool +like 'ngrep'. However, this will not work when TLS is active. If the problem is +not specific to TLS, you are advised to temporarily turn off TLS and sniff the +plain text protocol. If TLS is part of the issue, you can use Dovecot's +<rawlog> [Debugging.Rawlog.txt] facility to see what is going on if the client +is logged in. If the authentication is the problem, there is no real nice way +to obtain a transcript of the protocol. One way is to run managesieve from +inetd, wrapping it into a script that writes the protocol messages somewhere +(*FIXME*: This needs some checking and explanation). Alternatively, if +possible, the client can be altered to write its protocol messages somewhere. + +Refer to the <ManageSieve Clients page> [Pigeonhole.ManageSieve.Clients.txt] +for information on known client problems. + +Known Server Issues and Protocol Deviations +------------------------------------------- + + * The ANONYMOUS authentication mechanism is currently not supported and + explicitly denied. + +*NOTE*: If you add new issues to this list, notify the author or send an e-mail +to the Dovecot mailing list [http://dovecot.org/mailinglists.html]. In any +case, you must make sure that the issue is properly explained and that the +author can contact you for more information. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.ManageSieve.txt b/doc/wiki/Pigeonhole.ManageSieve.txt new file mode 100644 index 0000000..e025f00 --- /dev/null +++ b/doc/wiki/Pigeonhole.ManageSieve.txt @@ -0,0 +1,27 @@ +Pigeonhole ManageSieve Server +============================= + +The <Pigeonhole project> [Pigeonhole.txt] provides Sieve [http://sieve.info] +support for Dovecot, which allows users to filter incoming messages by writing +scripts specified in the Sieve language (RFC 5228). The PigeonholeManageSieve +service is used to manage a user's Sieve script collection. It has the +following advantages over doing it directly via filesystem: + + * No need to let users log in via FTP/SFTP/etc, which could be difficult + especially with virtual users. + * ManageSieve is a standard protocol [http://tools.ietf.org/html/rfc5804], so + users can manage their scripts using (hopefully) user-friendlyManageSieve + clients. Many webmails already include aManageSieve client. + * Scripts are compiled before they are installed, which guarantees that the + uploaded script is valid. This prevents a user from inadvertently installing + a broken Sieve script. + +Configuration and Use +--------------------- + + * <Download and Installation> [Pigeonhole.Installation.txt] + * <Configuration> [Pigeonhole.ManageSieve.Configuration.txt] + * <Troubleshooting> [Pigeonhole.ManageSieve.Troubleshooting.txt] + * <Client Issues> [Pigeonhole.ManageSieve.Clients.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Configuration.Dict.txt b/doc/wiki/Pigeonhole.Sieve.Configuration.Dict.txt new file mode 100644 index 0000000..6ec686f --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Configuration.Dict.txt @@ -0,0 +1,187 @@ +Pigeonhole Sieve: Dict Lookup for Sieve Scripts +=============================================== + +Sieve scripts can be obtained from a number of different <types of locations> +[Pigeonhole.Sieve.Configuration.txt]. This page shows how to retrieve them from +a Dovecot dictionary [http://wiki2.dovecot.org/Dictionary] (abbreviated as +'dict'), which can have either a file or database backend. + +To retrieve a Sieve script from the dict database, two lookups are performed. +First, the name of the Sieve script is queried from the dict path +'/priv/sieve/name/<name>'. If the Sieve script exists, this yields a data ID +which in turn points to the actual script text. The script text is subsequently +queried from the dict path '/priv/sieve/data/<dict-id>'. + +The second query is only necessary when no compiled binary is available or when +the script has changed and needs to be recompiled. The data ID is used to +detect changes in the dict's underlying database. Changing a Sieve script in +the database must be done by first making a new script data item with a new +data ID. Then, the mapping from name to data ID must be changed to point to the +new script text, thereby changing the data ID returned from the name lookup, +i.e. the first query mentioned above. Script binaries compiled from Sieve +scripts contained in a dict database record the data ID. While the data ID +contained in the binary is identical to the one returned from the dict lookup, +the binary is assumed up-to-date. When the returned data ID is different, the +new script text is retrieved using the second query and compiled into a new +binary containing the updated data ID. + +Note that, by default, compiled binaries are not stored at all for Sieve +scripts retrieved from a dict database. The ';bindir=<path>' option needs to be +specified in the <location specification> [Pigeonhole.Sieve.Configuration.txt]. + +Configuration +------------- + +The script location syntax is specified as follows: + +---%<------------------------------------------------------------------------- +sieve = dict:<dict-uri>[;<option>[=<value>][;...]] +---%<------------------------------------------------------------------------- + +The following additional options are recognized: + +user=<username> : + Overrides the user name used for the dict lookup. Normally, the name of the + user running the Sieve interpreter is used. + +If the name of the Script is left unspecified and is not otherwise provided by +the Sieve interpreter, the name defaults to `default'. + +Examples +-------- + +Using a flat file backend +------------------------- + +Flat file example 1 +------------------- + +To retrieve the Sieve script named "keep" from the dict file +/etc/dovecot/sieve.dict: + +---%<------------------------------------------------------------------------- +sieve = dict:file:/etc/dovecot/sieve.dict;name=keep +---%<------------------------------------------------------------------------- + +The file /etc/dovecot/sieve.dict might look like this. Note that with the above +configuration, only the "keep" script will be used. + +---%<------------------------------------------------------------------------- +priv/sieve/name/keep +1 +priv/sieve/name/discard +2 +priv/sieve/data/1 +keep; +priv/sieve/data/2 +discard; +---%<------------------------------------------------------------------------- + +Flat file example 2 +------------------- + +Following on from example 1, a more advanced script. This notifies an external +email address when new mail has arrived. Note that the script all needs to be +on one line. + +---%<------------------------------------------------------------------------- +priv/sieve/name/notify +5 +priv/sieve/data/5 +require ["enotify", "variables"]; if header :matches "From" "*" { set "from" +"${1}";} notify :importance "3" :message "New email from ${from}" +"mailto:other@domain.com?body=New%20email%20has%20arrived."; +---%<------------------------------------------------------------------------- + +Using a SQL backend +------------------- + +For greater flexibility, it's possible to use a SQL backend for your dict +scripts. First, set up a configuration file (such as +/etc/dovecot/dict-sieve-sql.conf) with your database configuration. This should +consist of the following parts: + +---%<------------------------------------------------------------------------- +# The database connection params +connect = host=localhost dbname=dovecot user=dovecot password=password + +# The name mapping that yields the ID of the Sieve script +map { + pattern = priv/sieve/name/$script_name # The name of the script, as per +the "sieve" config parameter + table = user_sieve_scripts # The database table + username_field = username # The field in the table to query +on + value_field = id # The field which contains the +return value of the script ID + fields { + script_name = $script_name # FIXME: The other database field +to query? + } +} + +# The name mapping that yields the script content from ID +{ + pattern = priv/sieve/data/$id # The ID, obtained from above + table = user_sieve_scripts # The database table + username_field = username # The field in the table to query + value_field = script_data # The field which contains the +script + fields { + id = $id # FIXME: The other database field +to query? + } +} +---%<------------------------------------------------------------------------- + +Next, create a dict proxy service. Normally in /etc/dovecot/dovecot.conf: + +---%<------------------------------------------------------------------------- +dict { + sieve = pgsql:/etc/dovecot/dict-sieve-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +Finally, configure Sieve to check the dict (e.g. in +/etc/dovecot/conf.d/90-sieve.conf). This looks up a script called "active" in +the database. + +---%<------------------------------------------------------------------------- +plugin { + sieve = dict:proxy::sieve;name=active +} +---%<------------------------------------------------------------------------- + +As with the flat file, the database query will need to return the Sieve script +all in one line, otherwise the subsequent lines will be ignored. + +Note: you might need to <configure the proxy permissions> [Dict.txt] + +Caching the compiled Sieve binaries +----------------------------------- + +With the configuration described above, the Sieve binaries will be compiled +each time they are called. To improve performance, it is preferable to cache +them, which can be done using the bindir parameter, which is added to the Sieve +configuration. For example: + +---%<------------------------------------------------------------------------- +{ + sieve = dict:file:/etc/dovecot/sieve.dict;name=keep;bindir=~/.sieve-bin +} +---%<------------------------------------------------------------------------- + +Or: + +---%<------------------------------------------------------------------------- +{ + sieve = +dict:file:/etc/dovecot/sieve.dict;name=keep;bindir=/var/sieve-scripts/%u +} +---%<------------------------------------------------------------------------- + +*Note:* Sieve uses the ID number as its cache index and to detect the need to +compile. Therefore, if a script is changed, then its ID must also be changed +for it to be reloaded. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Configuration.File.txt b/doc/wiki/Pigeonhole.Sieve.Configuration.File.txt new file mode 100644 index 0000000..0b95abb --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Configuration.File.txt @@ -0,0 +1,85 @@ +Pigeonhole Sieve: File Location for Sieve Scripts +================================================= + +The 'file' <location type> [Pigeonhole.Sieve.Configuration.txt] is used to +retrieve Sieve scripts from the file system. This is the default type if the +type specifier is omitted from the location specification. The location can +either point to a directory or to a regular file. If the location points to a +directory, a script called 'name' is retrieved by reading a file from that +directory with the file name 'name.sieve'. + +When this location type is involved in a 'sieve_before' or 'sieve_after' script +sequence and the location points to a directory, all files in that directory +with a '.sieve' extension are part of the sequence. The sequence order of the +scripts in that directory is determined by the file names, using a normal 8bit +per-character comparison. + +Unless overridden using the ';bindir=<path>' location option, compiled binaries +for scripts retrieved from the 'file' location type are by default stored in +the same directory as where the script file was found if possible. + +Configuration +------------- + +The script location syntax is specified as follows: + +---%<------------------------------------------------------------------------- +location = file:<path>[;<option>[=<value>][;...]] +---%<------------------------------------------------------------------------- + +The following additional options are recognized: + +active=<path> : + When <ManageSieve> [Pigeonhole.ManageSieve.txt] is used, one script in the + storage can be active; i.e., evaluated at delivery. For the 'file' location + type, the active script in the storage directory is pointed to by a symbolic + link. This option configures where this symbolic link is located. If the + 'file' location path points to a regular file, this setting has no effect + (andManageSieve cannot be used). + +Example +------- + +---%<------------------------------------------------------------------------- +plugin { + ... + sieve = file:~/sieve;active=~/.dovecot.sieve + + sieve_default = file:/var/lib/dovecot/;name=default +} +---%<------------------------------------------------------------------------- + +On <Pigeonhole.Sieve.txt> 2.2.13-12~deb8u4 combined with <ManageSieve> +[Pigeonhole.ManageSieve.txt] on Debian the active option didn't worked and it +lead to following errors/warnings: + +---%<------------------------------------------------------------------------- +Warning: sieve-storage: Active sieve script symlink /var/mail/xxx/xxx/xxx/sieve +is no symlink +---%<------------------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +Error: sieve: sieve file backend: invalid option `active=~/.dovecot.sieve' +---%<------------------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +managesieve(<MAILBOX>): Error: sieve-storage: Active sieve script file +'/var/mail/xxx/xxx/xxx/sieve' is no symlink nor a regular file. This needs to +be fixed manually. +---%<------------------------------------------------------------------------- + +Therefore we just used it without the active option as follows and then it +worked: + +(We adjusted of course the file paths to our needs) + +---%<------------------------------------------------------------------------- +plugin { + ... + sieve = file:~/sieve; + + sieve_default = file:/var/lib/dovecot/;name=default +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Configuration.LDAP.txt b/doc/wiki/Pigeonhole.Sieve.Configuration.LDAP.txt new file mode 100644 index 0000000..e122156 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Configuration.LDAP.txt @@ -0,0 +1,131 @@ +Pigeonhole Sieve: LDAP Lookup for Sieve Scripts +=============================================== + +The 'ldap' <location type> [Pigeonhole.Sieve.Configuration.txt] is used to +retrieve Sieve scripts from an LDAP database. To retrieve a Sieve script from +the LDAP database, at most two lookups are performed. First, the LDAP entry +containing the Sieve script is searched using the specified LDAP search filter. +If the LDAP entry changed since it was last retrieved (or it was never retieved +before), the attribute containing the actual Sieve script is retrieved in a +second lookup. In the first lookup, a special attribute is read and checked for +changes. Usually, this is the 'modifyTimestamp' attribute, but an alternative +can be configured. + +Note that, by default, compiled binaries are not stored at all for Sieve +scripts retrieved from an LDAP database. The ';bindir=<path>' option needs to +be specified in the <location specification> +[Pigeonhole.Sieve.Configuration.txt]. + +Depending on how Pigeonhole was configured and compiled (refer to INSTALL file +for more information), LDAP support may only be available when a plugin called +'sieve_storage_ldap' is loaded. + +Configuration +------------- + +If support for the 'ldap' location type is compiled as a plugin, it needs to be +added to the sieve_plugins setting before it can be used, e.g.: + +---%<------------------------------------------------------------------------- +sieve_plugins = sieve_storage_ldap +---%<------------------------------------------------------------------------- + +The 'ldap' script location syntax is specified as follows: + +---%<------------------------------------------------------------------------- +location = ldap:<config-file>[;<option>[=<value>][;...]] +---%<------------------------------------------------------------------------- + +The '<config-file>' is a filesystem path that points to a configuration file +containing the actual configuration for this 'ldap' script location. + +The following additional location options are recognized: + +user=<username> : + Overrides the user name used for the lookup. Normally, the name of the user + running the Sieve interpreter is used. + +If the name of the Script is left unspecified and not otherwise provided by the +Sieve interpreter, the name defaults to `'default''. + +The configuration file is based on the auth userdb/passdb LDAP configuration +[http://wiki2.dovecot.org/AuthDatabase/LDAP]. The following options are +specific to the Sieve ldap location type: + +sieve_ldap_filter = (&(objectClass=posixAccount)(uid=%u)) : + The LDAP search filter that is used to find the entry containing the Sieve + script. + +sieve_ldap_script_attr = mailSieveRuleSource : + The name of the attribute containing the Sieve script itself. + +sieve_ldap_mod_attr = modifyTimestamp : + The name of the attribute used to detect modifications to the LDAP entry. + +Example +------- + +The dovecot configuration: + +---%<------------------------------------------------------------------------- +plugin { + sieve = ldap:/etc/dovecot/sieve-ldap.conf;bindir=~/.sieve-bin/ +} +---%<------------------------------------------------------------------------- + +The contents of sieve-ldap.conf: + +---%<------------------------------------------------------------------------- +# This file needs to be accessible by the Sieve interpreter running in +LDA/LMTP. +# This requires acces by the mail user. Don't use privileged LDAP credentials +# here as these may likely leak. Only search and read access is required. + +# Space separated list of LDAP hosts to use. host:port is allowed too. +hosts = localhost + +# Distinguished Name - the username used to login to the LDAP server. +# Leave it commented out to bind anonymously. +dn = cn=sieve,ou=Programs,dc=example,dc=org + +# Password for LDAP server, if dn is specified. +dnpass = secret + +# Simple binding. +sasl_bind = no + +# No TLS +tls = no + +# LDAP library debug level as specified by LDAP_DEBUG_* in ldap_log.h. +# -1 = everything. You may need to recompile OpenLDAP with debugging enabled +# to get enough output. +debug_level = 0 + +# LDAP protocol version to use. Likely 2 or 3. +ldap_version = 3 + +# LDAP base +base = dc=mail,dc=example,dc=org + +# Dereference: never, searching, finding, always +deref = never + +# Search scope: base, onelevel, subtree +scope = subtree + +# Filter for user lookup. Some variables can be used: +# %u - username +# %n - user part in user@domain, same as %u if there's no domain +# %d - domain part in user@domain, empty if there's no domain +# %{name} - name of the Sieve script +sieve_ldap_filter = (&(objectClass=posixAccount)(uid=%u)) + +# Attribute containing the Sieve script +sieve_ldap_script_attr = mailSieveRuleSource + +# Attribute used for modification tracking +sieve_ldap_mod_attr = modifyTimestamp +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Configuration.txt b/doc/wiki/Pigeonhole.Sieve.Configuration.txt new file mode 100644 index 0000000..c54a0a8 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Configuration.txt @@ -0,0 +1,643 @@ +Pigeonhole Sieve Configuration +============================== + +Contents + + + 1. Pigeonhole Sieve Configuration + + 1. Script Locations + + 2. Basic Configuration + + 3. Configurable Limits + + 4. Extension-specific Configuration + + 5. Per-user Sieve script location + + 6. Executing Multiple Scripts Sequentially + + 7. Visible Default Script + + 8. Trace Debugging + + 9. Deprecated Settings + + 10. Migration + + 1. General Dovecot 2.0 changes + + 2. From CMUSieve (Dovecot v1.0/v1.1) + + 3. From Dovecot Sieve v0.1.x (Dovecot v1.2) + +Script Locations +---------------- + +The Sieve interpreter can retrieve Sieve scripts from several types of +locations. The default 'file' location type is a directory containing one or +more Sieve script files with a symlink pointing to the active one. More complex +setups can use other location types such as 'ldap' or 'dict' to fetch Sieve +scripts from remote databases. + +All settings that specify the location of one or more Sieve scripts accept the +following syntax:: + +---%<------------------------------------------------------------------------- +<setting> = [<type>:]path[;<option>[=<value>][;...]] +---%<------------------------------------------------------------------------- + +The following script location types are implemented by default: + +file : + The location path is a file system path pointing to a directory containing + one or more script files with names structured as '<script-name>.sieve' with + the active option (default ~/.dovecot.sieve) specifying a symlink to the one + that will be used, or without the active option specified, it may be a script + file instead of a directory. Read <this wiki page> + [Pigeonhole.Sieve.Configuration.File.txt] for more information and examples. + +dict : + The location path is a Dovecot dict uri. Read <this wiki page> + [Pigeonhole.Sieve.Configuration.Dict.txt] for more information and examples. + +ldap : + LDAP database lookup. The location path is a configuration file with LDAP + options. Read <this wiki page> [Pigeonhole.Sieve.Configuration.LDAP.txt] for + more information and examples. + +If the type prefix is omitted, the script location type is 'file' and the +location is interpreted as a local filesystem path pointing to a Sieve script +file or directory. + +The following options are defined for all location types: + +name=<script-name> : + Set the name of the Sieve script that this location points to. If the name of + the Sieve script is not contained in the location path and the location of a + single script is specified, this option is required (e.g. for 'dict' + locations that must point to a particular script). If the name of the script + is contained in the location path, the value of the name option overrides the + name retrieved from the location. If the Sieve interpreter explicitly queries + for a specific name (e.g. to let the Sieve include extension + [http://tools.ietf.org/html/draft-ietf-sieve-include-05] retrieve a script + from the 'sieve_global=' location), this option has no effect. + +bindir=<dirpath> : + Points to the directory where the compiled binaries for this script location + are stored. This directory is created automatically if possible. If this + option is omitted, the behavior depends on the location type. For 'file' type + locations, the binary is then stored in the same directory as where the + script file was found if possible. For `dict' type locations, the binary is + not stored at all in that case. Don't specify the same directory for + different script locations, as this will result in undefined behavior. + Multiple mail users can share a single script directory if the script + location is the same and all users share the same system credentials (uid, + gid). + +*NOTE*: Pigeonhole versions before v0.3.1 do not support the location syntax +described here. These versions only support bare filesystem paths pointing to +files or directores as script storage location. Also, in that case a few +additional <deprecated settings> [Pigeonhole.Sieve.Configuration.txt] are +needed for configuration. + +Basic Configuration +------------------- + +To use Sieve, you will first need to make sure you are using Dovecot <LDA.txt> +or <LMTP.txt> for delivering incoming mail to users' mailboxes. Then, you need +to enable the Pigeonhole Sieve plugin in your configuration: + +---%<------------------------------------------------------------------------- +protocol lda { + mail_plugins = $mail_plugins sieve +} +protocol lmtp { + mail_plugins = $mail_plugins sieve +} +---%<------------------------------------------------------------------------- + +The sieve plugin recognizes the following configuration options in the 'plugin' +section of the config file (default values are shown if applicable): + +sieve = file:~/sieve;active=~/.dovecot.sieve : + The location of the user's main Sieve script or script storage. The + <LDA.txt> Sieve plugin uses this to find the active script for Sieve + filtering at delivery. The Sieve include extension + [http://tools.ietf.org/html/draft-ietf-sieve-include-05] uses this location + for retrieving ":personal" scripts. + +: + This location is also where the <ManageSieve> [Pigeonhole.ManageSieve.txt] + service will store the user's scripts, if supported by the location type. For + the 'file' location type, the location will then be the path to the storage + directory for all the user's personal Sieve scripts. <ManageSieve> + [Pigeonhole.ManageSieve.txt] maintains a symbolic link pointing to the + currently active script (the script executed at delivery). The location of + this symbolic link can be configured using the ';active=<path>' option. + +: + For Pigeonhole versions before v0.3.1, this setting can only be a filesystem + path pointing to a script file, or - when <ManageSieve> + [Pigeonhole.ManageSieve.txt] is used - it is the location of the symbolic + link pointing to the active script in the storage directory. That storage + directory is then configured using the deprecated 'sieve_dir' setting. + +sieve_default = (v0.3+) : + The location of the default personal sieve script file which gets executed + ONLY if user's private Sieve script does not exist, + e.g.'file:/var/lib/dovecot/default.sieve' (check the <multiscript section> + [Pigeonhole.Sieve.Configuration.txt] for instructions on running global Sieve + scripts before and after the user's personal script). This is usually a + global script, so be sure to pre-compile the specified script manually in + that case using the 'sievec' command line tool, as explained <here> + [Pigeonhole.Sieve.Usage.txt]. This setting used to be called + 'sieve_global_path', but that name is now deprecated. + +sieve_default_name = (v0.4.8+) : + The name by which the default Sieve script is visible to <ManageSieve> + [Pigeonhole.ManageSieve.txt] clients. Normally, it is not visible at all. + Refer to the <visible default script section> + [Pigeonhole.Sieve.Configuration.txt] for more information. + +sieve_global = : + Location for :global include scripts for the Sieve include extension + [http://tools.ietf.org/html/draft-ietf-sieve-include-05]. This setting used + to be called 'sieve_global_dir', but that name is now deprecated. + +sieve_discard = (v0.4.16+) : + The location of a Sieve script that is run for any message that is about to + be discarded; i.e., it is not delivered anywhere by the normal Sieve + execution. This only happens when the "implicit keep" is canceled, by e.g. + the "discard" action, and no actions that deliver the message are executed. + This "discard script" can prevent discarding the message, by executing + alternative actions. If the discard script does nothing, the message is still + discarded as it would be when no discard script is configured. + +sieve_extensions = : + Which Sieve language extensions are available to users. By default, all + supported extensions are available, except for deprecated extensions, + extensions that add the ability to change messages, extensions that require + explicit configuration or extensions that are still under development. Some + system administrators may want to disable certain Sieve extensions or enable + those that are not available by default. All supported extensions are listed + <here> [Pigeonhole.Sieve.txt]. Normally, all enabled extensions must be + listed for this setting, but starting with Sieve version 0.1.7, this setting + can use '+' and '-' to specify differences relative to the default. For + example 'sieve_extensions = +imapflags' will enable the deprecated imapflags + extension [http://tools.ietf.org/html/draft-melnikov-sieve-imapflags-03] in + addition to all extensions enabled by default. + +sieve_global_extensions = (v0.3+) : + Which Sieve language extensions are ONLY avalable in global scripts. This can + be used to restrict the use of certain Sieve extensions to administrator + control, for instance when these extensions can cause security concerns. This + setting has higher precedence than the 'sieve_extensions' setting (above), + meaning that the extensions enabled with this setting are never available to + the user's personal script no matter what is specified for the + 'sieve_extensions' setting. The syntax of this setting is similar to the + 'sieve_extensions' setting, with the difference that extensions are enabled + or disabled for exclusive use in global scripts. Currently, no extensions are + marked as such by default. + +sieve_implicit_extensions = (v0.4.13+) : + Which Sieve language extensions are implicitly available to users. The + extensions listed in this setting do not need to be enabled explicitly using + the Sieve "require" command. This behavior directly violates the Sieve + standard, but can be necessary for compatibility with some existing + implementations of Sieve (notably jSieve). Do not use this setting unless you + really need to! The syntax and semantics of this setting are otherwise + identical to the 'sieve_extensions' setting. + +sieve_plugins = : + The Pigeonhole Sieve interpreter can have plugins of its own. Using this + setting, the used plugins can be specified. Check the <Sieve plugins page> + [Pigeonhole.Sieve.Plugins.txt] for available plugins. + +sieve_user_email = (v0.4.14+) : + The primary e-mail address for the user. This is used as a default when no + other appropriate address is available for sending messages. If this setting + is not configured, either the postmaster or null "<>" address is used as a + sender, depending on the action involved. This setting is important when + there is no message envelope to extract addresses from, such as when the + script is executed in IMAP. + +sieve_user_log = : + The path to the file where the user log file is written. If not configured, a + default location is used. If the main user's personal Sieve (as configured + with 'sieve=') is a file, the logfile is set to '<filename>.log' by default. + If it is not a file, the default user log file is '~/.dovecot.sieve.log'. + +recipient_delimiter = +: + The separator that is expected between the :user and :detail address parts + introduced by the subaddress extension [http://tools.ietf.org/html/rfc5233/]. + This may also be a sequence of characters (e.g. '--'). The current + implementation looks for the separator from the left of the localpart and + uses the first one encountered. The :user part is left of the separator and + the :detail part is right. This setting is also used by Dovecot's <LMTP.txt> + service with identical semantics. + +sieve_redirect_envelope_from = sender (v0.4.4+): + Specifies what envelope sender address is used for redirected messages. + Normally, the Sieve "redirect" command copies the sender address for the + redirected message from the processed message. So, the redirected message + appears to originate from the original sender. The following values are + supported for this setting: + "sender" : + The sender address is used (default) + + "recipient" : + The final recipient address is used + + "orig_recipient" : + The original recipient is used + + "user_email" (v0.4.14+): + The user's primary address is used. This is configured with the + "sieve_user_email" setting. If that setting is not configured, "user_mail" + is equal to "sender" (the default). + + "postmaster" : + The postmaster_address configured for LDA/LMTP. + + "<user@domain>" : + Redirected messages are always sent from user@domain. The angle brackets + are mandatory. The null "<>" address is also supported. + +: + When the envelope sender of the processed message is the null address "<>", + the envelope sender of the redirected message is also always "<>", + irrespective of what is configured for this setting. + +For example: + +---%<------------------------------------------------------------------------- +plugin { +... + # The location of the user's main script storage. The active script + # in this storage is used as the main user script executed during + # delivery. The include extension fetches the :personal scripts + # from this location. When ManageSieve is used, this is also where + # scripts are uploaded. This example uses the file system as + # storage, with all the user's scripts located in the directory + # `~/sieve' and the active script (symbolic link) located at + # `~/.dovecot.sieve'. + sieve = file:~/sieve;active=~/.dovecot.sieve + + # If the user has no personal active script (i.e. if the location + # indicated in sieve= does not exist or has no active script), use + # this one: + sieve_default = /var/lib/dovecot/sieve/default.sieve + + # The include extension fetches the :global scripts from this + # location. + sieve_global = /var/lib/dovecot/sieve/global/ +} +---%<------------------------------------------------------------------------- + +Configurable Limits +------------------- + +sieve_max_script_size = 1M : + The maximum size of a Sieve script. The compiler will refuse to compile any + script larger than this limit. If set to 0, no limit on the script size is + enforced. + +sieve_max_actions = 32 : + The maximum number of actions that can be performed during a single script + execution. If set to 0, no limit on the total number of actions is enforced. + +sieve_max_redirects = 4 : + The maximum number of redirect actions that can be performed during a single + script execution. The meaning of 0 differs based on your version. For + versions v0.3.0 and beyond this means that redirect is prohibited. For older + versions, however, this means that the number of redirects is /unlimited/, so + be careful. + +Extension-specific Configuration +-------------------------------- + +The following Sieve language extensions have specific configuration +options/needs: + + * <duplicate> [Pigeonhole.Sieve.Extensions.Duplicate.txt] + * <editheader> [Pigeonhole.Sieve.Extensions.Editheader.txt] (configuration + required) + * <imapsieve> [Pigeonhole.Sieve.Plugins.IMAPSieve.txt] (plugin configuration + required) + * <include> [Pigeonhole.Sieve.Extensions.Include.txt] + * <spamtest and virustest> [Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt] + (configuration required) + * <vacation and vacation-seconds> [Pigeonhole.Sieve.Extensions.Vacation.txt] + * <variables> [Pigeonhole.Sieve.Extensions.Variables.txt] + +Per-user Sieve script location +------------------------------ + +By default, the Dovecot Sieve plugin looks for the user's Sieve script file in +the user's home directory ('~/.dovecot.sieve'). This requires that the <home +directory> [VirtualUsers.txt] is set for the user. + +If you want to store the script elsewhere, you can override the default using +the 'sieve' setting, which specifies the path to the user's script file. This +can be done in two ways: + + 1. Define the 'sieve' setting in the plugin section of 'dovecot.conf'. + 2. Return 'sieve' extra field from <userdb extra fields> + [UserDatabase.ExtraFields.txt]. + +For example, to use a Sieve script file named '<username>.sieve' in +'/var/sieve-scripts', use: + +---%<------------------------------------------------------------------------- +plugin { +... + + sieve = /var/sieve-scripts/%u.sieve +} +---%<------------------------------------------------------------------------- + +You may use templates like %u, as shown in the example. See all <variables> +[Variables.txt]. + +A relative path (or just a filename) will be interpreted to point under the +user's home directory. + +Executing Multiple Scripts Sequentially +--------------------------------------- + +The Dovecot Sieve plugin allows executing multiple Sieve scripts sequentially. +The extra scripts can be executed before and after the user's private script. +For example, this allows executing global Sieve policies before the user's +script. This is not possible using the 'sieve_default' setting, because that is +only used when the user's private script does not exist. The following settings +in the 'plugin' section of the Dovecot config file control the execution +sequence: + +sieve_before = : + +sieve_before2 = : + +sieve_before3 = (etc..) : + Location Sieve of scripts that need to be executed before the user's personal + script. If a 'file' location path points to a directory, all the Sieve + scripts contained therein (with the proper '.sieve' extension) are executed. + The order of execution within that directory is determined by the file names, + using a normal 8bit per-character comparison. Multiple script locations can + be specified by appending an increasing number to the setting name. The Sieve + scripts found from these locations are added to the script execution sequence + in the specified order. Reading the numbered sieve_before settings stops at + the first missing setting, so no numbers may be skipped. + +sieve_after = : + +sieve_after2 = : + +sieve_after3 = (etc..) : + Identical to 'sieve_before', but the specified scripts are executed after the + user's script (only when keep is still in effect, as explained below). + +The script execution ends when the currently executing script in the sequence +does not yield a "keep" result: when the script terminates, the next script is +only executed if an implicit or explicit "keep" is in effect. Thus, to end all +script execution, a script must not execute keep and it must cancel the +implicit keep, e.g. by executing "'discard; stop;'". This means that the +command "'keep;'" has different semantics when used in a sequence of scripts. +For normal Sieve execution, "'keep;'" is equivalent to "'fileinto "INBOX";'", +because both cause the message to be stored in INBOX. However, in sequential +script execution, it only controls whether the next script is executed. Storing +the message into INBOX (the default folder) is not done until the last script +in the sequence executes (implicit) keep. To force storing the message into +INBOX earlier in the sequence, the fileinto command can be used (with "':copy'" +or together with "'keep;'"). + +Apart from the keep action, all actions triggered in a script in the sequence +are executed before continuing to the next script. This means that when a +script in the sequence encounters an error, actions from earlier executed +scripts are not affected. The sequence is broken however, meaning that the +script execution of the offending script is aborted and no further scripts are +executed. An implicit keep is executed in stead. + +Just as for executing a single script the normal way, the Dovecot Sieve plugin +takes care never to duplicate deliveries, forwards or responses. When vacation +actions are executed multiple times in different scripts, the usual error is +not triggered: the subsequent duplicate vacation actions are simply discarded. + +For example: + +---%<------------------------------------------------------------------------- +plugin { +... + # Global scripts executed before the user's personal script. + # E.g. handling messages marked as dangerous + sieve_before = /var/lib/dovecot/sieve/discard-virusses.sieve + + # Domain-level scripts retrieved from LDAP + sieve_before2 = ldap:/etc/dovecot/sieve-ldap.conf;name=ldap-domain + + # User-specific scripts executed before the user's personal script. + # E.g. a vacation script managed through a non-ManageSieve GUI. + sieve_before3 = /var/vmail/%d/%n/sieve-before + + # User-specific scripts executed after the user's personal script. + # (if keep is still in effect) + # E.g. user-specific default mail filing rules + sieve_after = /var/vmail/%d/%n/sieve-after + + # Global scripts executed after the user's personal script + # (if keep is still in effect) + # E.g. default mail filing rules. + sieve_after2 = /var/lib/dovecot/sieve/after.d/ +} +} +---%<------------------------------------------------------------------------- + +*IMPORTANT*: Be sure to manually pre-compile the scripts specified by +'sieve_before' and 'sieve_after' using the 'sievec' tool, as explained <here> +[Pigeonhole.Sieve.Usage.txt]. + +Visible Default Script +---------------------- + +The 'sieve_default=' setting specifies the location of a default script that is +executed when the user has no active personal script. Normally, this default +script is invisible to the user; i.e., it is not listed in <ManageSieve> +[Pigeonhole.ManageSieve.txt]. To give the user the ability to see and read the +default script, it is possible to make it visible under a specific configurable +name using the 'sieve_default_name' setting. This feature is only supported for +Pigeonhole versions 0.4.8 and higher. + +ManageSieve will magically list the default script under that name, even though +it does not actually exist in the user's normal script storage location. This +way, theManageSieve client can see that it exists and it can retrieve its +contents. If no normal script is active, the default is always listed as +active. The user can replace the default with a custom script, by uploading it +under the default script's name. If that custom script is ever deleted, the +default script will reappear from the shadows implicitly. + +This way, ManageSieve clients will not need any special handling for this +feature. If the name of the default script is equal to the name the client uses +for the main script, it will initially see and read the default script when the +user account is freshly created. The user can edit the script, and when the +edited script is saved through theManageSieve client, it will will override the +default script. If the user ever wants to revert to the default, the user only +needs to delete the edited script and the default will reappear. + +The name by which the default script will be known is configured using the +'sieve_default_name' setting. Of course, the 'sieve_default' setting needs to +point to a valid script location as well for this to work. If the default +script does not exist at the indicated location, it is not shown. + +For example: + +---%<------------------------------------------------------------------------- +plugin { +... + sieve = file:~/sieve;active=~/.dovecot.sieve + + sieve_default = /var/lib/dovecot/sieve/default.sieve + sieve_default_name = roundcube +} +---%<------------------------------------------------------------------------- + +Trace Debugging +--------------- + +Trace debugging provides detailed insight in the operations performed by the +Sieve script. Messages about what the Sieve script is doing are written to the +specified directory. This feature is only supported for Pigeonhole versions +0.4.14 and higher. + +*WARNING*: On a busy server, this functionality can quickly fill up the trace +directory with a lot of trace files. Enable this only temporarily and as +selective as possible; e.g., enable this only for a few users by returning the +settings below from userdb as <userdb extra fields> +[UserDatabase.ExtraFields.txt], rather than enabling these for everyone. + +The following settings apply to both the LDA/LMTP Sieve plugin and the +<IMAPSieve> [Pigeonhole.Sieve.Plugins.IMAPSieve.txt] plugin: + +sieve_trace_dir = : + The directory where trace files are written. Trace debugging is disabled if + this setting is not configured or if the directory does not exist. If the + path is relative or it starts with "~/" it is interpreted relative to the + current user's home directory. + +sieve_trace_level = : + The verbosity level of the trace messages. Trace debugging is disabled if + this setting is not configured. Possible values are: + "actions" : + Only print executed action commands, like keep, fileinto, reject and + redirect. + + "commands" : + Print any executed command, excluding test commands. + + "tests" : + Print all executed commands and performed tests. + + "matching" : + Print all executed commands, performed tests and the values matched in + those tests. + +sieve_trace_debug = no : + Enables highly verbose debugging messages that are usually only useful for + developers. + +sieve_trace_addresses = no : + Enables showing byte code addresses in the trace output, rather than only the + source line numbers. + +Deprecated Settings +------------------- + +These settings are deprecated in newer versions, but still recognized: + +sieve_global_path = (< v0.2): + The deprecated name for the 'sieve_default' setting. + +sieve_dir = ~/sieve (< v0.3.1): + Directory for :personal include scripts for the include extension + [http://tools.ietf.org/html/draft-ietf-sieve-include-05]. The Sieve + interpreter only recognizes files that end with a '.sieve' extension, so the + include extension expects a file called 'name.sieve' to exist in the + 'sieve_dir' directory for a script called 'name'. When using <ManageSieve> + [Pigeonhole.ManageSieve.txt], this is also the directory where scripts are + uploaded. For recent Pigeonhole versions, this location is configured as part + of the 'sieve' setting. + +sieve_global_dir = (< v0.3.1): + Directory for :global include scripts for the include extension + [http://tools.ietf.org/html/draft-ietf-sieve-include-05]. The Sieve + interpreter only recognizes files that end with a '.sieve' extension, so the + include extension expects a file called 'name.sieve' to exist in the + 'sieve_global_dir' directory for a script called 'name'. For recent + Pigeonhole versions, a more generic version of this setting is called + 'sieve_global' and allows locations other than file system directories. + +Migration +--------- + +General Dovecot 2.0 changes +--------------------------- + + * Note that the Dovecot v2.0 <LDA.txt> does not create mailfolders + automatically by default anymore. If your configuration relies on this, you + need to enable the 'lda_mailbox_autocreate' setting for <LDA.txt> or start + using the Sieve mailbox extension's ':create' tag for *fileinto* commands. + * Dovecot v2.0 adds support for <LMTP.txt>. Much like the <Dovecot LDA> + [LDA.txt], it can make use of the Pigeonhole Sieve plugin. Since the + <LMTP.txt> service has its own 'prototocol lmtp' section in the config file, + you need to add the Sieve plugin to the 'mail_plugins' setting there too + when you decide to use <LMTP.txt>. + +From CMUSieve (Dovecot v1.0/v1.1) +--------------------------------- + +For the most part, migration from CMUSieve to Pigeonhole Sieve is just a matter +of changing the used plugin name from *cmusieve* to *sieve* in the +'mail_plugins' option in the 'protocol lda' section of the config file (as +explained <above> [Pigeonhole.Sieve.Configuration.txt]). However, there are a +few important differences in the supported Sieve language features: + + * The *imapflags* extension is now called *imap4flags*. The CMUSieve + implementation is based on an old draft specification + [http://tools.ietf.org/html/draft-melnikov-sieve-imapflags-03] that is not + completely compatible with the new version + [http://tools.ietf.org/html/rfc5232/]. Particularly, the *mark* and *unmark* + commands were removed from the new specification. For backwards + compatibility, support for the old imapflags extension can be enabled using + the 'sieve_extensions' setting (as explained <above> + [Pigeonhole.Sieve.Configuration.txt]). This is disabled by default. + * The *notify* extension is now called *enotify*. The CMUSieve implementation + is based on an old draft specification + [http://tools.ietf.org/html/draft-martin-sieve-notify-01] that is not + completely compatible with the new version + [http://tools.ietf.org/html/rfc5435/]. Particularly, the *denotify* command + and *$text$* substitutions were removed from the new specification. For + backwards compatibility, support for the old imapflags extension can be + enabled using the 'sieve_extensions' setting (as explained <above> + [Pigeonhole.Sieve.Configuration.txt]). This is disabled by default. + * The include extension [http://tools.ietf.org/html/draft-ietf-sieve-include] + now requires your script /file/ names to end with ".sieve". This means that + 'include :personal "myscript"' won't work unless your script file is called + "'myscript.sieve'" on disk. Also note that the "'.sieve'" extension has no + special meaning within the Sieve script; if you 'include "myscript.sieve"', + the Sieve interpreter will look for a script file called + 'myscript.sieve.sieve' and not 'myscript.sieve'. + * Be sure to use *UTF8* for the mailbox argument of the *fileinto* command. + Older CMUSieve installations used modified UTF7 (as IMAP does) for the + mailbox parameter. If not adjusted, the Pigeonhole Sieve plugin will use the + wrong folder name for storing the message. + +From Dovecot Sieve v0.1.x (Dovecot v1.2) +---------------------------------------- + + * The 'sieve_subaddress_sep' setting for the Sieve subaddress extension + [http://tools.ietf.org/html/rfc5233/] is now known as 'recipient_delimiter'. + Although 'sieve_subaddress_sep' is still recognized for backwards + compatibility, it is recommended to update the setting to the new name, + since the <LMTP.txt> service also uses the 'recipient_delimiter' setting. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Examples.txt b/doc/wiki/Pigeonhole.Sieve.Examples.txt new file mode 100644 index 0000000..fb578df --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Examples.txt @@ -0,0 +1,423 @@ +Pigeonhole Sieve examples +========================= + +Contents + + + 1. Pigeonhole Sieve examples + + 1. Mail filtering by various headers + + 2. Flagging or Highlighting your mail + + 3. Spam/Virus rules + + 1. Direct filtering using message header + + 2. Filtering using the spamtest and virustest extensions + + 4. Plus Addressed mail filtering + + 5. Vacation auto-reply + + 6. Include scripts + + 7. Archiving a Mailinglist by Date + + 8. Emulating lmtp_save_to_detail_mailbox=yes + + 9. Translation from Procmail + +Below are some simple Sieve code examples, more can be found at +http://sieve.info/examplescripts. + +Mail filtering by various headers +--------------------------------- + +Use if/elsif/else to store messages into various folders/subfolders: + + * ---%<---------------------------------------------------------------------- + require ["fileinto", "envelope"]; + if address :is "to" "dovecot@dovecot.org" { + fileinto "Dovecot-list"; + } elsif envelope :is "from" "owner-cipe-l@inka.de" { + fileinto "lists.cipe"; + } elsif anyof (header :contains "X-listname" "lugog@cip.rz.fh-offenburg.de", + header :contains "List-Id" "Linux User Group Offenburg") { + fileinto "ml.lugog"; + } else { + # The rest goes into INBOX + # default is "implicit keep", we do it explicitly here + keep; + } + ---%<---------------------------------------------------------------------- + + "anyof" means logical OR, "allof" is AND. + +Forward mails with "order" or "buy" in their subject to another address: + + * ---%<---------------------------------------------------------------------- + if header :contains "subject" ["order", "buy"] { + redirect "orders@company.dom"; + } + ---%<---------------------------------------------------------------------- + +Message-ID and recipient of forwarded message are stored in a +'.dovecot.lda-dupes' at users home directory to prevent mail loops. + +Flagging or Highlighting your mail +---------------------------------- + +Some mail readers use these flags: + +---%<------------------------------------------------------------------------- +require "imap4flags"; +require "regex"; +if anyof (exists "X-Cron-Env", + header :regex ["subject"] [".* security run output", + ".* monthly run output", + ".* daily run output", + ".* weekly run output"]) { + addflag "$label1"; # ie 'Important'/red label within Thunderbird + +# Other flags: +# addflag "$label1"; # Important: #ff0000 => red +# addflag "$label2"; # Work: #ff9900 => orange +# addflag "$label3"; # personal: #009900 => green +# addflag "$label4"; # todo: #3333ff => blue +# addflag "$label5"; # later: #993399 => violet +# +} +---%<------------------------------------------------------------------------- + +Local copy of your emails: + +---%<------------------------------------------------------------------------- +require ["envelope", "imap4flags"]; +if envelope "from" "my_address@my_domain.com" +{ + setflag "\\seen"; +} +---%<------------------------------------------------------------------------- + +/Useful, when you want sieve to manage your incoming *and* outgoing email (you +must ask your mail reader to Bcc your mail to your dovecot in this case)./ + +Spam/Virus rules +---------------- + +Most spam and virus scanners add a special header to mail messages, so that +users can apply filtering accordingly. Depending on how the Sieve interpreter +is configured, filtering can either be performed by evaluating these headers +directly, or using the spamtest and virustest extensions. + +Direct filtering using message header +------------------------------------- + +Evaluating the headers directly is always possible as long as the headers are +actually added to the messages by the scanner software. For example, to +fileSpamAssassin-tagged mails into a folder called "Spam": + +---%<------------------------------------------------------------------------- +require "fileinto"; +if header :contains "X-Spam-Flag" "YES" { + fileinto "Spam"; +} +---%<------------------------------------------------------------------------- + +The following example discards SpamAssassin-tagged mails with level higher than +or equal to 10: + +---%<------------------------------------------------------------------------- +if header :contains "X-Spam-Level" "**********" { + discard; + stop; +} +---%<------------------------------------------------------------------------- + +Some spam scanners only produce a numeric score in a header. Then, the test +becomes more involved: + +---%<------------------------------------------------------------------------- +require ["comparator-i;ascii-numeric","relational"]; +if allof ( + not header :matches "x-spam-score" "-*", + header :value "ge" :comparator "i;ascii-numeric" "x-spam-score" "10" ) +{ + discard; + stop; +} +---%<------------------------------------------------------------------------- + +*NOTE:* Be very careful when matching against spam score headers using the +relational extension and the i;ascii-numeric comparator. This comparator can +only be used to match unsigned integers. Strings that do not begin with a digit +character represent positive infinity and will therefore always be larger than +any score mentioned in your rule! That is why the above example first checks +the minus sign explicitly. + +Filtering using the spamtest and virustest extensions +----------------------------------------------------- + +When the spamtest [http://tools.ietf.org/html/rfc5235#section-3.2] and +virustest [http://tools.ietf.org/html/rfc5235#section-3.3] extensions are +configured on the server ( <here> +[Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt] is explained how), users +(and GUIs) can have a much easier way to filter spam and virus messages +respectively. To filter spam, the spamtest extension can for example be used as +follows: + +---%<------------------------------------------------------------------------- +require "spamtestplus"; +require "fileinto"; +require "relational"; +require "comparator-i;ascii-numeric"; + +/* If the spamtest fails for some reason, e.g. spam header is missing, file + * file it in a special folder. + */ +if spamtest :value "eq" :comparator "i;ascii-numeric" "0" { + fileinto "Unclassified"; + +/* If the spamtest score (in the range 1-10) is larger than or equal to 3, + * file it into the spam folder: + */ +} elsif spamtest :value "ge" :comparator "i;ascii-numeric" "3" { + fileinto "Spam"; + +/* For more fine-grained score evaluation, the :percent tag can be used. The + * following rule discards all messages with a percent score + * (relative to maximum) of more than 85 %: + */ +} elsif spamtest :value "gt" :comparator "i;ascii-numeric" :percent "85" { + discard; +} + +/* Other messages get filed into INBOX */ +---%<------------------------------------------------------------------------- + +The virustest extension can be used in a similar manner: + +---%<------------------------------------------------------------------------- +require "virustest"; +require "fileinto"; +require "relational"; +require "comparator-i;ascii-numeric"; + +/* Not scanned ? */ +if virustest :value "eq" :comparator "i;ascii-numeric" "0" { + fileinto "Unscanned"; + +/* Infected with high probability (value range in 1-5) */ +} if virustest :value "eq" :comparator "i;ascii-numeric" "4" { + /* Quarantine it in special folder (still somewhat dangerous) */ + fileinto "Quarantine"; + +/* Definitely infected */ +} elsif virustest :value "eq" :comparator "i;ascii-numeric" "5" { + /* Just get rid of it */ + discard; +} +---%<------------------------------------------------------------------------- + +Plus Addressed mail filtering +----------------------------- + +Using the subaddress [http://tools.ietf.org/html/rfc5233/] extension, it is +possible to match against the 'detail' part of an e-mail address, e.g. a +''+tag'' suffix to the local part of the address. This is for example useful +when you don't want just any +tag to create a directory, but you want to use +tagged addresses such as with amavisd-new. This example would place email +addressed to user+spam@example.com into user's Spam folder. + +---%<------------------------------------------------------------------------- +require ["fileinto", "envelope", "subaddress"]; +if envelope :detail "to" "spam"{ + fileinto "Spam"; +} +---%<------------------------------------------------------------------------- + +The following more advanced example uses the subaddress +[http://tools.ietf.org/html/rfc5233/] extension to handle recipient addresses +structured as 'sales+<name>@company.com' in a special way. The '<name>' part is +extracted from the address using variables +[http://tools.ietf.org/html/rfc5229/] extension, transformed into a format with +the first letter in upper case and subsequently used to create the folder name +where the message is stored. The folder name is structured as 'users/<name>'. +If the '+<name>' detail is omitted from the recipient address, the message is +filed in the 'sales' folder. + +---%<------------------------------------------------------------------------- +require ["variables", "envelope", "fileinto", "subaddress"]; + +if envelope :is :user "to" "sales" { + if envelope :matches :detail "to" "*" { + /* Save name in ${name} in all lowercase except for the first letter. + * Joe, joe, jOe thus all become 'Joe'. + */ + set :lower :upperfirst "name" "${1}"; + } + + if string :is "${name}" "" { + /* Default case if no detail is specified */ + fileinto "sales"; + } else { + /* For sales+joe@ this will become users/Joe */ + fileinto "users/${name}"; + } +} +---%<------------------------------------------------------------------------- + +To work with Postfix, this requires that the envelope "to" still contains the +full address, so pass it with the -a flag. + +---%<------------------------------------------------------------------------- +dovecot unix - n n - - pipe + flags=DRhu user=mail:mail argv=/usr/local/libexec/dovecot/dovecot-lda + -f ${sender} -d ${user}@${nexthop} -a ${original_recipient} +---%<------------------------------------------------------------------------- + +or + +---%<------------------------------------------------------------------------- +mailbox_command = /usr/lib/dovecot/dovecot-lda -a "$RECIPIENT" +---%<------------------------------------------------------------------------- + +Vacation auto-reply +------------------- + +Auto-responder functionality is implemented using the vacation +[http://tools.ietf.org/html/rfc5230/] extension. The following script sends +out-of-office replies when the message is not spam: + +---%<------------------------------------------------------------------------- +require ["fileinto", "vacation"]; +# Move spam to spam folder +if header :contains "X-Spam-Flag" "YES" { + fileinto "spam"; + # Stop here so that we do not reply on spams + stop; +} +vacation + # Reply at most once a day to a same sender + :days 1 + :subject "Out of office reply" + # List of additional recipient addresses which are included in the auto +replying. + # If a mail's recipient is not the envelope recipient and it's not on this +list, + # no vacation reply is sent for it. + :addresses ["j.doe@company.dom", "john.doe@company.dom"] +"I'm out of office, please contact Joan Doe instead. +Best regards +John Doe"; +---%<------------------------------------------------------------------------- + +It's also possible to include the original subject using the variables +[http://tools.ietf.org/html/rfc5229/] extension: + +---%<------------------------------------------------------------------------- +require ["variables", "vacation"]; +# Store old Subject line so it can be used in vacation message +if header :matches "Subject" "*" { + set "subjwas" ": ${1}"; +} +vacation + :days 1 + :subject "Out of office reply${subjwas}" + :addresses ["j.doe@company.dom", "john.doe@company.dom"] +"I'm out of office, please contact Joan Doe instead. +Best regards +John Doe"; +---%<------------------------------------------------------------------------- + +Include scripts +--------------- + +It's possible to include other Sieve scripts in your script: + +---%<------------------------------------------------------------------------- +require ["include"]; +include :global "global-spam"; +include :personal "my-own-spam"; +---%<------------------------------------------------------------------------- + +The lookup directories can be specified with: + +---%<------------------------------------------------------------------------- +plugin { + # Directory for :personal include scripts. The default is to use home +directory. + sieve_dir = %h/sieve + + # Directory for :global include scripts (not to be confused with +sieve_global_path). + # If unset, the include fails. + sieve_global_dir = /etc/dovecot/sieve/ +} +---%<------------------------------------------------------------------------- + +Both 'sieve_dir' and 'sieve_global_dir' may also be overridden by <userdb extra +fields> [UserDatabase.ExtraFields.txt]. + +It's not currently possible to use subdirectories for the scripts. Having a '/' +character in the script name always fails the include. This is just an extra +check to avoid potential problems with including scripts within mail +directories. + +Archiving a Mailinglist by Date +------------------------------- + +You can archive messages from mailing lists in a date-structured folder tree as +follows: + +---%<------------------------------------------------------------------------- +require ["variables","date","fileinto","mailbox"]; + +# Extract date info +if currentdate :matches "year" "*" { set "year" "${1}"; } +if currentdate :matches "month" "*" { set "month" "${1}"; } + +# Archive Dovecot mailing list items by year and month. +# Create folder when it does not exist. +if header :is "list-id" "dovecot.dovecot.org" { + fileinto :create "INBOX.Lists.${year}.${month}.dovecot"; +} +---%<------------------------------------------------------------------------- + +For example, in March 2013 this puts messages from the Dovecot mailing list in +a folder called 'INBOX.Lists.2013.03.dovecot'. It combines the date +[http://tools.ietf.org/html/rfc5260#section-4] and variables +[http://tools.ietf.org/html/rfc5229/] extensions to extract the required date +strings. Using the ':create' argument for the 'fileinto' command, the indicated +folder is created automatically if it doesn't exist. The ':create' argument is +provided by the mailbox [http://tools.ietf.org/html/rfc5490#section-3] +extension. + +Emulating lmtp_save_to_detail_mailbox=yes +----------------------------------------- + +If you can't turn this option on, you can emulate the behaviour to some extent +with following code. + +---%<------------------------------------------------------------------------- +require ["variables", "fileinto", "envelope", "subaddress", "mailbox"]; + +if envelope :matches :detail "to" "*" { + # you can prefix with INBOX/ or INBOX. if necessary + # remove :create if you want to permit only existing mailboxes + fileinto :create "${1}"; +} +---%<------------------------------------------------------------------------- + +Translation from Procmail +------------------------- + +There exists a script which attempts to translate simple Procmail rules into +Sieve rules:http://www.earth.ox.ac.uk/~steve/sieve/procmail2sieve.pl +(dovecot.org mirror [http://dovecot.org/tools/procmail2sieve.pl]) + +Here's the original post announcing it: +http://dovecot.org/list/dovecot/2007-March/020895.html + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Extensions.Duplicate.txt b/doc/wiki/Pigeonhole.Sieve.Extensions.Duplicate.txt new file mode 100644 index 0000000..dbd0c28 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Extensions.Duplicate.txt @@ -0,0 +1,47 @@ +Pigeonhole Sieve: Duplicate Extension +===================================== + +The *duplicate* extension RFC 7353 [http://tools.ietf.org/html/rfc7352] adds a +new test command called 'duplicate' to the Sieve language. This test adds the +ability to detect duplications. The main application for this new test is +handling duplicate deliveries commonly caused by mailing list subscriptions or +redirected mail addresses. The detection is normally performed by matching the +message ID to an internal list of message IDs from previously delivered +messages. For more complex applications, the 'duplicate' test can also use the +content of a specific header field or other parts of the message. + +Previously, this extension was Dovecot-specific and available under the name +'vnd.dovecot.duplicate'. Specification for old version available here +[http://hg.rename-it.nl/dovecot-2.1-pigeonhole/raw-file/tip/doc/rfc/spec-bosch-sieve-duplicate.txt]. +That implementation differs significantly from what is now published as RFC +7353 [http://tools.ietf.org/html/rfc7352], but the original extension is still +supported for backwards compatibility. + +Configuration +------------- + +The *duplicate* extension is available by default. The *duplicate* extension +has its own specific settings. The following settings are available (default +values are indicated): + +sieve_duplicate_default_period = 14d : + +sieve_duplicate_max_period = 7d : + These options respectively specify the default and the maximum value for the + period after which tracked values are purged from the duplicate tracking + database. The period is specified in s(econds), unless followed by a d(ay), + h(our) or m(inute) specifier character. + +Example +------- + +---%<------------------------------------------------------------------------- +plugin { + sieve = ~/.dovecot.sieve + + sieve_duplicate_default_period = 1h + sieve_duplicate_max_period = 1d +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Extensions.Editheader.txt b/doc/wiki/Pigeonhole.Sieve.Extensions.Editheader.txt new file mode 100644 index 0000000..f7b3869 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Extensions.Editheader.txt @@ -0,0 +1,61 @@ +Pigeonhole Sieve: Editheader Extension +====================================== + +The *editheader* extension (RFC5293 [http://tools.ietf.org/html/rfc5293/]) +enables Sieve scripts to delete and add message header fields, thereby allowing +interaction with other components that consume or produce header fields. + +Configuration +------------- + +The *editheader* extension is not available by default and needs to be enabled +explicitly by adding it to the 'sieve_extensions' setting. + +The following settings can be configured for the *editheader* extension +(default values are indicated): + +sieve_editheader_max_header_size = 2048 : + The maximum size in bytes of a header field value passed to the 'addheader' + command. The minimum value for this setting is 1024 bytes. The value is in + bytes, unless followed by a k(ilo). + +sieve_editheader_forbid_add = : + A space-separated list of headers that cannot be added to the message header. + Addition of the 'Subject:' header cannot be prohibited, as required by the + RFC specification. Therefore, adding this header to this setting has no + effect. + +sieve_editheader_forbid_delete = : + A space-separated list of headers that cannot be deleted from the message + header. Deleting the 'Received:' and 'Auto-Submitted:' fields is always + forbidden, while removing the 'Subject:' header cannot be prohibited, as + required by the RFC specification. Therefore, adding one of these headers to + this setting has no effect. + +sieve_editheader_protected = : + A space-separated list of headers that cannot be added to or deleted from the + message header. This setting is provided for backwards compatibility. It is a + combination of the 'sieve_editheader_forbid_add' and + 'sieve_editheader_forbid_delete' settings. The same limitations apply. + +Invalid values for the settings above will make the Sieve interpreter log a +warning and revert to the default values. + +Example +------- + +---%<------------------------------------------------------------------------- +plugin { + # Use editheader + sieve_extensions = +editheader + + # Header fiels must not exceed one kilobyte + sieve_editheader_max_header_size = 1k + + # Protected special headers + sieve_editheader_forbid_add = X-Verified + sieve_editheader_forbid_delete = X-Verified X-Seen +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Extensions.Include.txt b/doc/wiki/Pigeonhole.Sieve.Extensions.Include.txt new file mode 100644 index 0000000..e5b820b --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Extensions.Include.txt @@ -0,0 +1,28 @@ +Pigeonhole Sieve: Include Extension +=================================== + +The Sieve *include* extension (RFC 6609 [http://tools.ietf.org/html/rfc6609]) +permits users to include one Sieve script into another. This can make managing +large scripts or multiple sets of scripts much easier, and allows a site and +its users to build up libraries of scripts. Users are able to include their own +personal scripts or site-wide scripts. + +Included scripts can include more scripts of their own, yielding a tree of +included scripts with the main script (typically the user's personal script) at +its root. + +Configuration +------------- + +The *include* extension is available by default. The *include* extension has +its own specific settings. The following settings can be configured for the +*include* extension (default values are indicated): + +sieve_include_max_includes = 255 : + The maximum number of scripts that may be included. This is the total number + of scripts involved in the include tree. + +sieve_include_max_nesting_depth = 10 : + The maximum nesting depth for the include tree. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt b/doc/wiki/Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt new file mode 100644 index 0000000..fbc7225 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt @@ -0,0 +1,146 @@ +Pigeonhole Sieve: Spamtest and Virustest Extensions +=================================================== + +Using the *spamtest* and *virustest* extensions (RFC 5235 +[http://tools.ietf.org/html/rfc5235/]), the Sieve language provides a uniform +and standardized command interface for evaluating spam and virus tests +performed on the message. Users no longer need to know what headers need to be +checked and how the scanner's verdict is represented in the header field value. +They only need to know how to use the *spamtest* (*spamtestplus*) and +*virustest* extensions. This also gives GUI-based Sieve editors the means to +provide a portable and easy to install interface for spam and virus filter +configuration. The burden of specifying which headers need to be checked and +how the scanner output is represented falls onto the Sieve administrator. + +Configuration +------------- + +The *spamtest*, *spamtestplus* and *virustest* extensions are not enabled by +default and thus need to be enabled explicitly using the 'sieve_extensions' +setting. + +The following settings need to be configured for using the *spamtest* and +*spamtestplus* extensions. The *virustest* extension has identical +configuration settings, but with a ''sieve_virustest_'' prefix instead of a +''sieve_spamtest_'' prefix: + +sieve_spamtest_status_type = "score" / "strlen" / "text": + This specifies the type of status result that the spam/virus scanner + produces. This can either be a numeric score ('score'), a string of identical + characters ('strlen'), e.g. ''*******'', or a textual description ('text'), + e.g.'{{{Spam}}}' or ''Not Spam''. + +sieve_spamtest_status_header = <header-field> [ ":" <regexp> ]: + This specifies the header field that contains the result information of the + spam scanner and it may express the syntax of the content of the header. If + no matching header is found in the message, the spamtest command will match + against "0". + +: + This is a structured setting. The first part specifies the header field name. + Optionally, a POSIX regular expression follows the header field name, + separated by a colon. Any white space directly following the colon is not + part of the regular expression. If the regular expression is omitted, any + header content is accepted and the full header value is used. When a regular + expression is used, it must specify one match value (inside brackets) that + yields the desired spam scanner result. If the header does not match the + regular expression or if no value match is found, the 'spamtest' test will + match against "0" during Sieve script execution. + +sieve_spamtest_max_value =: + This statically specifies the maximum value a numeric spam score can have. + +sieve_spamtest_max_header = <header-field> [ ":" <regexp> ]: + Some spam scanners include the maximum score value in one of their status + headers. Using this setting, this maximum can be extracted from the message + itself in stead of specifying the maximum manually using the setting + 'sieve_spamtest_max_value' explained above. The syntax is identical to the + 'sieve_spamtext_status_header' setting. + +sieve_spamtest_text_valueX =: + When the 'sieve_spamtest_status_type' setting is set to 'text', these + settings specify that the 'spamtest' test will match against the value "'X'" + when the specified string is equal to the text (extracted) from the status + header. For *spamtest* and *spamtestplus*, values of X between 0 and 10 are + recognized, while *virustest* only uses values between 0 and 5. + +Examples +-------- + +This section shows several configuration examples. Each example shows a +specimen of valid virus/spam test headers that the given configuration willwork +on. + +Example 1 +--------- + +Spam header: 'X-Spam-Score: No, score=-3.2' + +---%<------------------------------------------------------------------------- +plugin { + sieve_extensions = +spamtest +spamtestplus + + sieve_spamtest_status_type = score + sieve_spamtest_status_header = \ + X-Spam-Score: [[:alnum:]]+, score=(-?[[:digit:]]+\.[[:digit:]]) + sieve_spamtest_max_value = 5.0 +} +---%<------------------------------------------------------------------------- + +Example 2 +--------- + +Spam header: 'X-Spam-Status: Yes' + +---%<------------------------------------------------------------------------- +plugin { + sieve_extensions = +spamtest +spamtestplus + + sieve_spamtest_status_type = text + sieve_spamtest_status_header = X-Spam-Status + sieve_spamtest_text_value1 = No + sieve_spamtest_text_value10 = Yes +} +---%<------------------------------------------------------------------------- + +Example 3 +--------- + +Spam header: 'X-Spam-Score: sssssss' + +---%<------------------------------------------------------------------------- +plugin { + sieve_extensions = +spamtest +spamtestplus + + sieve_spamtest_status_header = X-Spam-Score + sieve_spamtest_status_type = strlen + sieve_spamtest_max_value = 5 +} +---%<------------------------------------------------------------------------- + +Example 4 +--------- + +Spam header: 'X-Spam-Score: status=3.2 required=5.0' + +Virus header: 'X-Virus-Scan: Found to be clean.' + +---%<------------------------------------------------------------------------- +plugin { + sieve_extensions = +spamtest +spamtestplus +virustest + + sieve_spamtest_status_type = score + sieve_spamtest_status_header = \ + X-Spam-Score: score=(-?[[:digit:]]+\.[[:digit:]]).* + sieve_spamtest_max_header = \ + X-Spam-Score: score=-?[[:digit:]]+\.[[:digit:]] +required=([[:digit:]]+\.[[:digit:]]) + + sieve_virustest_status_type = text + sieve_virustest_status_header = X-Virus-Scan: Found to be (.+)\. + sieve_virustest_text_value1 = clean + sieve_virustest_text_value5 = infected +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Extensions.Vacation.txt b/doc/wiki/Pigeonhole.Sieve.Extensions.Vacation.txt new file mode 100644 index 0000000..16bf772 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Extensions.Vacation.txt @@ -0,0 +1,106 @@ +Pigeonhole Sieve: Vacation Extension +==================================== + +The Sieve vacation extension (RFC5230 [http://tools.ietf.org/html/rfc5230/]) +defines a mechanism to generate automatic replies to incoming email messages. +It takes various precautions to make sure replies are only sent when +appropriate. Script authors can specify how often replies can be sent to a +particular contact. In the original vacation extension, this interval is +specified in days with a minimum of one day. When more granularity is necessary +and particularly when replies must be sent more frequently than one day, the +vacation-seconds extension (RFC6131 [http://tools.ietf.org/html/rfc5230/]) can +be used. This allows specifying the minimum reply interval in seconds with a +minimum of zero (a reply is then always sent), depending on administrator +configuration. + +Configuration +============= + +The *vacation* extension is available by default. In contrast, the +*vacation-seconds* extension - which implies the vacation extension when used - +is not available by default and needs to be enabled explicitly by adding it to +the 'sieve_extensions' setting. The configuration also needs to be adjusted +accordingly to allow a non-reply period of less than a day. + +The *vacation* and *vacation-seconds* extensions have their own specific +settings. The settings that specify a period (currently all of them) are +specified in *s*(econds), unless followed by a *d*(ay), *h*(our) or *m*(inute) +specifier character. + +The following settings can be configured for the vacation extension in the +'plugin' section (default values are indicated): + +sieve_vacation_min_period = 1d : + This specifies the minimum period that can be specified for the :days and + :seconds tags of the vacation command. A minimum of 0 indicates that users + are allowed to make the Sieve interpreter send a vacation response message + for every incoming message that meets the other reply criteria (refer to + RFC5230). A value of zero is however not recommended. + +sieve_vacation_max_period = 0 : + This specifies the maximum period that can be specified for the :days tag of + the vacation command. The configured value must be larger than the + sieve_vacation_min_period setting. A value of 0 has a special meaning: it + indicates that there is no upper limit. + +sieve_vacation_default_period = 7d : + This specifies the default period that is used when no :days or :seconds tag + is specified. The configured value must lie between the + sieve_vacation_min_period and sieve_vacation_max_period. + +sieve_vacation_use_original_recipient = no : + This specifies whether the original envelope recipient should be used in the + check for implicit delivery. The vacation command checks headers of the + incoming message, such as To: and Cc: for the address of the recipient, to + verify that the message is explicitly addressed at the recipient. If the + recipient address is not found, the vacation action will not trigger a + response to prevent sending a reply when it is not appropriate. Normally only + the final recipient address is used in this check. This setting allows + including the original recipient specified in the SMTP session if available. + This is useful to handle mail accounts with aliases. Use this option with + caution: if you are using aliases that point to more than a single account, + senders can get multiple vacation responses for a single message. Use the + <LDA.txt> '-a' option or the <LMTP.txt>/ <LDA.txt> + 'lda_original_recipient_header' setting to make the original SMTP recipient + available to Sieve. + +sieve_vacation_dont_check_recipient = no : + This disables the checks for implicit delivery entirely. This means that the + vacation command does not verify that the message is explicitly addressed at + the recipient. Use this option with caution. Specifying 'yes' will violate + the Sieve standards and can cause vacation replies to be sent for messages + not directly addressed at the recipient. + +sieve_vacation_send_from_recipient = no : + This setting determines whether vacation messages are sent with the SMTP MAIL + FROM envelope address set to the recipient address of the Sieve script owner. + Normally this is set to<>, which is the default as recommended in the + specification. This is meant to prevent mail loops. However, there are + situations for which a valid sender address is required and this setting can + be used to accommodate for those. + +Invalid values for the settings above will make the Sieve interpreter log a +warning and revert to the default values. + +See also <how vacation auto-reply uses addresses> [Pigeonhole.Sieve.Usage.txt]. + +Example +------- + +---%<------------------------------------------------------------------------- +plugin { + # Use vacation-seconds + sieve_extensions = +vacation-seconds + + # One hour at minimum + sieve_vacation_min_period = 1h + + # Ten days default + sieve_vacation_default_period = 10d + + # Thirty days at maximum + sieve_vacation_max_period = 30d +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Extensions.Variables.txt b/doc/wiki/Pigeonhole.Sieve.Extensions.Variables.txt new file mode 100644 index 0000000..ee7a7f7 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Extensions.Variables.txt @@ -0,0 +1,26 @@ +Pigeonhole Sieve: Variables Extension +===================================== + +The Sieve *variables* extension (RFC5229 [http://tools.ietf.org/html/rfc5229/]) +adds the concept of variables to the Sieve language. + +Configuration +============= + +The *variables* extension is available by default. The *variables* extension +has its own specific settings. The following settings can be configured for the +*variables* extension (default values are indicated): + +sieve_variables_max_scope_size = 255 (v0.5.0+) : + The maximum number of variables that can be declared in a scope. There are + currently two variable scopes: the normal script scope and the global scope + created by the "include" extension. The minimum value for this setting is + 128. + +sieve_variables_max_variable_size = 4k (v0.5.0+) : + The maximum allowed size for the value of a variable. If exceeded at runtime, + the value is always truncated to the configured maximum. The minimum value + for this setting is 4000 bytes. The value is in bytes, unless followed by a + k(ilo). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Extensions.txt b/doc/wiki/Pigeonhole.Sieve.Extensions.txt new file mode 100644 index 0000000..1c0cd1d --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Extensions.txt @@ -0,0 +1,14 @@ +Pigeonhole Sieve Extensions +=========================== + +The following Sieve language extensions have a dedicated wiki page with +specific configuration and usage information: + + * <duplicate> [Pigeonhole.Sieve.Extensions.Duplicate.txt] + * <editheader> [Pigeonhole.Sieve.Extensions.Editheader.txt] + * <include> [Pigeonhole.Sieve.Extensions.Include.txt] + * <spamtest and virustest> [Pigeonhole.Sieve.Extensions.SpamtestVirustest.txt] + + * <vacation and vacation-seconds> [Pigeonhole.Sieve.Extensions.Vacation.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Plugins.Extdata.txt b/doc/wiki/Pigeonhole.Sieve.Plugins.Extdata.txt new file mode 100644 index 0000000..1b7ecdb --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Plugins.Extdata.txt @@ -0,0 +1,155 @@ +Pigeonhole Sieve Extdata Plugin +=============================== + +The extdata plugin adds the *vnd.dovecot.extdata* extension to the Sieve +language. It allows a Sieve script to lookup information from a datasource +external to the script. This makes use of Dovecot's dict mechanism in a +read-only manner, meaning that scripts cannot update dict data sources. + +Getting the sources +------------------- + +Currently, the sources of the extdata plugin are not released, but you can get +them from the their Git repository. + +For Pigeonhole v0.4: + +---%<------------------------------------------------------------------------- +git clone -b core-0.4 https://github.com/stephanbosch/sieve-extdata-plugin.git +---%<------------------------------------------------------------------------- + +For Pigeonhole v0.5: + +---%<------------------------------------------------------------------------- +git clone -b core-0.5 https://github.com/stephanbosch/sieve-extdata-plugin.git +---%<------------------------------------------------------------------------- + +Compiling +--------- + +If you downloaded the sources of this plugin using Git, you will need to +execute './autogen.sh' first to build the automake structure in your source +tree. This process requires autotools and libtool to be installed. + +If you installed Dovecot from sources, the plugin's configure script should be +able to find the installed 'dovecot-config' automatically, along with the +Pigeonhole development headers: + +---%<------------------------------------------------------------------------- +./configure +make +sudo make install +---%<------------------------------------------------------------------------- + +If this doesn't work, you can use '--with-dovecot=<path>' configure option, +where the path points to a directory containing 'dovecot-config' file. This can +point to an installed file: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=/usr/local/lib/dovecot +make +sudo make install +---%<------------------------------------------------------------------------- + +The above example should also find the necessary Pigeonhole development headers +implicitly. You can also compile by pointing to compiled Dovecot and Pigeonhole +source trees: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=../dovecot-2.3.2/ +--with-pigeonhole=../dovecot-2.3-pigeonhole-0.5.2 +make +sudo make install +---%<------------------------------------------------------------------------- + +Configuration +------------- + +This package builds and installs the sieve_extdata plugin for Pigeonhole Sieve. +The plugin is activated by adding it to the sieve_plugins setting + +---%<------------------------------------------------------------------------- +sieve_plugins = sieve_extdata +---%<------------------------------------------------------------------------- + +The following configuration settings are used: + +sieve_extdata_dict_uri = : + Specifies the uri of the dict that is used for extdata lookups. + +Example: + +---%<------------------------------------------------------------------------- +plugin { + + sieve = ~/.dovecot.sieve + sieve_plugins = sieve_extdata + + sieve_extdata_dict_uri = file:/etc/dovecot/pigeonhole-sieve.dict +} +---%<------------------------------------------------------------------------- + +Usage +----- + +Sieve scripts can use the new 'vnd.dovecot.extdata' extension as follows: + +---%<------------------------------------------------------------------------- +require ["variables", "vacation", "vnd.dovecot.extdata"]; + +vacation :days 30 :subject "${extdata.vacation_subject}" +"${extdata.vacation_message}"; +keep; +---%<------------------------------------------------------------------------- + +where "priv/vacation_subject" & "priv/vacation_message" would be looked up in +the Dovecot dict. See below for some example dicts: + +Dict with flat file backend +--------------------------- + +To use a flat file backend for the above example, create a dict file with the +following format (for example /etc/dovecot/sieve-extdata-lookup.dict): + +---%<------------------------------------------------------------------------- +priv/vacation_message +Sorry I am out of the office +---%<------------------------------------------------------------------------- + +Dict with a SQL backend +----------------------- + +To use a SQL backend for the above example, first set up a dict proxy. In +/etc/dovecot.conf: + +---%<------------------------------------------------------------------------- +dict { + sieve = mysql:/etc/dovecot/pigeonhole-sieve.dict +} +---%<------------------------------------------------------------------------- + +And in /etc/dovecot/pigeonhole-sieve.dict: + +---%<------------------------------------------------------------------------- +connect = host=localhost dbname=dovecot user=dovecot password=password + +map { + pattern = priv/vacation_message # The dict value to lookup + table = virtual_users # The SQL table to perform the lookup in + username_field = email # The username field to search on in the +table + value_field = vacation_msg # The database value to return +} +---%<------------------------------------------------------------------------- + +Finally configure extdata to use the proxy: + +---%<------------------------------------------------------------------------- +sieve_extdata_dict_uri = proxy::sieve +---%<------------------------------------------------------------------------- + +Read the (preliminary) specification +[https://github.com/stephanbosch/sieve-extdata-plugin/blob/core-0.5/doc/rfc/spec-bosch-sieve-external-data.txt] +for more information. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Plugins.Extprograms.txt b/doc/wiki/Pigeonhole.Sieve.Plugins.Extprograms.txt new file mode 100644 index 0000000..78ffcdc --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Plugins.Extprograms.txt @@ -0,0 +1,337 @@ +Pigeonhole Sieve Extprograms Plugin +=================================== + +The "sieve_extprograms" plugin provides an extension to the Sieve filtering +language [http://www.sieve.info] adding new action commands for invoking a +predefined set of external programs. Messages can be piped to or filtered +through those programs and string data can be input to and retrieved from those +programs. To mitigate the security concerns, the external programs cannot be +chosen arbitrarily; the available programs are restricted through administrator +configuration. + +This plugin is only available for <Pigeonhole.txt> v0.3 and higher (available +for Dovecot v2.1). For <Pigeonhole.txt> v0.4 this plugin is part of the +release. This an evolution of the <Pipe plugin> +[Pigeonhole.Sieve.Plugins.Pipe.txt] for Pigeonhole v0.2 and now provides the +"filter" and "execute" commands (and corresponding extensions) in addition to +the "pipe" command that was provided earlier by the Pipe plugin. + +Getting the sources +------------------- + +Starting with Pigeonhole v0.4 for Dovecot v2.2, the plugin is included in the +release and therefore it is implicitly compiled and installed with Pigeonhole +itself. + +For Pigeonhole v0.3, the plugin was never released, but you can get the sources +from its Mercurial repository: + +---%<------------------------------------------------------------------------- +hg clone http://hg.rename-it.nl/pigeonhole-0.3-sieve-extprograms/ +---%<------------------------------------------------------------------------- + +Compiling for Pigeonhole v0.3 +----------------------------- + +If you downloaded the sources of this plugin using Mercurial, you will need to +execute './autogen.sh' first to build the automake structure in your source +tree. This process requires autotools and libtool to be installed. + +If you installed Dovecot from sources, the plugin's configure script should be +able to find the installed 'dovecot-config' automatically, along with the +Pigeonhole development headers: + +---%<------------------------------------------------------------------------- +./configure +make +sudo make install +---%<------------------------------------------------------------------------- + +If this doesn't work, you can use '--with-dovecot=<path>' configure option, +where the path points to a directory containing 'dovecot-config' file. This can +point to an installed file: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=/usr/local/lib/dovecot +make +sudo make install +---%<------------------------------------------------------------------------- + +The above example should also find the necessary Pigeonhole development headers +implicitly. You can also compile by pointing to compiled Dovecot and Pigeonhole +source trees: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=../dovecot-2.1.0/ +--with-pigeonhole=../dovecot-2.1-pigeonhole-0.3.0 +make +sudo make install +---%<------------------------------------------------------------------------- + +Configuration +------------- + +The plugin is activated by adding it to the sieve_plugins setting: + +---%<------------------------------------------------------------------------- +sieve_plugins = sieve_extprograms +---%<------------------------------------------------------------------------- + +This plugin registers the 'vnd.dovecot.pipe', 'vnd.dovecot.filter' and +'vnd.dovecot.execute' extensions with the Sieve interpreter. However, these +extensions are not enabled by default and thus need to be enabled explicitly. +It is recommended to restrict the use of these extensions to global context by +adding these to the 'sieve_global_extensions' setting. If personal user scripts +also need to directly access external programs, the extensions need to be added +to the 'sieve_extensions' setting. + +The commands introduced by the Sieve language extensions in this plugin can +directly pipe a message or string data to an external program (typically a +shell script) by forking a new process. Alternatively, these can connect to a +unix socket behind which a Dovecot script service is listening to start the +external program, e.g. to execute as a different user or for added security. + +The program name specified for the new Sieve 'pipe', 'filter' and 'execute' +commands is used to find the program or socket in a configured directory. +Separate directories are specified for the sockets and the directly executed +binaries. The socket directory is searched first. Since the use of "/" in +program names is prohibited, it is not possible to build a hierarchical +structure. + +Directly forked programs are executed with a limited set of environment +variables:'HOME', 'USER', 'SENDER', 'RECIPIENT' and 'ORIG_RECIPIENT'. Programs +executed through the script-pipe socket service currently have no environment +set at all. + +If a shell script is expected to read a message or string data, it must fully +read the provided input until the data ends with EOF, otherwise the Sieve +action invoking the program will fail. The action will also fail when the shell +script returns a nonzero exit code. Standard output is available for returning +a message (for the filter command) or string data (for the execute command) to +the Sieve interpreter. Standard error is written to the LDA log file. + +The three extensions introduced by this plugin - 'vnd.dovecot.pipe', +'vnd.dovecot.filter' and 'vnd.dovecot.execute' - each have separate but similar +configuration. The following configuration settings are used, for which +"<extension>" in the setting name is replaced by either 'pipe', 'filter' or +'execute' depending on which extension is being configured: + +sieve_<extension>_socket_dir = : + Points to a directory relative to the Dovecot base_dir where the plugin looks + for script service sockets. + +sieve_<extension>_bin_dir = : + Points to a directory where the plugin looks for programs (shell scripts) to + execute directly and pipe messages to. + +sieve_<extension>_exec_timeout = 10s : + Configures the maximum execution time after which the program is forcibly + terminated. + +sieve_<extension>_input_eol = crlf : + Determines the end-of-line character sequence used for the data piped to + external programs. The default is currently "crlf", which represents a + sequence of the carriage return (CR) and line feed (LF) characters. This + matches the Internet Message Format (RFC5322) and what Sieve itself uses as a + line ending. Set this setting to "lf" to use a single LF character instead. + +Configuration Example 1: socket service for "pipe" and "execute" +---------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +plugin { + sieve = ~/.dovecot.sieve + + sieve_plugins = sieve_extprograms + sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute + + # pipe sockets in /var/run/dovecot/sieve-pipe + sieve_pipe_socket_dir = sieve-pipe + + # execute sockets in /var/run/dovecot/sieve-execute + sieve_execute_socket_dir = sieve-execute +} + +service sieve-pipe-script { + # This script is executed for each service connection + executable = script /usr/lib/dovecot/sieve-extprograms/sieve-pipe-action.sh + + # use some unprivileged user for execution + user = dovenull + + # socket name is program-name in Sieve (without sieve-pipe/ prefix) + unix_listener sieve-pipe/sieve-pipe-script { + } +} + +service sieve-execute-action { + # This script is executed for each service connection + executable = script +/usr/lib/dovecot/sieve-extprograms/sieve-execute-action.sh + + # use some unprivileged user for execution + user = dovenull + + # socket name is program-name in Sieve (without sieve-execute/ prefix) + unix_listener sieve-execute/sieve-execute-action { + } +} +---%<------------------------------------------------------------------------- + +Configuration Example 2: direct execution for "pipe" and "filter" +----------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +plugin { + sieve = ~/.dovecot.sieve + + sieve_plugins = sieve_extprograms + sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.filter + + # This directory contains the scripts that are available for the pipe +command. + sieve_pipe_bin_dir = /usr/lib/dovecot/sieve-pipe + + # This directory contains the scripts that are available for the filter + # command. + sieve_filter_bin_dir = /usr/lib/dovecot/sieve-filter +} +---%<------------------------------------------------------------------------- + +Usage +----- + +Read the specification (v0.3 plugin +[http://hg.rename-it.nl/pigeonhole-0.3-sieve-extprograms/raw-file/tip/doc/rfc/spec-bosch-sieve-extprograms.txt]/v0.4+ +[https://github.com/dovecot/pigeonhole/blob/master/doc/rfc/spec-bosch-sieve-extprograms.txt]) +for detailed information on how to use the new language extensions. + +Full Examples +------------- + +Example 1 +--------- + +This simple example shows how to use the "vnd.dovecot.execute" extension to +perform some sort of test on the incoming message. + +Relevant configuration: + +---%<------------------------------------------------------------------------- +plugin { + sieve_extensions = +vnd.dovecot.execute + + sieve_plugins = sieve_extprograms + sieve_execute_bin_dir = /usr/lib/dovecot/sieve-execute +} +---%<------------------------------------------------------------------------- + +The sieve script: + +---%<------------------------------------------------------------------------- +require "vnd.dovecot.execute"; + +if not execute :pipe "hasfrop.sh" { + discard; + stop; +} +---%<------------------------------------------------------------------------- + +At the location '/usr/lib/dovecot/sieve-execute', create the executable script +'hasfrop.sh'. In this example, the 'hasfrop.sh' checks whether the message +contains the literal text "FROP" anywhere in the message. The Sieve script +shown above discards the message if this scripts ends with an exit code other +than 0, which happens when "FROP" was found. + +---%<------------------------------------------------------------------------- +# Something that reads the whole message and inspects it for some +# property. Not that the whole message needs to be read from input! +N=`cat | grep -i "FROP"` # Check it for the undesirable text "FROP" +if [ ! -z "$N" ]; then + # Result: deny + exit 1; +fi + +# Result: accept +exit 0 +---%<------------------------------------------------------------------------- + +Example 2 +--------- + +This example shows how to use the "vnd.dovecot.execute" extension for +querying/updating a MySQL database. This is used to redirect messages only once +every 300s for a particular sender. Note that this particular use case could +also be implemented using the Sieve " <duplicate> +[Pigeonhole.Sieve.Extensions.Duplicate.txt]" extension + +Relevant configuration: + +---%<------------------------------------------------------------------------- +plugin { + sieve_extensions = +vnd.dovecot.execute + + sieve_plugins = sieve_extprograms + sieve_execute_bin_dir = /usr/lib/dovecot/sieve-execute +} +---%<------------------------------------------------------------------------- + +The sieve script: + +---%<------------------------------------------------------------------------- +require ["variables", "copy", "envelope", "vnd.dovecot.execute"]; + +# put the envelope-from address in a variable +if envelope :matches "from" "*" { set "from" "${1}"; } + +# execute the vacationcheck.sh program and redirect the message based on its +exit code +if execute :output "vacation_message" "vacationcheck.sh" ["${from}","300"] +{ + redirect + :copy "foo@bar.net"; +} +---%<------------------------------------------------------------------------- + +At the location '/usr/lib/dovecot/sieve-execute', create the executable script +'vacationcheck.sh'. In this example, the 'vacationcheck.sh' script needs two +parameters: the sender address and a time interval specified in seconds. The +time interval is used to specify the minimum amount of time that needs to have +passed since the sender was last seen. If the script returns exit code 0, then +message is redirected in the Sieve script shown above. + +---%<------------------------------------------------------------------------- +USER=postfixadmin +PASS=pass +DATABASE=postfixadmin + +# DB STRUCTURE +#CREATE TABLE `sieve_count` ( +# `from_addres` varchar(254) NOT NULL, +# `date` datetime NOT NULL +#) ENGINE=InnoDB DEFAULT CHARSET=latin1; +# +#ALTER TABLE `sieve_count` +# ADD KEY `from_addres` (`from_addres`); + +MAILS=$(mysql -u$USER -p$PASS $DATABASE --batch --silent -e "SELECT count(*) as +ile FROM sieve_count WHERE from_addres='$1' AND DATE_SUB(now(),INTERVAL $2 +SECOND) < date;") +ADDRESULT=$(mysql -u$USER -p$PASS $DATABASE --batch --silent -e "INSERT INTO +sieve_count (from_addres, date) VALUES ('$1', NOW());") + +# uncoment below to debug +# echo Uset $1 sent $MAILS in last $2 s >> +/usr/lib/dovecot/sieve-pipe/output.txt +# echo Add result : $ADDRESULT >> /usr/lib/dovecot/sieve-pipe/output.txt +# echo $MAILS + +if [ "$MAILS" = "0" ] +then +exit 0 +fi + +exit 1 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Plugins.IMAPFilterSieve.txt b/doc/wiki/Pigeonhole.Sieve.Plugins.IMAPFilterSieve.txt new file mode 100644 index 0000000..d9f111a --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Plugins.IMAPFilterSieve.txt @@ -0,0 +1,49 @@ +Pigeonhole IMAP FILTER=SIEVE Plugin +=================================== + +Normally, Sieve filters can either be applied at initial mail delivery or +triggered by certain events in the Internet Message Access Protocol +('IMAPSIEVE';RFC 6785 [http://tools.ietf.org/html/rfc6785]). The user can +configure which Sieve scripts to run at these instances, but it is not possible +to trigger the execution of Sieve scripts manually. However, this could be very +useful; e.g, to test new Sieve rules and to re-filter messages that were +erroneously handled by an earlier version of the Sieve scripts involved. + +<Pigeonhole.txt> provides the 'imap_filter_sieve' plugin, which provides a +vendor-defined IMAP extension called 'FILTER=SIEVE'. This adds a new 'FILTER' +command that allows applying a mail filter (a Sieve script) on a set of +messages that match the specified IMAP searching criteria. + +The latest draft of the specification for this IMAP capability is available +here +[https://github.com/dovecot/pigeonhole/blob/master/doc/rfc/draft-bosch-imap-filter-sieve-00.txt]. +This plugin is experimental and the specification is likely to change. Use the +specification included in your current release to obtain the matching +specification for your release. + +This plugin is available for <Pigeonhole.txt> v0.4.24 and higher (available for +Dovecot v2.2.36), and v0.5.2 and higher (available for Dovecot v2.3.2). The +plugins are included in the Pigeonhole package and are therefore implicitly +compiled and installed with Pigeonhole itself. + +Configuration +------------- + +The IMAP FILTER Sieve plugin is activated by adding it to the mail_plugins +setting for the imap protocol: + +---%<------------------------------------------------------------------------- +protocol imap { + mail_plugins = $mail_plugins imap_filter_sieve +} +---%<------------------------------------------------------------------------- + +Note that enabling this plugin allows users to specify the Sieve script content +as a parameter to the FILTER command, not just run existing stored scripts. + +Currently, no other settings specific to this plugin are defined. It uses the +normal configuration settings used by the LDA Sieve plugin at delivery. + +The sieve_before and sieve_after scripts are currently ignored by this plugin. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Plugins.IMAPSieve.txt b/doc/wiki/Pigeonhole.Sieve.Plugins.IMAPSieve.txt new file mode 100644 index 0000000..7a04bc7 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Plugins.IMAPSieve.txt @@ -0,0 +1,109 @@ +Pigeonhole IMAPSieve Plugins +============================ + +As defined in the base specification (RFC 5228 +[http://tools.ietf.org/html/rfc5228]), the Sieve language is used only during +delivery. However, in principle, it can be used at any point in the processing +of an email message.RFC 6785 [http://tools.ietf.org/html/rfc6785] defines the +use of Sieve filtering in IMAP, operating when messages are created or their +attributes are changed. This feature extends both Sieve and IMAP. Therefore, +Pigeonhole provides both an IMAP plugin and a Sieve plugin. + +The 'sieve_imapsieve' plugin implements the 'imapsieve' extension for the Sieve +filtering language, adding functionality for using Sieve scripts from within +IMAP. The 'imap_sieve' plugin for IMAP adds the 'IMAPSIEVE' capability to the +'imap' service. The basic 'IMAPSIEVE' capability allows attaching a Sieve +script to a mailbox for any mailbox by setting a special IMAP METADATA entry. +This way, users can configure Sieve scripts that are run for IMAP events in +their mailboxes. + +Beyond the standard, the Pigeonhole implementation also adds the ability for +administrators to configure Sieve scripts outside the user's control, that are +run either before or after a user's script if there is one. + +This plugin is available for <Pigeonhole.txt> v0.4.14 and higher (available for +Dovecot v2.2.24). The plugins are included in the Pigeonhole package and are +therefore implicitly compiled and installed with Pigeonhole itself. + +Configuration +------------- + +The IMAP plugin is activated by adding it to the mail_plugins setting for the +imap protocol: + +---%<------------------------------------------------------------------------- +protocol imap { + mail_plugins = $mail_plugins imap_sieve +} +---%<------------------------------------------------------------------------- + +This will only enable support for administrator scripts. User scripts are only +supported when additionally a Sieve URL is configured using the imapsieve_url +plugin setting. This URL points to the <ManageSieve> +[Pigeonhole.ManageSieve.txt] server that users need to use to upload their +Sieve scripts. This URL will be shown to the client in the IMAP CAPABILITY +response as 'IMAPSIEVE=<URL>'. + +The Sieve plugin is activated by adding it to the sieve_plugins setting: + +---%<------------------------------------------------------------------------- +sieve_plugins = sieve_imapsieve +---%<------------------------------------------------------------------------- + +This plugin registers the 'imapsieve' extension with the Sieve interpreter. +This extension is enabled implicitly, which means that it does not need to be +added to the 'sieve_extensions' setting. + +Note that the 'imapsieve' extension can only be used in a Sieve script that is +invoked from IMAP. When it is used in the active delivery script, it will cause +runtime errors. To make a Sieve script suitable for both delivery and IMAP, the +availability of the extension can be tested using the 'ihave' test (RFC 5463 +[http://tools.ietf.org/html/rfc5463]) as usual. + +The following settings are recognized the "imap_sieve" plugin: + +imapsieve_url = : + If configured, this setting enables support for user Sieve scripts in IMAP. + So, leave this unconfigured if you don't want users to have the ability to + associate Sieve scripts with mailboxes. This has no effect on the + administrator-controlled Sieve scripts explained below. The value is an URL + pointing to the <ManageSieve> [Pigeonhole.ManageSieve.txt] server that users + must use to upload their Sieve scripts; e.g.,'sieve://sieve.example.com'. + +imapsieve_mailboxXXX_name = : + This setting configures the name of a mailbox for which administrator scripts + are configured. The `XXX' in this setting is a sequence number, which allows + configuring multiple associations between Sieve scripts and mailboxes. The + settings defined hereafter with matching sequence numbers apply to the + mailbox named by this setting. The sequence of configured mailboxes ends at + the first missing 'imapsieve_mailboxXXX_name' setting. This setting supports + wildcards with a syntax compatible with the IMAP LIST command, meaning that + this setting can apply to multiple or even all ("*") mailboxes. + +imapsieve_mailboxXXX_before = : + +imapsieve_mailboxXXX_after = : + When an IMAP event of interest occurs, these sieve scripts are executed + before and after any user script respectively. These settings each specify + the location of a single sieve script. The semantics of these settings are + very similar to the 'sieve_before' and 'sieve_after' settings: the specified + scripts form a sequence together with the user script in which the next + script is only executed when an (implicit) keep action is executed. + +imapsieve_mailboxXXX_causes = : + Only execute the administrator Sieve scripts for the mailbox configured with + 'imapsieve_mailboxXXX_name' when one of the listed 'IMAPSIEVE' causes + [https://tools.ietf.org/html/rfc6785#section-4.3] apply (currently either + 'APPEND', 'COPY', or 'FLAG'. This has no effect on the user script, which is + always executed no matter the cause. + +imapsieve_mailboxXXX_from = : + Only execute the administrator Sieve scripts for the mailbox configured with + 'imapsieve_mailboxXXX_name' when the message originates from the indicated + mailbox. This setting supports wildcards with a syntax compatible with the + IMAP LIST command + +See <Replacing antispam plugin with IMAPSieve> [HowTo.AntispamWithSieve.txt] as +example on how to use this. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Plugins.Pipe.txt b/doc/wiki/Pigeonhole.Sieve.Plugins.Pipe.txt new file mode 100644 index 0000000..b307037 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Plugins.Pipe.txt @@ -0,0 +1,187 @@ +Pigeonhole Sieve Pipe Plugin +============================ + +The sieve_pipe plugin adds the vnd.dovecot.pipe extension to the Sieve language +[http://www.sieve.info]. The extension adds a new action command for piping +messages to a pre-defined set of external programs. To mitigate the security +concerns, the external programs cannot be chosen arbitrarily; the available +programs are restricted through administrator configuration. + +This plugin is available for <Pigeonhole.txt> v0.2. This plugin is superseded +by the <Extprograms> [Pigeonhole.Sieve.Plugins.Extprograms.txt] plugin for +Pigeonhole v0.3 and beyond. + +Getting the sources +------------------- + +Currently, the sources of the sieve_pipe plugin are not released, but you can +get them from the the Mercurial repository: + +---%<------------------------------------------------------------------------- +hg clone http://hg.rename-it.nl/pigeonhole-0.2-sieve-pipe/ +---%<------------------------------------------------------------------------- + +Compiling +--------- + +If you downloaded the sources of this plugin using Mercurial, you will need to +execute './autogen.sh' first to build the automake structure in your source +tree. This process requires autotools and libtool to be installed. + +If you installed Dovecot from sources, the plugin's configure script should be +able to find the installed 'dovecot-config' automatically, along with the +Pigeonhole development headers: + +---%<------------------------------------------------------------------------- +./configure +make +sudo make install +---%<------------------------------------------------------------------------- + +If this doesn't work, you can use '--with-dovecot=<path>' configure option, +where the path points to a directory containing 'dovecot-config' file. This can +point to an installed file: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=/usr/local/lib/dovecot +make +sudo make install +---%<------------------------------------------------------------------------- + +The above example should also find the necessary Pigeonhole development headers +implicitly. You can also compile by pointing to compiled Dovecot and Pigeonhole +source trees: + +---%<------------------------------------------------------------------------- +./configure --with-dovecot=../dovecot-2.0.0/ +--with-pigeonhole=../dovecot-2.0-pigeonhole-0.2.0 +make +sudo make install +---%<------------------------------------------------------------------------- + +Configuration +------------- + +This package builds and installs the sieve_pipe plugin for Pigeonhole Sieve. +The plugin is activated by adding it to the sieve_plugins setting: + +---%<------------------------------------------------------------------------- +sieve_plugins = sieve_pipe +---%<------------------------------------------------------------------------- + +The plugin can directly pipe a message to an external program (typically a +shell script) by forking a new process. Alternatively, it can connect to a Unix +socket behind which a Dovecot script service is listening to start the external +program, e.g. to execute as a different user or for added security. + +The program name specified for the Sieve "pipe" command is used to find the +program or socket in a configured directory. Separate directories are specified +for the sockets and the directly executed binaries. The socket directory is +searched first. Since the Sieve "pipe" command refuses "/" in program names, it +is not possible to build a hierarchical structure. + +Directly forked programs are executed with a limited set of environment +variables: HOME, USER, SENDER, RECIPIENT and ORIG_RECIPIENT. Programs executed +through the script-pipe socket service currently have no environment set at +all. + +The following configuration settings are used by the sieve_pipe plugin: + +sieve_pipe_socket_dir = : + Points to a directory relative to the Dovecot base_dir where the sieve_pipe + plugin looks for the sockets. + +sieve_pipe_bin_dir = : + Points to a directory where the sieve_pipe plugin looks for programs (shell + scripts) to execute directly and pipe messages to. + +Example of socket service + +---%<------------------------------------------------------------------------- +plugin { + sieve = ~/.dovecot.sieve + + sieve_plugins = sieve_pipe + + sieve_pipe_socket_dir = sieve-pipe +} + +service sieve-custom-action { + executable = script-pipe /usr/lib/dovecot/sieve-pipe/sieve-custom-action.sh + + # use some unprivileged user for execution + user = dovenull + + # socket name is program-name in Sieve (without sieve-pipe/ prefix) + unix_listener sieve-pipe/sieve-custom-action { + } +} +---%<------------------------------------------------------------------------- + +Example of direct execution + +---%<------------------------------------------------------------------------- +plugin { + sieve = ~/.dovecot.sieve + + sieve_plugins = sieve_pipe + + # This directory contains the scripts that are available. + sieve_pipe_bin_dir = /usr/lib/dovecot/sieve-pipe +} +---%<------------------------------------------------------------------------- + +Usage +----- + +Sieve scripts can use the new 'vnd.dovecot.pipe' extension as follows: + +---%<------------------------------------------------------------------------- +require ["vnd.dovecot.pipe"]; + +pipe "external-program"; +---%<------------------------------------------------------------------------- + +Read the full specification +[http://hg.rename-it.nl/pigeonhole-0.2-sieve-pipe/raw-file/tip/doc/rfc/spec-bosch-sieve-pipe.txt] +for more information. + +Examples +-------- + +Example of a jabber notification notify.sieve: + +---%<------------------------------------------------------------------------- +require [ "vnd.dovecot.pipe", "copy", "variables" ]; +if header :matches "subject" "*" { set "subject" "${1}"; } +if header :matches "from" "*" { set "from" "${1}"; } +pipe :args [ "USER@DOMAIN.TLD", "${from}", "${subject}" ] :copy :try +"jabber_notify.sh" ; +---%<------------------------------------------------------------------------- + +The jabber_notify.sh: + +---%<------------------------------------------------------------------------- +USER="$1" +FROM="$2" +SUBJECT="$3" +# clix accepts this as a config search directory +# you can use clix - lua based +# http://code.matthewwild.co.uk/clix/summary +# or use any other jabber cmd line client that can send things +export XDG_CONFIG_HOME=/srv/mail +/srv/mail/clix.bin send -q --account=default --to="$USER" "New mail from +${FROM} about ${SUBJECT}" +# we don't care about the exit status in this case +exit 0 +---%<------------------------------------------------------------------------- + +The $XDG_CONFIG_HOME/.clixrc: + +---%<------------------------------------------------------------------------- +[default] +jid=USER@DOMAIN.TLD +password=PASSWORD +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Plugins.txt b/doc/wiki/Pigeonhole.Sieve.Plugins.txt new file mode 100644 index 0000000..3028b75 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Plugins.txt @@ -0,0 +1,45 @@ +Pigeonhole Sieve Plugins +======================== + +The Pigeonhole Sieve interpreter can be dynamically extended with new features +by means of plugins. Plugins can be configured using the 'sieve_plugins' +setting in the 'plugin' section of the Dovecot configuration, as explained on +the <Pigeonhole Sieve Configuration page> [Pigeonhole.Sieve.Configuration.txt]. + +The following plugins are currently available for the Pigeonhole Sieve +interpeter: + + * <Extdata> [Pigeonhole.Sieve.Plugins.Extdata.txt] /(custom language + extension, experimental)/ + * This plugin adds support for the Dovecot-specific 'vnd.dovecot.extdata' + extension, which allows access to external string data. Any type of Dovecot + dict lookup can be specified as data source. + * <Extprograms> [Pigeonhole.Sieve.Plugins.Extprograms.txt] [*Pigeonhole v0.3* + and beyond]/(custom language extensions)/ + * This plugin adds support for the Dovecot-specific 'vnd.dovecot.pipe', + 'vnd.dovecot.filter' and 'vnd.dovecot.execute' extensions. These extensions + add new action commands for invoking a predefined set of external programs + (typically shell scripts. Messages can be piped to or filtered through those + programs and string data can be input to and retrieved from those programs. + * <IMAPSieve> [Pigeonhole.Sieve.Plugins.IMAPSieve.txt] [*Pigeonhole v0.4.14* + and beyond]/(standard language extension, RFC 6785 + [https://tools.ietf.org/html/rfc6785])/ + * This set of plugins adds support for the 'imapsieve' extension in sieve and + the 'IMAPSIEVE' capability in IMAP. With these plugins it is possible to use + Sieve filtering in IMAP, operating when messages are created or their + attributes are changed. + * <IMAP FILTER=SIEVE> [Pigeonhole.Sieve.Plugins.IMAPFilterSieve.txt] + [*Pigeonhole v0.4.24* and beyond, *Pigeonhole v0.5.2* and beyond] /(custom + language extension, experimental)/ + * This plugin provides a vendor-defined IMAP extension called FILTER=SIEVE. It + adds a new FILTER command that allows applying a mail filter (a Sieve + script) on a set of messages that match the specified IMAP searching + criteria. + * <Pipe> [Pigeonhole.Sieve.Plugins.Pipe.txt] [*Pigeonhole v0.2*] /(custom + language extension, deprecated)/ + * This plugin adds support for the Dovecot-specific 'vnd.dovecot.pipe' + extension, which allows piping messages to a pre-defined set of external + programs. For Pigeonhole v0.3, the same functionality (and more) is provided + by the extprograms plugin. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Troubleshooting.txt b/doc/wiki/Pigeonhole.Sieve.Troubleshooting.txt new file mode 100644 index 0000000..758690d --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Troubleshooting.txt @@ -0,0 +1,143 @@ +Pigeonhole Sieve Troubleshooting +================================ + +This page explains how to approach problems with the <Sieve interpreter> +[Pigeonhole.Sieve.txt]. + +Contents + + + 1. Pigeonhole Sieve Troubleshooting + + 1. Troubleshooting Approach + + 2. Common Problems + + 1. Sieve Scripts are not Executed + + 2. Mailbox Names with non-ASCII Characters Cause Problems + +Troubleshooting Approach +------------------------ + +/This section should contain a step-wise approach to troubleshooting./ + +Common Problems +--------------- + +Common configuration problems and their solutions are described here. + +Sieve Scripts are not Executed +------------------------------ + +When Sieve scripts are not being executed, there are several possibilities: + +Your MTA is not using Dovecot LDA or LMTP : + Sieve scripts are executed by the Dovecot <LDA (Local Delivery Agent)> + [LDA.txt] and/or the Dovecot <LMTP.txt> service. That is why you first need + to check whether LDA or LMTP are actually being used. At least one of these + is supposed to be called/accessed from your <MTA.txt>, e.g. Exim or Postfix, + for local message delivery. Most MTAs have their own local delivery agent, + and without explicit configuration this is what is used. In that case, your + Sieve scripts are simply ignored. When you set 'mail_debug=yes' in your + configuration, your logs will show details of LDA and/or LMTP execution. The + following is an example of the first few log lines of an LDA delivery: + +: + ---%<----------------------------------------------------------------------- + dovecot: lda: Debug: Loading modules from directory: /usr/lib/dovecot/modules + dovecot: lda: Debug: Module loaded: + /usr/lib/dovecot/modules/lib90_sieve_plugin.so + dovecot: lda(hendrik): Debug: Effective uid=1000, gid=1000, + home=/home/hendrik + dovecot: lda(hendrik): Debug: Namespace inbox: type=private, prefix=, sep=, + inbox=yes, hidden=no, list=yes, subscriptions=yes location= + ---%<----------------------------------------------------------------------- + +: + The first lines show that LDA has found and loaded the Sieve plugin module. + Then it shows for what user it is delivering and where his INBOX is located. + LMTP produces similar log lines. If you don't see lines such as the above, + your MTA is probably not using Dovecot for local delivery. You can verify + whether Dovecot is working correctly by executing 'dovecot-lda' manually. + +The Sieve plugin is not enabled : + The Dovecot <LDA.txt> and <LMTP.txt> services do not provide Sieve support by + default. Sieve support is provided as a separate plugin that needs to be + enabled by adding it to the 'mail_plugins' setting in the 'protocol lda {...} + ' section for the LDA and the 'protocol lmtp {...} ' section for LMTP. If + this is omitted, Sieve scripts are ignored. Check the <configuration page> + [Pigeonhole.Sieve.Configuration.txt] for more information. When you set + 'mail_debug=yes' in your configuration, your logs will show details of Sieve + execution. The following is an example of the log lines produced for a simple + Sieve execution: + +: + ---%<----------------------------------------------------------------------- + dovecot: lda(hendrik): Debug: sieve: using sieve path for user's script: + /home/hendrik/.dovecot.sieve + dovecot: lda(hendrik): Debug: sieve: opening script + /home/hendrik/.dovecot.sieve + dovecot: lda(hendrik): Debug: sieve: binary save: not saving binary + /home/hendrik/.dovecot.svbin, because it is already stored + dovecot: lda(hendrik): Debug: sieve: executing script from + /home/hendrik/.dovecot.svbin + dovecot: lda(hendrik): sieve: msgid=2234234@host.example.com: stored mail + into mailbox 'INBOX' + ---%<----------------------------------------------------------------------- + + Without actually running LDA, you can also check if 'doveconf -f service=lda + mail_plugins' includes "sieve". + +The Sieve plugin is misconfigured or the involved Sieve scripts contain errors + : + If there is a configuration error or when a Sieve script cannot be compiled + and executed, an error is always logged. + +Mailbox Names with non-ASCII Characters Cause Problems +------------------------------------------------------ + +This problem most often manifests with the following error message: + +---%<------------------------------------------------------------------------- +error: msgid=<234234.234234@example.com>: failed to store into mailbox +'INBOX/Co&APY-rdineren' (INBOX/Co&-APY-rdineren): Mailbox doesn't exist: +INBOX.Co&-APY-rdineren. +---%<------------------------------------------------------------------------- + +The Sieve script causing this error contained the following command: + +---%<------------------------------------------------------------------------- +fileinto "INBOX/Co&APY-rdineren"; +---%<------------------------------------------------------------------------- + +The specified mailbox contains the non-ASCII character 'ö'. Unfortunately, the +author of this script used the wrong encoding. This is modified UTF-7 such as +used by IMAP. However, Sieve expects UTF-8 for mailbox names. Depending on +version and configuration, Dovecot uses modified UTF-7 internally. The Sieve +interpreter expects UTF-8 and converts that to UTF-7 when necessary. When the +mailbox is encoded in UTF-7 by the user, the '&' will just be escaped into '&-' +during the UTF-7 conversion, yielding an erroneous mailbox name for Dovecot. +That is what causes the error message presented above. Instead, the 'fileinto' +command should have looked as follows: + +---%<------------------------------------------------------------------------- +fileinto "INBOX/Coördineren"; +---%<------------------------------------------------------------------------- + +The old CMUSieve plugin did use UTF-7 for folder names. Therefore, this problem +could have emerged after migrating from CMUSieve to Pigeonhole. In that case +you should carefully read the migration instructions +[http://wiki.dovecot.org/LDA/Sieve/Dovecot#Migration_from_CMUSieve] again. + +Often the 'author' of such scripts is an older or misconfigured Sieve GUI +editor. For example, the SieveRules +[https://github.com/JohnDoh/Roundcube-Plugin-SieveRules-Managesieve#readme] +plugin for the RoundCube webmail IMAP client [http://roundcube.net/] has a +configuration option to enable the correct behavior: + +---%<------------------------------------------------------------------------- +$sieverules_config['folder_encoding'] = 'UTF-8'; +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.Usage.txt b/doc/wiki/Pigeonhole.Sieve.Usage.txt new file mode 100644 index 0000000..98f49b4 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.Usage.txt @@ -0,0 +1,130 @@ +Pigeonhole Sieve Usage +====================== + +Mailbox Names +------------- + +Regarding separators, you need to specify mailbox names in Sieve scripts the +same way as IMAP clients see them. For example if you want to deliver mail to +the "Customers" mailbox which exists under "Work" mailbox: + + * Namespace with 'prefix=""', 'separator=.' (Maildir default): + +---%<------------------------------------------------------------------------- +require "fileinto"; +fileinto "Work.Customers"; +---%<------------------------------------------------------------------------- + + * Namespace with 'prefix=INBOX.', 'separator=.' (Courier migration): + +---%<------------------------------------------------------------------------- +require "fileinto"; +fileinto "INBOX.Work.Customers"; +---%<------------------------------------------------------------------------- + + * Namespace with 'prefix=""', 'separator=/' (mbox, dbox default): + +---%<------------------------------------------------------------------------- +require "fileinto"; +fileinto "Work/Customers"; +---%<------------------------------------------------------------------------- + +However, Sieve uses UTF8 encoding for mailbox names, while IMAP uses modified +UTF7. This means that non-ASCII characters contained in mailbox names are +represented differently between IMAP and Sieve scripts. + +Vacation auto-reply +------------------- + +Vacation uses envelope sender and envelope recipient. They're taken from: + + * Envelope sender: -f parameter to dovecot-lda if given, otherwise + Return-Path: header in the message. + * Envelope recipient: -a parameter to dovecot-lda if given, otherwise -d + parameter to dovecot-lda. If neither is given (delivering to system users), + the $USER environment is used. + +The vacation replies are sent to the envelope sender. + +List of autoreplied senders is stored in '.dovecot.lda-dupes' file in user's +home directory. When you're testing the vacation feature, it's easy to forget +that the reply is sent only once in the number of configured days. If you've +problems getting the vacation reply, try deleting this file. If that didn't +help, make sure the problem isn't related to sending mails in general by trying +the "reject" Sieve command. + +The automatic replies aren't sent if any of the following is true: + + * The envelope sender is not available (equal to <>) + * The envelope sender and envelope recipient are the same + * The sender recently (within :days days; default 7) got a reply from the same + vacation command + * The message contains at least one of the mailing list headers "list-id", + "list-owner", "list-subscribe", "list-post", "list-unsubscribe", + "list-help", or "list-archive" + * Auto-Submitted: header exists with any value except "no" + * Precedence: header exists with value "junk", "bulk" or "list" + * The envelope sender is considered a system address, which either: + * begins with "MAILER-DAEMON" (case-insensitive), + * begins with "LISTSERV" (case-insensitive), + * begins with "majordomo" (case-insensitive), + * contains the string "-request" anywhere within it (case-sensitive), or + * begins with "owner-" (case-sensitive) + * The envelope recipient and alternative addresses specified with the vacation + command's :addresses tag are not found in the message's To:, Cc:, Bcc:, + Resent-To:, Resent-Cc: or Resent-Bcc: fields. + +Manually Compiling Sieve Scripts +-------------------------------- + +When the Sieve plugin executes a script for the first time (or after it has +been changed), it is compiled and stored in binary form (byte code) to avoid +compiling the script again for each subsequent mail delivery. The Pigeonhole +Sieve implementation uses the '.svbin' extension to store compiled Sieve +scripts (e.g.'.dovecot.svbin'). To store the binary, the plugin needs write +access in the directory in which the script is located. + +A problem occurs when a global script is encountered by the plugin. For +security reasons, global script directories are not supposed to be writable by +the user. Therefore, the plugin cannot store the binary when the script is +first compiled. Note that this doesn't mean that the old compiled version of +the script is used when the binary cannot be written: it compiles and uses the +current script version. The only real problem is that the plugin will not be +able to update the binary on disk, meaning that the global script needs to be +recompiled each time it needs to be executed, i.e. for every incoming message, +which is inefficient. + +To mitigate this problem, the administrator must manually pre-compile global +scripts using the 'sievec' command line tool. For example: + +---%<------------------------------------------------------------------------- +sievec /var/lib/dovecot/sieve/global/ +---%<------------------------------------------------------------------------- + +This is necessary for scripts listed in the 'sieve_global_path', 'sieve_before' +and 'sieve_after' settings. For global scripts that are only included in other +scripts using the Sieve include extension, this step is not necessary, since +included scripts are incorporated into the binary produced for the main script. + +Compile and Runtime Logging +--------------------------- + +Log messages produced during script compilation or during script execution are +written to two locations by the LDA Sieve plugin: + + * A log file is written in the same directory as the user's main private + script (as specified by the 'sieve' setting). This log file bears the name + of that script file appended with ".log", e.g.'.dovecot.sieve.log'. If there + are errors or warnings in the script, the messages are appended to that log + file until it eventually grows too large (>10 kB currently). When that + happens, the old log file is moved to a ".log.0" file and an empty log file + is started. Informational messages are not written to this log file and the + log file is not created until messages are actually logged, i.e. when an + error or warning is produced. + * Messages that could be of interest to the system administrator are also + written to the Dovecot logging facility (usually syslog). This includes + informational messages that indicate what actions are executed on incoming + messages. Compile errors encountered in the user's private script are not + logged here. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.Sieve.txt b/doc/wiki/Pigeonhole.Sieve.txt new file mode 100644 index 0000000..43e9578 --- /dev/null +++ b/doc/wiki/Pigeonhole.Sieve.txt @@ -0,0 +1,290 @@ +Pigeonhole Sieve Interpreter +============================ + +The <Pigeonhole project> [Pigeonhole.txt] provides Sieve support as a plugin +for Dovecot's <Local Delivery Agent (LDA)> [LDA.txt] and also for its +<LMTP.txt> service. The plugin implements a Sieve [http://www.sieve.info] +interpreter, which filters incoming messages using a script specified in the +Sieve language (RFC 5228 [http://tools.ietf.org/html/rfc5228/]). The Sieve +script is provided by the user and, using that Sieve script, the user can +customize how incoming messages are handled. Messages can be delivered to +specific folders, forwarded, rejected, discarded, etc. + +Configuration and Use +--------------------- + + * <Download and Installation> [Pigeonhole.Installation.txt] + * <Sieve Interpreter Configuration> [Pigeonhole.Sieve.Configuration.txt] + * <Sieve Usage Information> [Pigeonhole.Sieve.Usage.txt] + * <Sieve Script Examples> [Pigeonhole.Sieve.Examples.txt] + * <Sieve Interpreter Plugins> [Pigeonhole.Sieve.Plugins.txt] + * <Sieve Troubleshooting> [Pigeonhole.Sieve.Troubleshooting.txt] + +Supported Features +------------------ + +Sieve language has various <extensions> [Pigeonhole.Sieve.Extensions.txt]. You +can find more information about the extensions from the Sieve Mail Filtering +Language Charter [http://www.ietf.org/html.charters/sieve-charter.html] or the +Sieve.info wiki page [http://www.sieve.info/]. + +Note that Sieve doesn't support running external programs. + +The Pigeonhole Sieve interpreter recognizes the following Sieve extensions: + + * ++-----------------------------------------------------------------+------------+----------+----------------+ +| *Extension* | *Support | *Default | *Purpose* | +| | Status* | Enabled* | | ++-----------------------------------------------------------------+------------+----------+----------------+ +| body [http://tools.ietf.org/html/rfc5173/] | supported | yes | Allows | +| | | | evaluating the | +| | | | body of a | +| | | | message | ++-----------------------------------------------------------------+------------+----------+----------------+ +| copy [http://tools.ietf.org/html/rfc3894/] | supported | yes | Allows storing | +| | | | and forwarding | +| | | | messages | +| | | | without | +| | | | canceling the | +| | | | implicit keep | ++-----------------------------------------------------------------+------------+----------+----------------+ +| date [http://tools.ietf.org/html/rfc5260#section-4] | supported | yes | Adds the | +| | (v0.1.12+) | | ability to test| +| | | | date and time | +| | | | values in | +| | | | various ways | ++-----------------------------------------------------------------+------------+----------+----------------+ +| duplicate [http://tools.ietf.org/html/rfc7352] | supported | yes | Allows | +| | (v0.4.3+) | | detecting | +| | | | duplicate | +| | | | message | +| | | | deliveries | ++-----------------------------------------------------------------+------------+----------+----------------+ +| editheader [http://tools.ietf.org/html/rfc5293/] | supported | no | Adds the | +| | (v0.3.0+) | | ability to add | +| | | | and remove | +| | | | message header | +| | | | fields | ++-----------------------------------------------------------------+------------+----------+----------------+ +| encoded-character | supported | yes | Allows encoding| +| [http://tools.ietf.org/html/rfc5228#section-2.4.2.4] | | | special | +| | | | characters | +| | | | numerically | ++-----------------------------------------------------------------+------------+----------+----------------+ +| enotify [http://tools.ietf.org/html/rfc5435/] | supported | yes | Provides the | +| | (v0.1.3+) | | ability to send| +| | | | notifications | +| | | | by various | +| | | | means | +| | | | (currently only| +| | | | mailto) | ++-----------------------------------------------------------------+------------+----------+----------------+ +| envelope [http://tools.ietf.org/html/rfc5228#section-5.4] | supported | yes | Allows | +| | | | evaluating | +| | | | envelope parts,| +| | | | i.e. sender and| +| | | | recipient | ++-----------------------------------------------------------------+------------+----------+----------------+ +| environment [http://tools.ietf.org/html/rfc5183/] | supported | yes | Allows testing | +| | (v0.4.0+) | | against various| +| | | | labeled values | +| | | | from the | +| | | | execution | +| | | | environment | ++-----------------------------------------------------------------+------------+----------+----------------+ +| fileinto [http://tools.ietf.org/html/rfc5228#section-4.1] | supported | yes | Allows storing | +| | | | messages in | +| | | | folders other | +| | | | than INBOX | ++-----------------------------------------------------------------+------------+----------+----------------+ +| foreverypart [http://tools.ietf.org/html/rfc5703#section-3] | supported | yes | Allows | +| | (v0.4.14+) | | iterating | +| | | | through the | +| | | | message's MIME | +| | | | parts | ++-----------------------------------------------------------------+------------+----------+----------------+ +| ihave [http://tools.ietf.org/html/rfc5463/] | supported | yes | Adds the | +| | (v0.2.4+) | | ability to test| +| | | | for support of | +| | | | Sieve | +| | | | extensions and | +| | | | dynamically | +| | | | invoke their use| ++-----------------------------------------------------------------+------------+----------+----------------+ +| imap4flags [http://tools.ietf.org/html/rfc5232/] | supported | yes | Allows adding | +| | | | IMAP flags to | +| | | | stored messages| ++-----------------------------------------------------------------+------------+----------+----------------+ +| imapsieve [http://tools.ietf.org/html/rfc6785/] | supported | no | Provides access| +| | (v0.4.14+) | (plugin) | to special | +| | | | environment | +| | | | items when | +| | | | executing at | +| | | | IMAP events | ++-----------------------------------------------------------------+------------+----------+----------------+ +| include [http://tools.ietf.org/html/rfc6609/] | supported | yes | Allows | +| | (v0.4.0+) | | including other| +| | | | Sieve scripts | ++-----------------------------------------------------------------+------------+----------+----------------+ +| index [http://tools.ietf.org/html/rfc5260#section-6] | supported | yes | Allows matching| +| | (v0.4.7+) | | specific header| +| | | | field instances| +| | | | by index | ++-----------------------------------------------------------------+------------+----------+----------------+ +| mailbox [http://tools.ietf.org/html/rfc5490#section-3] | supported | yes | Provides a | +| | (v0.1.10+) | | mailbox | +| | | | existence check| +| | | | and allows | +| | | | creating | +| | | | mailboxes upon | +| | | | fileinto | ++-----------------------------------------------------------------+------------+----------+----------------+ +| mboxmetadata [http://tools.ietf.org/html/rfc5490] | supported | no | Provides access| +| | (v0.4.7+) | | to mailbox | +| | | | METADATA entries| ++-----------------------------------------------------------------+------------+----------+----------------+ +| mime [http://tools.ietf.org/html/rfc5703#section-4] | supported | yes | Allows testing | +| | (v0.4.14+) | | parts of | +| | | | structured MIME| +| | | | header fields | ++-----------------------------------------------------------------+------------+----------+----------------+ +| extracttext [http://tools.ietf.org/html/rfc5703#section-7] | supported | yes | Allows | +| | (v0.4.14+) | | extracting text| +| | | | from individual| +| | | | message MIME | +| | | | parts | ++-----------------------------------------------------------------+------------+----------+----------------+ +| regex | supported | yes | Provides | +| [http://tools.ietf.org/html/draft-murchison-sieve-regex-08/] | | | regular | +| | | | expression | +| | | | match support | ++-----------------------------------------------------------------+------------+----------+----------------+ +| reject [http://tools.ietf.org/html/rfc5429#section-2.2] | supported | yes | Allows | +| | | | rejecting | +| | | | messages with a| +| | | | rejection | +| | | | bounce message | ++-----------------------------------------------------------------+------------+----------+----------------+ +| relational [http://tools.ietf.org/html/rfc5231/] | supported | yes | Provides | +| | | | relational | +| | | | match support | ++-----------------------------------------------------------------+------------+----------+----------------+ +| servermetadata [http://tools.ietf.org/html/rfc5490] | supported | no | Provides access| +| | (v0.4.7+) | | to server | +| | | | METADATA entries| ++-----------------------------------------------------------------+------------+----------+----------------+ +| spamtest [http://tools.ietf.org/html/rfc5235/] | supported | no | Implements a | +| | (v0.1.16+) | | uniform way to | +| | | | test against | +| | | | headers added | +| | | | by spam filters| ++-----------------------------------------------------------------+------------+----------+----------------+ +| subaddress [http://tools.ietf.org/html/rfc5233/] | supported | yes | Allows testing | +| | | | against | +| | | | delimited | +| | | | elements of the| +| | | | local part of | +| | | | addresses | ++-----------------------------------------------------------------+------------+----------+----------------+ +| vacation [http://tools.ietf.org/html/rfc5230/] | supported | yes | Provides | +| | | | auto-responder | +| | | | functionality, | +| | | | e.g. for when | +| | | | the user is on | +| | | | vacation | ++-----------------------------------------------------------------+------------+----------+----------------+ +| vacation-seconds [http://tools.ietf.org/html/rfc6131/] | supported | no | Extends | +| | (0.2.3+) | | vacation | +| | | | extension with | +| | | | the ability to | +| | | | send vacation | +| | | | responses with | +| | | | intervals of | +| | | | seconds rather | +| | | | than days | ++-----------------------------------------------------------------+------------+----------+----------------+ +| variables [http://tools.ietf.org/html/rfc5229/] | supported | yes | Adds variables | +| | | | support to the | +| | | | language | ++-----------------------------------------------------------------+------------+----------+----------------+ +| virustest [http://tools.ietf.org/html/rfc5235/] | supported | no | Implements a | +| | (v0.1.16+) | | uniform way to | +| | | | test against | +| | | | headers added | +| | | | by virus | +| | | | scanners | ++-----------------------------------------------------------------+------------+----------+----------------+ +| imapflags(obsolete draft | deprecated | no | Old version of | +| [http://tools.ietf.org/html/draft-melnikov-sieve-imapflags-03]) | | | imap4flags (for| +| | | | backwards | +| | | | compatibility | +| | | | with CMU Sieve)| ++-----------------------------------------------------------------+------------+----------+----------------+ +| notify (obsolete draft | deprecated | no | Old version of | +| [http://tools.ietf.org/html/draft-martin-sieve-notify-01]) | | | enotify (for | +| | | | backwards | +| | | | compatibility | +| | | | with CMU Sieve)| ++-----------------------------------------------------------------+------------+----------+----------------+ + +The following Dovecot-specific Sieve extensions are available for the +Pigeonhole Sieve interpreter: + + * ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| *Extension* | *Support Status* | *Default | *Purpose* | +| | | Enabled* | | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| vnd.dovecot.debug | supported (v0.3.0+) | no | Allows logging debug messages | +| [https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-debug.txt] | | | | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| vnd.dovecot.environment | supported (v0.4.14+) | no | Extends the standard "environment" | +| [https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-dovecot-environment.txt] | | | extension with extra items and a | +| | | | variables namespace for direct access | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| vnd.dovecot.execute | <plugin> | no | Implements executing a pre-defined set | +| [https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-extprograms.txt] | [Pigeonhole.Sieve.Plugins.Extprograms.txt] | | of external programs with the option to| +| | (v0.3+) | | process string data through the | +| | | | external program | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| vnd.dovecot.extdata | <plugin> | no | Allows a Sieve script to lookup | +| [http://hg.rename-it.nl/pigeonhole-0.2-sieve-extdata/raw-file/tip/doc/rfc/spec-bosch-sieve-external-data.txt] | [Pigeonhole.Sieve.Plugins.Extdata.txt] | | information from a datasource external | +| | | | to the script | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| vnd.dovecot.filter | <plugin> | no | Implements filtering messages through a| +| [https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-extprograms.txt] | [Pigeonhole.Sieve.Plugins.Extprograms.txt] | | pre-defined set of external programs | +| | (v0.3+) | | | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| vnd.dovecot.pipe | <plugin> | no | Implements piping messages to a | +| [https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-extprograms.txt] | [Pigeonhole.Sieve.Plugins.Pipe.txt] (v0.2),| | pre-defined set of external programs | +| | <plugin> | | | +| | [Pigeonhole.Sieve.Plugins.Extprograms.txt] | | | +| | (v0.3+) | | | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ +| vnd.dovecot.report | supported (v0.4.14+) | no | Implements sending Messaging Abuse | +| [https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-report.txt] | | | Reporting Format (MARF) reports (RFC | +| | | | 5965 | +| | | | [http://tools.ietf.org/html/rfc5965/]) | ++----------------------------------------------------------------------------------------------------------------+--------------------------------------------+----------+----------------------------------------+ + +Please note that not all extensions are enabled by default, as shown in the +table above. Deprecated extensions, extensions that add the ability to change +messages, extensions that require explicit configuration and extensions that +are still under development are not enabled without explicit <configuration> +[Pigeonhole.Sieve.Configuration.txt]. This means that the 'sieve_extensions' or +'sieve_global_extensions' settings need to be adjusted accordingly. Also, for +<plugins> [Pigeonhole.Sieve.Plugins.txt] it is not enough to add the plugin +name to the 'sieve_plugins' setting; the extensions introduced by the plugin +also need to be enabled explicitly. + +ManageSieve server +------------------ + +To give users the ability to upload their own Sieve scripts to your server, +i.e. without the need for shell or FTP access, you can use the Manage ''Sieve +protocol. This is also provided by the <Pigeonhole.txt> project. It is +available as a separate Dovecot service. Its configuration and use is explained +on the <Pigeonhole ManageSieve page> [Pigeonhole.ManageSieve.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Pigeonhole.txt b/doc/wiki/Pigeonhole.txt new file mode 100644 index 0000000..7666b25 --- /dev/null +++ b/doc/wiki/Pigeonhole.txt @@ -0,0 +1,45 @@ +Pigeonhole Sieve support +======================== + +Introduction +------------ + +The <Pigeonhole project> [Pigeonhole.txt] provides mail filtering facilities at +time of final message delivery using the Sieve (RFC 5228 +[https://www.ietf.org/rfc/rfc5228.txt]) language. By writing Sieve scripts, +users can customize how messages are delivered, e.g. whether they are forwarded +or stored in special folders. The Sieve language is meant to be simple, +extensible and system independent. And, unlike most other mail filtering script +languages, it does not allow users to execute arbitrary programs. This is +particularly useful to prevent virtual users from having full access to the +mail store. The intention of the language is to make it impossible for users to +do anything more complex (and dangerous) than write simple mail filters. See +Sieve wiki [https://sieve.info/] for more comprehensive information about the +Sieve language itself. + +Installation +------------ + +Refer to the <installation page> [Pigeonhole.Installation.txt]. + +Configuration and Use +--------------------- + +The Pigeonhole package provides the following items: + + * The <Sieve interpreter plugin> [Pigeonhole.Sieve.txt] for Dovecot's <Local + Delivery Agent (LDA)> [LDA.txt] or its <LMTP.txt> server: This facilitates + the actual Sieve filtering upon delivery. + * The <ManageSieve Service> [Pigeonhole.ManageSieve.txt]: This implements the + ManageSieve protocol through which users can remotely manage Sieve scripts + on the server. + +Contact Info +------------ + + * Author: Stephan Bosch, stephan@rename-it.nl + * IRC: Freenode [https://freenode.net/], #dovecot, S[r]us + * Please use the Dovecot mailing list [https://www.dovecot.org/mailing-lists] + for questions about the Pigeonhole. You don't have to subscribe to it. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Apparmor.txt b/doc/wiki/Plugins.Apparmor.txt new file mode 100644 index 0000000..d68f05d --- /dev/null +++ b/doc/wiki/Plugins.Apparmor.txt @@ -0,0 +1,34 @@ +Apparmor plugin +=============== + +A simple plugin which allows changing "hat" (apparmor context) when user is +loaded. Context is changed back to default on user deinit. Multiple hats are +supported, and passed to apparmor_change_hatv function. Since v2.2.32. + +Configuration +------------- + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins apparmor + +plugin { + apparmor_hat = hat_name + apparmor_hat2 = another_hat +} +---%<------------------------------------------------------------------------- + +You can also specify hats from user or password database. If you provide from +passdb, use userdb_apparmor_hat=hat and subsequent hats as userdb_apparmor_hat2 +and so forth. From userdb, you can omit the userdb_ prefix. + +It's also possible to combine these, so that you can provide some of the hats +from config and some from passdb/userdb configuration. If you want to provide +apparmor_hat2 from config, make sure you provide apparmor_hat from userdb or +passdb always, otherwise apparmor_hat2 won't be seen. + +Debugging +--------- + +Set mail_debug=yes to see context changes. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Autocreate.txt b/doc/wiki/Plugins.Autocreate.txt new file mode 100644 index 0000000..e1b2eee --- /dev/null +++ b/doc/wiki/Plugins.Autocreate.txt @@ -0,0 +1,28 @@ +Autocreate plugin (v2.0 and older) +================================== + +With v2.1+ you don't need this plugin. Use <mailbox { auto } setting> +[MailboxSettings.txt] instead. + +This plugin allows administrator to specify mailboxes that must always exist +for all users. They can optionally also be subscribed. The mailboxes are +created and subscribed always after user logs in. Namespaces are fully +supported, so namespace prefixes need to be used where necessary. + +Example: + +---%<------------------------------------------------------------------------- +protocol imap { + mail_plugins = $mail_plugins autocreate +} +plugin { + autocreate = Trash + autocreate2 = Spam + #autocreate3 = ..etc.. + autosubscribe = Trash + autosubscribe2 = Spam + #autosubscribe3 = ..etc.. +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.CharsetAlias.txt b/doc/wiki/Plugins.CharsetAlias.txt new file mode 100644 index 0000000..b62e151 --- /dev/null +++ b/doc/wiki/Plugins.CharsetAlias.txt @@ -0,0 +1,20 @@ +Charset Alias plugin +==================== + +Requires v2.2.34+. This plugin allows treating the specified source charset as +a different charset when decoding to UTF-8. For instance, when decoding from +shift_jis to UTF-8, using cp932 (or sjis-win) instead of shift_jis may be +preferable to handle Microsoft extended chars properly. + +Configuration +------------- + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins charset_alias +plugin { + charset_aliases = shift_jis=sjis-win euc-jp=eucjp-win +iso-2022-jp=iso-2022-jp-3 +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Compress.txt b/doc/wiki/Plugins.Compress.txt new file mode 100644 index 0000000..d3cbd80 --- /dev/null +++ b/doc/wiki/Plugins.Compress.txt @@ -0,0 +1,16 @@ +IMAP COMPRESS plugin +==================== + +The goal of the extension is to reduce the bandwidth usage of IMAP. + +Configuration: + +---%<------------------------------------------------------------------------- +#mail_plugins = $mail_plugins zlib # Required for v2.1 and older only + +protocol imap { + mail_plugins = $mail_plugins imap_zlib +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Expire.txt b/doc/wiki/Plugins.Expire.txt new file mode 100644 index 0000000..12b9f08 --- /dev/null +++ b/doc/wiki/Plugins.Expire.txt @@ -0,0 +1,354 @@ +Expire plugin +============= + +(See also <autoexpunge> [MailboxSettings.txt] setting, which partially +obsoletes this plugin.) + +Typically this plugin is used to optimize expunging old mails from users' +mailboxes. For example: + +---%<------------------------------------------------------------------------- +doveadm expunge -A mailbox Trash savedbefore 30d +---%<------------------------------------------------------------------------- + +Note that: + + * *This command runs fine even without the expire plugin.* Then it just goes + through all users rather than those users who have something to actually + expunge. + * The plugin actually works for all doveadm mail commands, not just expunge. + So for testing you could use e.g.'doveadm search -A ...' to see which + messages it matches. + * The optimization is done only when -A parameter is used to go through all + users. If you use -u parameter to separately go through users, the plugin + does nothing. + * The optimization is done only when "savedbefore" is used in the search + query. It's possible to have further restrictions also, but the user + filtering is based only on the "savedbefore" timestamp. + * The optimization is done only when the given mailbox or mailboxes match the + list of expire mailboxes specified in 'plugin { expire* } ' settings. + +So the expire plugin's job is to reduce disk I/O when figuring out which users +have work to do. Unless you have thousands of users, you probably shouldn't +bother with this plugin. + +Details +------- + +When expire plugin is enabled, it keeps track of the "oldest mail's saved +timestamp" for the specified mailboxes. When doveadm command uses "savedbefore" +search key, the timestamps in the database can be used to figure out which +users have matching mails. + +Expire plugin tracks/optimizes only "savedbefore" search query, i.e. the date +when message was *saved or copied to the mailbox* (*NOT the time message was +originally received*) while expire plugin was loaded. If mailbox contained +existing messages before the plugin was loaded for the first time, they'll get +expunged eventually when the first message saved/copied after expire plugin was +enabled gets expunged. + +The save/copy date may not be exact if it's not cached in +'dovecot.index.cache': + + * mbox: The current lookup time is used and added to cache. + * maildir: File's ctime is used. + * dbox: Save/copy time is taken from the dbox file if it exists (it normally + should), fallbacking to file's ctime if not. + +You need to configure a list of mailboxes that are tracked. Mailbox patterns +can contain IMAP LIST command-compatible wildcards: + + * "*" works in a standard way: It matches any number of characters. + * "%" works by matching any number of characters, but it stops at the + hierarchy separator. Currently the separator is hardcoded to "/", so it + doesn't work correctly if you've configured separator to something else + (e.g. "." is the default for Maildir). + +Example configuration +--------------------- + +Make sure you enable the plugin globally, not just for specific protocols. + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins expire + +plugin { + expire = Trash + expire2 = Trash/* + expire3 = Spam + + # Enable caching of dict value in dovecot.index file. This significantly +reduces + # the number of dict lookups. It makes initial testing more confusing though, +so + # it's better to enable it only after you've verified that the expire plugin +is + # working as wanted. (v2.2.16+) + expire_cache = yes +} +---%<------------------------------------------------------------------------- + + * See <Dict.txt> for more information about how to use dict service, + especially about permission issues. + * You need to get 'doveadm -A' working using your userdb. See iterate settings + for <SQL> [AuthDatabase.SQL.txt] and <LDAP> [AuthDatabase.LDAP.txt]. + +MySQL Backend +------------- + +dovecot.conf: + +---%<------------------------------------------------------------------------- +plugin { + expire_dict = proxy::expire +} +dict { + expire = mysql:/etc/dovecot/dovecot-dict-expire.conf.ext +} +---%<------------------------------------------------------------------------- + +Create the table like this: + +---%<------------------------------------------------------------------------- +CREATE TABLE expires ( + username varchar(75) not null, + mailbox varchar(255) not null, + expire_stamp integer not null, + primary key (username, mailbox) +); +---%<------------------------------------------------------------------------- + +dovecot-dict-expire.conf.ext: + +---%<------------------------------------------------------------------------- +connect = host=localhost dbname=mails user=sqluser password=sqlpass + +map { + pattern = shared/expire/$user/$mailbox + table = expires + value_field = expire_stamp + + fields { + username = $user + mailbox = $mailbox + } +} +---%<------------------------------------------------------------------------- + +PostgreSQL Backend +------------------ + +Like MySQL configuration above, but you'll also need to create a trigger: + +---%<------------------------------------------------------------------------- +CREATE OR REPLACE FUNCTION merge_expires() RETURNS TRIGGER AS $$ +BEGIN + UPDATE expires SET expire_stamp = NEW.expire_stamp + WHERE username = NEW.username AND mailbox = NEW.mailbox; + IF FOUND THEN + RETURN NULL; + ELSE + RETURN NEW; + END IF; +END; +$$ LANGUAGE plpgsql; + +CREATE TRIGGER mergeexpires BEFORE INSERT ON expires + FOR EACH ROW EXECUTE PROCEDURE merge_expires(); +---%<------------------------------------------------------------------------- + +SQLite Backend +-------------- + +Like MySQL configuration above, but you'll also need to create a trigger: + +---%<------------------------------------------------------------------------- +CREATE TRIGGER mergeexpires BEFORE INSERT ON expires FOR EACH ROW +BEGIN + UPDATE expires SET expire_stamp=NEW.expire_stamp + WHERE username = NEW.username AND mailbox = NEW.mailbox; + SELECT raise(ignore) + WHERE (SELECT 1 FROM expires WHERE username = NEW.username AND +mailbox = NEW.mailbox) IS NOT NULL; +END; +---%<------------------------------------------------------------------------- + +Vpopmail (+MySQL) +----------------- + +Configure expire plugin like MySQL configuration. From here, there are two +cases: + +If you are using the mysql database for authentication directly, you only need +to make iterate work for <MySQL> [AuthDatabase.SQL.txt]. + +When using Vpopmail backend, doveadm -A won't work. To be able to use the +expire database, you can use a bash script like this: + +---%<------------------------------------------------------------------------- +#!/bin/bash + +# SQL connection data +HOST="localhost" +USER="dovecot" +PWD="yourpassword" +DBNAME="vpopmail" +TBLNAME="expires" +# expunge messages older than this (days) +TTL=30 + +expiredsql=`cat <<EOF + USE $DBNAME; + SELECT CONCAT(username, "~", mailbox) + FROM $TBLNAME + WHERE expire_stamp>0 AND expire_stamp<UNIX_TIMESTAMP()-86400*$TTL; +EOF` + +for row in `echo "$expiredsql" | mysql -h $HOST -u $USER -p$PWD -N`; do + username=`echo $row | cut -d"~" -f1` + mailbox=`echo $row | cut -d"~" -f2-` + + # Check if mailbox name is empty. Just in case. + if [ -n $mailbox ]; then + echo "Expunging: "$row + doveadm expunge -u $username mailbox "$mailbox" savedbefore $TTL"d" + fi +done +---%<------------------------------------------------------------------------- + +Example #1 timeline +------------------- + +FIXME: expire-tool no longer exists, update these examples. Let's say Trash is +configured to expire in 7 days and today is 2009-07-10. Initially the database +and the Trash mailbox is empty. + +User moves the first message to Trash. The expires table is updated: + +---%<------------------------------------------------------------------------- +mysql> select mailbox, from_unixtime(expire_stamp), username from expires; ++---------+-----------------------------+----------+ +| mailbox | from_unixtime(expire_stamp) | username | ++---------+-----------------------------+----------+ +| Trash | 2009-07-17 15:57:36 | tss | ++---------+-----------------------------+----------+ +---%<------------------------------------------------------------------------- + +The expire_stamp contains the date when expire-tool will look into that mailbox +and try to find messages to expunge. Until then it skips the mailbox. + +A day later user moves another message to Trash. The expire_stamp isn't +updated, because the second message's save date is newer than the first one's. +Checking Trash's contents via IMAP you can see something like: + +---%<------------------------------------------------------------------------- +1 fetch 1:* (internaldate x-savedate) +* 1 FETCH (INTERNALDATE "16-Dec-2008 09:52:38 -0500" X-SAVEDATE "10-Jul-2009 +15:57:36 -0400") +* 2 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 +16:03:11 -0400") +1 OK Fetch completed. +---%<------------------------------------------------------------------------- + +Note how the message's INTERNALDATE (received date) can be very old compared to +the save date. Now, running expire-tool --test: + +---%<------------------------------------------------------------------------- +Info: tss/Trash: stop, expire time in future: Fri Jul 17 15:57:36 2009 +---%<------------------------------------------------------------------------- + +So it does nothing, because the expire time is in future. Fast forward 6 more +days into future. Running expire-tool --test: + +---%<------------------------------------------------------------------------- +Info: tss/Trash: seq=1 uid=1: Expunge +Info: tss/Trash: timestamp 1247860656 (Fri Jul 17 15:57:36 2009) -> 1247947391 +(Sat Jul 18 16:03:11 2009) +---%<------------------------------------------------------------------------- + +The first message would be expunged and the second message's timestamp would +become the new expire_stamp in database. After running expire-tool without +--test, the database is updated: + +---%<------------------------------------------------------------------------- +mysql> select mailbox, from_unixtime(expire_stamp), username from expires; ++---------+-----------------------------+----------+ +| mailbox | from_unixtime(expire_stamp) | username | ++---------+-----------------------------+----------+ +| Trash | 2009-07-18 16:03:11 | tss | ++---------+-----------------------------+----------+ +---%<------------------------------------------------------------------------- + +Also you can see the first message has been expunged from Trash: + +---%<------------------------------------------------------------------------- +2 fetch 1:* (internaldate x-savedate) +* 1 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 +16:03:11 -0400") +2 OK Fetch completed. +---%<------------------------------------------------------------------------- + +Example #2 timeline +------------------- + +Again you have Trash configured for 7 days, but this time you have an existing +message there before expire plugin has been enabled. Initially the expire +database is empty. Today is 2009-07-20. + +---%<------------------------------------------------------------------------- +1 fetch 1:* (internaldate x-savedate) +* 1 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 +16:03:11 -0400") +1 OK Fetch completed. +---%<------------------------------------------------------------------------- + +If you run expire-tool, you'll notice that it does nothing for the mailbox. +There's nothing in expire database, so expire-tool doesn't even mention it when +running with --test. + +After user moves the first message to Trash, the database gets updated: + +---%<------------------------------------------------------------------------- +mysql> select mailbox, from_unixtime(expire_stamp), username from expires; ++---------+-----------------------------+----------+ +| mailbox | from_unixtime(expire_stamp) | username | ++---------+-----------------------------+----------+ +| Trash | 2009-07-27 16:32:11 | tss | ++---------+-----------------------------+----------+ +---%<------------------------------------------------------------------------- + +The messages in Trash are: + +---%<------------------------------------------------------------------------- +2 fetch 1:* (internaldate x-savedate) +* 1 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 +16:03:11 -0400") +* 2 FETCH (INTERNALDATE "16-Dec-2002 11:02:39 -0500" X-SAVEDATE "20-Jul-2009 +16:32:11 -0400") +2 OK Fetch completed. +---%<------------------------------------------------------------------------- + +So the first message should be expiring already, right? No. It doesn't because +the timestamp in database is still in future. expire-tool --test says: + +---%<------------------------------------------------------------------------- +Info: tss/Trash: stop, expire time in future: Mon Jul 27 16:32:11 2009 +---%<------------------------------------------------------------------------- + +OK, let's see what happens when we finally reach July 27th: + +---%<------------------------------------------------------------------------- +Info: tss/Trash: seq=1 uid=3: Expunge +Info: tss/Trash: seq=2 uid=4: Expunge +Info: tss/Trash: no messages left +---%<------------------------------------------------------------------------- + +They both got expunged! The expire database's timestamp simply tells +expire-tool when to start looking into messages in that mailbox. After that +expire-tool looks at the actual save dates and figures out which messages +exactly need to be expunged. + +After running expire-tool without --test you'll see that the Trash mailbox is +empty and the database row is deleted. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.FTS.Lucene.txt b/doc/wiki/Plugins.FTS.Lucene.txt new file mode 100644 index 0000000..0ddc392 --- /dev/null +++ b/doc/wiki/Plugins.FTS.Lucene.txt @@ -0,0 +1,74 @@ +Lucene Full Text Search Indexing +================================ + +*NOTE*: Although the fts-lucene plugin works, it's using CLucene library, which +is very old and has some bugs. It's a much better idea to use <fts-solr> +[Plugins.FTS.Solr.txt] instead, which has much more features and is more +stable. + +Requires Dovecot v2.1+ to work properly. The CLucene version must be v2.3 (not +v0.9).Dovecot builds only a single Lucene index for all mailboxes. The Lucene +indexes are stored in 'lucene-indexes/' directory under the mail root index +directory (e.g.'~/Maildir/lucene-indexes/'). + +Compilation +----------- + +If you compile Dovecot yourself, you must add the following switches to your +configure command for the plugin to be built: + +---%<------------------------------------------------------------------------- +--with-lucene --with-stemmer +---%<------------------------------------------------------------------------- + +The second switch is only required if you have compiled libstemmer yourself or +if it's included in the CLucene you are using. + +Configuration +------------- + +Into 10-mail.conf (note add existing plugins to string) + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins fts fts_lucene +---%<------------------------------------------------------------------------- + +Into 90-plugins.conf + +---%<------------------------------------------------------------------------- +plugin { + fts = lucene + # Lucene-specific settings, good ones are: + fts_lucene = whitespace_chars=@. +} +---%<------------------------------------------------------------------------- + +The fts-lucene settings include: + + * whitespace_chars=<chars>: List of characters that are translated to + whitespace. You may want to use "@." so that e.g. in + "'first.last@example.org'" it won't be treated as a single word, but rather + you can search separately for "first", "last" and "example". + * default_language=<lang>: Default stemming language to use for mails. The + default is english. Requires that Dovecot is built with libstemmer, which + also limits the languages that are supported. + * textcat_conf=<path> textcat_dir=<path>: If specified, enable guessing the + stemming language for emails and search keywords. This is a little bit + problematic in practice, since indexing and searching languages may differ + and may not find even exact words because they stem differently. + * no_snowball: Support normalization of indexed words even without stemming + and libstemmer (Snowball). (v2.2.3+) + * mime_parts: Index each MIME part separately and include the MIME part number + in the "part" field. In future versions this will allowing showing which + attachment matched the search result. (v2.2.13+) + +Libraries +--------- + + * CLucene [http://sourceforge.net/projects/clucene/files/]: Get v2.3.3.4 (not + v0.9) + * libstemmer [http://snowball.tartarus.org/download.php]: Builds libstemmer.o, + which you can rename to libstemmer.a + * textcat [http://textcat.sourceforge.net/] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.FTS.Solr.txt b/doc/wiki/Plugins.FTS.Solr.txt new file mode 100644 index 0000000..e888a71 --- /dev/null +++ b/doc/wiki/Plugins.FTS.Solr.txt @@ -0,0 +1,289 @@ +Solr Full Text Search Indexing +============================== + +Solr [https://lucene.apache.org/solr/] is a Lucene indexing server. Dovecot +communicates to it using HTTP/XML queries. + +The steps described in this wiki page are tested for Solr 7.7.0. For other +versions, this these steps may need to be adjusted. + +Compiling +--------- + +Dovecot is not compiled with Solr FTS support by default. To enable it, you +need to add the '--with-solr' parameter to your invocation of the 'configure' +script. You will also need to have libexpat installed, including development +headers (typically from a separate development package). Configuration will +fail if '--with-solr' is enabled while libexpat headers cannot be found. Older +versions of Dovecot also required libcurl for Solr support, but recent versions +of Dovecot include a custom HTTP client. + +Configuration +------------- + +Solr Installation +----------------- + +First, the Solr server needs to be installed. Most operating systems will have +packages for this. The latest version can be downloaded and installed from +official website, and here are instructions to install 7.7.0 based on the howto +How to Install Apache Solr 7.5 on Debian 9/8 +[https://tecadmin.net/install-apache-solr-on-debian/]: + +---%<------------------------------------------------------------------------- +wget https://www-eu.apache.org/dist/lucene/solr/7.7.0/solr-7.7.0.tgz +tar xzf solr-7.7.0.tgz solr-7.7.0/bin/install_solr_service.sh +--strip-components=2 +sudo bash ./install_solr_service.sh solr-7.7.0.tgz +---%<------------------------------------------------------------------------- + +To use Solr with Dovecot, it needs to configured specifically for use with +Dovecot. + +---%<------------------------------------------------------------------------- +sudo -u solr /opt/solr/bin/solr create -c dovecot +---%<------------------------------------------------------------------------- + +The location of the files for the newly created instance on the filesystem +varies between operating systems and installation methods. For example, in +Archlinux, the config files are located in '/opt/solr/server/solr/dovecot/conf' +and data files can be found in '/opt/solr/server/solr/dovecot/data'. When +installed from tarball, these directories can be found in +'/var/solr/data/dovecot/'. + +Once the instance is created, you can start Solr. The means of starting, +stopping and querying the status of the 'solr' service varies between systems. +For systemd, these commands are as follows: + +---%<------------------------------------------------------------------------- +sudo systemctl stop solr +sudo systemctl start solr +sudo systemctl status solr +---%<------------------------------------------------------------------------- + +By default, the Solr administation page for the newly created instance is +located at https://localhost:8983/solr/#/~cores/dovecot. It can be used to +check the status of the Solr instance. Configuration errors are often most +conveniently viewed here. Solr also writes log files. For a tarball +installation, these can be found at '/var/solr/logs/'. + +Solr Configuration +------------------ + +There are three primary configuration files that need to be changed to +accommodate the Dovecot FTS needs: the instance configuration file +'solrconfig.xml' and the schema files 'schema.xml' and 'managed-schema' used by +the instance. These files are both located in the 'conf' directory of the Solr +instance (e.g.,'/var/solr/data/dovecot/conf/'). + +Remove default core configuration files +--------------------------------------- + +---%<------------------------------------------------------------------------- +rm -f /var/solr/data/dovecot/conf/schema.xml +rm -f /var/solr/data/dovecot/conf/managed-schema +rm -f /var/solr/data/dovecot/conf/solrconfig.xml +---%<------------------------------------------------------------------------- + +Install schema.xml and solrconfig.xml +------------------------------------- + +Copy doc/solr-config-7.7.0.xml +[https://raw.githubusercontent.com/dovecot/core/master/doc/solr-config-7.7.0.xml] +and doc/solr-schema-7.7.0.xml +[https://raw.githubusercontent.com/dovecot/core/master/doc/solr-schema-7.7.0.xml] +(Since Dovecot 2.3.6+) to '/var/solr/data/dovecot/conf/' as 'solrconfig.xml' +and 'schema.xml'. The 'managed-schema' file is generated based on 'schema.xml'. + +Dovecot Plugin +-------------- + +On Dovecot's side add: + +Into 10-mail.conf (note add existing plugins to string) + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins fts fts_solr +---%<------------------------------------------------------------------------- + +Into 90-plugins.conf + +---%<------------------------------------------------------------------------- +plugin { + fts = solr + fts_solr = url=https://solr.example.org:8983/solr/dovecot/ +} +---%<------------------------------------------------------------------------- + +Fields listed in 'fts_solr' plugin setting are space separated. They can +contain: + + * url=<solr url> : Required base URL for Solr. (remember to add your core name + if using solr 7+ : "/solr/dovecot/"). The default URL for Solr 7+ is + https://localhost:8983/solr/dovecot + * debug : Enable HTTP debugging. Writes to debug log. + * break-imap-search : Use Solr also for indexing TEXT and BODY searches. This + makes your server non-IMAP-compliant. (This is always enabled in v2.1+, and + removed since v2.3+ as it's default behaviour) + * rawlog_dir=<directory> : For debugging, store HTTP exchanges between Dovecot + and Solr in this directory. (2.3.6+) + * batch_size : Configure the number of mails sent in single requests to Solr, + default is 1000. (2.3.6+) + * with fts_autoindex=yes, each new mail gets separately indexed on arrival, + so batch_size only matters when doing the initial indexing of a mailbox. + * with fts_autoindex=no, new mails don't get indexed on arrival, so + batch_size is used when indexing gets triggered. + * soft_commit=yes|no : Control whether new mails are immediately searchable + via Solr, default to yes. When using no, it's important to set autoCommit or + autoSoftCommit time in solrconfig.xml so mails eventually become searchable. + (2.3.6+) + +Important notes: + + * Some mail clients will not submit any search requests for certain fields if + they index things locally eg. Thunderbird will not send any requests for + fields such as sender/recipients/subject when Body is not included as this + data is contained within the local index. + +Solr commits & optimization +--------------------------- + +Solr indexes should be optimized once in a while to make searches faster and to +remove space used by deleted mails. Dovecot never asks Solr to optimize, so you +should do this yourself. Perhaps a cronjob that sends the optimize-command to +Solr every n hours. + +With v2.2.3+ Dovecot only does soft commits to the Solr index to improve +performance. You must run a hard commit once in a while or Solr will keep +increasing its transaction log sizes. For example send the commit command to +Solr every few minutes. + +---%<------------------------------------------------------------------------- +# Optimize should be run somewhat rarely, e.g. once a day +curl https://<hostname/ip>:<port|default +8983>/solr/dovecot/update?optimize=true +# Commit should be run pretty often, e.g. every minute +curl https://<hostname/ip>:<port|default 8983>/solr/dovecot/update?commit=true +---%<------------------------------------------------------------------------- + +You may not need those if you are using a recent Solr (7+) or <SolrCloud.txt>. +The default configuration of Solr is to auto-commit every once in a while +(~15sec) so commit is not necessary. Also, the default / +<TieredMergePolicy.txt>/ in Solr will automatically purge removed documents +later, so optimize is not necessary. + +Re-index mailbox +---------------- + +If you require to force dovecot to reindex a whole mailbox you can run the +command shown, this will only take action when a search is done and will apply +to the whole mailbox. + +---%<------------------------------------------------------------------------- +doveadm fts rescan -u <username> +---%<------------------------------------------------------------------------- + +If you want to index a single mailbox/all mailboxes you can run the command +shown, this will happen immediately and will block until the action is +completed. + +---%<------------------------------------------------------------------------- +doveadm index [-u <user>|-A] [-S <socket_path>] [-q] [-n <max recent>] <mailbox +mask> +---%<------------------------------------------------------------------------- + +Sorting by relevancy +-------------------- + +Solr/Lucene supports returning a relevancy score for search results. If you +want to sort the search results by the score, use Dovecot's non-standard +X-SCORE sort key: + +---%<------------------------------------------------------------------------- +1 SORT (X-SCORE) UTF-8 <search parameters> +---%<------------------------------------------------------------------------- + +Indexes +------- + +Dovecot creates the following fields: + + * id: Unique ID consisting of uid/uidv/user/box. + * Note that your user names really shouldn't contain '/' character. + * uid: Message's IMAP UID. + * uidv: Mailbox's UIDVALIDITY. This changes if mailbox gets recreated. + * box: Mailbox name + * user: User name who owns the mailbox, or empty for public namespaces + * hdr: Indexed message headers + * body: Indexed message body + * any: "Copy field" from hdr and body, i.e. searching based on this will + search from both headers and bodies. + +Lucene does duplicate suppression based on the "id" field, so even if Dovecot +sends the same message multiple times to Solr it gets indexed only once. This +might happen currently if multiple searches are started at the same time. + +You might want to build a cronjob to go through the Lucene indexes once in a +while to delete indexed messages (or entire mailboxes) that no longer exist on +the filesystem. It shouldn't normally find any such messages though. + +Testing +------- + +---%<------------------------------------------------------------------------- +# telnet localhost imap +* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT +SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN +NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 ESEARCH ESORT SEARCHRES WITHIN +CONTEXT=SEARCH LIST-STATUS STARTTLS AUTH=PLAIN AUTH=LOGIN] I am ready. +1 login username password +2 select Inbox +3 SEARCH text "test" +---%<------------------------------------------------------------------------- + +Sharding +-------- + +If you have more users than fit into a single Solr box, you can split users off +to different servers. A couple of different ways you could do it are: + + * Have some HTTP proxy redirecting the connections based on the URL + * Configure Dovecot's userdb lookup to return a different host for 'fts_solr' + setting using <extra fields> [UserDatabase.ExtraFields.txt]. + * LDAP: 'user_attrs = ..., + solrHost=fts_solr=url=https://%$:8983/solr/dovecot/' + * MySQL: 'user_query = SELECT concat('url=https://', solr_host, + ':8983/solr/dovecot/') AS fts_solr, ...' + +You can also use SolrCloud +[https://lucene.apache.org/solr/guide/7_6/solrcloud.html], the clustered +version of Solr, that allows you to scale up, and adds failover / high +availability to your FTS system. Dovecot-solr works fine with a <SolrCloud.txt> +cluster as long as the solr schema is the right one. + +External Tutorials +------------------ + +External sites with tutorials on using Solr under Dovecot + + * Installing Apache Solr with Dovecot for fulltext search results (ATmail + support guide) + [https://help.atmail.com/hc/en-us/articles/201566404-Installing-Apache-Solr-with-Dovecot-for-fulltext-search-results] + * FreeBSD: https://mor-pah.net/2016/08/15/dovecot-2-2-with-solr-6-or-5/ + * Substring searches with ngrams: + https://dovecot.org/list/dovecot/2011-May/059338.html + +Tips +---- + +Some additional things which might help you configuring Solr search: + + * If you are using Tomcat: Set 'maxHttpHeaderSize="65536"' (connector + definition for port 8080 in '/etc/tomcat7/server.xml') to accept long search + query strings (iPhones tend to send multi-kilobyte-sized queries) + * Set 'df' to 'hdr' in '/etc/solr/conf/solrconfig.xml' ('/select' request + handler) to avoid strange 'undefined field text' errors. + * Please keep in mind that you will have to change the Solr URL to include the + core name (ie:'dovecot': 'https://localhost:8939/solr/dovecot'). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.FTS.Squat.txt b/doc/wiki/Plugins.FTS.Squat.txt new file mode 100644 index 0000000..07dc867 --- /dev/null +++ b/doc/wiki/Plugins.FTS.Squat.txt @@ -0,0 +1,128 @@ +Squat Full Text Search Indexing +=============================== + +NOTE: The Squat code is quite slow for large mailboxes. There are also a few +bugs that are unlikely to be fixed. In v2.1+ it's recommended to use +<fts-lucene> [Plugins.FTS.Lucene.txt] instead. There's also a "New Squat" +redesign/reimplementation going on, which will likely end up having a new name. + +The main difference between Squat indexes and the others is that Squat provides +support for substring searches, while pretty much all other FTS indexes support +only matching from the beginning of words. By strictly reading the IMAP RFC it +requires substring matching, so to optimize regular TEXT and BODY searches you +must use Squat with Dovecot v2.0. The other indexes are used only for Dovecot's +non-standard X-TEXT-FAST and X-BODY-FAST searches. However, almost all other +commonly used IMAP servers no longer care about this requirement, so Dovecot +v2.1 also no longer makes this distinction. In Dovecot v2.1 TEXT and BODY +searches are done using any indexes available. + + * The Squat indexes are currently updated only when SEARCH TEXT or SEARCH BODY + command is used. It's planned that in future they could be updated + immediately when messages are being saved. + * The Squat indexes take about 30% of the mailbox size (with partial=4 + full=4). + * The initial Squat index building for large mailboxes can be very CPU and + memory hungry. + * The Squat indexes are stored among Dovecot's other index files. They're + called 'dovecot.index.search' and 'dovecot.index.search.uids'. It's safe to + delete both of these files to trigger a rebuild. + +Some statistics with Core2 duo 6600, 2 GB memory (partial=4 full=4): + + * 1,4 GB mbox with 368000 messages: SEARCH BODY asdfgh + * Without Squat: 2 min 35 sec CPU, process RSS size 24 MB + * Initial Squat build: 6 min 36 sec CPU, process RSS size peaks at 800 MB, + drops to 96 MB at end. + * Subsequent searches: 0.10 sec CPU, 1.18 sec real with cold cache, process + RSS size 2 MB. + * Most of the time was spent verifying Squat results. Searching for a + 1..4 letter word gives results in 0.00 CPU secs and 0.80 real secs + with cold cache. + + * 32 MB mbox with 7500 messages: SEARCH BODY asdfgh + * Without Squat: 2.3 sec CPU, 3.6 sec real with cold cache, process RSS + size 5 MB + * Initial Squat build: 8.9 sec CPU, 10.0 sec real with cold cache, process + RSS size peaks at 61 MB, drops to 20 MB at end. + * 48 MB maildir with 10000 messages: SEARCH BODY asdfgh + * Without Squat: 2.6 sec CPU, 9.7 sec real with cold cache + * Initial Squat build: 9.7 sec CPU, 15.3 sec real with cold cache + +Internal workings +----------------- + +Squat works by building a trie of all 1..4 character combinations of all words +in messages. For example given a word "world" it indexes "worl", "orld", "rld", +"ld" and "d". Matching message UIDs are stored to each trie node, so Squat +supports giving definitive answers to any searches with a length of 1..4 +characters. For longer search words the word is looked up in 4 letter pieces +and the results are merged. The resulting list is a list of messages where the +word *may* exist. This means that those messages are still opened and the word +is searched from them the slow way. + +Squat also supports indexing more characters from the beginning of words. For +example if 5 first characters are indexed, the word "character" would be cut to +"chara". If a user then searches for "chara", Dovecot does: + + 1. Search for messages having "char" (reply example: 1,5,10) + 2. Search for messages having "hara" (reply example: 3,5,10) + 3. Store the intersection of the above searches to "maybe UIDs" (example: + 5,10) + 4. Search for full word "chara" and store it as "definite UIDs" (reply + example: 5) + 5. Remove definite UIDs (5) from maybe UIDs (5,10 -> 10) + +In the above example Dovecot would know that message UID 5 has the word +"chara", but it still has to check if UID 10 has it as a substring (e.g. in +word "achara") by reading the message contents. If the user's search words +match mostly non-substrings (which is common), using long enough full word +indexing can improve the search performance a lot, especially when the word +matches a lot of messages. + +The Squat name comes from Cyrus IMAP [http://cyrusimap.web.cmu.edu/] which +implements slightly similar Squat indexes ("Search QUery Answer Tool"). +Dovecot's implementation and file format however is completely different. The +main visible difference is that Dovecot allows updating the index incrementally +instead of requiring to re-read the entire mailbox to build it. Cyrus Squat +also doesn't support 1..3 characters long searches. + +Configuration +------------- + +'fts_squat' setting can be used to change Squat options: + + * partial=n: Length of the substring blocks to index. Default is 4 characters + and it's probably not a good idea to change it. + * full=n: Index n first characters from the beginning of words. Default is 4, + but it could be useful to increase this to e.g. 10 or so. However larger + values take more disk space. + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins fts fts_squat + +plugin { + fts = squat + fts_squat = partial=4 full=10 +} +---%<------------------------------------------------------------------------- + +Using both Squat and non-Squat (v2.0-only) +------------------------------------------ + +It's possible to use both Squat and non-Squat indices at the same time, so that +TEXT/BODY are looked up from Squat indexes and X-TEXT-FAST/X-BODY-FAST are +looked up from the non-Squat index. This of course means that indices will have +to be built and updated for both, so it might not be that good idea. + +---%<------------------------------------------------------------------------- +protocol imap { +.. + mail_plugins = fts fts_squat fts_solr +} +... +plugin { + fts = squat solr +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.FTS.txt b/doc/wiki/Plugins.FTS.txt new file mode 100644 index 0000000..e960b4c --- /dev/null +++ b/doc/wiki/Plugins.FTS.txt @@ -0,0 +1,102 @@ +Full text search indexing +========================= + +The following FTS indexers (in preferred order) are supported: + + * <Solr> [Plugins.FTS.Solr.txt] communicates with Lucene's Solr server + [http://lucene.apache.org/solr/]. + * <Lucene> [Plugins.FTS.Lucene.txt] uses Lucene's C++ library. (Requires + v2.1+) + * <fts-dovecot> [Plugins.FTS.Dovecot.txt] is Dovecot Pro's new search index, + and is not available without commercial agreement. (Requires v2.2+) + * <Squat> [Plugins.FTS.Squat.txt] is Dovecot's own search index. (Obsolete in + v2.1+) + * fts-xapian [https://github.com/grosjo/fts-xapian] is Xapian + [https://xapian.org] based plugin maintained by '<jom AT NOSPAM grosjo DOT + net>'. (Requires v2.3+) + +Indexing +-------- + +By default the FTS indexes are updated *only* while searching, so neither the +<LDA.txt> nor an IMAP APPEND command updates the indexes immediately. This +means that if user has received a lot of mail since the last indexing (== +search operation), it may take a while to index all the mails before replying +to the search command. Dovecot sends periodic "* OK Indexed n% of the mailbox" +updates which can be caught by webmail implementations to implement a progress +bar. + +In v2.2.9+ the indexing can be done automatically with 'fts_autoindex=yes' +setting (see below). + +The indexing can be done manually (e.g. cronjob) or by a LDA script by running: + + * v2.1: 'doveadm index -u user@domain -q INBOX' + * v2.0: 'printf "a select INBOX\nb search text xyzzy\nc logout\n" | + /usr/local/libexec/dovecot/imap -u user@domain' + +Of course the INBOX needs to be replaced with whatever mailbox needs to be +indexed. + +Indexing Attachments (v2.1+) +---------------------------- + +Attachments can be indexed either via a script that translates the attachment +to UTF-8 plaintext or Apache Tika server. + + * 'fts_decoder = <service>': Decode attachments to plaintext using this + service and index the resulting plaintext. See the 'decode2text.sh' script + included in Dovecot for how to use this. (v2.1+) + * 'fts_tika = http://tikahost:9998/tika/': This URL needs to be running Apache + Tika server (e.g. started with 'java -jar + tika-server/target/tika-server-1.5.jar') (v2.2.13+) + +Rescan (v2.1+) +-------------- + +Since v2.1 Dovecot keeps track of indexed messages in the dovecot.index files. +If this becomes out of sync with the actual FTS indexes (either too many or too +few mails), you'll need to do a rescan: + +---%<------------------------------------------------------------------------- +doveadm fts rescan -u user@domain +---%<------------------------------------------------------------------------- + +Other Settings +-------------- + +All the FTS settings go inside 'plugin {} ' section of 90-plugin.conf. + + * 'fts_autoindex=yes': Index new messages immediately after they've been + saved/copied. (v2.2.9+) + * 'fts_autoindex_exclude=pattern1', 'fts_autoindex_exclude2=pattern2', ...: + Exclude given mailboxes, one pattern per setting. Supports "*" and "?" + wildcards. If a name starts with '\', it's treated as a case-insensitive + special-use flag. (v2.2.25+) + * Example: + + ---%<------------------------------------------------------------------- + plugin { + fts_autoindex_exclude = \Junk + fts_autoindex_exclude2 = \Trash + fts_autoindex_exclude3 = DUMPSTER + } + ---%<------------------------------------------------------------------- + + * 'fts_autoindex_max_recent_msgs=n': Skip autoindexing the mailbox if it has + more than n \Recent messages (implying that the mailbox is never actually + being accessed). (v2.2.9+) + * 'fts_enforced': + * no (default): All body searches will index all missing mails in FTS. + Header searches will use FTS if the mails are indexed, otherwise fallback + to parsing the headers (usually from dovecot.index.cache). If FTS search + fails, fallback to reading and parsing all mails. + * yes: All header and body searches will index all missing mails in FTS. If + FTS search fails, error is returned to client. + * 'fts_index_timeout': When SEARCH notices that index isn't up to date, it + tells indexer to index the mails and waits until it is finished. This + setting adds a maximum timeout to this wait. If the timeout is reached, the + SEARCH fails with:'NO [INUSE] Timeout while waiting for indexing to finish' + (v2.1+) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.LastLogin.txt b/doc/wiki/Plugins.LastLogin.txt new file mode 100644 index 0000000..65ae6a8 --- /dev/null +++ b/doc/wiki/Plugins.LastLogin.txt @@ -0,0 +1,54 @@ +Last Login Plugin +================= + +Requires v2.2.14+. This plugin can be used to easily update user's last-login +timestamp in the configured <dictionary> [Dictionary.txt]. Configuration: + +---%<------------------------------------------------------------------------- +protocol imap { + mail_plugins = $mail_plugins last_login +} +protocol pop3 { + mail_plugins = $mail_plugins last_login +} + +plugin { + last_login_dict = redis:host=127.0.0.1:port=6379 + #last_login_key = last-login/%u # default +} +---%<------------------------------------------------------------------------- + +Note that we enable last_login plugin explicitly only for imap & pop3 +protocols. If enabled globally, it'll also update the timestamp whenever new +mails are delivered via lda/lmtp or when doveadm is run for the user. This can +also be thought of as a feature, so if you want to update a different timestamp +for user when new mails are delivered, you can do that by enabling the +last_login plugin also for lda/lmtp and changing the last_login_key setting. + +Example dict config with schema + +---%<------------------------------------------------------------------------- +CREATE TABLE `users` ( + `userid` varchar(255) NOT NULL, + +... + + `last_login` int(11) DEFAULT NULL, + PRIMARY KEY (`userid`) +) +---%<------------------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +map { + pattern = shared/last-login/$user + table = users + value_field = last_login + value_type = uint + + fields { + userid = $user + } +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Lazyexpunge.txt b/doc/wiki/Plugins.Lazyexpunge.txt new file mode 100644 index 0000000..a85e24a --- /dev/null +++ b/doc/wiki/Plugins.Lazyexpunge.txt @@ -0,0 +1,202 @@ +Lazy expunge +============ + +The idea behind this plugin is that expunged mails and deleted mailboxes stay +around for a while, so that user can undelete them without assistance from +sysadmin. The expunged mails won't be counted in user's quota. The plugin +itself doesn't clean up the expunged messages, you'll have to do it some other +way (see below). + +The plugin is configured by defining namespaces where the mails are moved. You +can decide if you want the namespaces to be visible to clients, or if you want +to show them only via some special webmail interface. + +1 mailbox (v2.2.24+) +-------------------- + +You create a single mailbox. All messages that are expunged from all the +mailboxes are moved there. This is the simplest configuration. The mailbox is +created automatically. You probably also want to hide it with an ACL. + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins lazy_expunge acl +plugin { + lazy_expunge = .EXPUNGED + acl = vfile:/etc/dovecot/dovecot.acl + + # Expunged messages most likely don't want to be included in quota: + quota_rule2 = .EXPUNGED:ignore +} +---%<------------------------------------------------------------------------- + +Where '/etc/dovecot/dovecot.acl' contains: + +---%<------------------------------------------------------------------------- +.EXPUNGED owner rwstipekxa +---%<------------------------------------------------------------------------- + +You could also leave the permissions empty if you don't want to allow clients +to access it at all. + +1 namespace +----------- + +You create only a single namespace. When a message is expunged from mailbox +/<name>/, it's moved to a mailbox /<name>/ in the expunge namespace. When an +entire mailbox /<name>/ is deleted, it's also moved to this namespace as +/<name>/. If it already exists, their contents are merged. + +Example configuration: + +---%<------------------------------------------------------------------------- +# the default namespace +namespace { + prefix = + separator = / + inbox = yes +} + +# namespace for lazy_expunge plugin: +namespace { + prefix = .EXPUNGED/ + hidden = yes + list = no + separator = / + location = maildir:~/Maildir/expunged +} + +mail_plugins = $mail_plugins lazy_expunge +plugin { + lazy_expunge = .EXPUNGED/ +} +---%<------------------------------------------------------------------------- + +3 namespaces (obsolete, v2.0 only) +---------------------------------- + +The namespaces are: + + 1. Expunged messages namespace. Whenever a message is expunged in mailbox + /<name>/, it's moved to a mailbox /<name>/ in this namespace. The mailboxes + are created automatically as needed. + 2. Deleted mailboxes namespace. Whenever a mailbox /<name>/ is deleted, it's + moved here with name /<name>-YYYMMDD-hhmmss/. The timestamp is there so + that the mailbox can be deleted multiple times. If the mailbox is deleted + multiple times within a second, random 16bit hex value is appended to it. + 3. Expunged messages in a deleted mailbox namespace. When a mailbox is deleted + and it has messages in its expunged namespace, the mailbox is moved from + the expunged namespace to this namespace. The destination mailbox name is + the same as in the 2nd namespace (ie. contains the same timestamp). + +Example configuration: + +---%<------------------------------------------------------------------------- +# the default namespace +namespace { + prefix = + separator = / + inbox = yes +} + +# namespaces for lazy_expunge plugin: +namespace { + prefix = .EXPUNGED/ + separator = / + location = maildir:~/Maildir/expunged +} +namespace { + prefix = .DELETED/ + separator = / + location = maildir:~/Maildir/deleted +} +namespace { + prefix = .DELETED/.EXPUNGED/ + separator = / + location = maildir:~/Maildir/deleted/expunged +} + +mail_plugins = $mail_plugins lazy_expunge +plugin { + lazy_expunge = .EXPUNGED/ .DELETED/ .DELETED/.EXPUNGED/ +} +---%<------------------------------------------------------------------------- + +Multi-dbox +---------- + +With multi-dbox use different MAILBOXDIRs (so copying between namespaces works +quickly within the same storage), but otherwise exactly the same paths (index, +control): + +---%<------------------------------------------------------------------------- +# the default namespace +namespace { + prefix = + inbox = yes + location = mdbox:~/mdbox:INDEX=/var/index/%d/%n +} + +# lazy_expunge namespace(s): +namespace { + prefix = .EXPUNGED/ + hidden = yes + list = no + subscriptions = no + + location = mdbox:~/mdbox:INDEX=/var/index/%d/%n:MAILBOXDIR=expunged + # If mailbox_list_index=yes is enabled, it needs a separate index file +(v2.2.28+): + #location = +mdbox:~/mdbox:INDEX=/var/index/%d/%n:MAILBOXDIR=expunged:LISTINDEX=expunged.list.index +} +---%<------------------------------------------------------------------------- + +Copy only the last instance (v2.2+) +----------------------------------- + +If mail has multiple copies (via IMAP COPY), each copy is normally moved to +lazy expunge namespace when it's expunged. With v2.2+ you can set 'plugin { +lazy_expunge_only_last_instance = yes }' to copy only the last instance and +immediately expunge the others. This may be useful if you want to provide a +flat list of all expunged mails without duplicates in your webmail. With many +clients this means that the last instance is always in the Trash mailbox. + +Cleaning up +----------- + +doveadm +------- + +---%<------------------------------------------------------------------------- +doveadm expunge mailbox 'deleted/*' savedsince 1d +---%<------------------------------------------------------------------------- + +cronjob +------- + +Run something like this for each user every night (not actually tested if it +works, and vulnerable to TOCTTOU if the user can place symlinks in any of the +directories find is traversing): + +---%<------------------------------------------------------------------------- +# delete a day old mails +find Maildir/expunged Maildir/deleted Maildir/deleted/expunged \ + -type f ! -cmin 1440 -print0 | xargs -0 rm +---%<------------------------------------------------------------------------- + +Trash plugin +------------ + +<Trash plugin> [Plugins.Trash.txt] with some help from <Quota plugin> +[Quota.txt] could probably be used to keep the expunged and deleted mailboxes +under a specified size (not tested). + +Expire plugin +------------- + +<Expire plugin> [Plugins.Expire.txt] was created to keep track of mails in +specific mailboxes, and expunge them when they've been there for a specified +amount of time. It keeps an internal database (e.g. SQL) of all such mailboxes, +so it doesn't have to go through all the mailboxes for all the users. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Listescape.txt b/doc/wiki/Plugins.Listescape.txt new file mode 100644 index 0000000..514978f --- /dev/null +++ b/doc/wiki/Plugins.Listescape.txt @@ -0,0 +1,64 @@ +Listescape plugin +================= + +The Listescape plugin allows users to use characters in mailboxes names that +would otherwise be illegal (due to the underlying mailbox storage), for +example: + + * Maildir++ layout disallows using the '.' character (unless LAYOUT=fs is + used), since it's used internally as the folder hierarchy separator. + * The '~' character at the beginning of the mailbox name is disallowed, + because of the possibility that it gets expanded to user's home directory. + * The '/' character is disallowed on POSIX systems. + +The Listescape plugin allows you to use all of these characters, as long as the +virtual separator (i.e. what is set by the separator= setting and used as such +by the IMAP protocol) is changed to something else, which means that the plugin +does *not* make it possible to use the virtual separator in folder names. + +The characters are escaped to the mailbox name as \NN hex codes. + +So what would be a good hierarchy separator to use? + + * '.' and '/' are very commonly used and should work everywhere + * '\' is used by Exchange, and should also work everywhere (when specifying + this in the separator= setting it must be quoted, so one sets separator = + "\\") + * '^' is used internally by Thunderbird and causes some trouble with it + * others? + +Examples +-------- + +Allow '.' characters with Maildir++ layout when virtual hierarchy separator is +changed to '/' (it could be anything else except '.' itself): + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins listescape + +namespace private { + separator = / + inbox = yes +} + +plugin { + # The default escape character is '\', but you can change it. + # Note that even here the expansion of % takes place, thus you need to + # use "%%" if you want to have the % sign as the escape character. + #listescape_char = "\\" +} +---%<------------------------------------------------------------------------- + +Allow both '.' and '/' characters when virtual hierarchy separator is changed +to '$'. The '$' has to be quoted to avoid variable expansion.: + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins listescape + +namespace private { + separator = "$" + inbox = yes +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.MailCrypt.txt b/doc/wiki/Plugins.MailCrypt.txt new file mode 100644 index 0000000..1cab338 --- /dev/null +++ b/doc/wiki/Plugins.MailCrypt.txt @@ -0,0 +1,522 @@ +mail-crypt-plugin +================= + +Contents + + + 1. mail-crypt-plugin + + 1. Introduction + + 1. Functional Overview + + 2. Encryption Technologies + + 2. Technical Requirements + + 3. Settings for mail crypt plugin + + 4. Modes of operation + + 5. Folder keys + + 1. Unencrypted user keys + + 2. Encrypted user keys + + 6. Global keys + + 1. RSA key + + 2. EC key + + 3. Converting EC key to PKEY + + 4. Base64 encoded keys + + 7. New dcrypt format (mail_crypt_save_version = 2) + + 8. Old dcrypt format (mail_crypt_save_version = 1) + + 9. Read-only mode (mail_crypt_save_version = 0) + + 10. mail-crypt-plugin and ACLs + + 11. decrypting files encrypted with mail-crypt plugin + + 12. fs-crypt and fs-mail-crypt + + 2. doveadm plugin + + 1. doveadm mailbox cryptokey generate + + 2. doveadm mailbox cryptokey list + + 3. doveadm mailbox cryptokey export + + 4. doveadm mailbox cryptokey password + +Introduction +------------ + +The Mail crypt plugin is used to secure email messages stored in a Dovecot +system. Messages are encrypted before written to storage and decrypted after +reading. Both operations are transparent to the user. + +In case of unauthorized access to the storage backend, the messages will, +without access to the decryption keys, be unreadable to the offending party. + +There can be a single encryption key for the whole system or each user can have +a key of their own. The used cryptographical methods are widely usedstandards +and keys are stored in portable formats, when possible. + +Functional Overview +------------------- + +The use of Mail crypt plugin depends on a user having a keypair, a private and +a public key, for asymmetric cryptography. These keys are provisioned in a +variable via the user database or directly from Dovecot configuration files. + +The public half of the provisioned keypairs are used to generate and encrypt +keys for symmetric encryption. The symmetric keys are used to encrypt and +decrypt individual files. Symmetric encryption is faster and more suitable for +block mode storage encryption. The symmetric key used to encrypt a file is +stored, after being encrypted with the public asymmetric key, together with the +file. + +Encryption Technologies +----------------------- + +The Mail crypt plugin provides encryption at rest for emails. Encryption of the +messages is performed using the symmetric Advanced Encryption Standard (AES) +algorithm in Galois/Counter Mode (GCM) with 256 bit keys. Integrity of the data +is ensured using Authenticated Encryption with Associated Data (AEAD) with +SHA256 hashing. The encryption keys for the symmetric encryption are randomly +generated. These keys in turn are encrypted using a key derived with from the +provisioned private key. Provisioned private keys can be Elliptic Curve (EC) +keys or RSA Encryption is done using the Integrated Encryption Scheme (IES). +This algorithm is usable both with EC and RSA keys. + +Technical Requirements +---------------------- + +This feature is available in v2.2.27+. Using per-folder keys is not considered +production quality, but global keys are fine. + +*NB! Improper configuration or use can make your emails unrecoverable. Treat +encryption with care and backups.* + +mail-crypt-plugin encrypts and decrypts mail. The plugin has an older version, +and the extent of this version's backward compatibility is controlled by the +setting *mail_crypt_save_version*. The setting has three valid values, of which +one must be set for the plugin to do anything. The values are 0 and 2. With +*mail_crypt_save_version = 2*, mails are saved in dcrypt version 2 format, and +this is the value that should be used. With *mail_crypt_save_version = 0*, the +plugin does not write encrypted mails, but can still read them. To provide +*mail_crypt_global_private_key* and *mail_crypt_global_public_key* as userdb +attributes, you can base64 encode the original contents, such as PEM file. For +example, + +---%<------------------------------------------------------------------------- +cat ecprivkey.pem | base64 -w0 +---%<------------------------------------------------------------------------- + +Settings for mail crypt plugin +------------------------------ + +These all go into userdb environment or under plugin { } + + * *mail_crypt_save_version* - Save format, 0 = read only, 2 = current version + * *mail_crypt_curve* - EC curve to use for key generation + * *mail_crypt_global_private_key(_n)* - Private key to decrypt files, you can + specify many + * *mail_crypt_global_public_key* - Public key to use to encrypt files, you can + specify one + * *mail_crypt_private_key* - Private key to decrypt user's master key, can be + base64 encoded + * *mail_crypt_private_password* - Password to decrypt user's master key or + environment private key + * *mail_crypt_acl_require_secure_key_sharing* - Require secure key sharing + * *mail_crypt_require_encrypted_user_key* - Require user key encryption with + password + +All external keys must be in PEM format, using pkey format. + +Modes of operation +------------------ + +Mail crypt plugin can operate using *either* global keys *or* folder keys. +Using both is not supported. To perform any +encryption,*mail_crypt_save_version* must be specified and non-zero. + +Folder keys +----------- + +In this mode, the user is generated a key pair, and each folder is generated a +key pair, which is encrypted using the user's key pair. A user can have more +than one key pair but only one can be active. You must use save version 2. You +must also specify *mail_crypt_curve*. Any valid curve supported by underlying +cryptographic library is supported.*mail_attribute_dict* has to be set since it +is used to store the keys. + +Unencrypted user keys +--------------------- + +In this version of the folder keys mode, the users private key is stored +unencrypted on the server. + +Example config for folder keys with Maildir: + +---%<------------------------------------------------------------------------- +mail_attribute_dict = file:%h/Maildir/dovecot-attributes + +mail_plugins = $mail_plugins mail_crypt + +plugin { + mail_crypt_curve = secp521r1 + mail_crypt_save_version = 2 +} +---%<------------------------------------------------------------------------- + +Encrypted user keys +------------------- + +In this version of the folder keys mode, the users private key is stored +encrypted on the server. + +Example config for mandatory encrypted folder keys with Maildir: + +---%<------------------------------------------------------------------------- +mail_attribute_dict = file:%h/Maildir/dovecot-attributes + +mail_plugins = $mail_plugins mail_crypt + +plugin { + mail_crypt_curve = secp521r1 + mail_crypt_save_version = 2 + mail_crypt_require_encrypted_user_key = yes +} +---%<------------------------------------------------------------------------- + +The password that is used to decrypt the users master/private key, must be +provided via password query: + +---%<------------------------------------------------------------------------- +# File: /etc/dovecot/dovecot-sql.conf.ext + +password_query = SELECT \ +email as user, password, \ +'%w' AS userdb_mail_crypt_private_password \ +FROM virtual_users WHERE email='%u'; +---%<------------------------------------------------------------------------- + +Global keys +----------- + +In this mode, all keying material is taken from plugin environment. You can use +either EC keys (recommended) or RSA keys. No key generation is performed. + +RSA key +------- + +Use of RSA keys is discouraged, please use Elliptic Curve keys instead. + +You can generate an unencrypted RSA private key in the pkey format with the +command: + +---%<------------------------------------------------------------------------- +openssl genpkey -algorithm RSA -out rsaprivkey.pem +---%<------------------------------------------------------------------------- + +Alterantively, you can generate a password encrypted private key with: + +---%<------------------------------------------------------------------------- +openssl genpkey -algorithm RSA -out rsaprivkey.pem -aes-128-cbc -pass +pass:qwerty +---%<------------------------------------------------------------------------- + +This does make the password show up in the process listing, so it can be +visible for everyone on the system. + +Regardless of whether you generated an unencrypted or password encrypted +private key, you can generate a public key out of it with: + +---%<------------------------------------------------------------------------- +openssl pkey -in rsaprivkey.pem -pubout -out rsapubkey.pem +---%<------------------------------------------------------------------------- + +These keys can then be used by mail-crypt-plugin with the configuration: + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins mail_crypt + +plugin { + mail_crypt_global_private_key = <rsaprivkey.pem + mail_crypt_global_private_password = qwerty + mail_crypt_global_public_key = <rsapubkey.pem + mail_crypt_save_version = 2 +} +---%<------------------------------------------------------------------------- + +EC key +------ + +In order to generate an EC key, you must first choose a curve from the outputof +this command: + +---%<------------------------------------------------------------------------- +openssl ecparam -list_curves +---%<------------------------------------------------------------------------- + +If you choose the curve prime256v1, generate and EC key with the command: + +---%<------------------------------------------------------------------------- +openssl ecparam -name prime256v1 -genkey | openssl pkey -out ecprivkey.pem +---%<------------------------------------------------------------------------- + +Then generate a public key out of your private EC key + +---%<------------------------------------------------------------------------- +openssl pkey -in ecprivkey.pem -pubout -out ecpubkey.pem +---%<------------------------------------------------------------------------- + +These keys can now be used with mail-crypt-plugin with the configuration: + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins mail_crypt + +plugin { + mail_crypt_global_private_key = <ecprivkey.pem + mail_crypt_global_public_key = <ecpubkey.pem + mail_crypt_save_version = 2 +} +---%<------------------------------------------------------------------------- + +Converting EC key to PKEY +------------------------- + +If you have an EC private key which begins with something like: + +---%<------------------------------------------------------------------------- +-----BEGIN EC PRIVATE KEY----- +---%<------------------------------------------------------------------------- + +With possibly parameters like this before that: + +---%<------------------------------------------------------------------------- +-----BEGIN EC PARAMETERS----- +BgUrgQQACg== +-----END EC PARAMETERS----- +---%<------------------------------------------------------------------------- + +You must convert it to pkey format with: + +---%<------------------------------------------------------------------------- +openssl pkey -in oldkey.pem -out newkey.pem +---%<------------------------------------------------------------------------- + +Then newkey.pem can be used with mail-crypt-plugin. + +Base64 encoded keys +------------------- + +Mail-crypt plugin can read keys that are base64 encoded. This is intended +mostly for providing PEM keys via userdb. + +Hence, this is possible: + +---%<------------------------------------------------------------------------- +openssl ecparam -name secp256k1 -genkey | openssl pkey | base64 -w0 > +ecprivkey.pem +base64 -d ecprivkey.pem | openssl ec -pubout | base64 -w0 > ecpubkey.pem +---%<------------------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = password=pass mail_crypt_global_public_key=<content of ecpubkey.pem> +mail_crypt_global_private_key=<content of ecprivkey.pem> +} + +mail_plugins = $mail_plugins mail_crypt + +plugin { + mail_crypt_save_version = 2 +} +---%<------------------------------------------------------------------------- + +New dcrypt format (mail_crypt_save_version = 2) +----------------------------------------------- + +The recommended setting of *mail_crypt_save_version* for new installations of +mail-crypt-plugin is 2. + +Old dcrypt format (mail_crypt_save_version = 1) +----------------------------------------------- + +Do not use this. It is supported for legacy reasons only and should not be used +to create new files. It will not work without a global key. + +Read-only mode (mail_crypt_save_version = 0) +-------------------------------------------- + +If you have encrypted mailboxes that you need to read, but no longer want to +encrypt new mail, use *mail_crypt_save_version = 0*: + +---%<------------------------------------------------------------------------- +plugin { + mail_crypt_save_version = 0 + mail_crypt_global_private_key = <server.key +} +---%<------------------------------------------------------------------------- + +mail-crypt-plugin and ACLs +-------------------------- + +If you are using global keys, mails can be shared within the key scope. The +global key can be provided with several different scopes: + + * Global scope (key is configured in dovecot.conf file) + * Per-user(group) scope(key is configured in userdb file) + +With folder keys, key sharing can be done to single user, or multiple users. +When key is shared to single user, and the user has public key available, the +folder key is encrypted to recipient's public key. If you have +*mail_crypt_acl_require_secure_key_sharing* plugin setting, you can't share the +key to groups or someone with no public key. + +decrypting files encrypted with mail-crypt plugin +------------------------------------------------- + +You can use decrypt.rb +[https://gist.github.com/cmouse/882f2e2a60c1e49b7d343f5a6a2721de] to decrypt +encrypted files. + +fs-crypt and fs-mail-crypt +-------------------------- + +The fs-crypt is a lib-fs wrapper that can encrypt and decrypt files. It works +similarly to the fs-compress wrapper. It can be used to encrypt e.g.: + + * FTS index objects (fts_dovecot_fs) + * External mail attachments (mail_attachment_fs) + +fs-crypt comes in two flavors, mail-crypt and crypt. mail-crypt is intended to +be used with user context, while crypt can be used elsewhere. + +Currently the fs-crypt plugin requires that all the files it reads are +encrypted. If it sees an unencrypted file it'll fail to read it. The plan is to +fix this later. + +FS driver syntax: +*crypt:[algo=<s>:][set_prefix=<n>:][private_key_path=/path:][public_key_path=/path:][password=password:]<parent +fs>* where: + + * *algo*: Encryption algorithm. Default is aes-256-gcm-sha256. + * *set_prefix*: Read _public_key and _private_key under this prefix. Default + is "mail_crypt_global". + * *private_key_path*: Path to private key + * *public_key_path*: Path to public key + * *password*: Password for decrypting public key + +Example: + +---%<------------------------------------------------------------------------- +plugin { + fts_index_fs = crypt:set_prefix=fscrypt_index:posix:prefix=/tmp/fts + fscrypt_index_public_key = <server.pub + fscrypt_index_private_key = <server.key +} +---%<------------------------------------------------------------------------- + +To encrypt/decrypt files manually, you can use + +---%<------------------------------------------------------------------------- +doveadm fs get/put crypt +private_key_path=foo:public_key_path=foo2:posix:prefix=/path/to/files/root +path/to/file +---%<------------------------------------------------------------------------- + +doveadm plugin +============== + +Following commands are made available via doveadm. + +doveadm mailbox cryptokey generate +---------------------------------- + +---%<------------------------------------------------------------------------- +doveadm [-o plugin/mail_crypt_private_password=some_password] mailbox cryptokey +generate [-u username | -A] [-Rf] [-U] mailbox-mask [mailbox-mask ...] +---%<------------------------------------------------------------------------- + +Generate new keypair for user or folder. + + * -o - Dovecot option, needed if you use password protected keys + * -u - Username or mask to operate on + * -A - All users + * -R - Re-encrypt all folder keys with current active user key + * -f - Force keypair creation, normally keypair is only created if none found + * -U - Operate on user keypair only + +To generate new active user key and re-encrypt all your keys with it can be +done with + +---%<------------------------------------------------------------------------- +doveadm mailbox cryptokey generate -u username -UR + +This can be used to generate new user keypair and re-encrypt and create folder +keys. +---%<------------------------------------------------------------------------- + +Note that you must provide password if you want to generate password-protected +keypair right away. You can also use doveadm mailbox cryptokey password to +secure it. + +doveadm mailbox cryptokey list +------------------------------ + +---%<------------------------------------------------------------------------- +doveadm mailbox cryptokey list [-u username | -A] [-U] mailbox-mask +[mailbox-mask ...] +---%<------------------------------------------------------------------------- + + * -u - Username or mask to operate on + * -A - All users + * -U - Operate on user keypair only + +Will list all keys for user or mailbox. + +doveadm mailbox cryptokey export +-------------------------------- + +---%<------------------------------------------------------------------------- +doveadm [-o plugin/mail_crypt_private_password=some_password] mailbox cryptokey +export [-u username | -A] [-U] mailbox-mask [mailbox-mask ...] +---%<------------------------------------------------------------------------- + + * -u - Username or mask to operate on + * -A - All users + * -U - Operate on user keypair only + +Exports user or folder private keys. + +doveadm mailbox cryptokey password +---------------------------------- + +---%<------------------------------------------------------------------------- +doveadm mailbox cryptokey password [-u username | -A] [-N | -n password] [-O | +-o password] [-C] +---%<------------------------------------------------------------------------- + + * -u - Username or mask to operate on + * -A - All users + * -N - Ask new password + * -n - New password + * -O - Ask old password + * -o - Old password + * -C - Clear password + +Sets, changes or clears password for user's private key. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.MailFilter.txt b/doc/wiki/Plugins.MailFilter.txt new file mode 100644 index 0000000..5526239 --- /dev/null +++ b/doc/wiki/Plugins.MailFilter.txt @@ -0,0 +1,80 @@ +Mail filter plugin +================== + +Mail filter plugin can be used to filter written and/or read mails via a +script, for example to encrypt/decrypt mails. Currently the filtering must not +modify the message in any way: mail -> write filter -> read filter -> must +produce exactly the original mail back. (TODO: Modifying the mail during +writing would be possible with some code changes.) + +Note that IMAP protocol requires that emails never change, so the read filter +must always produce the same output for the message. If the output changes +you'll probably see some errors about Dovecot's cache file being corrupted and +the IMAP client may also become confused if it has already cached some of the +mail data. + +Configuration +------------- + +Add to 'dovecot.conf': + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins mail_filter + +plugin { + # Read filter: + mail_filter = mail-filter %u # %u = username given to the script as first +parameter + # Write filter: + mail_filter_out = mail-filter-out %u +} + +service mail-filter { + executable = script /usr/local/bin/mail-filter.sh + user = dovecot # run unprivileged + unix_listener mail-filter { + # enough permissions to give imap/pop3/etc processes access to this socket + mode = 0600 + user = vmail + } +} +service mail-filter-out { + executable = script /usr/local/bin/mail-filter-out.sh + user = dovecot # run unprivileged + unix_listener mail-filter { + # enough permissions to give imap/pop3/etc processes access to this socket + mode = 0600 + user = vmail + } +} +---%<------------------------------------------------------------------------- + +Example scripts +--------------- + +Here's a minimal example of how gpg could be used to encrypt and decrypt mails. +All the key handling details are left out. + +The mail is read from stdin and written to stdout. Note that the plugin +currently can't handle asynchronously reading+writing data, so the script +cannot write any data to stdout before it has read everything from stdin. This +is most easily done by first saving the stdin to a temporary file. + +'mail-filter.sh': + +---%<------------------------------------------------------------------------- +cat > tempfile +gpg -d tempfile +rm -f tempfile +---%<------------------------------------------------------------------------- + +'mail-filter-out.sh': + +---%<------------------------------------------------------------------------- +USER=$1 +cat > tempfile +gpg -e -r $USER tempfile +rm -f tempfile +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.MailLog.txt b/doc/wiki/Plugins.MailLog.txt new file mode 100644 index 0000000..4ee9eea --- /dev/null +++ b/doc/wiki/Plugins.MailLog.txt @@ -0,0 +1,54 @@ +Mail logger plugin +================== + +This plugin can be used to log several actions done in a mail session: + + * Setting and removing \Deleted flag + * Expunging (includes autoexpunge) + * Copying mails to another mailbox + * Mailbox creations + * Mailbox deletions + * Mailbox renames + * Any flag changes + * Saves + +Messages' UID and Message-ID header is logged for each action. Here's an +example: + +---%<------------------------------------------------------------------------- +imap(user): copy -> Trash: uid=908, msgid=<123.foo@bar> +imap(user): delete: uid=908, msgid=<123.foo@bar> +imap(user): expunged: uid=908, msgid=<123.foo@bar> +---%<------------------------------------------------------------------------- + +You can enable the plugin globally for all services by setting: + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins mail_log notify +---%<------------------------------------------------------------------------- + +The notify plugin is required for the mail_log plugin's operation, so be +certain it's also enabled. + +Configuration +------------- + +You can configure what and how mail_log plugin logs: + +---%<------------------------------------------------------------------------- +plugin { + # Events to log. Defined in src/plugins/mail-log/mail-log-plugin.c - also +available: flag_change save mailbox_create + # autoexpunge is included in expunge + mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename + + # Also available: Defined in src/plugins/mail-log/mail-log-plugin.c - flags +vsize from subject + mail_log_fields = uid box msgid size + + # Don't log fields that require opening the email (v2.2.28+). + #mail_log_cached_only = yes +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.MailboxAlias.txt b/doc/wiki/Plugins.MailboxAlias.txt new file mode 100644 index 0000000..465c170 --- /dev/null +++ b/doc/wiki/Plugins.MailboxAlias.txt @@ -0,0 +1,46 @@ +Mailbox alias plugin +==================== + +Requires v2.1.10+ (and currently mailbox_list_index=no). This plugin can be +used to configure mailbox aliases, which on the filesystem level are symlinks +to other mailboxes. This doesn't magically solve the problem of showing clients +e.g. multiple Sent mailboxes, but it can be used to make sure that all of the +different variants will have the same mails in them. Unfortunately it also +means that some clients will download the same mails to local cache multiple +times. + +The way it works is that: + + * The aliases won't be visible until the mailbox is CREATEd + * When alias is CREATEd, a symlink is created to the original mailbox. If the + original mailbox didn't exist yet, it's also created. + * If a mailbox with the same name as alias was already created before this + plugin was enabled, its behavior won't change unless it's deleted. + * When alias is DELETEd, the symlink is removed without deleting any of the + mails. + * The original mailbox can't be DELETEd while it still has aliases. + * Mailbox can't be RENAMEd if it's an alias or if it has aliases. + +Example configuration where "Sent" is the real mailbox and it has aliases "Sent +Messages" and "Sent Items": + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins mailbox_alias +plugin { + mailbox_alias_old = Sent + mailbox_alias_new = Sent Messages + mailbox_alias_old2 = Sent + mailbox_alias_new2 = Sent Items +} + +# Usually you want the Sent mailbox to be autocreated and advertised as +SPECIAL-USE \Sent: +namespace inbox { + mailbox Sent { + auto = create # or subscribe + special_use = \Sent + } +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Notify.txt b/doc/wiki/Plugins.Notify.txt new file mode 100644 index 0000000..a0a7ac6 --- /dev/null +++ b/doc/wiki/Plugins.Notify.txt @@ -0,0 +1,9 @@ +Notify plugin +============= + +The notify plugin can be used to easily develop other plugins that need to do +some work when something in user's mailboxes change. See <mail_log> +[Plugins.MailLog.txt] plugin as an example how to develop a plugin based on the +notify plugin. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.NotifyStatus.txt b/doc/wiki/Plugins.NotifyStatus.txt new file mode 100644 index 0000000..2fbce16 --- /dev/null +++ b/doc/wiki/Plugins.NotifyStatus.txt @@ -0,0 +1,69 @@ +notify_status plugin (v2.2.33+) +=============================== + +This plugin updates a dict every time a mailbox changes. The status can contain +total message count and unseen count. It will update key priv/status/<mailbox +name>. See <Dictionary.txt> for how to configure dict. + +Configuration +------------- + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins notify notify_status + +plugin { + notify_status_dict = <dict uri> # For example proxy:dict-async:notify_status + + # Value written to dict: + #notify_status_value = {"messages":%%{messages},"unseen":%%{unseen}} + + # By default all mailboxes are added to dict. This can be limited with: + #notify_status_mailbox = INBOX + #notify_status_mailbox2 = pattern2/* + #... +} +---%<------------------------------------------------------------------------- + +If SQL dict is used, you can use this map: + +---%<------------------------------------------------------------------------- +map { + pattern = priv/status/$box + table = mailbox_status + value_field = status + username_field = username + + fields { + mailbox = $box + } +} +---%<------------------------------------------------------------------------- + +The matching SQL schema is: + +---%<------------------------------------------------------------------------- +CREATE TABLE mailbox_status ( + username VARCHAR(255) NOT NULL, + mailbox VARCHAR(255) NOT NULL, + status VARCHAR(255), + PRIMARY KEY (username, mailbox) +); +---%<------------------------------------------------------------------------- + +Supported fields +---------------- + +These fields can be used as %%{variables} in notify_status_value setting: + + * username - Username (user@domain) + * mailbox - name of mailbox + * messages - n. of messages + * unseen - n. unseen message + * recent - n. recent messages (not accurate) + * uidvalidity - current UID validity + * uidnext - predicted next UID value + * first_recent_uid - first recent UID + * highest_modseq - higest modification sequence number + * highest_pvt_modseq - highest private modification sequence number + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.PushNotification.txt b/doc/wiki/Plugins.PushNotification.txt new file mode 100644 index 0000000..ba1e240 --- /dev/null +++ b/doc/wiki/Plugins.PushNotification.txt @@ -0,0 +1,505 @@ +Push Notification Framework +=========================== + +Contents + + + 1. Push Notification Framework + + 1. Introduction + + 2. Usage + + 3. Drivers + + 1. DLOG (Debug log) + + 2. OX (Open-Xchange) driver + + 1. Metadata + + 2. Example Payload + + 3. Lua + + 1. Configuration + + 2. Example script + + 1. Simple example + + 2. Example with event code (v2.3.4+) + + 3. Overview + + 4. Transactions + + 5. Mailbox events + + 6. Message events + +Introduction +------------ + +Dovecot's Push Notification framework exposes RFC 5423 (Internet Message Store +Events) [https://tools.ietf.org/html/rfc5423] events that occur in Dovecot to a +system that can be used to report these events to external services. + +These events are available within the notification framework, although a driver +may not implement all: + + * FlagsClear + * FlagsSet + * MailboxCreate + * MailboxDelete + * MailboxRename + * MailboxSubscribe + * MailboxUnsubscribe + * MessageAppend + * MessageExpunge + * MessageNew + * MessageRead + * MessageTrash + +These events are not supported by the notification framework: + + * Login (handled by <Authentication.txt>) + * Logout (handled by <Authentication.txt>) + * QuotaExceed (handled by <Quota.txt>) + * QuotaWithin (handled by <Quota.txt>) + +Usage +----- + +To use push notifications, both the "notify" and the "push_notification" +plugins need to be activated. For LMTP delivery, this is required: + +---%<------------------------------------------------------------------------- +protocol lmtp { + mail_plugins = $mail_plugins notify push_notification +} +---%<------------------------------------------------------------------------- + +If you also want push notifications to work for LDA-based delivery, you would +need additional configuration: + +---%<------------------------------------------------------------------------- +protocol lda { + mail_plugins = $mail_plugins notify push_notification +} +---%<------------------------------------------------------------------------- + +Drivers +------- + +DLOG (Debug log) +---------------- + +---%<------------------------------------------------------------------------- +plugin { + push_notification_driver = dlog +} +---%<------------------------------------------------------------------------- + +This will cause notifications to end up in your debug log. + +OX (Open-Xchange) driver +------------------------ + + The OX backend supports sending notifications on MessageNew events (i.e. mail +deliveries, not IMAP APPENDs). + +The HTTP end-point (URL + authentication information) to use is configured in +the Dovecot configuration file. The appropriate configuration options will +contain the HTTP URL denoting the end-point to connect to as well as the +authentication information for Basic Authentication as configured by properties +"com.openexchange.rest.services.basic-auth.login" and +"com.openexchange.rest.services.basic-auth.password". The URL to configure in +Dovecot configuration follows this pattern. + +---%<------------------------------------------------------------------------- +<http|https> + "://" + <login> + ":" + <password> + "@" + <host> + ":" + <port> ++ "/preliminary/http-notify/v1/notify" +---%<------------------------------------------------------------------------- + +E.g. + +---%<------------------------------------------------------------------------- +plugin { + push_notification_driver = +ox:url=http://login:pass@node1.domain.tld:8009/preliminary/http-notify/v1/notify +} +---%<------------------------------------------------------------------------- + +For HTTPS endpoints, system CAs are trusted by default, but internal CAs might +need further configuration. + +Furthermore, it is also possible to specify more than one HTTP end-point to +connect to if a new message delivery occurs. Thus the configuration section +mentioned above may be extended by additional "push_notification_driver" +entries; e.g. push_notification_driver2, push_notification_driver3, etc. + +Please note that the path "/preliminary/http-notify/v1/notify" denotes the +internal REST API of the Open-Xchange Middleware, which is not publicly +accessible. The administrator can decide whether to add that path to the Apache +configuration (see alsoAppSuite:Apache_Configuration and AppSuite:Grizzly) +through a Location/ProxyPass directive: + +---%<------------------------------------------------------------------------- +<Location /preliminary> + Order Deny,Allow + Deny from all + # Only allow access from servers within the network. Do not expose this + # location outside of your network. In case you use a load balancing service +in front + # of your Apache infrastructure you should make sure that access to +/preliminary will + # be blocked from the internet / outside clients. Examples: + # Allow from 192.168.0.1 + # Allow from 192.168.1.1 192.168.1.2 + # Allow from 192.168.0. + ProxyPass /preliminary balancer://oxcluster/preliminary +</Location> +---%<------------------------------------------------------------------------- + +In case the "user=" sent by OX in the push_notification_driver url data does +not match the IMAP login of a user, Dovecot ignores it. This can be overridden +by defining "user_from_metadata" in the push_notification_driver url, e.g. + +---%<------------------------------------------------------------------------- +push_notification_driver = ox:url=http://example.com/ user_from_metadata +---%<------------------------------------------------------------------------- + +Metadata +-------- + +The push notifications are enabled separately for each user using METADATA. +Normally <AppSuite.txt> does this internally, but for e.g. testing purposes you +can do this yourself: + +---%<------------------------------------------------------------------------- +doveadm mailbox metadata set -u user@example.com -s "" +/private/vendor/vendor.dovecot/http-notify user=11@3 +---%<------------------------------------------------------------------------- + +Example Payload +--------------- + +See +https://github.com/dovecot/core/blob/master/src/plugins/push-notification/push-notification-driver-ox.c. + +Push notification sent in JSON format with the following fields: + + * *user*: User identifier (string) + * *event*: RFC 5423 event type (string; currently only "'MessageNew'") + * *folder*: Mailbox name (string) + * *imap-uidvalidity*: RFC 3501 UIDVALIDITY value of the mailbox (integer) + * *imap-uid*: UID of the message, if applicable (integer) + * *from*: RFC 2822 address of the message sender, if applicable (string) + * *subject*: Subject of the message, if applicable (string) + * *snippet*: Snippet of the message body, if applicable (string) + * *unseen*: RFC 3501 UNSEEN value of the mailbox (integer) + +---%<------------------------------------------------------------------------- +Content-Type: application/json; charset=utf-8 + +{ + "user": "4@464646669", + "imap-uidvalidity": 123412341, + "imap-uid": 2345, + "folder": "INBOX", + "event": "MessageNew", + "from": "alice@barfoo.org", + "subject": "Test", + "snippet": "Hey guys\nThis is only a test...", + "unseen": 2 +} +---%<------------------------------------------------------------------------- + +Lua +--- + +Since v2.3.4+ you can use Lua to write custom push notification handlers. See +<Design.Lua.txt> for general information about Lua in Dovecot. If you have +mail_lua_script (a global script for storage) it will be used if no script is +specified. + +Configuration +------------- + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins mail_lua notify push_notification +push_notification_lua + +plugin { + push_notification_driver = lua:file=/path/to/lua/script + # you can omit the script name if you want to use mail_lua_script script +instead + #mail_lua_script=/path/to/common/script.lua +} +---%<------------------------------------------------------------------------- + +Example script +-------------- + +Simple example +-------------- + +---CodeArea------------------------------------------------------------------- +-- To use 1 +-- 2 +-- plugin { 3 +-- push_notification_driver = lua:file=/home/cmouse/empty.lua 4 +-- push_lua_url = http://push.notification.server/handler 5 +-- } 6 +-- 7 +-- server is sent a POST message to given url with parameters 8 +-- 9 + 10 +local http = require("socket.http") 11 +local url = require("socket.url") 12 + 13 +function table_get(t, k, d) 14 + return t[k] or d 15 +end 16 + 17 +function dovecot_lua_notify_begin_txn(user) 18 + return {messages={}, ep=user:plugin_getenv("push_lua_url"), +username=user.username} 19 +end 20 + 21 +function dovecot_lua_notify_end_txn(ctx, success) 22 + local i, msg = next(ctx["messages"], nil) 23 + while i do 24 + local r, c = http.request(ctx["ep"], "from=" .. url.escape(table_get(msg, +"from", "")) .. "&to=" .. url.escape(table_get(msg, "to", "")) .. "&subject=" +..url.escape(table_get(msg, "subject", "")) .. "&snippet=" .. +url.escape(table_get(msg, "snippet", "")) .. "&user=" .. +url.escape(ctx["username"])) 25 + if r and c/100 ~= 2 then 26 + dovecot.i_error("lua-push: Remote error " .. tostring(c) .. " handling +push notication") 27 + end 28 + if r == nil then 29 + dovecot.i_error("lua-push: " .. c) 30 + end 31 + i, msg = next(ctx["messages"], i) 32 + end 33 +end 34 + 35 +function dovecot_lua_notify_event_message_append(ctx, event) 36 + table.insert(ctx["messages"], event) 37 +end 38 + 39 +function dovecot_lua_notify_event_message_new(ctx, event) 40 + table.insert(ctx["messages"], event) 41 +end 42 +---CodeArea------------------------------------------------------------------- + +Example with event code (v2.3.4+) +--------------------------------- + +---CodeArea------------------------------------------------------------------- +-- To use 1 +-- 2 +-- plugin { 3 +-- push_notification_driver = lua:file=/home/cmouse/empty.lua 4 +-- push_lua_url = http://push.notification.server/handler 5 +-- } 6 +-- 7 +-- server is sent a POST message to given url with parameters 8 +-- 9 + 10 +local http = require "socket.http" 11 +local ltn12 = require "ltn12" 12 +local url = require "socket.url" 13 + 14 +function table_get(t, k, d) 15 + return t[k] or d 16 +end 17 + 18 +function script_init() 19 + return 0 20 +end 21 + 22 +function dovecot_lua_notify_begin_txn(user) 23 + return {user=user, event=dovecot.event(), +ep=user:plugin_getenv("push_lua_url"), states={}, messages={}} 24 +end 25 + 26 +function dovecot_lua_notify_event_message_new(ctx, event) 27 + -- get mailbox status 28 + local mbox = ctx.user:mailbox(event.mailbox) 29 + mbox:sync() 30 + local status = mbox:status(dovecot.storage.STATUS_RECENT, +dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) 31 + mbox:free() 32 + ctx.states[event.mailbox] = status 33 + table.insert(ctx.messages, +{from=event.from,subject=event.subject,mailbox=event.mailbox}) 34 +end 35 + 36 +function dovecot_lua_notify_event_message_append(ctx, event, user) 37 + dovecot_lua_notify_event_message_new(ctx, event, user) 38 +end 39 + 40 +function dovecot_lua_notify_end_txn(ctx) 41 + -- report all states 42 + for i,msg in ipairs(ctx.messages) do 43 + local e = dovecot.event(ctx.event) 44 + e:set_name("lua_notify_mail_finished") 45 + reqbody = "mailbox=" .. url.escape(msg.mailbox) .. "&from=" .. +url.escape(table_get(msg, "from", "")) .. "&subject=" .. +url.escape(table_get(msg, "subject", "")) 46 + e:log_debug(ctx.ep .. " - sending " .. reqbody) 47 + res, code = http.request({method="POST", 48 + url=ctx.ep, 49 + source=ltn12.source.string(reqbody), 50 + headers={ 51 + ["content-type"] = "application/x-www-form-url.escaped", +52 + ["content-length"] = tostring(#reqbody) 53 + } 54 + }) 55 + e:add_int("result_code", code) 56 + e:log_info("Mail notify status " .. tostring(code)) 57 + end 58 + for box,state in pairs(ctx.states) do 59 + local e = dovecot.event() 60 + e:set_name("lua_notify_mailbox_finished") 61 + reqbody = "mailbox=" .. url.escape(state.mailbox) .. "&recent=" .. +tostring(state.recent) .. "&unseen=" .. tostring(state.unseen) .. "&messages=" +.. tostring(state.messages) 62 + e:log_debug(ctx.ep .. " - sending " .. reqbody) 63 + res, code = http.request({method="POST", 64 + url=ctx.ep, 65 + source=ltn12.source.string(reqbody), 66 + headers={ 67 + ["content-type"] = "application/x-www-form-url.escaped", +68 + ["content-length"] = tostring(#reqbody) 69 + } 70 + }) 71 + e:add_int("result_code", code) 72 + e:log_info("Mailbox notify status " .. tostring(code)) 73 + end 74 +end 75 +---CodeArea------------------------------------------------------------------- + +Overview +-------- + +The Lua driver hooks into all events, and calls matching functions when found +in Lua script. + +Currently it supports + + * mailbox create, delete, rename, subscribe and unsubscribe + * message new, append, expunge, read and trash, flags set, flags clear + +All events are called within a transaction. The event is called with context +and an event table, which contains the event parameters.All events contain at +least + + * name - name of the event + * user - current mail user + +Events are always called after the fact. + +There has to be at least one event handler, or the transaction begin and end +functions are never called.This is optimization to avoid roundtrip to Lua when +it's not needed. + +Transactions +------------ + + * dovecot_lua_notify_begin_txn(user) + +Start transaction. Return value is used as transaction context and is treated +as opaque value by Lua driver. The user parameter is mail_user object. + + * dovecot_lua_notify_end_txn(context, success) + +End transaction, context is unreferenced. + +Mailbox events +-------------- + +All mailbox events contain 'mailbox' parameter, which is the name of the +affected mailbox. + + * dovecot_lua_notify_event_mailbox_create(context, {name, mailbox}) + +Called when mailbox has been created. + + * dovecot_lua_notify_event_mailbox_delete(context, {name, mailbox}) + +Called when mailbox has been deleted. + + * dovecot_lua_notify_event_mailbox_rename(context, {name, mailbox, + mailbox_old}) + +Called when mailbox has been renamed, old name is retained in mailbox_old +attribute. + + * dovecot_lua_notify_event_mailbox_subscribe(context, {name, mailbox}) + +Called when mailbox has been subscribed to. The mailbox does not necessarily +exist. + + * dovecot_lua_notify_event_mailbox_unsubscribe(context, {name, mailbox}) + +Called when mailbox has been unsubscribed from. The mailbox does not +necessarily exist. + +Message events +-------------- + +All message events contain following parameters ++--------------+----------------------+ +| mailbox | Mailbox name | ++--------------+----------------------+ +| uid | Message UID | ++--------------+----------------------+ +| uid_validity | Mailbox UID validity | ++--------------+----------------------+ + + * dovecot_lua_notify_event_message_new(context, {name, mailbox, uid, + uid_validity, date, tz, from, to, subject, snippet}) + +Called when message is delivered. + + * dovecot_lua_notify_event_message_append(context, {name, mailbox, uid, + uid_validity, from, to, subject, snippet}) + +Called when message is APPENDed to a mailbox. + + * dovecot_lua_notify_event_message_read(context, {name, mailbox, uid, + uid_validity}) + +Called when message is marked as Seen. + + * dovecot_lua_notify_event_message_trash(context, {name, mailbox, uid, + uid_validity}) + +Called when message is marked Deleted. + + * dovecot_lua_notify_event_message_expunge(context, {name, mailbox, uid, + uid_validity}) + +Called when message is EXPUNGEd. + + * dovecot_lua_notify_event_flags_set(context, {name, mailbox, uid, + uid_validity, flags, keywords_set}) + +Called when message flags or keywords are set. flags is a bitmask. keywords_set +is a table of strings of the keywords set by the event. + + * dovecot_lua_notify_event_flags_clear(context, {name, mailbox, uid, + uid_validity, flags, keywords_clear, keywords_old}) + +Called when message flags or keywords are removed. flags is a bitmask. +keywords_clear contains the keywords cleared, keywords_old is the table of +keywords that were set before the event. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.QuotaClone.txt b/doc/wiki/Plugins.QuotaClone.txt new file mode 100644 index 0000000..56325f7 --- /dev/null +++ b/doc/wiki/Plugins.QuotaClone.txt @@ -0,0 +1,37 @@ +Quota Clone Plugin +================== + +(Requires v2.2.17+) + +Quota clone plugin is useful when you want to store everybody's current quota +usage to a database, but you don't want to use the database as the +authoritative quota database. For example you might want to access everybody's +quota via Redis (or with SQL database, Redis is example here), but you don't +store the Redis database permanently so it could become empty once in a while. +In this case you can use e.g. <dict-file> [Quota.Dict.txt] or <count> +[Quota.Count.txt] as the authoritative quota database and make a copy of the +quota usage to Redis. From Redis you could then once in a while gather +everybody's current quota usage and send it to yet another place (e.g. for +statistics handling). + +Every time quota is updated, the value is updated to the cloned dict. There are +race conditions with it so the quota may not always be 100% correct. The old +value is always replaced with the new one though (not just +incremented/decremented) so the cloned quota is never too much wrong. + +The keys that are written to: + + * priv/quota/storage - storage usage in bytes + * priv/quota/messages - count of messages + +Configuration +------------- + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins quota quota_clone +plugin { + quota_clone_dict = redis:host=127.0.0.1:port=6379 +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Snarf.txt b/doc/wiki/Plugins.Snarf.txt new file mode 100644 index 0000000..a53e9ba --- /dev/null +++ b/doc/wiki/Plugins.Snarf.txt @@ -0,0 +1,63 @@ +Snarf Plugin +============ + +Requires v2.0.8+. + +This plugin can be used to move mails from a "snarf mailbox" to user's real +INBOX. For example from '/var/mail/user' mbox file to user's final INBOX (e.g. +'~/mbox' or '~/Maildir'). Dovecot moves as many messages as it can until it +runs out of messages or out of quota. + +Configuration +------------- + +You need a default and a snarf namespace, for example: + +---%<------------------------------------------------------------------------- +#mail_location = mbox:~/mail/:INBOX=~/mbox +#mail_location = maildir:~/Maildir + +namespace default { + prefix = + separator = / + inbox = yes +} +namespace snarf { + prefix = /snarf + separator = / + location = mbox:/var/run/dovecot/empty:INBOX=/var/mail/%u:INDEX=MEMORY + list = no + hidden = yes +} +---%<------------------------------------------------------------------------- + +Then tell snarf to get the mails from '/snarf/INBOX': + +---%<------------------------------------------------------------------------- +plugin { + snarf = /snarf/INBOX +} +---%<------------------------------------------------------------------------- + +Use hierarchy separator as the first character in snarf namespace prefix to +make sure that there won't be any conflicts with user's mailbox names. + +UW-IMAP style optional snarfing +------------------------------- + +UW-IMAP does snarfing from '/var/mail/user' to '~/mbox' file, but only if +'~/mbox' exists. Normally Dovecot does the snarfing always if it's enabled. To +enable it only optionally, set also: + +---%<------------------------------------------------------------------------- +# keep mail_location etc. settings as above + +plugin { + mbox_snarf = ~/mbox +} +---%<------------------------------------------------------------------------- + +Now the snarfing is done only if '~/mbox' file exists. If it's not, the +'/var/mail/user' is used directly as the INBOX. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Stats.txt b/doc/wiki/Plugins.Stats.txt new file mode 100644 index 0000000..b9f67ad --- /dev/null +++ b/doc/wiki/Plugins.Stats.txt @@ -0,0 +1,2 @@ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Trash.txt b/doc/wiki/Plugins.Trash.txt new file mode 100644 index 0000000..a37905d --- /dev/null +++ b/doc/wiki/Plugins.Trash.txt @@ -0,0 +1,41 @@ +Trash Plugin +============ + +This requires <quota plugin> [Quota.txt] to be loaded and configured to use +non- <FS quota> [Quota.FS.txt]. + +Normally if a message can't be saved/copied because it would bring user over +quota, the save/copy fails with "Quota exceeded" error. The trash plugin can be +used to avoid such situations by making Dovecot automatically expunge oldest +messages from configured mailboxes until the message can be saved. If the new +message is large enough that it wouldn't fit even if all messages from +configured mailboxes were expunged, then none are and user gets the "Quota +exceeded" error. + +The configuration file is a text file where each line is in format: '<priority> +<mailbox name>'. Mails are deleted in lowest -> highest priority number order. + +dovecot.conf: + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins quota trash + +plugin { + trash = /etc/dovecot/dovecot-trash.conf.ext +} +---%<------------------------------------------------------------------------- + +dovecot-trash.conf.ext: + +---%<------------------------------------------------------------------------- +# Spam mailbox is emptied before Trash +1 Spam +# Trash mailbox is emptied before Sent +2 Trash +# If both Sent and "Sent Messages" mailboxes exist, the next oldest message +# to be deleted is looked up from both of the mailboxes. +3 Sent +3 Sent Messages +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.VarExpandCrypt.txt b/doc/wiki/Plugins.VarExpandCrypt.txt new file mode 100644 index 0000000..4bf43d9 --- /dev/null +++ b/doc/wiki/Plugins.VarExpandCrypt.txt @@ -0,0 +1,45 @@ +var_expand_crypt plugin +======================= + +This plugins provides generic encrypt/decrypt facility for var_expand. It +requires functional lib-dcrypt backend. For dovecot-auth process this plugin is +automatically usable. + +Syntax +------ + +---%<------------------------------------------------------------------------- +args=encrypted_value=%{encrypt;key=value,iv=value,noiv=yes,algo=algorithm,format=base64|hex:field} +args=decrypted_value=%{decrypt;key=value,iv=value,noiv=yes,algo=algorithm,format=base64|hex:field} +---%<------------------------------------------------------------------------- + + * key - hex-encoded value + * iv - hex-encoded value + * noiv - whether iv is included in return value + * algo - algorithm name (defaults to aes-256-cbc) + * format - return format + +decrypt expects input in base64 or hex format. + +NOTE: It is usually best to leave iv management to dovecot, and not use iv and +noiv options at all. + +Return formats +-------------- + +Without noiv encrypt returns iv$encrypted$. With noiv, just encrypted data is +returned. Field(s) are encoded using format. + +key and iv must be the length required by the given algorithm. + +Example +------- + +---%<------------------------------------------------------------------------- +%{encrypt;key=f1f2f3f4f5f6f7f8f1f2f3f4f5f6f7f8f1f2f3f4f5f6f7f8f1f2f3f4f5f6f7f8:password} += 93736a0f910df27f89210e096e1d639a$966c2b4f3e7487f6acdb836f8d1dc3e0$ +%{decrypt;key=f1f2f3f4f5f6f7f8f1f2f3f4f5f6f7f8f1f2f3f4f5f6f7f8f1f2f3f4f5f6f7f8:encrypted} += pass +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Virtual.txt b/doc/wiki/Plugins.Virtual.txt new file mode 100644 index 0000000..67f5e06 --- /dev/null +++ b/doc/wiki/Plugins.Virtual.txt @@ -0,0 +1,284 @@ +Virtual mailboxes +================= + +First, you'll have to load the plugin: + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins virtual +---%<------------------------------------------------------------------------- + +Namespace configuration +----------------------- + +Then, you'll have to create a <namespace> [Namespaces.txt] for the virtual +mailboxes, for example: + +---%<------------------------------------------------------------------------- +namespace { + prefix = virtual/ + separator = / + location = virtual:~/Maildir/virtual +} +---%<------------------------------------------------------------------------- + +After this you can create virtual mailboxes under '~/Maildir/virtual'. By +default it uses the "fs" layout, so you can create directories such as: + + * INBOX: '~/Maildir/virtual/INBOX/' + * Sub/mailbox: '~/Maildir/virtual/Sub/mailbox/' + +If you prefer to use Maildir++ layout instead, you can simply append +':LAYOUT=maildir++' to the location. + +Virtual mailbox configuration +----------------------------- + +For each virtual directory you need to create a 'dovecot-virtual' file. Its +syntax is like: + +---%<------------------------------------------------------------------------- +<1+ mailbox patterns> + <search program> +[<more mailbox patterns> + <search program for these mailboxes> +[etc..]] +---%<------------------------------------------------------------------------- + +Mailbox patterns can contain IMAP LIST-compatible +[http://tools.ietf.org/html/rfc3501#section-6.3.8] "*" and "%" wildcards. They +are currently evaluated only when the virtual mailbox is being selected, so if +more mailboxes are created during that they aren't noticed. + +"*" wildcard matches only one namespace at a time based on the namespace +prefix. For example if you have namespaces with an empty prefix and a prefix +"mail/": + + * '*' matches only mailboxes from the namespace with empty prefix + * 'mail*' matches mailboxes beginning with name "mail" from the namespace with + empty prefix + * 'mail/*' matches only mailboxes from the mail/ namespace + +Beware that "*" will not match any mailbox which already has a more specialised +match! + +The mailbox names have special prefixes: + + * '-': Don't include this mailbox. + * '+': Drop \Recent flags from the backend mailbox when opening it. + * '!': Save new mails to this mailbox (see below). + +If you need to actually include a mailbox name that contains such prefix, you +can currently just kludge it by using '+' prefix (if you don't care about the +\Recent flags) and adding the mailbox name after that (e.g. "+-box"). + +Search program is compatible with IMAP SEARCH command +[http://tools.ietf.org/html/rfc3501#section-6.4.4]. Besides the standard SEARCH +key you may want to use X-MAILBOX key which matches the message's original +mailbox. Note the leading whitespace in front of search specifications. + +Saving mails to virtual mailboxes +--------------------------------- + +It's possible to configure virtual mailbox so that it's possible to save/copy +messages there. This is done by specifying a single physical mailbox where the +message is really saved by prefixing it with '!', e.g.: + +---%<------------------------------------------------------------------------- +!INBOX +work/* + unseen +---%<------------------------------------------------------------------------- + +Note however that nothing guarantees that the saved mail will actually show up +in the virtual mailbox. If a message was saved with \Seen flag to the above +virtual mailbox, it wouldn't show up there. This also means it's problematic to +support IMAP UIDPLUS extension for virtual mailboxes, and currently Dovecot +doesn't even try (no [APPENDUID] or [COPYUID] is sent to client). + +The !-prefixed virtual mailbox is also selected from; you don't need to list it +again without an ! or you'll get two copies of your messages in the virtual +mailbox. + +Sieve filters with virtual mailboxes +------------------------------------ + +Using the sieve plugin with virtual mailboxes will cause dovecot to output a +fatal exception error in it's logs and crash. This is because sieve can't tell +the difference between a virtual location and a maildir/mbox location due to +the way it detects actions in the mailboxes. + +If you use virtual mailboxes that are configured in sieve, make sure that they +point to the namespace which has a maildir/mbox location and a unique prefix. +If you don't, sieve will crash trying to copy a message to a virtual mailbox. + +Mailbox selection base on metadata (v2.2.22+) +--------------------------------------------- + +Instead of a mailbox name, you can specify a metadata filter: + +---%<------------------------------------------------------------------------- +[-]/<metadata-entry-name>:<value-wildcard> +---%<------------------------------------------------------------------------- + +There can be multiple metadata entries. All the entries must match. + +For example: + +---%<------------------------------------------------------------------------- +* +/private/vendor/vendor.dovecot/virtual:* +-/private/vendor/vendor.dovecot/virtual:ignore + all +---%<------------------------------------------------------------------------- + +This matches all mailboxes, which contain a virtual METADATA entry that has any +value except "ignore". + +Examples +-------- + + * List all messages with \Deleted flag in all mailboxes: + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/Trash/dovecot-virtual +* + deleted +---%<------------------------------------------------------------------------- + + * List all unseen INBOX and work/* messages: + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/unseen/dovecot-virtual +INBOX +work/* + unseen +---%<------------------------------------------------------------------------- + + * Create a GMail-style "conversation view" for INBOX which shows all threads + that have messages in INBOX, but shows all messages in the thread regardless + of in what mailbox they physically exist in: + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/all/dovecot-virtual +* + all +---%<------------------------------------------------------------------------- + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/INBOX/dovecot-virtual +virtual/all + inthread refs x-mailbox INBOX +---%<------------------------------------------------------------------------- + + * Create a mailbox containing messages from all mailboxes except Trash and its + children: + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/all/dovecot-virtual +* +-Trash +-Trash/* + all +---%<------------------------------------------------------------------------- + + * Create a virtual Sentmail folder that includes Sent*: + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/Sentmail/dovecot-virtual +Sent* + all +---%<------------------------------------------------------------------------- + + * List messages from past 48 hours (syntax is in seconds): + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/recent/dovecot-virtual +INBOX +work/* + all younger 172800 +---%<------------------------------------------------------------------------- + + * List unseen messages from foo and flagged messages from all mailboxes + (including foo): + +---%<------------------------------------------------------------------------- +# ~/Maildir/virtual/example/dovecot-virtual +foo + or unseen flagged +* + flagged +---%<------------------------------------------------------------------------- + +Virtual POP3 INBOX +------------------ + +If you want POP3 INBOX to contain some or all mailboxes, you can do this in the +following way: + +Namespace configuration: + +---%<------------------------------------------------------------------------- +# The default namespace that is visible to IMAP clients +namespace inbox { + prefix = + separator = / + list = yes +} + +# Virtual namespace for the virtual INBOX. Use a global directory for +dovecot-virtual files. +namespace virtual { + prefix = virtual/ + separator = / + location = virtual:/etc/dovecot/virtual:INDEX=~/Maildir/virtual + list = no + hidden = yes +} + +# Copy of the inbox namespace. We'll use this in dovecot-virtual file. +namespace real { + prefix = RealMails/ + separator = / + list = no + hidden = yes +} +---%<------------------------------------------------------------------------- + +Note that none of the namespaces have inbox=yes. This is because for IMAP users +you want the inbox namespace to have inbox=yes, but for POP3 users you want the +virtual namespace to have inbox=yes. This requires setting the inbox=yes in +<userdb extra fields> [UserDatabase.ExtraFields.txt]. For example with MySQL +you can can do this like: + +---%<------------------------------------------------------------------------- +user_query = SELECT ..., \ + CASE '%s' WHEN 'pop3' THEN NULL ELSE 'yes' END AS 'namespace/inbox/inbox', \ + CASE '%s' WHEN 'pop3' THEN 'yes' ELSE NULL END AS 'namespace/virtual/inbox' \ + WHERE ... +---%<------------------------------------------------------------------------- + +Finally specify what the virtual INBOX looks like for POP3 users: + +'/etc/dovecot/virtual/INBOX/dovecot-virtual' : + +---%<------------------------------------------------------------------------- +RealMails +RealMails/* +-RealMails/Trash +-RealMails/Trash/* +-RealMails/Spam + all +---%<------------------------------------------------------------------------- + +You'll have to use the RealMails/ prefix if you want to use "*" wildcard, +otherwise it would match INBOX, which in turn would again lead to the virtual +INBOX and that would create a loop. + +Also to avoid accidental POP3 UIDL changes, you shouldn't base the UIDLs on +IMAP UIDs. Instead use for GUIDs (with maildir the same as base filename): + +---%<------------------------------------------------------------------------- +pop3_uidl_format = %g +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Welcome.txt b/doc/wiki/Plugins.Welcome.txt new file mode 100644 index 0000000..e5da2f6 --- /dev/null +++ b/doc/wiki/Plugins.Welcome.txt @@ -0,0 +1,27 @@ +Welcome plugin +============== + +Requires v2.2.25+. Call a script when the user logs in for the first time. This +is specifically done when the INBOX is (auto)created. The scripts are called +similarly to <quota warning scripts> [Quota.Configuration.txt]. + +---%<------------------------------------------------------------------------- +plugin { + welcome_script = welcome %u + # By default we run the script asynchronously, but with this option we + # wait for the script to finish. + #welcome_wait = yes +} + +service welcome { + executable = script /usr/local/bin/welcome.sh + user = dovecot + unix_listener welcome { + user = vmail + } +} + +mail_plugins = $mail_plugins welcome +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.Zlib.txt b/doc/wiki/Plugins.Zlib.txt new file mode 100644 index 0000000..9b38af2 --- /dev/null +++ b/doc/wiki/Plugins.Zlib.txt @@ -0,0 +1,100 @@ +Zlib plugin +=========== + +Zlib plugin can be used to read compressed mbox, maildir or dbox files. It can +be also used to write(via IMAP, <LDA.txt> and/or <LMTP.txt>) compressed +messages to <dbox> [MailboxFormat.dbox.txt] or Maildir mailboxes. Zlib plugin +supports compression using zlib/gzip, bzlib/bzip2, liblzma/xz (v2.2.9+) and +liblz4/lz4 (v2.2.11+). + +Configuration: + +---%<------------------------------------------------------------------------- +# Enable zlib plugin globally for reading/writing: +mail_plugins = $mail_plugins zlib + +# Enable these only if you want compression while saving: +plugin { + zlib_save_level = 6 # 1..9; default is 6 + zlib_save = gz # or bz2, xz or lz4 +} +---%<------------------------------------------------------------------------- + +mbox +---- + +Compressed mbox files can be accessed only as read-only. The compression is +detected based on the file name, so your compressed mboxes should end with .gz +or .bz2 extension. There is no support for compression during saving. + +dbox +---- + +Mails can be stored as compressed. Existing uncompressed mails can't currently +be directly compressed (or vice versa). You could, however, use <dsync> +[Tools.Dsync.txt] to copy all mails to another location (which saves them +compressed) and then replace the original location with the new compressed +location. You can do this by treating the operation the same as if you were +migrating from one mailbox format to another (see the dsync page examples). + +Maildir +------- + +When this plugin is loaded Dovecot can read both compressed and uncompressed +files from Maildir. If you've enabled both gzip and bzip2 support you can have +files compressed with either one of them in the Maildir. The compression is +detected by reading the first few bytes from the file and figuring out if it's +a valid gzip or bzip2 header. The file name doesn't matter. This means that an +IMAP client could also try to exploit security holes in zlib/bzlib by writing +specially crafted mails using IMAP's APPEND command. This is prevented by +Dovecot not allowing clients to save mails that are detected as compressed. + +All mails must have ',S=<size>' in their filename where <size> contains the +original uncompressed mail size, otherwise there will be problems with quota +calculation as well as other potential random failures. Note that if the +filename doesn't contain the ',S=<size>' before compression, adding it +afterwards changes the base filename and thus the message UID. The safest thing +to do is simply to not compress such files. + +You should also preserve the file's mtime so INTERNALDATE doesn't change. + +If you want to use dsync to convert to a compressed Maildir you may need -o +maildir_copy_with_hardlinks=no (this is set to yes by default and will prevent +compression). + +Compression +----------- + +You'll probably want to use some cronjob to compress old mails. However note +that to avoid seeing duplicate mails in rare race conditions you'll have to use +the included maildirlock utility. The idea is to: + + 1. Find the mails you want to compress in a single maildir. + * Skip files that don't have ',S=<size>' in the filename. + 2. Compress the mails to 'tmp/' + * Update the compressed files' mtimes to be the same as they were in the + original files (e.g. touch command) + 3. Run 'maildirlock <path> <timeout>'. It writes PID to stdout, save it. + * <path> is path to the directory containing Maildir's dovecot-uidlist + (the control directory, if it's separate) + * <timeout> specifies how long to wait for the lock before failing. + 4. If maildirlock grabbed the lock successfully (exit code 0) you can + continue. + 5. For each mail you compressed: + 1. Verify that it still exists where you last saw it. + 2. If it doesn't exist, delete the compressed file. Its flags may have + been changed or it may have been expunged. This happens rarely, so just + let the next run handle it. + 3. If the file does exist, 'rename()' (mv) the compressed file over the + original file. + * Dovecot can now read the file, but to avoid compressing it again on + the next run, you'll probably want to rename it again to include + e.g. a "Z" flag in the file name to mark that it was compressed + (e.g.'1223212411.M907959P17184.host,S=3271:2,SZ'). Remember that the + Maildir specifications [http://cr.yp.to/proto/maildir.html] require + that the flags are sorted by their ASCII value, although Dovecot + itself doesn't care about that. + 6. Unlock the maildir by sending a TERM signal to the maildirlock process + (killing the PID it wrote to stdout). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Plugins.txt b/doc/wiki/Plugins.txt new file mode 100644 index 0000000..efdac36 --- /dev/null +++ b/doc/wiki/Plugins.txt @@ -0,0 +1,102 @@ +Plugins +======= + +Distributed with Dovecot: + + * <acl> [ACL.txt]: Access Control Lists for mailboxes + * <apparmor> [Plugins.Apparmor.txt]: Apparmor support plugin + * <autocreate> [Plugins.Autocreate.txt]: Automatically create/subscribe + mailboxes when user logs in + * <expire> [Plugins.Expire.txt]: Delete mails from specified mailboxes after a + designated number of days + * <fts> [Plugins.FTS.txt]: Full text search indexing + * <last_login> [Plugins.LastLogin.txt]: Last-login timestamp updating + * <lazy_expunge> [Plugins.Lazyexpunge.txt]: Make EXPUNGE and DELETE commands + just rename the mails elsewhere + * <listescape> [Plugins.Listescape.txt]: Allow using characters in mailbox + names that would otherwise be illegal + * <mailbox_alias> [Plugins.MailboxAlias.txt]: Map multiple names to the same + mailbox. + * <mail_filter> [Plugins.MailFilter.txt]: Filter mails while reading/writing + through a script + * <mail_log> [Plugins.MailLog.txt]: Log several mail actions + * <notify> [Plugins.Notify.txt]: Wrapper plugin for easily developing other + plugins that act on mailbox changes. + * <notify_status> [Plugins.NotifyStatus.txt]: Plugin for notify that pushes + mailbox status to dict on change + * <push_notification> [Plugins.PushNotification.txt]: Push notification + framework + * <push_notification_driver_ox> [Plugins.PushNotification.txt]: + Open-Xchange driver for push notification + * <push_notification_driver_dlog> [Plugins.PushNotification.txt]: dlog + driver for push notification + * <snarf> [Plugins.Snarf.txt]: UW-IMAP style (mbox) snarfing, i.e. keeps + moving all mails from one mailbox to INBOX + * <quota> [Quota.txt]: Quota tracking and enforcing + * imap_quota: IMAP commands for requesting current quota + * <quota_clone> [Plugins.QuotaClone.txt]: Copy the current quota usage to a + dict. + * <stats> [Plugins.Stats.txt]: Send statistics from mail processes to the + stats process + * <trash> [Plugins.Trash.txt]: Delete mails from Trash when user would go over + quota + * <virtual> [Plugins.Virtual.txt]: Virtual mailboxes + * <welcome> [Plugins.Welcome.txt]: Call a script on user's first login. + * <zlib> [Plugins.Zlib.txt]: Access compressed mails + * <imap_zlib> [Plugins.Compress.txt]: Enable IMAP COMPRESS + [http://www.ietf.org/rfc/rfc4978.txt] extension + * <mail_crypt> [Plugins.MailCrypt.txt]: In-rest mail encryption + * <var_expand_crypt> [Plugins.VarExpandCrypt.txt]: Variable + encryption/decryption + +External: + + * <antispam> [Plugins.Antispam.txt]: Integrates spam learning into Dovecot + * drac [http://sourceforge.jp/projects/dovecot2-drac/]: Pop-before-SMTP plugin + using DRAC + * whoson [http://ftp.ufanet.ru/pub/boco/dovecot/whoson-plugin-2/]: + Pop-before-SMTP plugin using WHOSON protocol [http://whoson.sourceforge.net] + + * alert [http://dovecot.org/patches/1.0/imap-alert-plugin.c]: Send IMAP ALERT + message from '/etc/dovecot.alert'. + * <Pigeonhole Sieve> [Pigeonhole.Sieve.txt]: Use the Sieve + [http://sieve.info/] language to perform actions upon message delivery, such + as filtering, forwarding, vacation replies, notifications and many other + actions. See the <documentation> [Pigeonhole.Sieve.txt] for a full list. + * Fetchmail wakeup [http://github.com/marschap/fetchmail_wakeup]: Wakes + fetchmail when Dovecot looks for new mail. This is a heavily extended and + updated version that is tested to work with Dovecot 1.1.x and 1.2.x. + Versions 2.x including the freshly released version 2.0.3 work with Dovecot + 2.0.x. + * Fetchmail wake [http://guichaz.free.fr/misc/]: Wakes fetchmail when Dovecot + looks for new mail. This is the original version that only works with + dovecot 1.0.x + * <deleted-to-trash> [Plugins.deleted-to-trash.txt]: Automatically move + deleted item to trash folder, if client side doesn't do it, such as outlook + and evolution. + * extra-copies [https://github.com/vandry/dovecot-plugin-extra-copies]: allows + extra copies of messages to be made in other folders whenever a message is + added to a particular folder. + * fts-elasticsearch [https://github.com/ascendantcom/fts-elasticsearch]: + Allows for the use of <ElasticSearch.txt> as a full-text search backend. + * openchange + [https://github.com/openchange/openchange/tree/master/mapiproxy/services/plugins/dovecot]: + Allows <OpenChange.txt> to update Microsoft Outook mailboxes automatically + when new emails are received by Dovecot 2.x. + * TREES [https://0xacab.org/riseuplabs/trees]: A NaCL-based Dovecot encryption + plugin + * dovecot-xaps-plugin [https://github.com/st3fan/dovecot-xaps-plugin]: iOS + Push Email for Dovecot + * dovecot-libsodium-plugin + [https://github.com/LuckyFellow/dovecot-libsodium-plugin]: Libsodium + password hashing schemes plugin + +Experimental Plugins: + + * <xexec> [Plugins.xexec.txt]: Execute any server side application and + communicate with it through plugins over IMAP + +To enable / disable plugins per user you can make your userdb return +'mail_plugins' extra field. See <UserDatabase.ExtraFields.txt> for examples. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PostLoginScripting.txt b/doc/wiki/PostLoginScripting.txt new file mode 100644 index 0000000..390ae00 --- /dev/null +++ b/doc/wiki/PostLoginScripting.txt @@ -0,0 +1,247 @@ +Post-login scripting +==================== + +If you want to do something special after authentication, but before beginning +the IMAP or POP3 session, you can do this by telling imap/pop3 executable to +use post-login service by editing conf.d/10-master.conf: + +---%<------------------------------------------------------------------------- +service imap { + # tell imap to do post-login lookup using a socket called "imap-postlogin" + executable = imap imap-postlogin +} + +# The service name below doesn't actually matter. +service imap-postlogin { + # all post-login scripts are executed via script-login binary + executable = script-login /usr/local/bin/postlogin.sh + + # the script process runs as the user specified here (v2.0.14+): + user = $default_internal_user + # this UNIX socket listener must use the same name as given to imap +executable + unix_listener imap-postlogin { + } +} +---%<------------------------------------------------------------------------- + +You can run multiple post-login scripts by just giving multiple scripts as +parameters to 'script-login', for example: + +---%<------------------------------------------------------------------------- + executable = script-login rawlog /usr/local/bin/postlogin.sh +/usr/local/bin/postlogin2.sh +---%<------------------------------------------------------------------------- + +The scripts are run in the specified order. Remember that the post-login script +runs with the privileges of the "user" setting given to the service (root by +default). If you need the script to access user's mail files, change it to +whatever user owns the mails (e.g. vmail). If you're using multiple UNIX UIDs +(e.g. system users), use 'script-login -d' to drop to the UID/GID specified by +the userdb lookup (ignoring user/group/chroot service settings). + +It's not currently possible to run post-login scripts in <proxies> +[PasswordDatabase.ExtraFields.Proxy.txt], because they're not actually logging +in to the local Dovecot. An alternative method could be implemented some day, +maybe as a plugin. + +Running environment +------------------- + +Standard input and output file descriptors are redirected to the client's +network socket, so you can send data to client by simply writing to stdout. +Standard error fd is redirected to Dovecot's error log, you can write errors +there as well. + +The script can use environment variables: + + * USER: Username + * IP: Remote IP address + * LOCAL_IP: Local IP address + * Fields returned by userdb lookup with their keys uppercased (e.g. if userdb + returned home, it's stored in HOME). + +It's possible to add/modify userdb fields by adding them to environment and +adding the field to USERDB_KEYS. For example to change user's mail location: + +---%<------------------------------------------------------------------------- +#!/bin/sh + +export MAIL=maildir:/tmp/test +export USERDB_KEYS="$USERDB_KEYS mail" +exec "$@" +---%<------------------------------------------------------------------------- + +You can change any Dovecot settings using the above method. + +Changing user's password after login +------------------------------------ + +See <HowTo.ConvertPasswordSchemes.txt> + +Last-login tracking +------------------- + +If you want to know when the user last logged in, you can do it like this: + +---%<------------------------------------------------------------------------- +#!/bin/sh +# a) Filesystem based timestamp in user's home directory +touch ~/.last_login +# b) SQL based tracking. Beware of potential SQL injection holes if you allow +# users to have ' characters in usernames. Following is just an example: +#echo "UPDATE mailbox SET modified = now() WHERE username = '$USER'" | mysql +postfixadmin +exec "$@" +---%<------------------------------------------------------------------------- + +/Note: if creating a timestamp inside the Maildir itself, it's better to avoid +filenames which begin with a dot. The IMAP "list" command will show such files +as IMAP folders, unless you also set 'maildir_stat_dirs = yes' which generates +more I/O ops./ + +Custom mailbox location autodetection +------------------------------------- + +See <MailLocation.txt> for an example. + +Alerts +------ + +If you want to give the user's client some warning notification, you can do it +just by writing it to stdout. But note: + + * Not all clients show the alerts, even though IMAP RFC requires it. + * IMAP protocol requires CRLF (\r\n) line feeds. Some clients will break if + you only send LF. + +---%<------------------------------------------------------------------------- +#!/bin/sh +if [ -f ~/.out-of-office ]; then + printf "* OK [ALERT] You're still marked as being out of office.\r\n" +fi +exec "$@" +---%<------------------------------------------------------------------------- + +Use UNIX groups for ACL authorization +------------------------------------- + +---%<------------------------------------------------------------------------- +#!/bin/sh +ACL_GROUPS=`groups $USER | tr ' ' ','` +export ACL_GROUPS +export USERDB_KEYS="$USERDB_KEYS acl_groups" +exec "$@" +---%<------------------------------------------------------------------------- + +Denying connection from some IP/User +------------------------------------ + +You can use the IP and USER shell variables that are setup by dovecot in a bash +script in order to deny connection (after a successful login), like this: + +---%<------------------------------------------------------------------------- +if [ "$USER" = "myuser" ] ; then + printf "* NO [ALERT] The user '$USER' can not login\r\n" + exit 0 +fi + +if [ ! "$IP" = "192.168.1.1" ] ; then + printf "* NO [ALERT] Access not allowed from the Internet\r\n" + exit 0 +fi +exec "$@" +---%<------------------------------------------------------------------------- + +You can also use + + * http://www.linux.org.py/wiki/howto/dovecot_connect_acl + * TCP wrappers can be used with 'login_access_sockets = tcpwrap' + +Dynamically adding shared mailboxes according to filesystem permissions +----------------------------------------------------------------------- + +Additional namespaces can be dynamically added via environment variables: + +---%<------------------------------------------------------------------------- +use strict; + +my $SHAREDDIR= '/var/spool/mail/Shared'; + +if (! @ARGV) { + exit 1; +} + +# for testing... +#if ($ENV{USER} eq 'lemur') { +# # print "* OK [ALERT] Hello $ENV{'USER'}!\n"; +# &set_namespaces(); +# system("env >> /tmp/dovecot-env-$$"); +#} + +&set_namespaces(); + +exec(@ARGV) or die "Unable to exec @ARGV: $!"; + +sub set_namespaces { + my $mailbox; + local *D; + if (opendir(D, $SHAREDDIR)) { + my $dir; + my @namespaces = (); + while ($mailbox= readdir(D)) { + next if ($mailbox =~ /^\./); + if (-r "${SHAREDDIR}/${mailbox}") { + my $nsname = 'S-'.uc($mailbox); + push(@namespaces, lc($nsname)); + &log("adding NAMESPACE/${nsname}/PREFIX ${SHAREDDIR}/${mailbox}"); + $ENV{"NAMESPACE/${nsname}/LOCATION"} = + "maildir:$SHAREDDIR/$mailbox:INDEX=~/Maildir/index/Shared/$mailbox"; + $ENV{"NAMESPACE/${nsname}/PREFIX"} = "Shared/$mailbox/"; + $ENV{"NAMESPACE/${nsname}/TYPE"}= "public"; + $ENV{"NAMESPACE/${nsname}/SEPARATOR"}= "/"; + $ENV{"NAMESPACE/${nsname}/LIST"}= "yes"; + # $ENV{"NAMESPACE/${nsname}/SUBSCRIPTIONS"} = "no" + } + } + closedir D; + if (@namespaces) { + $ENV{"NAMESPACE"} = join(' ', @namespaces); + my @userdb_keys; + if ($ENV{'USERDB_KEYS'}) { + push(@userdb_keys, $ENV{'USERDB_KEYS'}); + } + push(@userdb_keys, grep(/^NAMESPACE/, keys(%ENV))); + $ENV{'USERDB_KEYS'} = join(' ', @userdb_keys); + } + } +} + +sub log { + print STDERR "@_\n"; +} +---%<------------------------------------------------------------------------- + +This adds environment variables like that: + +---%<------------------------------------------------------------------------- +NAMESPACE/S-SPAMREP/LIST=yes +NAMESPACE/S-SPAMREP/LOCATION=maildir:/var/spool/mail/Shared/spamrep:INDEX=~/Maildir/index/Shared/spamrep +NAMESPACE/S-SPAMREP/PREFIX=Shared/spamrep/ +NAMESPACE/S-SPAMREP/SEPARATOR=/ +NAMESPACE/S-SPAMREP/TYPE=public +NAMESPACE/S-TESTSHARED/LIST=yes +NAMESPACE/S-TESTSHARED/LOCATION=maildir:/var/spool/mail/Shared/testshared:INDEX=~/Maildir/index/Shared/testshared +NAMESPACE/S-TESTSHARED/PREFIX=Shared/testshared/ +NAMESPACE/S-TESTSHARED/SEPARATOR=/ +NAMESPACE/S-TESTSHARED/TYPE=public +NAMESPACE=s-testshared s-spamrep +USERDB_KEYS=SYSTEM_GROUPS_USER UID GID HOME NAMESPACE/S-SPAMREP/LIST NAMESPACE +NAMESPACE/S-TESTSHARED/SEPARATOR NAMESPACE/S-TESTSHARED/TYPE +NAMESPACE/S-TESTSHARED/PREFIX NAMESPACE/S-TESTSHARED/LIST +NAMESPACE/S-TESTSHARED/LOCATION NAMESPACE/S-SPAMREP/SEPARATOR +NAMESPACE/S-SPAMREP/TYPE NAMESPACE/S-SPAMREP/PREFIX +NAMESPACE/S-SPAMREP/LOCATION +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/PreAuth.txt b/doc/wiki/PreAuth.txt new file mode 100644 index 0000000..97ec73c --- /dev/null +++ b/doc/wiki/PreAuth.txt @@ -0,0 +1,22 @@ +Pre-Authenticated Sessions +========================== + +For debugging purpose or to perform IMAP actions on-behalf of an user, you can +use pre-authenticated sessions. You can do this by simply running: + +---%<------------------------------------------------------------------------- +/usr/local/libexec/dovecot/imap +---%<------------------------------------------------------------------------- + +And you can start talking IMAP via stdin/stdout. This doesn't change process's +UID, GID, or get any userdb settings. If you want to emulate a full regular +login, you can execute + +---%<------------------------------------------------------------------------- +/usr/local/libexec/dovecot/imap -u username +---%<------------------------------------------------------------------------- + +Of course, you may need to run the above command as root if it needs to change +process's uid/gid. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/QuickConfiguration.txt b/doc/wiki/QuickConfiguration.txt new file mode 100644 index 0000000..9e1be7a --- /dev/null +++ b/doc/wiki/QuickConfiguration.txt @@ -0,0 +1,209 @@ +Quick Configuration +=================== + +If you just want to get Dovecot running with typical configuration in a typical +environment, here's what you'll have to do: + +Contents + + + 1. Quick Configuration + + 1. TLDR; Just want it running + + 2. Configuration file + + 1. Installing from sources + + 2. Split configuration files + + 3. Hints about writing configuration files + + 3. Authentication + + 4. Mail Location + + 5. Mbox + + 6. Maildir + + 7. Client Workarounds + + 8. SSL and Plaintext Authentication + + 9. NFS + + 10. Running + +TLDR; Just want it running +-------------------------- + +Here is a very simple basic configuration with single vmail user to be placed +as dovecot.conf. Please note that some distros split configuration under +/etc/dovecot/conf.d, which, while it can be useful, is not required. + +You need to create group vmail and user vmail. + +---%<------------------------------------------------------------------------- +mail_home=/srv/mail/%Lu +mail_location=sdbox:~/Mail + +## this is sometimes needed +#first_valid_uid = uid-of-vmail-user + +# if you want to use system users +passdb { + driver = pam +} + +userdb { + driver = passwd + args = blocking=no + override_fields = uid=vmail gid=vmail +} + +ssl=yes +ssl_cert=</path/to/cert.pem +ssl_key=</path/to/key.pem +# if you are using v2.3.0-v2.3.2.1 (or want to support non-ECC DH algorithms) +# since v2.3.3 this setting has been made optional. +#ssl_dh=</path/to/dh.pem + +namespace { + inbox = yes + separator = / +} +---%<------------------------------------------------------------------------- + +Configuration file +------------------ + +Prebuilt packages usually install the configuration files into '/etc/dovecot/'. +You'll find the correct path by running: + +---%<------------------------------------------------------------------------- +doveconf -n | head -n 1 +---%<------------------------------------------------------------------------- + +It's a good idea to read through all the config files and see what settings you +might want to change. + +Installing from sources +----------------------- + +If you compiled and installed Dovecot from sources, Dovecot has installed only +a '/usr/local/etc/dovecot/README' file, which contains the path to the +installed example configuration files, usually +'/usr/local/share/doc/dovecot/example-config'. Copy them to etc/: + +---%<------------------------------------------------------------------------- +cp -r /usr/local/share/doc/dovecot/example-config/* /usr/local/etc/dovecot/ +---%<------------------------------------------------------------------------- + +Split configuration files +------------------------- + +The default configuration starts from 'dovecot.conf', which contains an +'!include conf.d/*.conf' statement to read the rest of the configuration. The +idea is that the settings are nicely grouped into different files to make it +easier for new admins to scan through related settings. It doesn't matter which +config file you add which setting. In the production system it's often easier +to just have a single 'dovecot.conf' file, which you can create easily using + +---%<------------------------------------------------------------------------- +doveconf -nP > dovecot.conf +---%<------------------------------------------------------------------------- + +Hints about writing configuration files +--------------------------------------- + + * Usually it does not matter in which file you write the setting, however, + later settings replace earlier ones. If you use the same section multiple + times, the settings are merged together. + * Before v2.3, boolean settings in the 'plugin' section interpret /any/ value + as *true*, even '0', 'no' and 'false'. + * To read the content of a file, for instance for the SSL certificate option, + prefix the filename with a '<', e.g.: + +---%<------------------------------------------------------------------------- +ssl_cert = </etc/ssl/certs/imap.pem +---%<------------------------------------------------------------------------- + +Authentication +-------------- + +You'll probably be using <PAM> [PasswordDatabase.PAM.txt] authentication. See +the <PAM> [PasswordDatabase.PAM.txt] page for how to configure it. A typical +configuration with Linux would be to create '/etc/pam.d/dovecot' which +contains: + +---%<------------------------------------------------------------------------- +auth required pam_unix.so +account required pam_unix.so +---%<------------------------------------------------------------------------- + +If you're using something else, see <password databases> [PasswordDatabase.txt] +and <user databases> [UserDatabase.txt]. + +Mail Location +------------- + +You can let Dovecot do its automatic mail location detection, but if that +doesn't work, you can set the location manually in 'mail_location' setting. See +<MailLocation.txt> for more information. + +Mbox +---- + +Make sure that all software accessing the mboxes are using the same locking +methods in the same order. The order is important to prevent deadlocking. From +Dovecot's side you can change these from 'mbox_read_locks' and +'mbox_write_locks' settings. See <MboxLocking.txt> for more information. + +If you're using '/var/mail/' directory for INBOXes, you may need to set +'mail_privileged_group = mail' so Dovecot can create dotlocks there. + +For better performance you may want to set 'mbox_very_dirty_syncs = yes' +option. + +Maildir +------- + +For better performance you may want to set 'maildir_very_dirty_syncs = yes' +option. + +Client Workarounds +------------------ + +Check 'imap_client_workarounds' and 'pop3_client_workarounds' and see if you +want to enable more of them than the defaults. + +SSL and Plaintext Authentication +-------------------------------- + +If you intend to use SSL, set 'ssl_cert' and 'ssl_key' settings. Otherwise set +'ssl = no'. Easiest way to get SSL certificates built is to use Dovecot's +'doc/mkcert.sh' script. See <SSL.txt>. + +By default 'disable_plaintext_auth = yes', which means that Dovecot will fail +the authentication if the client doesn't use SSL (or use <non-plaintext +authentication> [Authentication.Mechanisms.txt]). This is recommended in most +situations, since it prevents leaking passwords. However, if you don't offer +SSL for some reason, you'll probably want to set 'disable_plaintext_auth = no'. + +Since v2.3.3 you only need ssl_key and ssl_cert, leaving ssl_dh unset (and +removing ssl-parameters.dat if left over from 2.2) will prevent using non-EC DH +algorithms. + +NFS +--- + +If you're using NFS or some other remote filesystem that's shared between +multiple computers, you should read <NFS.txt>. + +Running +------- + +See <RunningDovecot.txt> and <Logging.txt>. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Quota.Configuration.txt b/doc/wiki/Quota.Configuration.txt new file mode 100644 index 0000000..75e78b2 --- /dev/null +++ b/doc/wiki/Quota.Configuration.txt @@ -0,0 +1,513 @@ +Quota Configuration +=================== + +Contents + + + 1. Quota Configuration + + 1. Quota rules + + 1. Limit configuration + + 2. Per-user quota + + 1. LDAP + + 2. MySQL + + 3. PostgreSQL, SQLite + + 4. passwd-file + + 5. passwd + + 3. Quota for public namespaces + + 4. Quota for private namespaces + + 5. Custom Quota Exceeded Message + + 6. Quota warnings + + 7. Overquota-flag (v2.2.16+) + + 8. Quota grace + + 9. Maximum saved mail size + + 10. Quota admin commands + +The configuration is split into multiple settings: + +First you have the quota root backend configuration. Quota root is a concept +from IMAP Quota specifications [http://www.rfc-editor.org/rfc/rfc2087.txt]. +Normally you'll have only one quota root, but in theory there could be e.g. +"user quota" and "domain quota" roots. It's unspecified how the quota roots +interact with each others (if at all). In some systems for example INBOX could +have a completely different quota root from the rest of the mailboxes (e.g. +INBOX in '/var/mail/' partition and others in '/home/' partition). + +Quota root configuration includes the backend name, quota root name and its +parameters, if there are any: + +---%<------------------------------------------------------------------------- +quota = <backend>[:<quota root name>[:<backend args>]] +---%<------------------------------------------------------------------------- + +The quota root name is just an arbitrary string that is sent to IMAP clients, +which in turn may show it to the user. The name has no meaning. By default an +empty string is used, but you may want to change that since some clients (Apple +Mail) break and don't show quota at all then. + +You can define multiple quota roots by appending an increasing number, for +example: + +---%<------------------------------------------------------------------------- +plugin { + quota = maildir:User quota + quota2 = fs:Disk quota + #quota3 = ... +} +---%<------------------------------------------------------------------------- + +Quota rules +----------- + +Quota rules configure the actual quota limits. The syntax is: + +---%<------------------------------------------------------------------------- +quota_rule = <mailbox name>:<limit configuration> +#quota_rule2 = ... +#quota_rule3 = ..etc.. +---%<------------------------------------------------------------------------- + +"*" as the mailbox name configures the default limit, which is applied on top +of a mailbox-specific limit if found. So for example: + +---%<------------------------------------------------------------------------- +quota_rule = *:storage=1G +quota_rule2 = Trash:storage=+100M +quota_rule3 = SPAM:ignore +---%<------------------------------------------------------------------------- + +This means that the user has 1GB quota, but when saving messages to Trash +mailbox it's possible to use up to 1.1GB of quota. The quota isn't specifically +assigned to Trash, so if you had 1GB of mails in Trash you could still save +100MB of mails to Trash, but nothing to other mailboxes. The idea of this is +mostly to allow the clients' move-to-Trash feature work while user is deleting +messages to get under quota. Additionally, any messages in the SPAM folder are +ignored per the 'ignore' directive and would not count against the quota. + +"?" as the mailbox name works almost like "*". The difference is that "?" is +used only if quota backend doesn't override the limit. For example with +Maildir++ [http://www.inter7.com/courierimap/README.maildirquota.html] quota if +'maildirsize' file exists the limits are taken from it, but if it doesn't exist +the "?" limits are used. + +"*" can't be used as a generic wildcard in mailbox names, so for example "box*" +wouldn't match "boxes". As shown in the above example, the first quota rule is +named 'quota_rule' while the following rules have an increasing digit in them. +You can have as many quota rules as you want. + +Limit configuration +------------------- + +The following limit names are supported: + + * *storage*: Quota limit in kilobytes, 0 means unlimited. + * *bytes*: Quota limit in bytes, 0 means unlimited. + * *messages*: Quota limit in number of messages, 0 means unlimited. This + probably isn't very useful. + * *backend*: Quota backend-specific limit configuration. + * *ignore*: Don't include the specified mailbox in quota at all. + +All of these support also b/k/M/G/T/% suffixes. So storage=100M and bytes=100M +both mean the exact same thing. messages=1k also means 1024 messages (not +1000). + +Percents are relative to the default rule. For example: + +---%<------------------------------------------------------------------------- +plugin { + quota = maildir:User quota + quota_rule = *:storage=1GB + # 10% of 1GB = 100MB + quota_rule2 = Trash:storage=+10%% + # 20% of 1GB = 200MB + quota_rule3 = Spam:storage=+20%% +} +---%<------------------------------------------------------------------------- + +Note that % is written twice to escape it, because <%variables> [Variables.txt] +are expanded in plugin section. <userdb> [UserDatabase.txt] configuration may +or may not require this escaping. + +Backend-specific configuration currently is used only with Maildir++ quota +backend. It means you can have the quota in Maildir++ format (e.g. +"10000000S"). + +Per-user quota +-------------- + +You can override quota rules in your <userdb> [UserDatabase.txt]'s <extra +fields> [UserDatabase.ExtraFields.txt]. Keep all the global settings in plugin +section and override only those settings you need to in your userdb. + +If you're wondering why per-user quota isn't working: + + * Check that 'dovecot-lda' is called with *-d* parameter. + * Check that you're not using <userdb static> [UserDatabase.Static.txt]. + * Check that 'quota_rule' setting is properly returned by userdb. Enable + 'auth_debug=yes' and 'mail_debug=yes' to see this. + +For example: + +---%<------------------------------------------------------------------------- +plugin { + quota = maildir:User quota + quota_rule = *:storage=1G + quota_rule2 = Trash:storage=+100M +} +---%<------------------------------------------------------------------------- + +Next override the default 1GB quota for users: + +LDAP +---- + +Quota limit is in quotaBytes field: + +---%<------------------------------------------------------------------------- +user_attrs = homeDirectory=home, quotaBytes=quota_rule=*:bytes=%$ +---%<------------------------------------------------------------------------- + +Remember that 'user_attrs' is used only if you use <userdb ldap> +[AuthDatabase.LDAP.txt]. + +MySQL +----- + +---%<------------------------------------------------------------------------- +user_query = SELECT uid, gid, home, \ + concat('*:bytes=', quota_limit_bytes) AS quota_rule \ + FROM users WHERE userid = '%u' + +# MySQL with userdb prefetch: Remember to prefix quota_rule with userdb_ +# (just like all other userdb extra fields): +password_query = SELECT userid AS user, password, \ + uid AS userdb_uid, gid AS userdb_gid, \ + concat('*:bytes=', quota_limit_bytes) AS userdb_quota_rule \ + FROM users WHERE userid = '%u' +---%<------------------------------------------------------------------------- + +Remember that 'user_query' is used only if you use <userdb sql> +[AuthDatabase.SQL.txt]. + +PostgreSQL, SQLite +------------------ + +---%<------------------------------------------------------------------------- +user_query = SELECT uid, gid, home, \ + '*:bytes=' || quota_limit_bytes AS quota_rule \ + FROM users WHERE userid = '%u' +---%<------------------------------------------------------------------------- + +Remember that 'user_query' is used only if you use <userdb sql> +[AuthDatabase.SQL.txt]. + +passwd-file +----------- + +Example passwd-file entries: + +---%<------------------------------------------------------------------------- +user:{plain}pass:1000:1000::/home/user::userdb_quota_rule=*:bytes=100M +user2:{plain}pass2:1001:1001::/home/user2::userdb_quota_rule=*:bytes=200M +user3:{plain}pass3:1002:1002::/home/user3::userdb_mail=maildir:~/Maildir +userdb_quota_rule=*:bytes=300M +---%<------------------------------------------------------------------------- + +passwd +------ + +The <passwd> [AuthDatabase.Passwd.txt] userdb doesn't support extra fields. +That's why you can't directly set users' quota limits to passwd file. One +possibility would be to write a script that reads quota limits from another +file, merges them with passwd file and produces another passwd-file, which you +could then use with Dovecot's <userdb passwd-file> +[AuthDatabase.PasswdFile.txt]. + +Quota for public namespaces +--------------------------- + +You can create a separate namespace-specific quota that's shared between all +users. This is done simply by adding ':ns=<namespace prefix>' parameter to +quota setting. For example you could have something like: + +---%<------------------------------------------------------------------------- +namespace { + type = public + prefix = Public/ + #location = .. +} + +plugin { + quota = maildir:User quota + quota2 = maildir:Shared quota:ns=Public/ + #quota_rules and quota2_rules.. +} +---%<------------------------------------------------------------------------- + +Quota for private namespaces +---------------------------- + +You can create a separate namespace-specific quota for a folder hierarchy. This +is done by adding another namespace and the ':ns=<namespace prefix>' parameter +to quota setting. For example you could have something like: + +---%<------------------------------------------------------------------------- +namespace { + type = private + prefix = Archive/ + #location = .. +} + +plugin { + # Maildir quota + quota = maildir:User quota:ns= + quota2 = maildir:Archive quota:ns=Archive/ + + # Dict quota + #quota = dict:User quota:%u.default:ns=:proxy::quota + #quota2 = dict:Archive quota:%u.archive:ns=Archive/:proxy::quota + + #quota_rules and quota2_rules.. +} +---%<------------------------------------------------------------------------- + +Note: If you're using dict quota, you need to make sure that the quota of the +'Archive' namespace is calculated for another "user" than the default +namespace. Either track different namespaces in different backends or make sure +the users differs.'%u.archive' defines '<username>.archive' as key to track +quota for the 'Archive' namespace, the '%u.default' tracks the quota of other +folders. See <Variables.txt> for further help on variables. + +Custom Quota Exceeded Message +----------------------------- + +You can configure Dovecot to send a custom string instead of the default quota +exceeded message. You could have something like: + +---%<------------------------------------------------------------------------- +plugin { + quota_exceeded_message = Quota exceeded, please go to +http://www.example.com/over_quota_help for instructions on how to fix this. +} +---%<------------------------------------------------------------------------- + +Dovecot can also read the quota exceeded message from a file: + +---%<------------------------------------------------------------------------- +plugin { + quota_exceeded_message = </path/to/quota_exceeded_message.txt +} +---%<------------------------------------------------------------------------- + +Quota warnings +-------------- + +You can configure Dovecot to run an external command when user's quota exceeds +a specified limit. Note that the warning is ONLY executed at the exact time +when the limit is being crossed, so when you're testing it you have to do it by +crossing the limit by saving a new mail. If something else besides Dovecot +updates quota so that the limit is crossed, the warning is never executed. The +syntax is: + +---%<------------------------------------------------------------------------- +quota_warning = <limit configuration> <quota-warning socket name> <parameters> +#quota_warning2 = ... +#quota_warning3 = ..etc.. +---%<------------------------------------------------------------------------- + +Limit configuration is almost exactly same as for rules, with the exception of +adding "-" before the value for "reverse" warnings where the script is called +when quota drops below the value. Usually you want to use percents instead of +absolute limits. Only the command for the first exceeded limit is executed, so +configure the highest limit first. The actual commands that are run need to be +created as services. + +An example configuration: + +---%<------------------------------------------------------------------------- +plugin { + quota_warning = storage=95%% quota-warning 95 %u + quota_warning2 = storage=80%% quota-warning 80 %u + quota_warning3 = -storage=100%% quota-warning below %u # user is no longer +over quota +} + +service quota-warning { + executable = script /usr/local/bin/quota-warning.sh + # use some unprivileged user for executing the quota warnings + user = vmail + unix_listener quota-warning { + } +} +---%<------------------------------------------------------------------------- + +With the above example when user's quota exceeds 80%, 'quota-warning.sh' is +executed with parameter 80. The same goes for when quota exceeds 95%. If user +suddenly receives a huge mail and the quota jumps from 70% to 99%, only the 95 +script is executed. + +You have to create the 'quota-warning.sh' script yourself. Here is an example +that sends a mail to the user: + +---%<------------------------------------------------------------------------- +#!/bin/sh +PERCENT=$1 +USER=$2 +cat << EOF | /usr/local/libexec/dovecot/dovecot-lda -d $USER -o +"plugin/quota=maildir:User quota:noenforcing" +From: postmaster@domain.com +Subject: quota warning + +Your mailbox is now $PERCENT% full. +EOF +---%<------------------------------------------------------------------------- + +The quota enforcing is disabled to avoid looping. You'll of course need to +change the 'plugin/quota' value to match the quota backend and other +configuration you use. Basically preserve your original "quota" setting and +just insert ":noenforcing" to proper location in it. For example with dict +quota, you can use something like:'-o "plugin/quota=dict:User +quota::noenforcing:proxy::quota"' + +Overquota-flag (v2.2.16+) +------------------------- + +Quota warnings scripts can be used to set an overquota-flag to userdb (e.g. +LDAP) when user goes over/under quota. This flag can be used by MTA to reject +mails to an user who is over quota already at SMTP RCPT TO stage. The only +problem with this has been that there are race conditions that in some rare +situations cause the overquota-flag to be set even when user is already under +quota. This situation doesn't get solved itself without manual admin +intervention or the new overquota-flag feature: This feature checks the flag's +value every time user logs in (or mail gets delivered or any other email access +to user) and compares it to the current actual quota usage. If the flag is +wrong, a script is executed that should fix up the situation. + +The overquota-flag name in userdb must be "quota_over_flag". There are two +settings to configure what to do: + +---%<------------------------------------------------------------------------- +plugin { + # If quota_over_flag=TRUE, the overquota-flag is enabled. Otherwise not. + quota_over_flag_value = TRUE + + # Any non-empty value for quota_over_flag means user is over quota. + # Wildcards can be used in a generic way, e.g. "*yes" or "*TRUE*" + #quota_over_flag_value = * + + # If set, overquota-flag is checked only when current quota usage is going to +be checked anyway. + # This can be used to optimize this check in case it's running too slowly. +(v2.2.25+) + #quota_over_flag_lazy_check = yes + + # Service script to execute if overquota-flag is wrong. Configured the + # same as quota_warning scripts. The current quota_over_flag's value is + # appended as the last parameter. + quota_over_script = quota-warning mismatch %u +} +---%<------------------------------------------------------------------------- + +Quota grace +----------- + +With v2.2+ by default the last mail can bring user over quota. This is useful +to allow user to actually unambiguously become over quota instead of fail some +of the last larger mails and pass through some smaller mails. Of course the +last mail shouldn't be allowed to bring the user hugely over quota, so by +default in v2.2+ this limit is 10% of the user's quota limit. (In v2.1 this is +disabled by default.) + +To change the quota grace, use: + +---%<------------------------------------------------------------------------- +plugin { + # allow user to become max 10% over quota + quota_grace = 10%% + # allow user to become max 50 MB over quota + quota_grace = 50 M +} +---%<------------------------------------------------------------------------- + +Maximum saved mail size +----------------------- + +(v2.2.29+) Dovecot allows specifying the maximum message size that is allowed +to be saved (e.g. by LMTP, IMAP APPEND or doveadm save). The default is 0, +which is unlimited. Since outgoing mail sizes are also typically limited on the +MTA size, it can be beneficial to prevent user from saving too large mails, +which would later on fail on the MTA side anyway. + +---%<------------------------------------------------------------------------- +plugin { + quota_max_mail_size = 100M +} +---%<------------------------------------------------------------------------- + +Quota admin commands +-------------------- + +The imap_quota plugin implements the SETQUOTA command, which allows changing +the logged in user's quota limit if the user is admin. Normally this means that +a master user must log in with 'userdb_admin=y' set in the master passdb. The +changing is done via dic_set() command, so you must configure the 'quota_set' +setting to point to some dictionary where your quota limit exists. Usually this +is in SQL, e.g.: + +dovecot.conf: + +---%<------------------------------------------------------------------------- +plugin { + quota_set = dict:proxy::sqlquota +} +dict { + sqlquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +dovecot-dict-sql.conf.ext: + +---%<------------------------------------------------------------------------- +# Use "host= ... pass=foo#bar" with double-quotes if your password has '#' +character. +connect = host=/var/run/mysqld/mysqld.sock dbname=mails user=admin +password=pass +# Alternatively you can connect to localhost as well: +#connect = host=localhost dbname=mails user=admin password=pass # port=3306 + +map { + pattern = priv/quota/limit/storage + table = quota + username_field = username + value_field = bytes +} +map { + pattern = priv/quota/limit/messages + table = quota + username_field = username + value_field = messages +} +---%<------------------------------------------------------------------------- + +Afterwards the quota can be changed with: + +---%<------------------------------------------------------------------------- +a SETQUOTA "User quota" (STORAGE 12345 MESSAGES 123) +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Quota.Count.txt b/doc/wiki/Quota.Count.txt new file mode 100644 index 0000000..4e461ec --- /dev/null +++ b/doc/wiki/Quota.Count.txt @@ -0,0 +1,44 @@ +Count quota +=========== + +(Requires Dovecot v2.2.19+) + +The quota is tracked internally within Dovecot's index files. Each mailbox's +quota is tracked separately and when the current quota usage is wanted to be +known, the mailboxes' quotas are summed up together. To get the best +performance mailbox list indexes should be enabled. + +*WARNING*: If you're switching from some other quota backend to count, make +sure that all the mails have their virtual sizes already indexed. Otherwise +there may be a significant performance hit when Dovecot starts opening all the +mails to get their sizes. You can help to avoid this by accessing the mailbox +vsizes for all the users before doing the configuration change:'doveadm mailbox +status -u user@domain vsize '*'' + +Configuration +------------- + +---%<------------------------------------------------------------------------- +mailbox_list_index = yes +# Avoid spending excessive time waiting for the quota calculation to finish +when +# mails' vsizes aren't already cached. If this many mails are opened, finish +the +# quota calculation on background in indexer-worker process. Mail deliveries +will +# be assumed to succeed, and explicit quota lookups will return internal error. +# (v2.2.28+) +mail_vsize_bg_after_count = 100 + +plugin { + # 10MB quota limit + quota = count:User quota + quota_rule = *:storage=10M + + # This is required - it uses "virtual sizes" rather than "physical sizes" for +quota counting: + quota_vsizes = yes +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Quota.Dict.txt b/doc/wiki/Quota.Dict.txt new file mode 100644 index 0000000..e346653 --- /dev/null +++ b/doc/wiki/Quota.Dict.txt @@ -0,0 +1,201 @@ +Dictionary quota +================ + +*NOTE:* Using the <count> [Quota.Count.txt] backend (possibly together with +<quota_clone> [Plugins.QuotaClone.txt] plugin) is now preferred to using this +backend. It has less of a chance of quota usage becoming wrong. + +The /dictionary/ quota backend supports both *storage* and *messages* quota +limits. The current quota is kept in the specified <dictionary> +[Dictionary.txt]. The available dictionaries include: + + * SQL + * Redis + * Flat files + +See <Dictionary.txt> for full description of the available backends. + +The quota root format is: + +---%<------------------------------------------------------------------------- +quota = dict:<quota root name>:<username>[:<option>[...]]:<dictionary URI> +---%<------------------------------------------------------------------------- + +If /username/ is left empty, the logged in username is used (this is typically +what you want). Another useful username is '%d' for supporting domain-wide +quotas. + +The supported options are: + + * noenforcing: Don't enforce quota limits, only track them. + * ignoreunlimited: If user has unlimited quota, don't track it. + * ns=<prefix>: This quota root is tracked only for the given namespace. + * hidden: Hide the quota root from IMAP GETQUOTA* commands. + * no-unset: When recalculating quota, don't unset the quota first. This is + needed if you wish to store the quota usage among other data in the same SQL + row - otherwise the entire row could get deleted. Note that the unset is + required with PostgreSQL or the merge_quota() trigger doesn't work + correctly. (v2.2.20+) + +NOTE: The dictionary stores only the current quota usage. The quota limits are +still configured in userdb the same way as with other quota backends. + +NOTE2: By default the quota dict may delete rows from the database when it +wants to rebuild the quota. You must use a separate table that contains only +the quota information, or you'll lose the other data. This can be avoided with +the "no-unset" parameter. + +Examples +-------- + +Simple per-user flat file +------------------------- + +This will create one quota-accounting file for each user. + +The Dovecot user process (imap, pop3 or lda) needs write access to this file, +so %h or mail_location are good candidates to store it. + +*Warning*: if a user has shell or file access to this location, he can mangle +his quota file, thus overcoming his quota limit by lying about his used +capacity. + +---%<------------------------------------------------------------------------- +plugin { + quota = dict:User quota::file:%h/mdbox/dovecot-quota + quota_rule = *:storage=10M:messages=1000 +} +---%<------------------------------------------------------------------------- + +Server-based dictionaries +------------------------- + +---%<------------------------------------------------------------------------- +plugin { + # SQL backend: + quota = dict:User quota::proxy::sqlquota + # Redis backend (v2.1.9+): + quota = dict:User quota::redis:host=127.0.0.1:prefix=user/ + + quota_rule = *:storage=10M:messages=1000 +} +dict { + sqlquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +The above SQL example uses dictionary proxy process (see below), because SQL +libraries aren't linked to all Dovecot binaries. The file and Redis examples +use direct access. + +Example 'dovecot-dict-sql.conf.ext': + +---%<------------------------------------------------------------------------- +connect = host=localhost dbname=mails user=sqluser password=sqlpass +map { + pattern = priv/quota/storage + table = quota + username_field = username + value_field = bytes +} +map { + pattern = priv/quota/messages + table = quota + username_field = username + value_field = messages +} +---%<------------------------------------------------------------------------- + +Create the table like this: + +---%<------------------------------------------------------------------------- +CREATE TABLE quota ( + username varchar(100) not null, + bytes bigint not null default 0, + messages integer not null default 0, + primary key (username) +); +---%<------------------------------------------------------------------------- + +MySQL uses the following queries to update the quota. You need suitable +privileges. + +---%<------------------------------------------------------------------------- +INSERT INTO table (bytes,username) VALUES ('112497180','foo@example.com') ON +DUPLICATE KEY UPDATE bytes='112497180'; +INSERT INTO table (messages,username) VALUES ('1743','foo@example.com') ON +DUPLICATE KEY UPDATE messages='1743'; +UPDATE table SET bytes=bytes-14433,messages=messages-2 WHERE username = +'foo@example.com'; +DELETE FROM table WHERE username = 'foo@example.com'; +---%<------------------------------------------------------------------------- + +If you're using SQLite, then take a look at the trigger in this post: +http://dovecot.org/pipermail/dovecot/2013-July/091421.html + +If you're using PostgreSQL, you'll need a trigger: + +---%<------------------------------------------------------------------------- +CREATE OR REPLACE FUNCTION merge_quota() RETURNS TRIGGER AS $$ +BEGIN + IF NEW.messages < 0 OR NEW.messages IS NULL THEN + -- ugly kludge: we came here from this function, really do try to insert + IF NEW.messages IS NULL THEN + NEW.messages = 0; + ELSE + NEW.messages = -NEW.messages; + END IF; + return NEW; + END IF; + + LOOP + UPDATE quota SET bytes = bytes + NEW.bytes, + messages = messages + NEW.messages + WHERE username = NEW.username; + IF found THEN + RETURN NULL; + END IF; + + BEGIN + IF NEW.messages = 0 THEN + INSERT INTO quota (bytes, messages, username) + VALUES (NEW.bytes, NULL, NEW.username); + ELSE + INSERT INTO quota (bytes, messages, username) + VALUES (NEW.bytes, -NEW.messages, NEW.username); + END IF; + return NULL; + EXCEPTION WHEN unique_violation THEN + -- someone just inserted the record, update it + END; + END LOOP; +END; +$$ LANGUAGE plpgsql; + +CREATE TRIGGER mergequota BEFORE INSERT ON quota + FOR EACH ROW EXECUTE PROCEDURE merge_quota(); +---%<------------------------------------------------------------------------- + +Dictionary proxy server +----------------------- + +To avoid each process making a new SQL connection, you can make all dictionary +communications go through a dictionary server process which keeps the +connections permanently open. + +The dictionary server is referenced with URI 'proxy:<dictionary server socket +path>:<dictionary name>'. The socket path may be left empty if you haven't +changed 'base_dir' setting in 'dovecot.conf'. Otherwise set it to +'<base_dir>/dict-server'. The dictionary names are configured in +'dovecot.conf'. For example: + +---%<------------------------------------------------------------------------- +dict { + quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext + expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +See <Dict.txt> for more information, especially about permission issues. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Quota.Dirsize.txt b/doc/wiki/Quota.Dirsize.txt new file mode 100644 index 0000000..ec28989 --- /dev/null +++ b/doc/wiki/Quota.Dirsize.txt @@ -0,0 +1,21 @@ +Dirsize quota +============= + +The /dirsize/ quota backend supports *storage* quota limits, but not *messages* +quota limits. The current quota is calculated by finding all files in +configured mail directories and summing up their sizes. This works pretty fast +with mboxes, but *dirsize is a really bad idea with maildirs*. It will end up +eating all your CPU and disk I/O. + +Example +------- + +---%<------------------------------------------------------------------------- +plugin { + # 10MB quota limit + quota = dirsize:User quota + quota_rule = *:storage=10M +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Quota.FS.txt b/doc/wiki/Quota.FS.txt new file mode 100644 index 0000000..7b59802 --- /dev/null +++ b/doc/wiki/Quota.FS.txt @@ -0,0 +1,122 @@ +Filesystem quota +================ + +Filesystem quota supports both local filesystems and rquota (NFS). + +If you are using systemd, please make sure you /turn off/ 'PrivateDevices=yes', +otherwise it won't work properly. Best way to do this is to use 'systemctl edit +dovecot' command or add file +'/etc/systemd/system/dovecot.service.d/override.conf' with + +---%<------------------------------------------------------------------------- +[Service] +PrivateDevices=off +---%<------------------------------------------------------------------------- + +Index files +----------- + +It's a good idea to keep index files in a partition where there are no +filesystem quota limits. The index files exist to speed up mailbox operations, +so Dovecot runs more slowly if it can't keep them updated. You can specify the +index file location by appending ':INDEX=/somewhere' to <mail_location> +[MailLocation.txt]. + +Dovecot can handle "out of disk space" errors in index file handling and +transparently move to in-memory indexes. It'll use the in-memory indexes until +the mailbox is re-opened. + +mbox +---- + +It's a good idea to have 'mbox_lazy_writes=yes' (default), otherwise Dovecot +might give "Not enough disk space" errors when opening the mailbox, making it +impossible to expunge any mails. + +If user has run out of quota and index files are also in memory (because +they're also over quota), it's possible that message flag changes are lost. +This should be pretty rare though because Dovecot keeps some extra space +allocated inside the mbox file for flag changes. + +Example preferred configuration: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=/var/no-quotas/index/%u +---%<------------------------------------------------------------------------- + +Maildir +------- + +Maildir needs to be able to add UIDs of new messages to 'dovecot-uidlist' file. +If it can't do this, it can give an error when opening the mailbox, making it +impossible to expunge any mails. + +Currently the only way to avoid this is to use a separate partition for the +uidlist files where there are no filesystem quota limits. You can do this by +appending ':CONTROL=/somewhere' to <mail_location> [MailLocation.txt]. + +Example preferred configuration: + +---%<------------------------------------------------------------------------- +mail_location = +maildir:~/Maildir:INDEX=/var/no-quotas/index/%u:CONTROL=/var/no-quotas/control/%u +---%<------------------------------------------------------------------------- + +Note that if you change the location of the control files, Dovecot will look in +the new CONTROL directory ('/var/no-quotas/control/%u') for the subscriptions +file. + +Parameters +---------- + +By default only user quota is shown, or if it doesn't exist, group quota is +used as fallback. + + * user: Report only user quotas, don't fallback to showing group quotas. + * group: Report only group quotas + * inode_per_mail: Report inode quota as "number of message" quota. This can be + useful with Maildir or single-dbox. + * noenforcing: Don't try to enforce quotas by calculating if saving would get + user over quota. Only handle write failures. + * mount=<path>: Report quota from given path. Default is to use the path for + the mail root directory. + +If you want to give multiple parameters, separate them with ':' (e.g. +"inode_per_mail:noenforcing"). + +Examples +-------- + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins quota +protocol imap { + mail_plugins = mail_plugins imap_quota +} + +plugin { + quota = fs:user +} +---%<------------------------------------------------------------------------- + +If you want to see both user and group quotas as separate quota roots, you can +use this: + +---%<------------------------------------------------------------------------- +plugin { + quota = fs:User quota:user + quota2 = fs:Group quota:group +} +---%<------------------------------------------------------------------------- + +If you have your mails in two filesystems, you can create two quota roots: + +---%<------------------------------------------------------------------------- +plugin { + # Assuming INBOX in /var/mail/ which is mounted to / + quota = fs:INBOX:mount=/ + # Assuming other mailboxes are in /home mount + quota2 = fs:Others:mount=/home +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Quota.Maildir.txt b/doc/wiki/Quota.Maildir.txt new file mode 100644 index 0000000..d6ccc85 --- /dev/null +++ b/doc/wiki/Quota.Maildir.txt @@ -0,0 +1,51 @@ +Maildir++ quota +=============== + +Maildir++ is the most commonly used quota backend with Maildir format. Note +that *Maildir++ quota works only with Maildir format*. With other mailbox +formats you should use <count quota> [Quota.Count.txt]. Dovecot implements the +standard Maildir++ specification +[http://www.courier-mta.org/imap/README.maildirquota.html], so it's compatible +with Courier [http://www.courier-mta.org/], maildrop +[http://www.courier-mta.org/maildrop/], Exim [http://www.exim.org], etc. + +There are two ways to configure Maildir++ quota limits: + + 1. Configure the limits in Dovecot. You most likely want to do this. See + <quota main page> [Quota.txt] for how to do this configuration. + 2. Make Dovecot get the limits from existing 'maildirsize' files. + +Only Maildir++-specific settings are described below. See <Quota.txt> for more +generic configuration. + +Maildir++ quota relies on 'maildirsize' file having correct information, so if +your users can modify the file in some way (e.g. shell access), you're relying +on the goodwill of your users for the quota to work. + +You can't rely on Dovecot noticing external changes to Maildir and updating +maildirsize accordingly. This happens eventually when quota is being +recalculated, but it may take a while. Quota recalculation also currently +doesn't trigger quota warning executions. + +Maildirsize file +---------------- + +The 'maildirsize' file in the Maildir root directory contains both the quota +limit information and the current quota status. It contains a header in format: + +---%<------------------------------------------------------------------------- +<storage limit in bytes>S,<messages limit>C +---%<------------------------------------------------------------------------- + +If you don't configure any quota limits in Dovecot ('quota=maildir' with no +other settings), Dovecot takes the limits from the header. If the file does not +exist, quota isn't enforced. + +If you configure quota limits in Dovecot, Dovecot makes sure that this header +is kept up to date. If the file does not exist, it's simply rebuilt. + +Once the 'maildirsize' reaches 5120 bytes, the quota is recalculated and the +file is recreated. This makes sure that if quota happens to be broken (e.g. +externally deleted files) it won't stay that way forever. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Quota.txt b/doc/wiki/Quota.txt new file mode 100644 index 0000000..6d82d2a --- /dev/null +++ b/doc/wiki/Quota.txt @@ -0,0 +1,134 @@ +Quota +===== + +Quota backend specifies the method how Dovecot keeps track of the current quota +usage. They don't (usually) specify users' quota limits, that's done by +<returning extra fields from userdb> [Quota.Configuration.txt]. There are +different quota backends that Dovecot can use: + + * <fs> [Quota.FS.txt]: Filesystem quota. + * <dirsize> [Quota.Dirsize.txt]: The simplest and slowest quota backend, but + it works quite well with mboxes. + * <dict> [Quota.Dict.txt]: Store quota usage in a dictionary (e.g. SQL, or + flat files). + * <maildir> [Quota.Maildir.txt]: Store quota usage in Maildir++ maildirsize + files. This is the most commonly used quota for virtual users. + * <count> [Quota.Count.txt]: Store quota usage within Dovecot's index files. + * <imapc> [MailboxFormat.imapc.txt]: Use quota from remote IMAP server with + imapc. + +Since v2.2.19 we recommend using <count> [Quota.Count.txt] for any new +installations. If you need usage data to an external database, consider using +<Plugins.QuotaClone.txt> for exporting the information. + +Quota service +------------- + +The quota service allows postfix to check quota before delivery: + +---%<------------------------------------------------------------------------- +service quota-status { + executable = quota-status -p postfix + inet_listener { + port = 12340 + # You can choose any port you want + } + client_limit = 1 +} +---%<------------------------------------------------------------------------- + +And then have postfix check_policy_service check that: + +---%<------------------------------------------------------------------------- +smtpd_recipient_restrictions = + ... + check_policy_service inet:mailstore.example.com:12340 +---%<------------------------------------------------------------------------- + +For more about this service see +https://sys4.de/en/blog/2013/04/08/postfix-dovecot-mailbox-quota/ + +Enabling quota plugins +---------------------- + +There are three quota related plugins: + + * quota: Implements the actual quota handling and includes also all the quota + backends. + * imap_quota: For reporting quota information via IMAP. + * quota_grace: Determines if and how far user can go over quota + +Enable them in configuration files, e.g.: + +conf.d/10-mail.conf: + +---%<------------------------------------------------------------------------- +# Space separated list of plugins to load for all services. Plugins specific to +# IMAP, LDA, etc. are added to this list in their own .conf files. +mail_plugins = $mail_plugins quota +---%<------------------------------------------------------------------------- + +conf.d/20-imap.conf: + +---%<------------------------------------------------------------------------- +protocol imap { + # Space separated list of plugins to load (default is global mail_plugins). + mail_plugins = $mail_plugins imap_quota +} +---%<------------------------------------------------------------------------- + +conf.d/90-quota.conf: (for use with the quota-status service) + +---%<------------------------------------------------------------------------- +plugin { + quota_grace = 10%% + # 10% is the default + quota_status_success = DUNNO + quota_status_nouser = DUNNO + quota_status_overquota = "552 5.2.2 Mailbox is full" +} +---%<------------------------------------------------------------------------- + +Configuration +------------- + +See <Quota.Configuration.txt> for backend-independent quota configuration. + +Quota recalculation +------------------- + +If your quotas are out of sync, you can use 'doveadm quota recalc -u <uid>' +command to recalculate them. + +Quota and Trash mailbox +----------------------- + +Standard way to expunge messages with IMAP works by: + + 1. Marking message with \Deleted flag + 2. Actually expunging the message using EXPUNGE command + +Both of these commands can be successfully used while user's quota is full. +However many clients use a "move-to-Trash" feature, which works by: + + 1. COPY the message to Trash mailbox + 2. Mark the message with \Deleted + 3. Expunge the message from the original mailbox. + 4. (Maybe later expunge the message from Trash when "clean trash" feature is + used) + +If user is over quota (or just under it), the first COPY command will fail and +user may get an unintuitive message about not being able to delete messages +because user is over quota. The possible solutions for this are: + + * Disable move-to-trash feature from client + * You can create a separate quota rule ignoring Trash mailbox's quota. Note + that this would allow users to store messages infinitely to the mailbox. + * You can create a separate quota rule giving Trash mailbox somewhat higher + quota limit (but not unlimited). + +To make sure users don't start keeping messages permanently in Trash you can +use a nightly <cronjob> [Plugins.Expire.txt] to expunge old messages from Trash +mailbox. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Replication.txt b/doc/wiki/Replication.txt new file mode 100644 index 0000000..58d25a9 --- /dev/null +++ b/doc/wiki/Replication.txt @@ -0,0 +1,382 @@ +Replication with dsync +====================== + +Dovecot supports master/master replication using dsync. It's recommended that +the same user always gets redirected to the same replica, but no changes get +lost even if the same user modifies mails simultaneously on both replicas, some +mails just might have to be redownloaded. The replication is done +asynchronously, so high latency between the replicas isn't a problem. The +replication is done by looking at Dovecot index files (not what exists in +filesystem), so no mails get lost due to filesystem corruption or an accidental +rm -rf, they will simply be replicated back. + +NOTE: v2.2 is highly recommended for this. Earlier versions can't do +incremental metadata syncing. This means that the more mails a mailbox has, the +slower it is to sync it. + +Replication works only between server pairs. If you have a large cluster, you +need multiple independently functioning Dovecot backend pairs. This means that +<director> [Director.txt] isn't currently supported with replication. The +replication in general is a bit resource intensive, so it's not recommended to +be used in multi-million user installations. + +*WARNING*: Shared folder replication doesn't work correctly right now – mainly +it can generate a lot of duplicate emails. This is because there's currently a +per-user lock that prevents multiple dsyncs from working simultaneously on the +same user. But with shared folders multiple users can be syncing the same +folder. So this would need additional locks (e.g. shared folders would likely +need to lock the owner user, and public folders would likely need a per-folder +lock or a maybe a global public folder lock). Fixing this is currently low +priority for Dovecot developers. + +Configuration +------------- + +Since v2.3.1 you can disable replication for a user by providing 'noreplicate' +<user database field> [UserDatabase.ExtraFields.txt]. Another way to disable +replication for some users is to return mail_replica field from userdb for +users you want to replicate. + +Make sure that user listing is configured for your userdb, this is required by +replication to find the list of users that are periodically replicated: + +---%<------------------------------------------------------------------------- +doveadm user '*' +---%<------------------------------------------------------------------------- + +Enable the replication plugin globally (most likely you'll need to do this in +10-mail.conf): + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins notify replication +---%<------------------------------------------------------------------------- + +Replicator process should be started at startup, so it can start replicating +users immediately: + +---%<------------------------------------------------------------------------- +service replicator { + process_min_avail = 1 +} +---%<------------------------------------------------------------------------- + +You need to configure how and where to replicate. Using SSH for example: + +---%<------------------------------------------------------------------------- +dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u +plugin { + mail_replica = remote:vmail@anotherhost.example.com +} +---%<------------------------------------------------------------------------- + +The mail processes need to have access to the replication-notify fifo and +socket. If you have a single vmail UID, you can do: + +---%<------------------------------------------------------------------------- +service aggregator { + fifo_listener replication-notify-fifo { + user = vmail + } + unix_listener replication-notify { + user = vmail + } +} +---%<------------------------------------------------------------------------- + +The replication-notify only notifies the replicator processes that there is +work to be done, so it's not terribly insecure either to just set 'mode=0666'. + +Enable doveadm replicator commands by setting: + +---%<------------------------------------------------------------------------- +service replicator { + unix_listener replicator-doveadm { + mode = 0600 + user = vmail + } +} +---%<------------------------------------------------------------------------- + +You can configure how many dsyncs can be run in parallel (10 by default): + +---%<------------------------------------------------------------------------- +replication_max_conns = 10 +---%<------------------------------------------------------------------------- + +Normally all replication is asynchronous. You can also optionally configure new +mail saving to be synchronous, with a timeout to avoid waiting too long. This +can be configured with: + +---%<------------------------------------------------------------------------- +plugin { + # When saving a new mail via IMAP or delivering a mail via LDA/LMTP, + # wait for the mail to be synced to the remote site. If it doesn't finish + # in 2 seconds, return success anyway. + replication_sync_timeout = 2s +} +---%<------------------------------------------------------------------------- + +dsync over TCP connections (v2.2+) +---------------------------------- + +Create a listener for doveadm-server: + +---%<------------------------------------------------------------------------- +service doveadm { + inet_listener { + port = 12345 + } +} +---%<------------------------------------------------------------------------- + +And tell doveadm client to use this port by default: + +---%<------------------------------------------------------------------------- +doveadm_port = 12345 +---%<------------------------------------------------------------------------- + +Both the client and the server also need to have a shared secret: + +---%<------------------------------------------------------------------------- +doveadm_password = secret +---%<------------------------------------------------------------------------- + +Now you can use 'tcp:hostname' as the dsync target. You can also override the +port with 'tcp:hostname:port'. + +---%<------------------------------------------------------------------------- +plugin { + mail_replica = tcp:anotherhost.example.com # use doveadm_port + #mail_replica = tcp:anotherhost.example.com:12345 # use port 12345 explicitly +} +---%<------------------------------------------------------------------------- + +SSL +--- + +You can also use SSL for the connection: + +---%<------------------------------------------------------------------------- +service doveadm { + inet_listener { + port = 12345 + ssl = yes + } +} +---%<------------------------------------------------------------------------- + +The doveadm listener will use the SSL certificate that is configured globally +for all SSL listeners, i.e. via the following settings at the top level of the +configuration file: + +---%<------------------------------------------------------------------------- +ssl_cert = </etc/ssl/dovecot.pem +ssl_key = </etc/ssl/dovecot.pem +---%<------------------------------------------------------------------------- + +'ssl_cert' is not a valid setting inside 'service' or 'inet_listener' blocks, +so you can't use a separate SSL certificate for the doveadm listener.You can, +however, use separate SSL certificates for the /other/ protocols, like so: + +---%<------------------------------------------------------------------------- +protocol imap { + ssl_cert = </etc/ssl/certs/imap.pem + ssl_key = </etc/ssl/private/imap.pem +} +protocol pop3 { + ssl_cert = </etc/ssl/certs/pop3.pem + ssl_key = </etc/ssl/private/pop3.pem +} +---%<------------------------------------------------------------------------- + +When one Dovecot instance connects to the other one in the replication pair, it +has to verify that the partner's SSL certificate is valid, so you need to +specify a directory or file containing valid SSL CA roots: + +---%<------------------------------------------------------------------------- +ssl_client_ca_dir = /etc/ssl/certs # Debian/Ubuntu +ssl_client_ca_file = /etc/pki/tls/cert.pem # RedHat +---%<------------------------------------------------------------------------- + +Now you can use 'tcps:hostname' or 'tcps:hostname:port' as the dsync target. + +Note that the SSL certificate must be signed by one of the CAs in the +'ssl_client_ca_dir' or 'ssl_client_ca_file'. You can't use a self-signed +certificate or a private CA, unless you correctly set them up into the CA +file/directory (see openssl documentation for details). + +You could point 'ssl_client_ca_file' to your private CA, but keep in mind that +'ssl_client_ca_file' and 'ssl_client_ca_dir' also affect other services where +Dovecot acts as an SSL client (e.g. the imapc feature), so be careful not to +break SSL for those services. + +dsync wrapper script for root SSH login (v2.2+) +----------------------------------------------- + +If you're using multiple UIDs, dsync needs to be started as root, which means +you need to log in as root with ssh (or use sudo). Another possibility is to +allow root to run only a wrapper script. There is some built-in support for +this in v2.2+ to make it easier: + +dovecot.conf: + +---%<------------------------------------------------------------------------- +dsync_remote_cmd = /usr/bin/ssh -i /root/.ssh/id_dsa.dsync %{host} +/usr/local/bin/dsync-in-wrapper.sh +plugin { + mail_replica = remoteprefix:vmail@anotherhost.example.com +} +---%<------------------------------------------------------------------------- + +/root/.ssh/authorized_keys: + +---%<------------------------------------------------------------------------- +command="/usr/local/bin/dsync-in-wrapper.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty +<ssh key> +---%<------------------------------------------------------------------------- + +/usr/local/bin/dsync-in-wrapper.sh: + +---%<------------------------------------------------------------------------- +read username +ulimit -c unlimited # for debugging any crashes +/usr/local/bin/doveadm dsync-server -u $username +---%<------------------------------------------------------------------------- + +dsync parameters +---------------- + +With v2.2.9+ you can configure what parameters replicator uses for the 'doveadm +sync' command: + +---%<------------------------------------------------------------------------- +replication_dsync_parameters = -d -N -l 30 -U +---%<------------------------------------------------------------------------- + +The '-f' and '-s' parameters are added automatically when needed. + +Usually the only change you may want to do is replace '-N' (= sync all +namespaces) with '-n <namespace>' or maybe just add '-x <exclude>' +parameter(s). + +Administration +-------------- + +'doveadm replicator status' provides a summary. For example: + +---%<------------------------------------------------------------------------- +Queued 'sync' requests 0 +Queued 'high' requests 0 +Queued 'low' requests 0 +Queued 'failed' requests 0 +Queued 'full resync' requests 90 +Waiting 'failed' requests 10 +Total number of known users 100 +---%<------------------------------------------------------------------------- + +The first 3 fields describe users who have a replication pending with a +specific priority. The same user can only be in one (or none) of these queues: + + * Queued 'sync' requests: This priority is used only for mail saves if + 'replication_sync_timeout' setting is used. + * Queued 'high' requests: This priority is used only for mail saves if + 'replication_sync_timeout' setting is not used, or if the sync request timed + out. + * Queued 'low' requests: This priority is used for everything else except mail + saves. + +The following fields are: + + * Queued 'failed' requests: Number of users who have a replication pending and + where the last sync attempt failed. These users are retried as soon as + higher priority users' replication has finished. + * Queued 'full resync' requests: Number of users who don't specifically have + any replication pending, but who are currently waiting for a periodic "full + sync". This is controlled by the 'replication_full_sync_interval' setting. + * Waiting 'failed' requests: Number of users whose last replication attempt + failed, and we're now waiting for the retry interval (5 mins) to pass before + another attempt. + * Total number of known users: Number of users that replicator knows about. + The users can be listed with:'doveadm replicator status '*'' + +The per-user replication status can be shown with 'doveadm replicator status +<username pattern>'. The username pattern can contain '*' and '?' wildcards. +The response contains for example: + +---%<------------------------------------------------------------------------- +username priority fast sync full sync success sync failed +test100 none 02:03:52 02:08:52 02:03:52 - +test1 none 00:00:01 00:43:33 03:20:46 y +test2 none 02:03:51 02:03:51 02:03:51 - +---%<------------------------------------------------------------------------- + +These fields mean: + + * priority: none, low, high, sync + * fast sync: How long time ago the last "fast sync" (non-full sync) attempt + was performed. Ideally this is close to the time when the user was last + modified. This doesn't mean that the sync succeeded necessarily. + * full sync: How long time ago the last "full sync" attempt was performed. + This should happen once per 'replication_full_sync_interval'. This doesn't + mean that the sync succeeded necessarily. + * success sync: Time when the last successful sync was performed. If the last + sync succeeded, this is the same as the "fast sync" or the "full sync" + timestap. + * failed: "y" if the last sync failed, "-" if not. + +The current dsync replication status can be looked up with 'doveadm replicator +dsync-status'. This shows the dsync replicator status for each potential dsync +connection, as configured by 'replication_max_conns'. An example output is: + +---%<------------------------------------------------------------------------- +username type status +test100 full Waiting for dsync to finish +test1 normal Waiting for handshake + - Not connected + - Not connected +---%<------------------------------------------------------------------------- + +Here there are 4 lines, meaning 'replication_max_conns=4'. Only two of the +dsync-connections are being used currently. + +The fields mean: + + * username: User currently being replicated. + * type: incremental, normal or full. Most of the replications are + "incremental", while full syncs are "full". A "normal" sync is done when + incremental syncing state isn't available currently. The "incremental" + matches doveadm sync's -s parameter, "full" is -f parameter and "normal" is + the default. + * status: Human-readable status of the connection. These are the current + values: + * Not connected + * Failed to connect to '%s' - last attempt %ld secs ago + * Idle + * Waiting for handshake + * Waiting for dsync to finish + +Failed replication attempts are always automatically retried, so any temporary +problems should get fixed automatically. In case of bugs it may be necessary to +fix something manually. These should be visible in the error logs. So if a user +is marked as failed, try to find any errors logged for the user and see if the +same error keeps repeating in the logs. If you want to debug the dsync, you can +manually trigger it with:'doveadm -D sync -u user@domain -d -N -l 30 -U' (the +parameters after "sync" should be the same as in 'replication_dsync_parameters' +setting). + +Notes +----- + +Random things to remember: + + * The replicas can't share the same quota database, since both will always + update it + * With mdbox format "doveadm purge" won't be replicated + * "doveadm force-resync", "doveadm quota recalc" and other similar fixing + commands don't get replicated + * The servers must have different hostnames or the locking doesn't work and + can cause replication problems. + * v2.2.6+: If you're having trouble, verify that 'dovecot --hostdomain' + returns different values for the servers. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/RunningDovecot.txt b/doc/wiki/RunningDovecot.txt new file mode 100644 index 0000000..0520846 --- /dev/null +++ b/doc/wiki/RunningDovecot.txt @@ -0,0 +1,140 @@ +Running Dovecot +=============== + +Starting +-------- + +Dovecot can simply be started by running 'dovecot' as root. If there are any +problems, they're usually written to terminal, but they may also be written to +<error log> [Logging.txt] as well. + + * A sample universal <init.d script> [DovecotInit.txt]. + * A sample Mac OS X 10.4 <launchd plist file> [LaunchdInstall.txt] + +Stopping +-------- + +Killing the Dovecot master process with a normal TERM signal does a clean +shutdown. This can be done easily with: + +---%<------------------------------------------------------------------------- +doveadm stop +---%<------------------------------------------------------------------------- + +'shutdown_clients' setting controls whether existing IMAP and POP3 sessions are +killed. + +Processes +--------- + +When Dovecot is running, it uses several processes: + +---%<------------------------------------------------------------------------- +# ps auxw|grep "dovecot" +root 7245 0.1 0.1 2308 1096 pts/0 S+ 19:53 0:00 dovecot +dovecot 7246 0.0 0.0 2084 824 pts/0 S+ 19:53 0:00 dovecot/anvil +root 7247 0.0 0.0 2044 908 pts/0 S+ 19:53 0:00 dovecot/log +root 7250 0.0 0.3 4988 3740 pts/0 S+ 19:53 0:00 dovecot/config +root 7251 0.0 0.2 10024 2672 pts/0 S+ 19:53 0:00 dovecot/auth +root 7303 0.6 0.3 10180 3116 pts/0 S+ 19:57 0:00 dovecot/auth +-w +vmail 7252 0.0 0.1 3180 1264 pts/0 S+ 19:53 0:00 dovecot/imap +vmail 7255 0.0 0.1 3228 1596 pts/0 S+ 19:54 0:00 dovecot/pop3 +dovenull 7260 0.0 0.1 4028 1940 pts/0 S+ 19:54 0:00 +dovecot/imap-login +dovenull 7262 0.0 0.1 4016 1916 pts/0 S+ 19:54 0:00 +dovecot/pop3-login +---%<------------------------------------------------------------------------- + + * 'dovecot' process is the Dovecot master process which keeps everything + running. + * 'anvil' keeps track of user connections + * 'log' writes to log files. All logging, except from master process, goes + through it. + * 'config' parses the configuration file and sends the configuration to other + processes. + * 'auth' handles all authentication. + * 'auth -w' process is an "authentication worker" process. It's used only with + some "blocking" authentication databases, such as MySQL. + * 'imap-login' and 'pop3-login' processes handle new IMAP and POP3 connections + until user has logged in. They also handle proxying SSL connections even + after login. + * 'imap' and 'pop3' processes handle the IMAP and POP3 connections after user + has logged in. + +Reloading Configuration +----------------------- + +Sending HUP signal to Dovecot reloads configuration. This can be done easily +with: + +---%<------------------------------------------------------------------------- +doveadm reload +---%<------------------------------------------------------------------------- + +An acknowledgement is written to log file: + +---%<------------------------------------------------------------------------- +Jun 14 19:59:59 master: Warning: SIGHUP received - reloading configuration +---%<------------------------------------------------------------------------- + +Running Multiple Invocations of Dovecot +--------------------------------------- + +You may wish to invoke a second session (or even multiple sessions) of Dovecot +for testing different functionality, configurations, etc. In order to run +multiple instances of Dovecot, you must: + + 1. Create a differently named copy of the dovecot.conf configuration file with + these changes: + 1. Change 'base_dir' to the new run directory + 2. Change services' inet_listener port numbers to new, unused values (in + '10-master.conf'). + 3. Optionally change 'instance_name' to show a different "dovecot/" prefix + in ps output. (v2.0.18+) + 4. If you're using authentication sockets (for SMTP AUTH or deliver), + you'll need to change them as well.'auth_socket_path' specifies the + socket path for deliver. + * Alternatively if all the instances have identical authentication + configuration, you can have only a single Dovecot instance serve the + auth sockets and have the other instances use them. + 2. Invoke 'dovecot' (and 'dovecot-lda') with the '-c' parameter and the + modified configuration file, e.g.:'dovecot -c /usr/local/etc/dovecot2.conf' + + 3. In order to tell the logs apart, you can set different log facilities for + the instances, e.g.'syslog_facility=local6', then configure syslogd to + write local6 into "dovecot-otherinstance.log". Alternatively specify the + log paths directly in 'log_path' and related settings. + +Rotating Log Files +------------------ + +If you specified log file paths manually in 'dovecot.conf' instead of using +syslog, you can send USR1 signal to Dovecot to make it close and reopen the log +files. This can be done easily with: + +---%<------------------------------------------------------------------------- +doveadm log reopen +---%<------------------------------------------------------------------------- + +Troubleshooting +--------------- + +If you can't see the Dovecot processes running after starting 'dovecot', +something is most likely wrong in your 'dovecot.conf'. Look at the error from +Dovecot's log file. See <Logging.txt> for how to find the log. + +If you really can't find any error messages from any logs, try starting Dovecot +with 'dovecot -F'. If you see it crash like: + +---%<------------------------------------------------------------------------- +sh: segmentation fault (core dumped) dovecot -F +---%<------------------------------------------------------------------------- + +Then it's a bug in Dovecot. Please report it with your configuration file. + +If it simply quits without giving any error, then it wrote the error to a log +file and you just didn't find it. Try specifying the log file manually and make +sure you're really looking at the correct file. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SSL.CertificateClientImporting.txt b/doc/wiki/SSL.CertificateClientImporting.txt new file mode 100644 index 0000000..94be09e --- /dev/null +++ b/doc/wiki/SSL.CertificateClientImporting.txt @@ -0,0 +1,65 @@ +SSL certificate importing to clients +==================================== + +You may import either the server's self-signed certificate or the CA +certificate (see <SSL.CertificateCreation.txt>). + +Windows XP +---------- + +Import to Trusted Root Certification Authorities store: +http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sag_cmprocsimport.mspx?mfr=true + +Outlook and Outlook Express uses the Windows's certificate store. + +Mac OS X +-------- + + * Doubleclick the certificate + * Keychain should open + * Add the certificate to X509 Anchors keychain + +Apple Mail uses the OS X's certificate store. + +Thunderbird +----------- + +Preferences -> Privacy -> Security -> View Certificates -> Authorities -> +Import -> Trust this CA to identify email users. + +Opera Mail +---------- + +Preferences -> Advanced > Security > Certificates > Import certificate file. + +Evolution +--------- + +Preferences -> Certificates -> Authorities -> Import -> Trust this CA to +identify email users. + +Pine +---- + +http://www.madboa.com/geek/pine-ssl/ tells a story how to do this. Basically it +seems to be: + + 1. Find out your OPENSSLDIR: 'openssl version -d' + 2. Get a hash of your certificate: 'openssl x509 -in cert.pem -hash -noout' + 3. Copy the certificate to '$OPENSSLDIR/certs/$hash.0' + +This probably works only for self-signed certificates. + +KMail +----- + +Instructions needed. + +Claws Mail +---------- + +Configuration -> Edit accounts (Choose here your's one and press 'Edit'-button) + +Account -> SSL -> Certificate for receiving->Browse + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SSL.CertificateCreation.txt b/doc/wiki/SSL.CertificateCreation.txt new file mode 100644 index 0000000..59dcfe0 --- /dev/null +++ b/doc/wiki/SSL.CertificateCreation.txt @@ -0,0 +1,93 @@ +SSL certificate creation +======================== + +Self-signed SSL certificates +---------------------------- + +Self-signed SSL certificates are the easiest way to get your SSL server +working. However unless you take some action to prevent it,*this is at the cost +of security*: + + * The first time the client connects to the server, it sees the certificate + and asks the user whether to trust it. The user of course doesn't really + bother verifying the certificate's fingerprint, so a man-in-the-middle + attack can easily bypass all the SSL security, steal the user's password and + so on. + * If the client was lucky enough not to get attacked the first time it + connected, the following connections will be secure as long as the client + had permanently saved the certificate. Some clients do this, while others + have to be manually configured to accept the certificate. + +The only way to be fully secure is to import the SSL certificate to client's +(or operating system's) list of trusted CA certificates prior to first +connection. See <SSL.CertificateClientImporting.txt> how to do it for different +clients. + +Self-signed certificate creation +-------------------------------- + +Dovecot includes a script to build self-signed SSL certificates using OpenSSL. +In the source distribution this exists in doc/mkcert.sh +[http://dovecot.org/doc/mkcert.sh]. Binary installations usually create the +certificate automatically when installing Dovecot and don't include the script. + +The SSL certificate's configuration is taken from doc/dovecot-openssl.cnf +[http://dovecot.org/doc/dovecot-openssl.cnf] file. Modify the file before +running mkcert.sh. Especially important field is the CN (Common Name) field, +which should contain your server's host name. The clients will verify that the +CN matches the connected host name, otherwise they'll say the certificate is +invalid. It's also possible to use wildcards (eg. *.domain.com) in the host +name. They should work with most clients. + +By default the certificate is created to '/etc/ssl/certs/dovecot.pem' and the +private key file is created to '/etc/ssl/private/dovecot.pem'. Also by default +the certificate will expire in 365 days. If you wish to change any of these, +modify the mkcert.sh script. + +Certificate Authorities +----------------------- + +The correct way to use SSL is to have each SSL certificate signed by an +Certificate Authority (CA). The client has a list of trusted Certificate +Authorities, so whenever it sees a new SSL certificate signed by a trusted CA, +it will automatically trust the new certificate without asking the user any +questions. + +There are two ways to get a CA signed certificate: get it from an external CA, +or create your own CA. + +The clients have a built-in list of trusted CAs, so getting it from one of +those CAs will have the advantage of the certificate working without any client +configuration. There are already-trusted-CAs where you can buy the certificate, +and there are already-trusted-CAs where you can get your certificate for free. +On the oher hand, if you create your own CA, you'll have to install the CA +certificate to all the clients (see <SSL.CertificateClientImporting.txt>). + +So the two options end up being three: + + * a) Get it from an external CA (which already have the public keys installed + on the clients, so the clients trust the CA) + * a.1) Purchase the certificate. + * a.2) Get a free certificate. + * b) Create your own CA (in this case you'll have to add the CA public keys + into the clients, as you are bot trusted by default) + +If you choose "a.2)", there is letsencrypt.org [https://letsencrypt.org/] where +you need to do some technical effort to demonstrate to a robot that you own the +domains for which the certificate is issued. Let's encrypt is an initiative of +the Internet Security Research Group (ISRG), with board members from Cisco, +Mozilla and the University of Michingan among others and technical advisors +from Akamai, Google, Electronic Frontier Foundation, Internet Sociaty and +independents among others. Let's encrypt CA is trusted by default by many +clients. If you can control the entries in your DNS, you'll be able to +demonstrate to the robot that "you are you" for domain-based identities. There +is this specific conversation about letsencrypt + dovecot here +[https://community.letsencrypt.org/t/simple-guide-using-lets-encrypt-ssl-certs-with-dovecot/2921]. + +If you choose "b)", there are multiple different tools for managing your own +CA. The simplest way is to use a CA managing tool as gnoMint +[http://gnomint.sourceforge.net/] or TinyCA [http://tinyca.sm-zone.net/]. +However, if you need to tailor the properties of the CA, you always can use +OpenSSL, very much customizable, but however a bit cumbersome. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SSL.DovecotConfiguration.txt b/doc/wiki/SSL.DovecotConfiguration.txt new file mode 100644 index 0000000..fb70671 --- /dev/null +++ b/doc/wiki/SSL.DovecotConfiguration.txt @@ -0,0 +1,522 @@ +Dovecot SSL configuration +========================= + +Contents + + + 1. Dovecot SSL configuration + + 1. Multiple SSL certificates + + 1. Different certificates per algorithm + + 2. Different certificates per IP and protocol + + 3. With client TLS SNI (Server Name Indication) support + + 2. Password protected key files + + 3. Chained SSL certificates + + 4. SSL security settings + + 5. SSL verbosity + + 6. Client certificate verification/authentication + + 7. Testing + + 8. Testing CA + + 1. Testing CA On Debian + + 2. Testing CA On RHEL + + 3. Testing CA Success + + 9. Client connections + +The most important SSL settings are (in 'conf.d/10-ssl.conf'): + +---%<------------------------------------------------------------------------- +ssl = yes +# Preferred permissions: root:root 0444 +ssl_cert = </etc/ssl/certs/dovecot.pem +# Preferred permissions: root:root 0400 +ssl_key = </etc/ssl/private/dovecot.pem +---%<------------------------------------------------------------------------- + +The '<' is mandatory. It indicates that the variable should contain *contents* +of the file, instead of the file name. Not using it will cause an error. + +The certificate file can be world-readable, since it doesn't contain anything +sensitive (in fact it's sent to each connecting SSL client). The key file's +permissions should be restricted to only root (and possibly ssl-certs group or +similar if your OS uses such). Dovecot opens both of these files while still +running as root, so you don't need to give Dovecot any special permissions to +read them (in fact:*do not give dovecot user any permissions to the key file*). + +It's possible to keep the certificate and the key both in the same file: + +---%<------------------------------------------------------------------------- +# Preferred permissions: root:root 0400 +ssl_cert = </etc/ssl/dovecot.pem +ssl_key = </etc/ssl/dovecot.pem +---%<------------------------------------------------------------------------- + +It's also possible to use different certificates for IMAP and POP3. However its +important to note that "ssl = yes" must be set globally if you require SSL for +any protocol (or dovecot will not listen on the SSL ports), which in turn +requires that a certificate and key are specified globally even if you intend +to specify certificates per protocol. The per protocol certificate settings +override the global setting.: + +---%<------------------------------------------------------------------------- +protocol imap { + ssl_cert = </etc/ssl/certs/imap.pem + ssl_key = </etc/ssl/private/imap.pem +} +protocol pop3 { + ssl_cert = </etc/ssl/certs/pop3.pem + ssl_key = </etc/ssl/private/pop3.pem +} +---%<------------------------------------------------------------------------- + +There are a couple of different ways to specify when SSL/TLS is required: + + * 'ssl=no': SSL/TLS is completely disabled. + * 'ssl=yes' and 'disable_plaintext_auth=no': SSL/TLS is offered to the client, + but the client isn't required to use it. The client is allowed to login with + plaintext authentication even when SSL/TLS isn't enabled on the connection. + This is insecure, because the plaintext password is exposed to the internet. + * 'ssl=yes' and 'disable_plaintext_auth=yes': SSL/TLS is offered to the + client, but the client isn't required to use it. The client isn't allowed to + use plaintext authentication, unless SSL/TLS is enabled first. However, if + <non-plaintext authentication mechanisms> [Authentication.Mechanisms.txt] + are enabled they are still allowed even without SSL/TLS. Depending on how + secure they are, the authentication is either fully secure or it could have + some ways for it to be attacked. + * 'ssl=required': SSL/TLS is always required, even if non-plaintext + authentication mechanisms are used. Any attempt to authenticate before + SSL/TLS is enabled will cause an authentication failure. + * NOTE: If you have only plaintext mechanisms enabled (e.g. 'auth { mechanisms + = plain login }'), 'ssl=yes' and 'ssl=required' are completely equivalent + because in either case the authentication will fail unless SSL/TLS is + enabled first. + * NOTE2: With both 'ssl=yes' and 'ssl=required' it's still possible that the + client attempts to do a plaintext authentication before enabling SSL/TLS, + which exposes the plaintext password to the internet. Dovecot attempts to + indicate this to the IMAP clients via the LOGINDISABLED capability, but many + clients still ignore it and send the password anyway. There is unfortunately + no way for Dovecot to prevent this behavior. The POP3 standard doesn't have + an equivalent capability at all, so the POP3 clients can't even know if the + server would accept a plaintext authentication. + * The main difference between 'ssl=required' and 'disable_plaintext_auth=yes' + is that if 'ssl=required', it guarantees that the entire connection is + protected against eavesdropping (SSL/TLS encrypts the rest of the + connection), while 'disable_plaintext_auth=yes' only guarantees that the + password is protected against eavesdropping (SASL mechanism is encrypted, + but no SSL/TLS is necessarily used). Nowadays you most likely should be + using SSL/TLS anyway for the entire connection, since the cost of SSL/TLS is + cheap enough. Using both SSL/TLS and non-plaintext authentication would be + the ideal situation since it protects the plaintext password even against + man-in-the-middle attacks. + +Note that plaintext authentication is always allowed (and SSL not required) for +connections from localhost, as they're assumed to be secure anyway. This +applies to all connections where the local and the remote IP addresses are +equal. Also IP ranges specified by 'login_trusted_networks' setting are assumed +to be secure. + +Multiple SSL certificates +------------------------- + +Different certificates per algorithm +------------------------------------ + +Since v2.2.31+ you can specify alternative ssl certificate that will be used if +the algorithm differs from the primary certificate. This is useful when +migrating to e.g. ECDSA certificate. + +---%<------------------------------------------------------------------------- +ssl_alt_cert=</path/to/alternative/cert.pem +ssl_alt_key=</path/to/alternative/key.pem +---%<------------------------------------------------------------------------- + +Different certificates per IP and protocol +------------------------------------------ + +If you have multiple IPs available, this method is guaranteed to work with all +clients. + +---%<------------------------------------------------------------------------- +local 192.0.2.10 { # instead of IP you can also use hostname, which will be +resolved + protocol imap { + ssl_cert = </etc/ssl/dovecot/imap-01.example.com.cert.pem + ssl_key = </etc/ssl/dovecot/imap-01.example.com.key.pem + } + + protocol pop3 { + ssl_cert = </etc/ssl/dovecot/pop-01.example.com.cert.pem + ssl_key = </etc/ssl/dovecot/pop-01.example.com.key.pem + } +} + +local 192.0.2.20 { + protocol imap { + ssl_cert = </etc/ssl/dovecot/imap-02.example.com.cert.pem + ssl_key = </etc/ssl/dovecot/imap-02.example.com.key.pem + } + + protocol pop3 { + ssl_cert = </etc/ssl/dovecot/pop-02.example.com.cert.pem + ssl_key = </etc/ssl/dovecot/pop-02.example.com.key.pem + } +} +---%<------------------------------------------------------------------------- + +Note that you will still need a top-level "default" 'ssl_key' and 'ssl_cert' as +well, or you will receive errors. + +---%<------------------------------------------------------------------------- +# doveconf -n +doveconf: Error: ssl enabled, but ssl_cert not set +---%<------------------------------------------------------------------------- + +With client TLS SNI (Server Name Indication) support +---------------------------------------------------- + +/It is important to note that having multiple SSL certificates per IP will not +be compatible with all clients, especially mobile ones. It is a TLS SNI +limitation./ See <SSL.SNIClientSupport.txt> for list of clients known to (not) +support SNI. + +---%<------------------------------------------------------------------------- +local_name imap.example.org { + ssl_cert = </etc/ssl/certs/imap.example.org.crt + ssl_key = </etc/ssl/private/imap.example.org.key +} +local_name imap.example2.org { + ssl_cert = </etc/ssl/certs/imap.example2.org.crt + ssl_key = </etc/ssl/private/imap.example2.org.key +} +# ..etc.. +---%<------------------------------------------------------------------------- + +Password protected key files +---------------------------- + +SSL key files may be password protected. There are two ways to provide Dovecot +with the password: + + 1. Starting Dovecot with 'dovecot -p' asks the password. It's not stored + anywhere, so this method prevents Dovecot from starting automatically at + startup. + 2. 'ssl_key_password' setting. Note that 'dovecot.conf' is by default + world-readable, so you probably shouldn't place it there directly. Instead + you could store it in a different file, such as '/etc/dovecot-private.conf' + containing: + + ---%<--------------------------------------------------------------------- + ssl_key_password = secret + ---%<--------------------------------------------------------------------- + + and then use '!include_try /etc/dovecot-private.conf' in the main + 'dovecot.conf'. + +Chained SSL certificates +------------------------ + +Put all the certificates in the 'ssl_cert' file. For example when using a +certificate signed by TDC the correct order is: + + 1. Dovecot's public certificate + 2. TDC SSL Server CA + 3. TDC Internet Root CA + 4. Globalsign Partners CA + +SSL security settings +--------------------- + +When Dovecot starts up for the first time, it generates new 512bit and 1024bit +Diffie Hellman parameters and saves them into +'<prefix>/var/lib/dovecot/ssl-parameters.dat'. Dovecot v2.1.x and older +regenerated them every week by default, but because the extra security gained +by the regeneration is quite small, Dovecot v2.2 disabled the regeneration +feature completely. + + * Since v2.3.3+ Diffie-Hellman parameters have been made optional, and you are + encouraged to disable non-ECC DH algorithms completely. + +From and up to version 2.2, you can specify the wanted DH parameters length +using: + +---%<------------------------------------------------------------------------- +ssl_dh_parameters_length = 2048 +---%<------------------------------------------------------------------------- + +From version 2.3, you must specify path to DH parameters file using: + +---%<------------------------------------------------------------------------- +ssl_dh=</path/to/dh.pem +---%<------------------------------------------------------------------------- + +To generate new parameters file, you can use: + +---%<------------------------------------------------------------------------- +# This might take a very long time. Run it on a machine with sufficient +entropy. +openssl dhparam 4096 > dh.pem +---%<------------------------------------------------------------------------- + +You can also convert an old v2.2 parameters file with command: + +---%<------------------------------------------------------------------------- +dd if=/path/to/ssl-parameters.dat bs=1 skip=88 | openssl dhparam -inform DER +---%<------------------------------------------------------------------------- + +This should work most of the times. If not, generate new file. + +By default Dovecot's allowed ciphers list contains: + +---%<------------------------------------------------------------------------- +ssl_cipher_list = +ALL:!kRSA:!SRP:!kDHd:!DSS:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW@STRENGTH +---%<------------------------------------------------------------------------- + +Disallowing more won't really gain any security for those using better ciphers, +but it does prevent people from accidentally using insecure ciphers. See +http://www.openssl.org/docs/apps/ciphers.html for a list of the ciphers. + +You should usually prefer server ciphers and their order, so setting + +---%<------------------------------------------------------------------------- +ssl_prefer_server_ciphers=yes +---%<------------------------------------------------------------------------- + +is recommended. + +SSL verbosity +------------- + +---%<------------------------------------------------------------------------- +verbose_ssl = yes +---%<------------------------------------------------------------------------- + +This will make Dovecot log all the problems it sees with SSL connections. Some +errors might be caused by dropped connections, so it could be quite noisy. + +Client certificate verification/authentication +---------------------------------------------- + +If you want to require clients to present a valid SSL certificate, you'll need +these settings: + +---%<------------------------------------------------------------------------- +ssl_ca = </etc/ssl/ca.pem +ssl_verify_client_cert = yes + +auth_ssl_require_client_cert = yes +#auth_ssl_username_from_cert = yes +---%<------------------------------------------------------------------------- + +The CA file should contain the certificate(s) followed by the matching CRL(s). +Note that the CRLs are required to exist. For a multi-level CA place the +certificates in this order: + + 1. Issuing CA cert + 2. Issuing CA CRL + 3. Intermediate CA cert + 4. Intermediate CA CRL + 5. Root CA cert + 6. Root CA CRL + +The certificates and the CRLs have to be in PEM format. To convert a DER format +CRL (e.g.http://crl.cacert.org/class3-revoke.crl) into PEM format, use: + +---%<------------------------------------------------------------------------- +openssl crl -in class3-revoke.crl -inform DER -outform PEM > class3-revoke.pem +---%<------------------------------------------------------------------------- + +With the above settings if a client connects which doesn't present a +certificate signed by one of the CAs in the 'ssl_ca' file, Dovecot won't let +the user log in. This could present a problem if you're using Dovecot to +provide SASL authentication for an MTA (such as Postfix) which is not capable +of supplying client certificates for SASL authentication. If you need Dovecot +to provide SASL authentication to an MTA without requiring client certificates +and simultaneously provide IMAP service to clients while requiring client +certificates, you can put 'auth_ssl_require_client_cert = yes' inside of a +protocol block as shown below to make an exemption for SMTP SASL clients (such +as Postfix). + +---%<------------------------------------------------------------------------- +protocol !smtp { + auth_ssl_require_client_cert = yes +} +---%<------------------------------------------------------------------------- + +You may also force the username to be taken from the certificate by setting +'auth_ssl_username_from_cert = yes'. + + * The text is looked up from subject DN's specified field using OpenSSL's + 'X509_NAME_get_text_by_NID()' function. + * By default the 'CommonName' field is used. + * You can change the field with 'ssl_cert_username_field = name' setting + (parsed using OpenSSL's 'OBJ_txt2nid()' function). 'x500UniqueIdentifier' is + a common choice. + +You may also want to disable the password checking completely. Doing this +currently circumvents Dovecot's security model so it's not recommended to use +it, but it is possible by making the <passdb> [PasswordDatabase.txt] allow +logins using any password (typically requiring <"nopassword" extra field> +[PasswordDatabase.ExtraFields.txt] to be returned). + +Testing +------- + +Try out your new setup: + +---%<------------------------------------------------------------------------- +openssl s_client -servername mail.sample.com -connect mail.sample.com:pop3s +---%<------------------------------------------------------------------------- + +You should see something like this: + +---%<------------------------------------------------------------------------- +CONNECTED(00000003) +depth=2 /O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing +Authority/emailAddress=support@cacert.org +verify error:num=19:self signed certificate in certificate chain +verify return:0 +--- +Certificate chain + 0 s:/CN=mail.example.com + i:/O=CAcert Inc./OU=http://www.CAcert.org/CN=CAcert Class 3 Root + 1 s:/O=CAcert Inc./OU=http://www.CAcert.org/CN=CAcert Class 3 Root + i:/O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing +Authority/emailAddress=support@cacert.org + 2 s:/O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing +Authority/emailAddress=support@cacert.org + i:/O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing +Authority/emailAddress=support@cacert.org +--- +Server certificate +-----BEGIN CERTIFICATE----- +MIIE1DCCArygAwIBAgIDAMBPMA0GCSqGSIb3DQEBBAUAMFQxFDASBgNVBAoTC0NB +Y2VydCBJbmMuMR4wHAYDVQQLExVodHRwOi8vd3d3LkNBY2VydC5vcmcxHDAaBgNV +BAMTE0NBY2VydCBDbGFzcyAzIFJvb3QwHhcNMTAxMjIwMTM1NDQ1WhcNMTIxMjE5 +MTM1NDQ1WjAmMSQwIgYDjksadnjkasndjksandjksandjksandj5YXJlYS5vcmcw +ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3jOX3FC8wVqnb2r65Sfvk +cYUpJhlbhCfqPdN41c3WS0y1Jwwum1q4oMAJvdRnD5TMff1+fqTFy3lS1sYxIXiD +kBRo478eNqzXHMpBOqbvKjYp/UZgWUNA9ebI1nQtwd7rnjmm/GrtyItjahCsgzDS +qPAie+mXYzuT49ZoG+Glg7/R/jDcLMcJY0d5eJ7kufB1RLhvRitZD4FEbJVehqhY +aevf5bLk1BNFhzRBfLXmv6u/kfvWf2HjGAf0aFhaQyiAldDgnZrvaZOFjkToJk27 +p9MguvwGmbciao0DmMjcJhQ0smclFwy8Kj98Tz+nTkfAlU8jJdb1J/tIatJdpSRh +AgMBAAGjgdwwgdkwDAYDVR0TAQH/BAIwADA0BgNVHSUELTArBggrBgEFBQcDAgYI +KwYBBQUHAwEGCWCGSAGG+EIEAQYKKwYBBAGCNwoDAzALBgNVHQ8EBAMCBaAwMwYI +KwYBBQUHAQEEJzAlMCMGCCsGAQUFBzABhadodHRwOi8vb2NzcC5jYWNlcnQub3Jn +LzBRBgNVknsadkjasnjdksandjksandjsnNlY3VyaXR5YXJlYS5vcmegKQYIKwYB +BQUHCAWgHQwbbWFpbC5qb2ludC5zZWN1cml0eWFyZWEub3JnMA0GCSqGSIb3DQEB +BQUAA4ICAQAX8ceObvUZNKYTlNQ/cv0BiA1XweRsVNca1ILACNLdVPR9mvf+aXCh +ODkHaZAmGngj1DfD4fJsTbaydGWSPeVH91Qi9F+Pi6szhsxylI83NKbuXihcenuG +twnte8aIb5FelVHttLQPSKRR62E8YmDWk3KYivuFAuZqDaGnWc5yeneTBpsGter/ +4awqsgymBK2YEg1HIWMPaRBvwzCVN/yUyWhFH9Nj11f/xgZE87VXrjLHWT/73i2Z +S4uIZ2KHQUYuxMGldgpXm+QxFM8DGA6z1T1oPCVfW85cezlfr8QVvX6SXZrAUNL0 +3D5YPzQuevW+5CrqnGA+F5ff4mBMl8R8Sg0+0LoLqt5PbpGyTt9vS1INZCdfvtIA +/d7Ae7Xp9W8FVRqd7tvNMIy3ZA0/wNMDUczkhC/YtvHfMELpjtMJAGF15OtO7Vik +V+FZnBP1Yd7760dtEmd6bF8vjcXCvDdxwGtcAehAUpIgAWvkHHOt8+H56tkFENAP +/ZpJ+Wr+K3lxkkG+BN1bucxMuAdVyTpFyZfKDHRXIO/5e0hpPOaTO+obD3kifzdh +yy7KmdKvDclHTiPuonJBzEXeM3JQBjcDHbMSyA6+38yBcso27h9VqCQJB2cZmSlW +ArS/9wt2X21KgeuGHlTZ/8z9gXAjQKXhDYECWWd6LkWl98ZDBihslQ== +-----END CERTIFICATE----- +subject=/CN=mail.example.com +issuer=/O=CAcert Inc./OU=http://www.CAcert.org/CN=CAcert Class 3 Root +--- +No client certificate CA names sent +--- +SSL handshake has read 5497 bytes and written 293 bytes +--- +New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA +Server public key is 2048 bit +Secure Renegotiation IS supported +Compression: zlib compression +Expansion: zlib compression +SSL-Session: + Protocol : TLSv1 + Cipher : DHE-RSA-AES256-SHA + Session-ID: +114A22BE4625B33F6893124ACF640AE0628B48B5039E90B3B9A20ADF7FA691F3 + Session-ID-ctx: + Master-Key: +B8A55EC91A060575CFB29503FBF7160C2DC8BCBFE02D20A7F704882F72D8D00272D8D002CE5CCC4B94A492F43ED8F + Key-Arg : None + TLS session ticket: + 0000 - 86 c7 46 63 a5 b6 48 74-16 d8 e0 a7 e2 64 e8 89 ..Fc..Ht.....d.. + 0010 - 97 90 59 4b 57 f3 e2 b3-e2 d2 88 90 a8 aa b4 44 ..YKW..........D + 0020 - ea 24 08 5e b4 14 7f e1-2a 1a 1c 40 ca 85 e7 41 .$.^....*..@...A + 0030 - 9d 0d a8 4c f7 e3 db 1e-ef da 53 9c fe 43 cc 62 ...L......S..C.b + 0040 - 79 b6 ad ea 9d cf ca b2-37 41 b7 0f ea 7d 59 e8 y.......7A...}Y. + 0050 - 10 01 a0 eb dc c2 63 66-56 54 6a e8 3a 4b 93 49 ......cfVTj.:K.I + 0060 - 77 da e4 4b 21 e8 30 7e-bf 10 91 3a 2c f9 59 80 w..K!.0~...:,.Y. + 0070 - 01 1f 36 0b 92 85 67 55-c8 86 1d 44 b1 6f 0d ae ..6...gU...D.o.. + 0080 - 15 36 b6 49 3a ef 94 9a-ef 6d 27 f0 80 20 43 09 .6.I:....m'.. C. + 0090 - be 70 c5 30 15 3b 93 c6-c1 4c e9 7f 5c 34 98 dd .p.0.;...L..\4.. + + Compression: 1 (zlib compression) + Start Time: 1292857721 + Timeout : 300 (sec) + Verify return code: 19 (self signed certificate in certificate chain) +--- ++OK Dovecot ready. +---%<------------------------------------------------------------------------- + +Testing CA +---------- + +The above test procedure returns: + +---%<------------------------------------------------------------------------- + Verify return code: 19 (self signed certificate in certificate chain) +---%<------------------------------------------------------------------------- + +which is expected result since test command omits option to verify CA root +certificate. The following commands will enable CA root certificate +validation. + +Testing CA On Debian +-------------------- + +On Debian derived distributions try: + +---%<------------------------------------------------------------------------- +openssl s_client -CApath /etc/ssl/certs -connect mail.sample.com:pop3s +---%<------------------------------------------------------------------------- + +Testing CA On RHEL +------------------ + +On Red Hat Enterprise Linux derived distributions try: + +---%<------------------------------------------------------------------------- +openssl s_client -CAfile /etc/pki/tls/cert.pem -connect mail.sample.com:pop3s +---%<------------------------------------------------------------------------- + +Testing CA Success +------------------ + +---%<------------------------------------------------------------------------- + Verify return code: 0 (ok) +---%<------------------------------------------------------------------------- + +Client connections +------------------ + +Since v2.3.4 dovecot accepts default system CAs for outgoing connections. + +---%<------------------------------------------------------------------------- +ssl_client_ca_dir = /path/to/pem/certificates +ssl_client_ca_file = /path/to/pem/bundle +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SSL.SNIClientSupport.txt b/doc/wiki/SSL.SNIClientSupport.txt new file mode 100644 index 0000000..ba65cc7 --- /dev/null +++ b/doc/wiki/SSL.SNIClientSupport.txt @@ -0,0 +1,21 @@ +TLS SNI Client Support +====================== + +Works: + + * Thunderbird (Linux) + * K-9 on Android (merged Sept 2015 - https://github.com/k9mail/k-9/pull/718) + * Apple Mail (according to + https://forums.cpanel.net/threads/mail-ssl-sni.454592/ ) + * Mutt (ticket https://dev.mutt.org/trac/ticket/3923) + * <NeoMutt.txt> (since 2016-03-07 according to + https://www.neomutt.org/feature/tls-sni) + +Doesn't work: + + * K-9 on Droid X2 (maybe fixed in newer versions - see above) + * Apple Mail (Mac OS X 10.10 and lower AND iOS 9.3 and lower) + * Outlook for Mac version 15 (according to + https://forums.cpanel.net/threads/mail-ssl-sni.454592/ ) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SSL.txt b/doc/wiki/SSL.txt new file mode 100644 index 0000000..7e32883 --- /dev/null +++ b/doc/wiki/SSL.txt @@ -0,0 +1,65 @@ +SSL +=== + + * <Creating SSL certificates> [SSL.CertificateCreation.txt] + * <Configuring Dovecot to use SSL certificates> [SSL.DovecotConfiguration.txt] + + * <Importing CAs and self-signed SSL certificates to clients> + [SSL.CertificateClientImporting.txt] + * SSL works pretty much the same universally, so for more information about + SSL you can see for example Apache's documentation + [http://httpd.apache.org/docs/2.2/ssl/]. + * Dovecot uses OpenSSL, so whatever information you find about it applies also + to Dovecot. + +SSL, TLS and STARTTLS confusion +------------------------------- + +SSL and TLS terms are often used in confusing ways: + + * SSL (Secure Sockets Layer) is the original protocol implementation. SSLv3 is + still allowed by Dovecot, but it's rarely used. Some clients use SSL to mean + that they're going to connect to the imaps (993), pop3s (995) or smtps (465) + port, although they're still going to use TLSv1 protocol. + * TLS (Transport Layer Security) replaced the SSL protocol. TLSv1 protocol is + used practically always nowadays. Some clients use TLS to mean that they're + going to use STARTTLS command after connecting to the standard imap (143), + pop3 (110) or smtp port (25/587). Nothing would prevent using SSLv3 protocol + after STARTTLS command. + +Unfortunately there doesn't seem to be any clear and simple way to refer to +these different meanings. SSL term is much more widely understood than TLS, so +Dovecot configuration and this documentation only talks about SSL when in fact +it means both SSL/TLS. + +Originally SSL support was added to protocols by giving them a separate "SSL +port" (imaps, pop3s, etc.), where the SSL handshake starts immediately when +client connects, and only after the session is encrypted the regular protocol +handling begins. Using two separate ports for plaintext and SSL connections was +thought to be wasteful and adds complexity for clients which may wish to make +use of SSL when it is advertised, so STARTTLS command was added and intended to +deprecate the SSL ports. Clients using STARTTLS work by connecting to the +regular unencrypted port and immediately issue a STARTTLS command, after which +the session is encrypted. After SSL handshake there is no difference between +SSL port initiated connections and STARTTLS initiated connections. + +SSL port deprecation never really happened, probably because of a few reasons: + + * Some admins don't even know about STARTTLS. + * Some admins want to require SSL/TLS, but don't realize that this is also + possible with STARTTLS (Dovecot has 'disable_plaintext_auth=yes' and + 'ssl=required' settings). + * Some admins understand everything, but still prefer to allow only SSL ports + (maybe with a firewall). This could be because it makes it easier to ensure + that no information is leaked, because SSL/TLS handshake happens + immediately. Some clients unfortunately try to do plaintext authentication + without STARTTLS, even when IMAP server has told the client that it won't + work. Besides, it requires fewer round-trips if you begin SSL upon + connection when you know you want it, instead of connect, negotiate + capabilities, insist on TLS, then start all over again inside the encrypted + session. + * According to some reports (like this + [https://it.slashdot.org/story/14/11/11/2349244/isps-removing-their-customers-email-encryption]) + STARTTLS can not guarantee encrypted delivery of mail. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Sasl.txt b/doc/wiki/Sasl.txt new file mode 100644 index 0000000..940f187 --- /dev/null +++ b/doc/wiki/Sasl.txt @@ -0,0 +1,27 @@ +SASL +==== + +SASL stands for "Simple Authentication and Security Layer". SASL itself is +nothing more than a list of requirements for <authentication mechanisms> +[Authentication.Mechanisms.txt] and protocols to be SASL-compatible as +described in RFC 4422 [http://www.ietf.org/rfc/rfc4422.txt]. IMAP, POP3 and +SMTP protocols all have support for SASL. + +Many people confuse SASL with one specific SASL implementation: the Cyrus SASL +library. Dovecot has its own SASL implementation which may at some point be +separated from Dovecot itself to "compete" against Cyrus SASL library in server +side. + +Dovecot SASL can already be used with: + + * Postfix *v2.3* and later. See <HowTo.PostfixAndDovecotSASL.txt> for details. + + * Exim *v4.64* and later. See <HowTo.EximAndDovecotSASL.txt> for details. + * chasquid [https://blitiri.com.ar/p/chasquid] *v0.04* and later. See + <HowTo.ChasquidAndDovecotSASL.txt> for details. + * Prosody (with mod_auth_dovecot) + * ejabberd (with check_dovecot.pl) + +Hopefully more software will follow. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SecurityTuning.txt b/doc/wiki/SecurityTuning.txt new file mode 100644 index 0000000..90a2dfa --- /dev/null +++ b/doc/wiki/SecurityTuning.txt @@ -0,0 +1,22 @@ +Security tuning +=============== + +Dovecot is pretty secure out-of-the box. It uses multiple processes and +privilege separation to isolate different parts from each others in case a +security hole is found from one part. + +Some things you can do more: + + * Allocate each user their own UID and GID (see <UserIds.txt>) + * Use a separate /dovecot-auth/ user for authentication process (see + <UserIds.txt>) + * You can chroot authentication and mail processes (see <Chrooting.txt>) + * Compiling Dovecot with garbage collection ('--with-gc' configure option) + fixes at least in theory any security holes caused by double free()s. + However this hasn't been tested much and there may be problems. + * There are some security related SSL settings (see + <SSL.DovecotConfiguration.txt>) + * Set 'first/last_valid_uid/gid' settings to contain only the range actually + used by mail processes + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Services.txt b/doc/wiki/Services.txt new file mode 100644 index 0000000..67904ec --- /dev/null +++ b/doc/wiki/Services.txt @@ -0,0 +1,502 @@ +Service configuration +===================== + +Contents + + + 1. Service configuration + + 1. Service basics + + 2. Service privileges + + 3. Service limits + + 4. Service listeners + + 1. unix_listeners and fifo_listeners + + 2. inet_listeners + + 2. Default services + + 1. anvil + + 2. auth + + 3. auth-worker + + 4. config + + 5. dict + + 6. director + + 7. dns_client + + 8. doveadm + + 9. imap, pop3, submission, managesieve + + 10. imap-login, pop3-login, submission-login, managesieve-login + + 11. indexer + + 12. indexer-worker + + 13. ipc + + 14. lmtp + + 15. log + + 16. ssl-params + + 17. stats + +This page describes Dovecot's services comprehensively. Most admins don't need +to know these details. The important service settings are described in the +'example-config/conf.d/10-master.conf' file. + +Service basics +-------------- + +executable: + The binary path to execute and its parameters. If the path doesn't begin with + '/', it's relative to *base_dir*. + +type: + Type of this service: + * "" is the default. + * "startup" creates one process at startup. For example SSL parameters are + generated at startup because of this, instead of only after the first SSL + connection arrives. + * "login" is used by login processes. The login processes have "all + processes full" notification fd. It's used by the processes to figure out + when no more client connections can be accepted because client and process + limits have been reached. The login processes can then kill some of their + oldest connections that haven't logged in yet. + * "log", "config" and "anvil" are treated specially by these specific + processes. + +protocol: + If non-empty, this service is enabled only when the protocol name is listed + in *protocols* setting. + +idle_kill: + If a process doesn't appear to be doing anything after this much time, notify + it that it should kill itself if it's not doing anything.*process_min_avail* + setting overrides this. If set to 0,*default_idle_kill* is used. + +Service privileges +------------------ + +user: + UNIX user (UID) which runs this process. *default_login_user* setting's value + should be used for type=login processes and *default_internal_user* should be + used for other processes that don't require root privileges. + +group: + The primary UNIX group (GID) which runs this process. + +extra_groups: + Secondary UNIX groups that this process belongs to. + +privileged_group: + Secondary UNIX group, which is disabled by default, but can be enabled by the + process. This setting is probably never needed + directly.*mail_privileged_group* setting is a more user friendly way to use + this setting for mail processes. + +chroot: + The processes are chrooted to this directory at startup. Relative to + *base_dir*. + +drop_priv_before_exec: + Drop all privileges after forking, but before executing the binary. This is + mainly useful for dumping core files on non-Linux OSes, since the processes + are no longer in "setuid" mode. This setting can't be used with non-empty + chroot. + +Service limits +-------------- + +client_limit: + Maximum number of simultaneous client connections per process. Once this + number of connections is received, the next incoming connection will prompt + Dovecot to spawn another process. If set to 0,*default_client_limit* is used + instead. + +service_count: + Number of client connections to handle until the process kills itself. 0 + means unlimited. 1 means only a single connection is handled until the + process is stopped - this is the most secure choice since there's no way for + one connection's state to leak to the next one. For better performance this + can be set higher, but ideally not unlimited since more complex services can + have small memory leaks and/or memory fragmentation and the process should + get restarted eventually. For example 100..1000 can be good values. + +process_limit: + Maximum number of processes that can exist for this service. If set to 0, + *default_process_limit* is used instead. + +process_min_avail: + Minimum number of processes that always should be available to accept more + client connections. For service_count=1 processes this decreases the latency + for handling new connections. For service_count!=1 processes it could be set + to the number of CPU cores on the system to balance the load among them. + +vsz_limit: + Limit the process's address space (both RLIMIT_DATA and RLIMIT_AS if + available). When the space is reached, some memory allocations may start + failing with "Out of memory", or the kernel may kill the process with signal + 9. This setting is mainly intended to prevent memory leaks from eating up all + of the memory, but there can be also legitimate reasons why the process + reaches this limit. For example a huge mailbox may not be accessed if this + limit is too low. The default value (18446744073709551615 = 2^64-1) sets the + limit to *default_vsz_limit*, while 0 disables the limit entirely. + +There are 3 types of services that need to be optimized in different ways: + + 1. Master services (e.g. auth, anvil, indexer, director, log): + * Currently there isn't any easy way to optimize these. If these become a + bottleneck, typically you need to run another Dovecot server. In some + cases it may be possible to create multiple master processes and have + each one be responsible for only specific users/processes, although this + may also require some extra development. + 2. Services that do disk I/O or other blocking operations (e.g. imap, pop3, + lmtp): + * These should have *client_limit=1*, because any blocking operation will + block all the other clients and cause unnecessary delays and even + timeouts. + * This means that *process_limit* specifies the maximum number of + available parallel connections. + 3. Services that have no blocking operations (e.g. imap-login, pop3-login): + * For best performance (but a bit less safety), these should have + *process_limit* and *process_min_avail* set to the number of CPU cores, + so each CPU will be busy serving the process but without unnecessary + context switches. + * Then *client_limit* needs to be set high enough to be able to serve all + the needed connections (max connections =*process_limit* * + *client_limit*). + * *service_count* is commonly set to unlimited (0) for these services. + Otherwise when the *service_count* is beginning to be reached, the total + number of available connections will shrink. With very bad luck that + could mean that all the processes are simply waiting for the existing + connections to die away before the process can die and a new one can be + created. Although this could be made less likely by setting + *process_limit* higher than *process_min_avail*, but that's still not a + guarantee since each process could get a very long running connection + and the *process_limit* would be eventually reached. + +Service listeners +----------------- + +unix_listeners and fifo_listeners +--------------------------------- + +path: + Path to the file, relative to *base_dir* setting. This is also used as the + section name. + +user: + Owner of the file. Defaults to 0 (root). + +group: + Group of the file. Defaults to 0 (root/wheel). + +mode: + Mode of the file. Defaults to 0700. Note that 0700 is an octal value, while + 700 is a different decimal value. Setting mode to 0 disables the listener. + +inet_listeners +-------------- + +name: + Section name of this listener. It is meant to be descriptive for humans (e.g. + "imap", "imaps"). + +address: + Space separated list of IP addresses / host names to listen on. "*" means all + IPv4 addresses, "::" means all IPv6 addresses. Defaults to *listen* setting. + +port: + Port number where to listen. 0 disables the listener. + +ssl: + If yes, the listener does an immediate SSL/TLS handshake after accepting a + connection. This is needed for the legacy imaps and pop3s ports. + + All listeners with ssl=yes will be removed if global ssl is turned off + +haproxy (v2.2.19+): + If yes, this listener is configured for use with <HAProxy.txt>. It expects a + Proxy Protocol [http://blog.haproxy.com/haproxy/proxy-protocol/] header right + after accepting the connection. Connections are aborted immediately when this + protocol is violated. + +Default services +================ + +anvil +----- + +The anvil process tracks state of users and their connections. + + * *chroot=empty* and *user=$default_internal_user*, because anvil doesn't need + access to anything. + * *process_limit=1*, because there can be only one. + * *idle_kill=4294967295s*, because it should never die or all of its tracked + state would be lost. + * "doveadm who" and some other doveadm commands connect to anvil's UNIX + listener and request its state. + +auth +---- + +The master auth process. There are 4 types of auth client connections: + + 1. client: Only SASL authentication is allowed. This can be safely exposed to + entire world. + 2. userdb: userdb lookups and passdb lookups (without the password itself) can + be done for any user, and a list of users can be requested. This may or may + not be a security issue. Access to userdb lookup is commonly needed by + dovecot-lda, doveadm and other tools. + 3. login: Starts a two phase user login by performing authenticating (same as + "client" type). Used by login processes. + 4. master: Finishes the two phase user login by performing a userdb lookup + (similar to "userdb" type). Used by post-login processes (e.g. imap, pop3). + +With UNIX listeners the client type is selected based on the filename after the +last "-" in the filename. For example "anything-userdb" is of "userdb" type. +The default type is "client" for inet insteners and unrecognized UNIX +listeners. You can add as many client and userdb listeners as you want (and you +probably shouldn't touch the login/master listeners). + + * *client_limit* should be large enough to handle all the simultaneous + connections. Typically only login processes use long lasting auth + connections, while other processes do only quick lookups and disconnect + afterwards. + * *process_limit=1*, because there can be only one auth master process. + * *user=$default_internal_user*, because it typically doesn't need permissions + to do anything (PAM lookups are done by auth-workers). + * *chroot* could be set (to e.g. "empty") if passdb/userdb doesn't need to + read any files (e.g. SQL, LDAP config is read before chroot) + +auth-worker +----------- + +Auth master process connects to auth worker processes. It is mainly used by +passdbs and userdbs that do potentially long running lookups. For example MySQL +supports only synchronous lookups, so each query is run in a separate auth +worker process that does nothing else during the query. PostgreSQL and LDAP +supports asynchronous lookups, so those don't use worker processes at all. With +some passdbs and userdbs you can select if worker processes should be used. + + * *client_limit=1*, because only the master auth process connects to auth + worker. + * *service_count=1*, because auth master stops extra idling workers by + disconnecting from them. + * *process_limit* should be a bit higher than *auth_worker_max_count* setting. + + * *user=root* by default, because by default PAM authentication is used, which + usually requires reading '/etc/shadow'. If this isn't needed, it's a good + idea to change this to something else, such as *$default_internal_user*. + * *chroot* could also be set if possible. + +config +------ + +Config process reads and parses the 'dovecot.conf' file, and exports the parsed +data in simpler format to config clients. + + * *user=root*, because the process needs to be able to reopen the config files + during a config reload, and often some parts of the config having secrets + are readable only by root. + * Only root should be able to connect to its UNIX listener, unless there are + no secrets in the configuration. Passwords are obviously secrets, but less + obviously *ssl_key* is also a secret, since it contains the actual SSL key + data instead of only a filename. + +dict +---- + +Dovecot has a "lib-dict" API for doing simple key-value lookups/updates in +various backends (SQL, file, others in future). This is optionally used by +things like quota, expire plugin and other things in future. It would be +wasteful for each mail process to separately create a connection to SQL, so +usually they go through the "proxy" dict backend. These proxy connections are +the client connections of dict processes. + + * dict / Synchronous lookups (e.g. mysql): + * *client_limit=1*, because dict lookups are synchronous and the client is + supposed to disconnect immediately after the lookup. + * dict-async / Asynchronous lookups (e.g. pgsql, cassandra, ldap): + * *process_limit* should commonly be the same as number of CPU cores. + Although with Cassandra this may not be true, because Cassandra library + can use multiple threads. + * *user=$default_internal_user*, because the proxy dict lookups are typically + SQL lookups, which require no filesystem access. (The SQL config files are + read while still running as root.) + * The dict clients can do any kind of dict lookups and updates for all users, + so they can be rather harmful if exposed to an attacker. That's why by + default only root can connect to dict socket. Unfortunately that is too + restrictive for all setups, so the permissions need to be changed so that + Dovecot's mail processes (and only them) can connect to it. + +director +-------- + +<Director.txt> tracker process, which hooks into all auth-client and +auth-userdb connections. + + * *process_limit=1*, because only one process can keep track of everyone's + state. + * *user=$default_internal_user*, because director doesn't access any files. + * *chroot* can't be set, because it still needs to be connect to auth process. + + * Connections are basically proxying auth connections, so they have similar + security considerations. + +dns_client +---------- + +Used by "lib-dns" library to perform asynchronous DNS lookups. The dns-client +processes internally use the synchronous 'gethostbyname()' function. + + * *client_limit=1*, because the DNS lookup is synchronous. + * *user=$default_internal_user*, because typically no special privileged files + need to be read. + * *chroot* can be used only if it contains 'etc/resolv.conf' and other files + necessary for DNS lookups. + +doveadm +------- + +It's possible to run doveadm mail commands via doveadm server processes. This +is useful for running doveadm commands for multiple users simultaneously, and +it's also useful in a multiserver system where doveadm can automatically +connect to the correct backend to run the command. + + * *client_limit=1*, because doveadm command execution is synchronous. + * *service_count=1* just in case there were any memory leaks. This could be + set to some larger value (or 0) for higher performance. + * *user=root*, but the privileges are (temporarily) dropped to the mail user's + privileges after userdb lookup. If only a single UID is used, user can be + set to the mail UID for higher security, because the process can't gain root + privileges anymore. + +imap, pop3, submission, managesieve +----------------------------------- + +Post-login process for handling IMAP/POP3/ <Submission.txt>/ <ManageSieve> +[Pigeonhole.ManageSieve.txt] client connections. + + * *client_limit* may be increased from the default 1 to save some CPU and + memory, but it also increases the latency when one process serving multiple + clients it waiting for a long time for a lock or disk I/O. In future these + waits may be reduced or avoided completely, but for now it's not safe to set + this value higher than 1 in enterprise mail systems. For small mostly-idling + hobbyist servers a larger number may work without problems. + * *service_count* can be changed from 1 if only a single UID is used for mail + users. This is improves performance, but it's less secure, because bugs in + code may leak email data from another user's earlier connection. + * *process_limit* defaults to 1024, which means that the number of + simultaneous connections for the protocol that this service handles (IMAP, + POP3, <Submission.txt>, or <ManageSieve> [Pigeonhole.ManageSieve.txt]) is + limited by this setting. If you expect more connections, increase this + value. + +imap-login, pop3-login, submission-login, managesieve-login +----------------------------------------------------------- + +See <LoginProcess.txt>. + +indexer +------- + +Indexer master process, which tracks and prioritizes indexing requests from +mail processes. The actual indexing is done by indexer-worker processes. The +indexing means both updating Dovecot's internal index and cache files with new +messages and more importantly updating full text search indexes (if enabled). +The indexer master process guarantees that the FTS index is never modified by +more than one process. + + * *process_limit=1*, because only one process can keep the FTS guarantee. + * *user=$default_internal_user*, because the process doesn't need any + permissions. + * *chroot* could be set to *$base_dir* for extra security. It still needs to + be able to connect to indexer-worker socket. + +indexer-worker +-------------- + +Indexer worker process. + + * *client_limit=1*, because indexing is a synchronous operation. + * *process_limit* defaults to 10, because the FTS index updating can eat a lot + of CPU and disk I/O. You may need to adjust this value depending on your + system. + * *user=root*, but the privileges are (temporarily) dropped to the mail user's + privileges after userdb lookup. If only a single UID is used, user can be + set to the mail UID for higher security, because the process can't gain root + privileges anymore. + +ipc +--- + +IPC hub process. + + * *process_limit=1*, because there can be only one hub. + * *chroot=empty* and *user=$default_internal_user*, because it doesn't need + any files and there are no outbound connections. + * The "ipc" UNIX socket can be used to send any commands to other processes, + such as killing a specific user's connection. It is somewhat security + sensitive. + +lmtp +---- + +LMTP process for delivering new mails. + + * *client_limit=1*, because most of the time spent on an LMTP client is spent + waiting for disk I/O and other blocking operations. There's no point in + having more clients waiting around during that doing nothing. + * However, LMTP proxying is only writing to temporary files that normally + stay only in memory. So for LMTP proxying a client_limit above 1 could be + useful. + * *user=root*, but the privileges are (temporarily) dropped to the mail user's + privileges after userdb lookup. If only a single UID is used, user can be + set to the mail UID for higher security, because the process can't gain root + privileges anymore. + +log +--- + +All processes started via Dovecot master process log their messages via the +"log" process. This allows some nice features compared to directly logging via +syslog. + + * *process_limit=1*, because the log process keeps track of all the other + logging processes. + * *user=root*, because it guarantees being able to write to syslog socket and + to the log files directly. + +ssl-params +---------- + +Build SSL parameters every n days, based on *ssl_parameters_regenerate* +setting. Obsoleted in v2.3.0. + + * *type=startup* so that the (re)generation can be started immediately at + startup when needed, instead of waiting until the first SSL handshake + starts. + +stats +----- + +Mail process statistics tracking. Its behavior is very similar to the anvil +process, but anvil's data is of higher importance and lower traffic than stats, +so stats are tracked in a separate process. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SharedMailboxes.ClusterSetup.txt b/doc/wiki/SharedMailboxes.ClusterSetup.txt new file mode 100644 index 0000000..ae581d1 --- /dev/null +++ b/doc/wiki/SharedMailboxes.ClusterSetup.txt @@ -0,0 +1,51 @@ +Shared Mailboxes in Dovecot Cluster +=================================== + +As mentioned in <Director.txt>, you can't have multiple servers accessing the +same user at the same time or it will lead into trouble. This becomes +problematic with shared mailboxes, because two users who are sharing a folder +may run in different servers. The solution here is to access the shared folders +via IMAP protocol, which passes through the Dovecot proxies/directors so the +actual filesystem access is done only by one server. + +There are a couple of things still missing for this kind of use case: + + * imapc_* settings are global. You can't have two different namespaces with + different imapc settings yet. + * The imapc code doesn't support many IMAP features. Most importantly SEARCH + isn't supported, which may result in lower performance. + +Setting up user-shared folders +------------------------------ + +You'll need to setup master user logins to work for all the users. The logged +in user becomes the master user. This way the ACLs are applied correctly. + +---%<------------------------------------------------------------------------- +namespace { + type = shared + prefix = shared/%%u/ + location = imapc:~/shared/%%u/ # cache for shared indexes +} +imapc_host = director-ip +imapc_master_user = %u +#imapc_user = # leave this empty. It'll be automatically filled with the +destination username. +imapc_password = master-secret + +plugin { + acl_shared_dict = fs:posix:prefix=/nfs/shared-acls/ +} +---%<------------------------------------------------------------------------- + +The shared dictionary needs to be accessible from all the backends. The +possibilities for it are: + + * file: A single shared file in filesystem. This becomes a performance + bottleneck easily if there are many users sharing folders. + * fs posix: Shared directory in filesystem. This will create many small files + to the filesystem. + * sql: Shared SQL server + * Any other <shared dictionary> [Dictionary.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SharedMailboxes.Permissions.txt b/doc/wiki/SharedMailboxes.Permissions.txt new file mode 100644 index 0000000..d3772f6 --- /dev/null +++ b/doc/wiki/SharedMailboxes.Permissions.txt @@ -0,0 +1,168 @@ +Filesystem permissions in shared mailboxes +========================================== + +IMAP processes need filesystem level permissions to access shared/public +mailboxes. This means that: + + * If you use more than one <UNIX UID> [UserIds.txt] for your mail users (e.g. + you use system users), you'll need to make sure that all users can access + the mailboxes on filesystem level. ( <ACL plugin> [ACL.txt] won't help you + with this.) + * You can remove write permissions on purpose from public namespace root + directory to prevent users from creating new mailboxes under it. + +Dovecot never modifies permissions for existing mail files or directories. When +users share mailboxes between each others, the system must have been set up in +a way that filesystem permissions don't get in the way. The easiest way to do +that is to use only a single UID. Another possibility would be to use one or +more groups for all the mail files that may be shared to other users belonging +to the same group. For example if you host multiple domains, you might create a +group for each domain and allow mailbox sharing (only) between users in the +same domain. + +System user UNIX groups +----------------------- + +There's no requirement to use UNIX groups (i.e. typically defined in +'/etc/group') for anything. If you don't care about them, you can safely ignore +this section. + +If you use <passwd> [AuthDatabase.Passwd.txt] userdb, the IMAP process has +access to all the UNIX groups defined for that user. You may use these groups +when granting filesystem permissions. If you wish to use UNIX groups defined in +'/etc/group' but don't use passwd userdb, you can still do this by returning +'system_groups_user' <userdb extra field> [UserDatabase.ExtraFields.txt], which +contains the UNIX user name whose groups are read from the group file. + +You can also set up extra UNIX groups by listing them in 'mail_access_groups' +setting. To have per-user UNIX groups, return 'mail_access_groups' as userdb +extra field. The advantage of using this method is that only Dovecot mail +processes have access to the group, but nothing else, such as user's SSH +session. For example a simple way to set up shared mailbox access for all +system users is to make all mail dirs/files 0770/0660 mode and owned by group +"sharedmail" and then set 'mail_access_groups=sharedmail'. Using more fine +grained groups of course leaks less mail data in case there's a security hole +in Dovecot. + +Permissions for new mailboxes +----------------------------- + +When creating a new mailbox, Dovecot copies the permissions from the mailbox +root directory. For example with mboxes if you have directories: + +---%<------------------------------------------------------------------------- +drwx--xr-x 8 user group 4096 2009-02-21 18:31 /home/user/mail/ +drwxrwxrwx 2 user group 4096 2009-02-21 18:32 /home/user/mail/foo/ +---%<------------------------------------------------------------------------- + +When creating a new foo/bar/ directory, Dovecot gives it permissions: + +---%<------------------------------------------------------------------------- +drwx--xr-x 2 user group 4096 2009-02-21 18:33 /home/user/mail/foo/bar/ +---%<------------------------------------------------------------------------- + +As you can see, the file mode was copied from mail/ directory, not mail/foo/. +The group is also preserved. If this causes problems (e.g. different users +having different groups create mailboxes, causing permission denied errors when +trying to preserve the group) you can set the setgid bit for the root +directory: + +---%<------------------------------------------------------------------------- +chmod g+s /home/user/mail +---%<------------------------------------------------------------------------- + +This will cause the group to be automatically copied by the OS for all created +files/directories under it, even if the user doesn't belong to the group. + +Permissions for new files in mailboxes +-------------------------------------- + +When creating new files inside a mailbox, Dovecot copies the read/write +permissions from the mailbox's directory. For example if you have: + +---%<------------------------------------------------------------------------- +drwx--xr-x 5 user group 4096 2009-02-21 18:53 /home/user/Maildir/.foo/ +---%<------------------------------------------------------------------------- + +Dovecot creates files under it with modes: + +---%<------------------------------------------------------------------------- +drwx--xr-x 2 user group 4096 2009-02-21 18:54 cur/ +drwx--xr-x 2 user group 4096 2009-02-21 18:54 new/ +drwx--xr-x 2 user group 4096 2009-02-21 18:54 tmp/ +-rw----r-- 1 user group 156 2009-02-21 18:54 dovecot.index.log +-rw----r-- 1 user group 17 2009-02-21 18:54 dovecot-uidlist +---%<------------------------------------------------------------------------- + +Note how the g+x gets copied to directories, but for files it's simply ignored. +The group is copied the same way as explained in the previous section. + +When mails are copied between Maildirs, it's usually done by hard linking. If +the source and destination directory permissions are different, Dovecot create +a new file and copies data the slow way so that it can assign the wanted +destination permissions. The source and destination permission lookups are done +only by looking at the mailbox root directories' permissions, not individual +mail files. This may become a problem if the mail files' permissions aren't as +Dovecot expects. + +Permissions to new /domain/user directories +------------------------------------------- + +If each user has different UIDs and you have '/var/mail/domain/user/' style +directories, you run into a bit of trouble. The problem is that the first user +who creates '/var/mail/domain/' will create it as 0700 mode, and other users +can't create their own user/ directories under it anymore. The solution is to +use a common group for the users and set '/var/mail/' directory's permissions +properly (group-suid is required): + +---%<------------------------------------------------------------------------- +chgrp dovemail /var/mail +chmod 02770 /var/mail # or perhaps 03770 for extra security +---%<------------------------------------------------------------------------- + +and in dovecot.conf: + +---%<------------------------------------------------------------------------- +mail_location = maildir:/var/vmail/%d/%n/Maildir +mail_access_groups = dovemail +---%<------------------------------------------------------------------------- + +The end result should look like this: + +---%<------------------------------------------------------------------------- +drwxrwsr-x 3 user dovemail 60 Oct 24 12:04 domain.example.com/ +drwx--S--- 3 user user 60 Oct 24 12:04 domain.example.com/user/ +---%<------------------------------------------------------------------------- + +Note that this requires that the mail_location setting is in its explicit +format with %variables. Using 'maildir:~/Maildir' won't work, because Dovecot +can't really know how far down it should copy the permissions from. + +Permissions to new user home directories (v2.2+) +------------------------------------------------ + +When mail_location begins with '%h' or '~/', its permissions are copied from +the first existing parent directory if it has setgid-bit set. This isn't done +when the path contains any other %variables. + +Mail Delivery Agent permissions +------------------------------- + +When using Dovecot <LDA.txt>, it uses all the same configuration files as +IMAP/POP3, so you don't need to worry about it. + +When using an external MDA to deliver to a shared mailbox, you need to make +sure that the resulting files have proper permissions. For example with +Procmail + Maildir, set 'UMASK=007' in '.procmailrc' to make the delivered mail +files group-readable. To get the file to use the proper group, set the group to +the Maildir's 'tmp/' directory and also set its setgid bit ('chmod g+s'). + +Dictionary files +---------------- + +Created dictionary files (e.g. 'acl_shared_dict = file:...') also base their +initial permissions on parent directory's permissions. After the initial +creation, the permissions are permanently preserved. So if you want to use +different permissions, just chown/chmod the file. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SharedMailboxes.Public.txt b/doc/wiki/SharedMailboxes.Public.txt new file mode 100644 index 0000000..e958f19 --- /dev/null +++ b/doc/wiki/SharedMailboxes.Public.txt @@ -0,0 +1,241 @@ +Contents + + + 1. Public Mailboxes + + 1. Per-user \Seen flag (v2.2+) + + 2. Maildir: Per-user \Seen flag + + 1. Maildir: Keyword sharing + + 3. Subscriptions + + 4. Read-only mailboxes + + 1. Read-only mboxes + + 2. Read-only Maildirs + + 5. Example: Public mailboxes with ACLs + +Public Mailboxes +================ + +Public mailboxes are typically mailboxes that are visible to all users or to +large user groups. They are created by defining a public namespace, under which +all the shared mailboxes are. See <SharedMailboxes.Permissions.txt> for issues +related to filesystem permissions. See <Namespaces.txt> for details of how +namespaces are configured. + +For example to create a public Maildir mailboxes, use: + +---%<------------------------------------------------------------------------- +# User's private mail location +mail_location = maildir:~/Maildir + +# When creating any namespaces, you must also have a private namespace: +namespace { + type = private + separator = / + prefix = + #location defaults to mail_location. + inbox = yes +} + +namespace { + type = public + separator = / + prefix = Public/ + location = maildir:/var/mail/public + subscriptions = no +} +---%<------------------------------------------------------------------------- + +In the above example, you would then create Maildir mailboxes under the +'/var/mail/public/' directory. For example: + +---%<------------------------------------------------------------------------- +# ls -la /var/mail/public/ +drwxr-s--- 1 root mail 0 2007-03-19 03:12 . +drwxrws--- 1 root mail 0 2007-03-19 03:12 .lkml +drwxrws--- 1 root mail 0 2007-03-19 03:12 .bugtraq +-rw-rw---- 1 root mail 0 2007-03-19 03:12 dovecot-shared +---%<------------------------------------------------------------------------- + +Note that there are no 'cur/', 'new/' or 'tmp/' directories directly under the +'/var/mail/public/', because the Public/ namespace isn't a mailbox itself. (If +you create them manually, it does become a selectable mailbox. ) + +The 'dovecot-shared' file isn't directly used for either lkml or bugtraq +mailboxes, but if you create a new public mailbox via Dovecot it's +automatically copied there. + +Note that Dovecot uses Maildir++ layout by default for folders, where the +folder names must begin with a "." or Dovecot will ignore them. You can also +optionally use the <"fs" layout> [MailboxFormat.Maildir.txt] if you want the +directory structure to look like: + + * '/var/mail/public/' (root dir) + * '/var/mail/public/info/' (maildir folder) + * '/var/mail/public/company/' (maildir folder) + +Per-user \Seen flag (v2.2+) +--------------------------- + +Since v2.2 the recommended way to enable private flags for shared mailboxes is +to create private indexes with :INDEXPVT=<path>. This creates +dovecot.index.pvt[.log] files, which contain only the message UIDs and the +private flags. Currently the list of private flags is hardcoded only to the +\Seen flag. + +Example: + +---%<------------------------------------------------------------------------- +namespace { + type = public + separator = / + prefix = Public/ + location = maildir:/var/mail/public:INDEXPVT=~/Maildir/public + subscriptions = no +} +---%<------------------------------------------------------------------------- + +Maildir: Per-user \Seen flag +---------------------------- + +(With v2.2+ you should instead use the INDEXPVT as described above.) + +With Maildir a 'dovecot-shared' file controls if the \Seen flags are shared or +private. The file must be created separately inside each Maildir, although if +the file already exists in the Maildir root it's automatically copied for newly +created mailboxes. If 'dovecot-shared' file doesn't exist in Maildir, the \Seen +flags are shared. If it exists, the \Seen flag state is stored only in the +user's index files. By making each user have their own private index files, you +can make the \Seen flag private for the users. For example: + +---%<------------------------------------------------------------------------- +namespace { + type = public + separator = / + prefix = Public/ + location = maildir:/var/mail/public:INDEX=~/Maildir/public + subscriptions = no +} +---%<------------------------------------------------------------------------- + +Now when accessing e.g. "Public/lkml" mailbox, Dovecot keeps its index files in +'~/Maildir/public/.lkml/' directory. If it ever gets deleted, the \Seen flags +are lost. + +If you want to change what flags are shared when 'dovecot-shared' file exists, +currently you'll have to modify the source +code:'src/lib-storage/index/maildir/maildir-storage.c' maildir_open() has +'mbox->ibox.box.private_flags_mask = MAIL_SEEN;' Change the 'MAIL_SEEN' to any +flag combination you want. See 'src/lib-mail/mail-types.h' for list of valid +flags. + +Maildir: Keyword sharing +------------------------ + +Make sure you don't try to use per-user CONTROL directory. Otherwise +'dovecot-keywords' file doesn't get shared and keyword mapping breaks. + +Subscriptions +------------- + +Typically you want each user to have control over their own subscriptions for +mailboxes in public namespaces. This is why you should set 'subscriptions=no' +to the namespace. Dovecot will then use the parent namespace's subscriptions +file. Note that this practically means you must have a namespace with empty +prefix, otherwise there is no "parent namespace". + +Read-only mailboxes +------------------- + +Read-only mboxes +---------------- + +If you have a read-only directory structure containing mbox files, you'll need +to store index files elsewhere: + +---%<------------------------------------------------------------------------- +namespace { + type = public + prefix = Public/ + location = mbox:/var/mail/public/:INDEX=/var/indexes/public + subscriptions = no +} +---%<------------------------------------------------------------------------- + +In the above example all the users would still be sharing the index files, so +you might have problems with filesystem permissions. Alternatively you could +place the index files under user's home directory. + +Read-only Maildirs +------------------ + +If your Maildir is read-only, the control and index files still need to be +created somewhere. You can specify the path for these by appending +':CONTROL=<path>:INDEX=<path>' to mail location. The path may point to a +directory that is shared among all users, or to a per-user path. Note that if +the Maildir has any keywords, the per-user control directory breaks the +keywords since there is no 'dovecot-keywords' file. + +When configuring multiple namespaces, the CONTROL/INDEX path must be different +for each namespace. Otherwise if namespaces have identically named mailboxes +their control/index directories will conflict and cause all kinds of problems. + +If you put the control files to a per-user directory, you must also put the +index files to a per-user directory, otherwise you'll get errors. It is however +possible to use shared control files but per-user index files, assuming you've +set up permissions properly. + +---%<------------------------------------------------------------------------- +namespace { + type = public + separator = / + prefix = Public/ + location = +maildir:/var/mail/public:CONTROL=~/Maildir/public:INDEX=~/Maildir/public + subscriptions = no +} +namespace { + type = public + separator = / + prefix = Team/ + location = maildir:/var/mail/team:CONTROL=~/Maildir/team:INDEX=~/Maildir/team + subscriptions = no +} +---%<------------------------------------------------------------------------- + +Example: Public mailboxes with ACLs +----------------------------------- + +See <ACL.txt> for more information about ACLs. + +---%<------------------------------------------------------------------------- +namespace { + type = public + separator = . + prefix = public. + location = maildir:/var/mail/public + subscriptions = no + list = children +} + +plugin { + acl = vfile +} +---%<------------------------------------------------------------------------- + +It's important that the namespace type is "public" regardless of whether you +set the namespace prefix to "shared." or something else. + +After this you have to place 'dovecot-acl' files in every mailbox/folder below +'/var/mail/public' with rights for that folder (e.g. 'user=someone lr'). + +'acl_shared_dict' setting is not relevant for public mailboxes (only for +shared). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SharedMailboxes.Shared.txt b/doc/wiki/SharedMailboxes.Shared.txt new file mode 100644 index 0000000..879b4f4 --- /dev/null +++ b/doc/wiki/SharedMailboxes.Shared.txt @@ -0,0 +1,306 @@ +Mailbox sharing between users +============================= + +To enable mailbox sharing, you'll need to create a shared namespace. See +<ACL.txt> for more information about ACL-specific settings. + +---%<------------------------------------------------------------------------- +# User's private mail location. +mail_location = maildir:~/Maildir + +# When creating any namespaces, you must also have a private namespace: +namespace { + type = private + separator = / + prefix = + #location defaults to mail_location. + inbox = yes +} + +namespace { + type = shared + separator = / + prefix = shared/%%u/ + # a) Per-user seen flags. Maildir indexes are shared. (INDEXPVT requires +v2.2+) + location = maildir:%%h/Maildir:INDEXPVT=~/Maildir/shared/%%u + # b) Per-user seen flags. Maildir indexes are not shared. If users have +direct filesystem level access to their mails, this is a safer option: + #location = +maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u:INDEXPVT=~/Maildir/shared/%%u + subscriptions = no + list = children +} + +mail_plugins = acl +protocol imap { + mail_plugins = $mail_plugins imap_acl +} +plugin { + acl = vfile +} +---%<------------------------------------------------------------------------- + +This creates a shared/ namespace under which each user's mailboxes are. If you +have multiple domains and allow sharing between them, you might want to set +'prefix=shared/%%d/%%n/' instead (although %%u works just fine too). If you +don't, you might want to drop the domain part and instead use +'prefix=shared/%%n/'. + +'list=children' specifies that if no one has shared mailboxes to the user, the +"shared" directory isn't listed by the LIST command. If you wish it to be +visible always, you can set 'list=yes'. + +The 'location' setting specifies how to access other users' mailboxes. If you +use %%h, the user's home directory is asked from auth process via auth-userdb +socket. See <LDA.txt> for how to configure the socket. If the users' mailboxes +can be found using a template, it's faster not to use the %%h. For example: + +---%<------------------------------------------------------------------------- + location = maildir:/var/mail/%%d/%%n/Maildir:INDEXPVT=~/Maildir/shared/%%u +---%<------------------------------------------------------------------------- + +% vs %% +------- + +%var expands to the logged in user's variable, while %%var expands to the other +users' variables. For example if your name is "myself" and "someone1" and +"someone2" have shared mailboxes to you, the variables could be expanded like: + + * %u expands to "myself" + * %%u expands to "someone1" or "someone2" + * %h might expand to "/home/myself" + * %%h might expand to "/home/someone1" or "/home/someone2" + * ~/ equals %h/ + +Note that in e.g. mail_location setting you might need both. For example in: + +---%<------------------------------------------------------------------------- +mail_location = maildir:%%h/Maildir:INDEXPVT=%h/Maildir/shared/%%u +---%<------------------------------------------------------------------------- + +What it means is: + + * %%h/Maildir points to the other user's Maildir, e.g. "/home/someone1". + * :INDEXPVT=%h/Maildir/shared/%%u points to a per-user directory under your + own Maildir, e.g. "/home/myself/Maildir/someone1" or + "/home/myself/Maildir/someone2". This is necessary for storing per-user seen + flags. + +dbox +---- + +With dbox the index files are a very important part of the mailboxes. You must +not try to :INDEX= to have copies of index files. This will only result in +mailbox corruption. (INDEXPVT can be used though.) + +Filesystem permissions +---------------------- + +Dovecot assumes that it can access the other users' mailboxes. If you use +multiple UNIX UIDs, you may have problems setting up the permissions so that +the mailbox sharing works. Dovecot never modifies existing files' permissions. +See <SharedMailboxes.Permissions.txt> for more information. + +Shared mailbox listing +---------------------- + +With the above configuration it's possible to open shared mailboxes if you know +their name, but they won't be visible in the mailbox list. This is because +Dovecot has no way of knowing what users have shared mailboxes to whom. +Iterating through all users and looking inside their mail directories would be +horribly inefficient for more than a couple users. + +To overcome this problem Dovecot needs a dictionary, which contains the list of +users who have shared mailboxes and to whom they have shared. If the users +aren't properly listed in this dictionary, their shared mailboxes won't be +visible. Currently there's no way to automatically rebuild this dictionary, so +make sure it doesn't get lost. If it does, each user having shared mailboxes +must use the IMAP SETACL command (see below) to get the dictionary updated for +themselves. + +You could use any dictionary backend, including SQL, but a simple flat file +should work pretty well too: + +---%<------------------------------------------------------------------------- +plugin { + acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes.db +} +---%<------------------------------------------------------------------------- + +The IMAP processes must be able to write to the 'db/' directory. If you're +using system users, you probably want to make it mode 0770 and group +'sharedusers' and set 'mail_access_groups=sharedusers' (or something similar). + +If you use multiple domains and don't wish users to share their mailboxes to +users in other domains, you can use separate dict files for each domain: + +---%<------------------------------------------------------------------------- +plugin { + # assumes mailboxes are in /var/mail/%d/%n: + acl_shared_dict = file:/var/mail/%d/shared-mailboxes.db +} +---%<------------------------------------------------------------------------- + +Using SQL dictionary +-------------------- + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +plugin { + acl_shared_dict = proxy::acl +} + +dict { + acl = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +See <Dict.txt> for more information, especially about permission issues. + +Database tables: + +---%<------------------------------------------------------------------------- +CREATE TABLE user_shares ( + from_user varchar(100) not null, + to_user varchar(100) not null, + dummy char(1) DEFAULT '1', -- always '1' currently + primary key (from_user, to_user) +); +COMMENT ON TABLE user_shares IS 'User from_user shares folders to user +to_user.'; + +CREATE INDEX to_user +ON user_shares (to_user); -- because we always search for to_user + +CREATE TABLE anyone_shares ( + from_user varchar(100) not null, + dummy char(1) DEFAULT '1', -- always '1' currently + primary key (from_user) +); +COMMENT ON TABLE anyone_shares IS 'User from_user shares folders to anyone.'; +---%<------------------------------------------------------------------------- + +'/etc/dovecot/dovecot-dict-sql.conf.ext': + +---%<------------------------------------------------------------------------- +connect = host=localhost dbname=mails user=sqluser password=sqlpass +map { + pattern = shared/shared-boxes/user/$to/$from + table = user_shares + value_field = dummy + + fields { + from_user = $from + to_user = $to + } +} + +map { + pattern = shared/shared-boxes/anyone/$from + table = anyone_shares + value_field = dummy + + fields { + from_user = $from + } +} +---%<------------------------------------------------------------------------- + +Mailbox sharing +--------------- + +You can use <doveadm acl> [Tools.Doveadm.ACL.txt] to share mailboxes or it can +be done using IMAP SETACL command. It is the only way to update the shared +mailbox list dictionary. + +Below is a quick introduction to IMAP ACL commands. See RFC 4314 +[http://www.ietf.org/rfc/rfc4314.txt] for more details. + + * MYRIGHTS <mailbox>: Returns the user's current rights to the mailbox. + * GETACL <mailbox>: Returns the mailbox's all ACLs. + * SETACL <mailbox> <id> [+|-]<rights>: Give <id> the specified rights to the + mailbox. + * DELETEACL <mailbox> [-]<id>: Delete <id>'s ACL from the mailbox. + +<id> is one of: + + * anyone: Matches all users, including anonymous users. + * authenticated: Like "anyone", but doesn't match anonymous users. + * $group: Matches all users belonging to the group ($ is not part of the group + name). + * $!group: See group-override in <ACL.txt> (Dovecot-specific feature). + * user: Matches the given user. + +The $group syntax is not a standard, but it is mentioned in RFC 4314 examples +and is also understood by at least Cyrus IMAP. Having '-' before the identifier +specifies negative rights. + +See <ACL.txt> for list of <rights>. + +Sharing mailboxes to everyone +----------------------------- + +By default Dovecot doesn't allow using the IMAP "anyone" or "authenticated" +identifier, because it would be an easy way to spam other users in the system. +If you wish to allow it, set: + +---%<------------------------------------------------------------------------- +plugin { + acl_anyone = allow +} +---%<------------------------------------------------------------------------- + +Note that you can also do this only for some users by using the second table +"anyone_shares". Every user listed in this table shares his folders with +everyone. See also <userdb extra field> [UserDatabase.ExtraFields.txt]. + +IMAP ACL examples +----------------- + +Let's begin with some simple example that first gives "read" and "lookup" +rights, and later adds "write-seen" right: + +---%<------------------------------------------------------------------------- +1 SETACL Work user@domain rl +1 OK Setacl complete. + +2 SETACL Work user@domain +s +2 OK Setacl complete. + +3 GETACL Work +* ACL "Work" "user@domain" lrs "myself" lrwstipekxacd +3 OK Getacl completed. +---%<------------------------------------------------------------------------- + +Let's see how negative rights work by testing it on ourself. See how we +initially have "lookup" right, but later we don't: + +---%<------------------------------------------------------------------------- +1 MYRIGHTS Work +* MYRIGHTS "Work" lrwstipekxacd +1 OK Myrights completed. + +2 SETACL Work -myself l +2 OK Setacl complete. + +3 GETACL Work +* ACL "Work" "-myself" l "user@domain" lr "myself" lrwstipekxacd +3 OK Getacl completed. + +4 myrights Work +* MYRIGHTS "Work" rwstipekxacd +4 OK Myrights completed. +---%<------------------------------------------------------------------------- + +Troubleshooting +--------------- + + * Make sure the % and %% variables are specified correctly in the namespace + location.'mail_debug=yes' will help you see if Dovecot is trying to access + correct paths. + * 'doveadm acl debug -u user@domain shared/user/box' can be helpful in + figuring out why a mailbox can't be accessed. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SharedMailboxes.Symlinks.txt b/doc/wiki/SharedMailboxes.Symlinks.txt new file mode 100644 index 0000000..08cd903 --- /dev/null +++ b/doc/wiki/SharedMailboxes.Symlinks.txt @@ -0,0 +1,42 @@ +Mailbox sharing with symlinks +============================= + +It's possible to share mailboxes simply by symlinking them among user's private +mailboxes. See <SharedMailboxes.Permissions.txt> for issues related to +filesystem permissions. + +Maildir +------- + +---%<------------------------------------------------------------------------- +ln -s /home/user2/Maildir/.Work /home/user1/Maildir/.shared.user2 +ln -s /home/user3/Maildir/.Work /home/user1/Maildir/.shared.user3 +---%<------------------------------------------------------------------------- + +Now user1 has a "shared" directory containing "user2" and "user3" child +mailboxes, which point to those users' "Work" mailbox. + +With Maildir++ layout it's not possible to automatically share "mailbox and its +children". You'll need to symlink each mailbox separately. With v1.1+ you could +do this by using "fs" layout for mailboxes (requires converting existing +maildirs from Maildir++). + +mbox +---- + +Doing the same as in the above Maildir example: + +---%<------------------------------------------------------------------------- +mkdir /home/user1/mail/shared +ln -s /home/user2/mail/Work /home/user1/mail/shared/user2 +ln -s /home/user3/mail/Work /home/user1/mail/shared/user3 +---%<------------------------------------------------------------------------- + +One additional problem with mbox format is the creation of dotlock files. The +dotlocks would be created under user1's directory, which makes them useless. +Make sure the locking works properly with only fcntl or flock locking (See +<MboxLocking.txt>) and just disable dotlocks. Alternatively instead of +symlinking an mbox file, put the shared mailboxes inside a directory and +symlink the entire directory. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SharedMailboxes.txt b/doc/wiki/SharedMailboxes.txt new file mode 100644 index 0000000..8009553 --- /dev/null +++ b/doc/wiki/SharedMailboxes.txt @@ -0,0 +1,17 @@ +Shared mailboxes +================ + +Dovecot can support mailbox sharing in several different ways: + + * <Public mailboxes> [SharedMailboxes.Public.txt]: Shared mailboxes created by + administrators (ACLs can still restrict who sees them) + * <Shared mailboxes> [SharedMailboxes.Shared.txt]: Users sharing their + mailboxes to other users (using IMAP ACL commands) (v1.2+) + * <Symlinking mailboxes> [SharedMailboxes.Symlinks.txt]: Quick and dirty way + of sharing a few mailboxes. + * <Shared Mailboxes in Dovecot Cluster> [SharedMailboxes.ClusterSetup.txt] + +See <SharedMailboxes.Permissions.txt> for common filesystem related permission +problems that are common with all the sharing methods. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SocketUnavailable.txt b/doc/wiki/SocketUnavailable.txt new file mode 100644 index 0000000..b4068bc --- /dev/null +++ b/doc/wiki/SocketUnavailable.txt @@ -0,0 +1,84 @@ +UNIX Socket Resource Temporarily Unavailable +============================================ + +Commonly visible as: + +---%<------------------------------------------------------------------------- +imap-login: Error: net_connect_unix(imap) failed: Resource temporarily +unavailable +---%<------------------------------------------------------------------------- + +This means that there are more imap-login processes trying to connect to the +"imap" UNIX socket than there are imap processes accepting the connections. The +kernel's connection listener queue got full and it started rejecting further +connections. So what can be done about it? + +Wrong service settings +---------------------- + +This can happen if 'service imap { client_limit } ' is set to anything else +than 1. IMAP (and POP3 and other mail) processes do disk IO, lock waiting and +such, so if all the available imap processes are stuck waiting on something, +they can't accept new connections and they queue up in the kernel. For mail +processes only 'client_limit=1' is recommended, except maybe for very tiny +systems with a few users. + +It can also happen if 'service imap { process_limit } ' is reached. Dovecot +logs a warning if process_limit or client_limit is reached. + +Out of file descriptors +----------------------- + +If the "ulimit -n" is too low, kernel stops notifying the process about +incoming connections. Make sure that the limit is at least as high as the +client_limit. Dovecot also internally checks this, and if it's too low it +writes a warning to stderr at startup (and to log in v2.2.33+). + +Note that Dovecot is not using "dovecot" user's or PAM's limits in general. +Make sure the limits are correct with:'cat /proc/`pidof dovecot`/limits' + +Master process busy +------------------- + +Dovecot master process forks all of the new processes. If it's using 100% CPU, +it doesn't have time to fork enough new processes. Even if it's not constantly +using 100% CPU there may be fork bursts where it temporarily gets too busy. The +solution is to make it do less work by forking less processes: + + * Most importantly switch to <high-performance login process mode> + [LoginProcess.txt]. This alone might be enough. + * You can also switch (most of the) other commonly forked processes to be + reused. For example 'service imap { service_count = 100 } ' reuses the imap + process for 100 different IMAP connections before it dies. This is useful + mainly for imap, pop3 and managesieve services. It's better to avoid using + 'service_count=0' (unlimited) in case there are memory leaks. + * You can pre-fork some idling processes to handle bursts with 'service { + process_min_avail }'. + +See <Services.txt> before changing any service settings. Some services require +specific values to work correctly. + +Listener queue size +------------------- + +Dovecot uses the ' service { client_limit } '*' service { process_limit } ' as +the listener queue size. Dovecot v2.2.25 and older also had a hardcoded maximum +of 511, while later versions have no upper limit. Most OSes use an even lower +limit, typically 128. In Linux you can increase this from +'/proc/sys/net/core/somaxconn'. + +Dovecot v2.2.[0-6] bug +---------------------- + +Dovecot v2.2.0 - v2.2.6 were attempting to optimize host.domain lookups by +doing them only once in the master process. Unfortunately they were actually +doing the lookup every time when creating a new process. In some configuration +this lookup could have done a somewhat slow DNS lookup, causing the process +creation to become very slow and triggering this message. + +The fix is in v2.2.7 and you can also workaround this: + + * Add to dovecot.conf: 'import_environment = TZ DEBUG_OUTOFMEM + DOVECOT_HOSTDOMAIN=mailserver.example.com' (of course changing the value) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Statistics.Old.txt b/doc/wiki/Statistics.Old.txt new file mode 100644 index 0000000..210103a --- /dev/null +++ b/doc/wiki/Statistics.Old.txt @@ -0,0 +1,227 @@ +Statistics +========== + +Dovecot v2.1+ supports gathering statistics (CPU, disk usage, etc.) from mail +processes (IMAP, POP3, LMTP, etc.) to the stats process. The stats process can +later be queried what's going on in the system. With imap_stats plugin you can +get per-command level statistics for IMAP commands. + +There are different "zoom levels" you can look at the statistics: + + * command: Per-IMAP command + * session: Per IMAP/POP3 connection + * user: Per user (all of user's sessions summed up) + * domain: Per domain (all of domain's users summed up) + * ip: Per IP address (all sessions from the IP summed up) + * global: Everything summed up (2.2.16+) + +Basic Configuration +------------------- + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins stats +protocol imap { + mail_plugins = $mail_plugins imap_stats +} +plugin { + # how often to session statistics (must be set) + stats_refresh = 30 secs + # track per-IMAP command statistics (optional) + stats_track_cmds = yes +} +---%<------------------------------------------------------------------------- + +You'll also need to give enough permissions for mail processes to be able to +write to stats-mail fifo. For example if you use a single "vmail" user for mail +access: + +---%<------------------------------------------------------------------------- +service stats { + fifo_listener stats-mail { + user = vmail + mode = 0600 + } +} +---%<------------------------------------------------------------------------- + +Memory usage configuration +-------------------------- + +The stats process attempts to keep memory usage below a specified amount. This +value is only approximate because of extra overhead caused by malloc() itself. + +---%<------------------------------------------------------------------------- +stats_memory_limit = 16 M +---%<------------------------------------------------------------------------- + +Once the memory limit is reached, oldest statistics are freed from memory. +Different statistics levels have different timeout limits, which are configured +in: + +---%<------------------------------------------------------------------------- +stats_command_min_time = 1 mins +stats_domain_min_time = 12 hours +stats_ip_min_time = 12 hours +stats_session_min_time = 15 mins +stats_user_min_time = 1 hours +---%<------------------------------------------------------------------------- + +So for example the above means: + + * An IMAP command is kept in memory for at least 1 minute after it has + finished + * A user is kept in memory for 1 hour after its last session has disconnected. + +The stats process attempts to honor these min_time-settings, but if memory is +tight it can go below these values to honor the 'stats_memory_limit' setting. + +Statistics gathered +------------------- + +Statistics gathered internally by the stats process: + + * num_logins: Number of logins (2.2.14+) + * num_cmds: Number of IMAP commands run (2.2.14+) + * num_connected_sessions: Number of current IMAP sessions (2.2.14+) + +Statistics gathered using the 'getrusage()' system call: + + * user_cpu: User CPU (seconds.microseconds) + * sys_cpu: System CPU (seconds.microseconds) + * clock_time: Wall-clock time (seconds.microseconds). Doesn't include time + spent waiting in ioloop, which means it doesn't include (most of) the time + spent waiting on client network traffic. (v2.2.11+) + * min_faults: Minor page faults (page reclaims) + * maj_faults: Major page faults + * vol_cs: Voluntary context switches + * invol_cs: Involuntary context switches + * disk_input: Number of bytes read from disk + * disk_output: Number of bytes written to disk + +The disk_input and disk_output attempt to count the actual read/write bytes to +physical disk, so e.g. reads from OS's cache aren't counted. Note that not all +operating systems and filesystem support this, instead they simply show these +values always as 0. + +Statistics gathered from '/proc/self/io' output (Linux-only): + + * read_count: Number of read() syscalls + * write_count: Number of write() syscalls + * read_bytes: Number of bytes read using read() syscalls + * write_bytes: Number of bytes written using write() syscalls + +Note that the above numbers are not only about disk I/O, but also about network +I/O, Dovecot's IPC and every other kind of reads/writes as well. + +Statistics gathered by Dovecot's lib-storage internally: + + * mail_lookup_path: Number of open() and stat() calls (i.e. "path lookups") + * mail_lookup_attr: Number of stat() and fstat() calls + * mail_read_count: Number of read() calls for message data (e.g. index files + not counted) + * mail_read_bytes: Number of message bytes read() + * mail_cache_hits: Number of cache hits from 'dovecot.index.cache' file + +Note that statistics are collected only on backends so stats service doesn't do +anything on directors and proxies. + +doveadm stats +------------- + +top +--- + +'doveadm stats top [<sort field>]' + +The top command gives a very simple "top"-like view of connected sessions. The +optional sort field is one of: + + * disk: disk_input and disk_output summed up (default) + * cpu: user_cpu and sys_cpu summed up + * any other statistics field + +This "top" isn't very good, but a much better one can be found as a Perl +script:stats-top.pl [https://dovecot.org/tools/stats-top.pl], which also +requires stats.pl [https://dovecot.org/tools/stats.pl] and tab-formatter.pl +[https://dovecot.org/tools/tab-formatter.pl]. + +dump +---- + +'doveadm stats dump <level> [<filter>]' + +The dump command shows a raw output of the statistics. The level parameter is +one of the levels listed at the top of this page (e.g. "session"). The filter +can contain zero of more filters: + + * connected: The session must be currently connected (or the user/domain/ip + must have at least one session that is currently connected) + * since=<timestamp>: Last update was since this UNIX timestamp + * user=<wildcard>: Username matches this wildard + * domain=<wildcard>: Domain name matches this wildard + * ip=<ip>[/bits]: IP address matches this IP/network (e.g. 192.168.1.0/24) + +If nothing matches the filter, the output is a single empty line. Otherwise it +begins with a header line followed by data lines. Each line has a list of +fields separated by TABs. The header describes what the data fields are. The +list of fields depends on what level you're listing. Some of the fields are: + + * session: 128 bit session GUID in hex format. This uniquely identifies a + single session. Used by commands and sessions. + * connected: Is the client currently connected? 0=no, 1=yes. + * pid: Process ID of the session. If the session is no longer connected, the + PID may not exist anymore. + * last_update: UNIX timestamp of the last time this data was updated + * reset_timestamp: UNIX timestamp of when this user/domain/ip structure was + created. This is useful when you want to track incrementally what changed: + * If timestamp is the same as in your previous lookup, you can simply count + different = new_value - previous_value. + * If timestamp has changed since your previous lookup, the statistics were + reset to zero since and the difference = new_value. + +Stats protocol +-------------- + +You can connect to stats process via '$base_dir/stats' UNIX socket, or you can +simply add more UNIX/TCP listeners to the stats service, e.g.: + +---%<------------------------------------------------------------------------- +service stats { + inet_listener { + address = 127.0.0.1 + port = 24242 + } +} +---%<------------------------------------------------------------------------- + +The protocol is almost entirely identical to 'doveadm stats dump' command's +parameters and output. The only difference is that you prefix your request with +"EXPORT<tab>". For example: + +---%<------------------------------------------------------------------------- +EXPORT<tab>session<tab>connected<lf> +---%<------------------------------------------------------------------------- + +The output will be identical to 'doveadm stats dump session connected' command. + +Carbon support +-------------- + +Since v2.2.27, you can configure dovecot to send statistics periodically in +carbon format. To do this, configure + +---%<------------------------------------------------------------------------- +stats_carbon_server=ip:port # default port 2003 +stats_carbon_name=hostname # do not use dots +stats_carbon_interval=30s # default is 30 seconds + +service stats { + # this is needed if you want stats to be sent when no one is connected + process_min_avail=1 +} +---%<------------------------------------------------------------------------- + +this will send all available global statistics in carbon format +[https://graphite.readthedocs.io/en/latest/feeding-carbon.html]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Statistics.txt b/doc/wiki/Statistics.txt new file mode 100644 index 0000000..e299220 --- /dev/null +++ b/doc/wiki/Statistics.txt @@ -0,0 +1,109 @@ +Statistics +========== + +This page is about the statistics support for Dovecot v2.3. For v2.1 and v2.2 +see <Statistics.Old.txt>. + +See <Events.txt> for list of all events that can be used in statistics. + +Dovecot v2.3 supports gathering statistics from "events". Eventually all of the +log messages should be events, so it will be possible to configure Dovecot to +get statistics for anything that is logged. For debug messages it's possible to +get statistics even if the message itself isn't logged. + +Currently there are no statistics logged by default (but this might change). +You'll need to add some metrics: + +---%<------------------------------------------------------------------------- +metric name { + # Individual events can be identified either by their name or source +file:line location. + # The source location of course can change between Dovecot versions, so it +should be + # avoided. + event_name = example_event_name + #source_location = example.c:123 + + # Space-separated list of categories that must match the event (e.g. "mail" +or "storage") + #categories = + + # List of fields in event parameters that are included in the metrics. + # All events have a default "duration" field that doesn't need to be listed +here. + #fields = + + # List of key=value pairs that must match the event. The value can contain +'?' and '*' wildcards. + #filter { + # field_key = wildcard + #} +} +---%<------------------------------------------------------------------------- + +Listing Statistics +------------------ + +The gathered statistics are available by running: + +---%<------------------------------------------------------------------------- +doveadm stats dump +---%<------------------------------------------------------------------------- + +Each event has a "duration" field, which tracks in microseconds how long the +event existed. For example with imap_command_finished field it could be: + +---%<------------------------------------------------------------------------- +metric_name field count sum min max avg median %95 +imap_commands duration 35 1190122 162 340477 34003 244 188637 +---%<------------------------------------------------------------------------- + +The above means: + + * count: There have been 35 IMAP commands + * sum: The IMAP commands were running in total for 1190122 microseconds (= 1.1 + seconds) + * min: The fastest IMAP command took 162 microseconds + * max: The slowest IMAP command took 340477 microseconds + * avg: The average time spent on an IMAP commands was 34003 microseconds + * median: The median time spent on an IMAP command was 244 microseconds + * %95: 95% of the IMAP commands took 188637 microseconds or less + +The other fields (than duration) track whatever that field represents. For +example with imap_command_finished's bytes_in field could be tracking how many +bytes were being used by the IMAP commands. Non-numeric fields can also be +tracked, although only the "count" is relevant to those. + +The stats counters are reset whenever the stats process is started, which also +means a dovecot reload will reset statistics. Using 'doveadm stats -r' +parameter will also reset the statistics atomically after they're dumped. + +Examples +-------- + +---%<------------------------------------------------------------------------- +metric imap_select_no { + event_name = imap_command_finished + filter { + name = SELECT + tagged_reply_state = NO + } +} +metric imap_select_no_notfound { + event_name = imap_command_finished + filter { + name = SELECT + tagged_reply = NO*Mailbox doesn't exist:* + } +} + +metric storage_http_gets { + event_name = http_request_finished + categories = storage + filter { + method = get + } +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Submission.txt b/doc/wiki/Submission.txt new file mode 100644 index 0000000..46bafb3 --- /dev/null +++ b/doc/wiki/Submission.txt @@ -0,0 +1,191 @@ +Submission Server +================= + +As of version 2.3.0, Dovecot provides an SMTP submission service, also known as +a Mail Submission Agent (MSA) [https://tools.ietf.org/html/rfc6409]. It is +currently implemented as a proxy that acts as a front-end for any <MTA.txt>, +adding the necessary functionality required for a submission service: it adds +the required AUTH [https://tools.ietf.org/html/rfc4954] support, avoiding the +need to configure the MTA for SASL authentication. More SMTP capabilities like +CHUNKING [https://tools.ietf.org/html/rfc3030] and SIZE +[https://tools.ietf.org/html/rfc1870] are supported, without requiring the +backend MTA supporting these extensions. Other capabilities like 8BITMIME +[https://tools.ietf.org/html/rfc6152] and DSN +[https://tools.ietf.org/html/rfc3461] currently require support from the +backend/relay MTA. + +The most notable feature that the proxy adds is the BURL capability +[https://tools.ietf.org/html/rfc4468]. The main application of that +capabilitytogether with <IMAP> [IMAPServer.txt] URLAUTH +[https://tools.ietf.org/html/rfc4467]is avoiding a duplicate upload of +submitted e-mail messages; normally the message is both sent through SMTP and +uploaded to the "Sent" folder through IMAP. Using BURL, the client can first +upload the message to IMAP and then use BURL to make the SMTP server fetch the +message from IMAP for submission, thereby avoiding a second upload. Few clients +currently support the BURL capability, but once it becomes available on the +server side, client developers will at least have some incentive to provide +support for this feature. + +*NOTE:* Currently, the submission proxy is still pretty basic. However, it will +provide a basis for adding all kinds of functionality in the (not so distant) +future. For the first time, it will be possible to act upon message submission, +rather than only message retrieval; e.g. plugins can be devised that process +outgoing messages somehow. Examples of the things that could be implemented are +adding <Sieve> [Pigeonhole.txt] filtering support for outgoing messages, or +implicitly storing submitted messages to the Sent folder. Once a plugin API is +devised, you can create your own plugins. + +Features +-------- + +The following SMTP capabilities are supported by the Dovecot submission +service: + + * 8BITMIME [https://tools.ietf.org/html/rfc6152] - Only if relay MTA provides + support + * AUTH [https://tools.ietf.org/html/rfc4954] + * BURL [https://tools.ietf.org/html/rfc4468] + * CHUNKING [https://tools.ietf.org/html/rfc3030] + * DSN [https://tools.ietf.org/html/rfc3461] - Only if relay MTA provides + support + * ENHANCEDSTATUSCODES [https://tools.ietf.org/html/rfc2034] + * PIPELINING [https://tools.ietf.org/html/rfc2920] + * SIZE [https://tools.ietf.org/html/rfc1870] + * STARTTLS [https://tools.ietf.org/html/rfc3207] + * VRFY [https://tools.ietf.org/html/rfc5321] + * XCLIENT [http://www.postfix.org/XCLIENT_README.html] + +Configuration +------------- + +Submission Service +------------------ + +Just add 'submission' to the 'protocols=' setting and configure the relay MTA +server. The submission service is a login service, just like IMAP, POP3 and +<ManageSieve> [Pigeonhole.ManageSieve.txt], so clients are required to +authenticate. The same <authentication configuration> [Authentication.txt] will +also apply to submission, unless you're doing protocol-specific things, in +which case you may need to amend your configuration for the new protocol. BURL +support requires a working IMAP URLAUTH implementation. + +The following settings apply to the Submission service: + +submission_logout_format = in=%i out=%o: + The SMTP Submission logout format string. The following variable + substitutions are supported: + %i : + Total number of bytes read from client + + %o : + Total number of bytes sent to client + + %{command_count} : + Number of commands received from client + + %{reply_count} : + Number of replies sent to client + + %{session} : + Session ID of the login session + + %{transaction_id} : + ID of the current transaction, if any + +hostname : + The host name reported by the SMTP service (and other parts of Dovecot), for + example to the client in the initial greeting and to the relay server in the + HELO/EHLO command. Default is the system's real hostname@domain. + +submission_max_mail_size : + The maximum size of messages accepted for relay. This announced in the SMTP + SIZE capability. If not configured, this is either determined from the relay + server or left unlimited if no limit is known (the relay MTA will reply with + error if some unknown limit exists there, which is duly passed to Dovecot's + client). + +submission_max_recipients : + Maximum number of recipients accepted per connection (default: unlimited). + +Relay MTA +--------- + +The Dovecot SMTP submission service directly proxies the mail transaction to +the SMTP relay configured with the following settings: + +submission_relay_host : + Host name for the relay server (required). + +submission_relay_port = 25 : + Port for the relay server. + +submission_relay_trusted = no : + Is the relay server trusted? This determines whether we try to send + (Postfix-specific) XCLIENT data to the relay server. + +submission_relay_user = : + User name for authentication to the relay MTA if authentication is required + there (authorization ID). + +submission_relay_master_user = : + Master user name for authentication to the relay MTA if authentication is + required there (authentication ID). + +submission_relay_password = : + Password for authentication to the relay MTA if authentication is required + there. + +submission_relay_ssl = no : + Indicates whether TLS is used for the connection to the relay server. The + following values are defined for this setting: + no : + No SSL is used. + + smtps : + An SMTPS connection (immediate SSL) is used. + + starttls : + The STARTTLS command is used to establish the TLS layer. + +submission_relay_ssl_verify = yes : + Configures whether the TLS certificate of the relay server is to be verified. + +submission_relay_rawlog_dir : + Write protocol logs for relay connection to this directory for debugging. + +See <Debugging.Rawlog.txt> for more details on rawlogs. + +Login Proxy +----------- + +Like IMAP and POP3, the Submission login service supports proxying to multiple +backend Dovecot servers. The <proxy configuration wiki page> +[PasswordDatabase.ExtraFields.Proxy.txt] for POP3 and IMAP applies +automatically to Submission as well. + +*IMPORTANT*: Please note that the login proxy described here is configured +between two Dovecot servers (e.g. director frontend and mail storage backend). +This is not the way to configure the relay connection between the Dovecot +submission service and the MTA! That is configured using the relay settings +described in the <previous section> [Submission.txt]. If you get this wrong, +things will seem to work (at least to some extent), but the service provided by +Dovecot will be effectively bypassed. + +Client Support +-------------- + +As of April 2019 there aren't many clients that support the BURL extension. ++-------------+-----------+-------------------------------------------------------------------------------------------------------+ +| *Client* | *Supports | *Notes* | +| | BURL?* | | ++-------------+-----------+-------------------------------------------------------------------------------------------------------+ +| Trojita | Yes | Mentioned on the dovecot discussion list | +| | | [http://dovecot.2317879.n4.nabble.com/New-Dovecot-service-SMTP-Submission-RFC6409-tp62117p62122.html] | ++-------------+-----------+-------------------------------------------------------------------------------------------------------+ +| Thunderbird | No | Vote on bugzilla [https://bugzilla.mozilla.org/page.cgi?id=voting/user.html&bug_id=421779] | ++-------------+-----------+-------------------------------------------------------------------------------------------------------+ +| KDE Akonadi | No | KDE bugzilla [https://bugs.kde.org/show_bug.cgi?id=408732] | +| / KMail | | | ++-------------+-----------+-------------------------------------------------------------------------------------------------------+ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/SystemUsers.txt b/doc/wiki/SystemUsers.txt new file mode 100644 index 0000000..45b5fd9 --- /dev/null +++ b/doc/wiki/SystemUsers.txt @@ -0,0 +1,112 @@ +System Users +============ + +System users are typically defined in '/etc/passwd' file, but this isn't +necessary. Using NSS [http://en.wikipedia.org/wiki/Name_Service_Switch] you can +configure the lookups to be done from elsewhere (e.g. LDAP). See <passwd> +[AuthDatabase.Passwd.txt] userdb configuration for how to set this up. +Especially if you're using nss_ldap you must set 'blocking=yes'. + +System users usually have their own separate user IDs (UIDs). This is good from +security point of view, because it means that the kernel will also prevent +users from accessing each others' mails. + +If the users have direct write access to the mail files (eg. the users have +shell access), they can easily cause all sorts of mailbox corruptions. That may +generate all kinds of error messages to Dovecot's error logs, so it may be +sometimes difficult to tell if there really is a problem or if user is just +doing something stupid. + +If users are going to access the mailboxes with other software than Dovecot, +it's important to make sure that their mailbox accesses are compatible. This +mostly means that with mboxes you must make sure that everyone uses the <same +locking methods in the same order> [MailboxFormat.mbox.txt]. + +Authentication +-------------- + +Admins often wish to use different passwords for IMAP and POP3 than for other +services (eg. SSH), because IMAP and POP3 clients often send the password +unencrypted over the internet without even bothering to give users any +warnings. Dovecot can easily support non-system passwords for system users. + +If you wish to use system passwords, you'll want to use one of these passdbs: + + * <PAM> [PasswordDatabase.PAM.txt]: Most commonly used in Linux and BSDs + nowadays. + * <BSDAuth> [PasswordDatabase.BSDAuth.txt]: BSD authentication is used by + OpenBSD. + * <Passwd> [AuthDatabase.Passwd.txt]: System users (NSS, '/etc/passwd', or + similiar). This may work instead of PAM (mostly in some BSDs). + * <Shadow> [PasswordDatabase.Shadow.txt]: Shadow passwords for system users + (NSS,'/etc/shadow' or similiar). Deprecated by PAM nowadays, but it should + work with Linux and Solaris. + +If you wish to use non-system passwords, you can use pretty much any of the +Dovecot's <password databases> [PasswordDatabase.txt], but for simple +installations you'll probably want to use <passwd-file> +[AuthDatabase.PasswdFile.txt]. + +<User database> [UserDatabase.txt] for system users is always <passwd> +[AuthDatabase.Passwd.txt]. + +Mail Location +------------- + +Usually UNIX systems are configured by default to deliver mails to +'/var/mail/username' or '/var/spool/mail/username' mboxes. You may decide to +use these, or use <maildir> [MailboxFormat.Maildir.txt] format instead. + +Dovecot detects the mailbox format and location automatically if +'mail_location' setting isn't set, but it's still a good idea to explicitly +tell Dovecot where to find the mails. This makes it sure that Dovecot behaves +correctly also when the user's mailbox doesn't exist at the moment (eg. a new +user). If Dovecot can't figure out where the existing mails are, it simply +gives an error message and quits. It never tries to create a missing mailbox +when autodetection is used. + +See <MailLocation.txt> for more information how to configure the mailbox +location. Below are the highlights for mbox and maildir. + +mbox +---- + +The '/var/mail/username' mbox is called user's INBOX. IMAP protocol supports +multiple mailboxes however, so Dovecot needs some directory where to store the +other mailboxes. Typically they're stored in '~/mail/' or '~/Mail/' directory. +All of these locations are included in mailbox location autodetection. You can +specify them manually with: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +Remember that the first path after 'mbox:' is the mailbox root directory, never +try to give 'mbox:/var/mail/%u' because that just isn't going to work (unless +you really want to store mails under '/var/mail/%u/' directory). + +If you're also using other software than Dovecot to access mboxes, you should +try to figure out what locking methods exactly they're using and update +'mbox_read_locks' and 'mbox_write_locks' settings accordingly. See locking +section in <mbox> [MailboxFormat.mbox.txt] for more information. + +Maildir +------- + +Maildir is typically stored in '~/Maildir' directory. You can specify this +manually with: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +---%<------------------------------------------------------------------------- + +Chrooting +--------- + +Dovecot, including several other software, allow using "/./" in home directory +path to specify the chroot path. For example '/home/./user' would chroot to +'/home'. If you want to enable this for Dovecot, add the chroot path to +'valid_chroot_dirs' setting ('/home' in the previous example). If this isn't +done, Dovecot just ignores the "/./". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/TestInstallation.txt b/doc/wiki/TestInstallation.txt new file mode 100644 index 0000000..71416dc --- /dev/null +++ b/doc/wiki/TestInstallation.txt @@ -0,0 +1,239 @@ +Contents + + + 1. Check that it's running + + 2. Check that it's listening + + 3. Check that it's allowing logins + + 4. Check that it's allowing remote logins + + 5. Check that it finds INBOX + + 6. Check that it finds other mailboxes + + 7. Check out some other IMAP commands + + 8. Check that real mail clients work + + 9. Make a graceful exit + +For testing POP3 installation, see <TestPop3Installation.txt>. + +Check that it's running +----------------------- + +First check with 'ps' that 'dovecot' process is actually running. If it's not, +you had an error in 'dovecot.conf' and the error message was written to log. Go +back to <RunningDovecot.txt> and <Logging.txt> if you can't find it. + +Check that it's listening +------------------------- + +Next check that Dovecot is listening for connections: + +---%<------------------------------------------------------------------------- +# telnet localhost 143 +Trying 127.0.0.1... +Connected to localhost. +Escape character is '^]'. +* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS +AUTH=PLAIN] Dovecot ready. +---%<------------------------------------------------------------------------- + +If you got "connection refused", make sure that Dovecot is configured to serve +the imap protocol and listening on the expected interfaces/addresses.The +simplest way to do that would be using <doveconf(1)> [Tools.Doveconf.txt]: + +---%<------------------------------------------------------------------------- +# doveconf protocols listen +protocols = imap pop3 lmtp sieve +listen = *, :: +---%<------------------------------------------------------------------------- + +If the protocols setting don't contain 'imap' then add it. Also make sure, that +relevant '!include' or '!include_try' configuration lines are not commented. + +If the telnet fails and dovecot emits a log "/auth: Fatal: Support not compiled +in for passdb driver 'pam'/", then rebuild dovecot with package libpam0g-dev +installed. In that case you have to re-run the configure script, possibly +including option *--with-pam* to the configure command line. + +Next check that it also works from remote host: + +---%<------------------------------------------------------------------------- +# telnet imap.example.com 143 +Trying 1.2.3.4... +Connected to imap.example.com. +Escape character is '^]'. +* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS +AUTH=PLAIN] Dovecot ready. +---%<------------------------------------------------------------------------- + +If that didn't work, check all possible firewalls in between, and check that +'listen' setting is '*' in 'dovecot.conf'. + +If you have only imaps enabled, see "remote login" section below for how to +test using openssl s_client. + +Check that it's allowing logins +------------------------------- + +---%<------------------------------------------------------------------------- +# telnet localhost 143 +a login "username" "password" +---%<------------------------------------------------------------------------- + +Replace the username and password with the ones you added to 'passwd.dovecot' +in <BasicConfiguration.txt>. Note that all IMAP commands begin with a tag, +which is basically any string you want, but it must be there. So don't leave +out the "a" in the above example. If the password contains '"' character, +escape it with '\' (e.g.'"foo\"bar"'). + +You should get an "a OK Logged in." reply. If you get "Authentication failed" +error, set 'auth_verbose = yes' and 'auth_debug = yes' in 'dovecot.conf', +restart Dovecot and try again. The log file should now show enough information +to help you fix the problem. + +Check that it's allowing remote logins +-------------------------------------- + +You'll need to try this from another computer, since all local IPs are treated +as secure: + +---%<------------------------------------------------------------------------- +# telnet imap.example.com 143 +a login "username" "password" +---%<------------------------------------------------------------------------- + +If the connection is hanging instead of giving '* Dovecot ready', you have a +firewall that's preventing the connections. + +Otherwise, the only difference here compared to step above is that you might +get: + +---%<------------------------------------------------------------------------- +* BAD [ALERT] Plaintext authentication is disabled, but your client sent +password in plaintext anyway. If anyone was listening, the password was +exposed. +a NO Plaintext authentication disabled. +---%<------------------------------------------------------------------------- + +If this is the case, you didn't set 'disable_plaintext_auth = no'. You could +alternatively use OpenSSL to test that the server works with SSL: + + * Test using imaps port (assuming you haven't disabled imaps port): + ---%<---------------------------------------------------------------------- + # openssl s_client -connect imap.example.com:993 + * OK Dovecot ready. + ---%<---------------------------------------------------------------------- + + * Test using imap port and STARTTLS command (works also with imap port): + ---%<---------------------------------------------------------------------- + # openssl s_client -connect imap.example.com:143 -starttls imap + * OK Dovecot ready. + ---%<---------------------------------------------------------------------- + +Check that it finds INBOX +------------------------- + +After logging in, check that the INBOX is found: + +---%<------------------------------------------------------------------------- +b select inbox +* FLAGS (\Answered \Flagged \Deleted \Seen \Draft) +* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags +permitted. +* 1 EXISTS +* 1 RECENT +* OK [UIDVALIDITY 1106186941] UIDs valid +* OK [UIDNEXT 2] Predicted next UID +b OK [READ-WRITE] Select completed. +---%<------------------------------------------------------------------------- + +It should contain the mail that you sent to yourself in <FindMailLocation.txt> +step. + +If anything goes wrong, set 'mail_debug = yes' and try again. The log file +should now contain debugging information of where Dovecot is trying to find the +mails. Fix 'mail_location' setting and try again. + +Check that it finds other mailboxes +----------------------------------- + +If you already have other mailboxes created, you can check that Dovecot finds +them: + +---%<------------------------------------------------------------------------- +c list "" * +* LIST (\NoInferiors) "/" "test" +* LIST (\NoInferiors) "/" "INBOX" +c OK List completed. +---%<------------------------------------------------------------------------- + +If they weren't found, set 'mail_debug = yes' and look at the debugging +information. Fix 'mail_location' setting and try again. + +Check out some other IMAP commands +---------------------------------- + +If you already have some emails, you can try reading them: + +---%<------------------------------------------------------------------------- +1 SELECT INBOX +2 FETCH 1:* (FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (SUBJECT)]) +3 FETCH 1 BODY[TEXT] +---%<------------------------------------------------------------------------- + +'1:*' means all messages + +You can also try moving a mail to Trash: + +---%<------------------------------------------------------------------------- +4 CREATE Trash +5 COPY 1 Trash +6 STORE 1 +FLAGS \Deleted +7 EXPUNGE +---%<------------------------------------------------------------------------- + +Check that real mail clients work +--------------------------------- + +Since mail clients can be configured in various ways, please check first if the +problem is with Dovecot configuration or with the client's configuration. You +can rule out it being Dovecot's problem with the "telnet" methods described +above. + +If you can't log in, + + * Make sure SSL/TLS settings are correct. + * Make sure the client uses plaintext authentication method, unless you've + specifically configured Dovecot to accept others. + +If you can see only INBOX, + + * Clear out any "IMAP namespace prefix" or similar settings from clients. + * Check if client is configured to show only "subscribed mailboxes". If so, + you'll have to subscribe to the mailboxes you wish to see. You can see a + list of subscribed mailboxes with: + ---%<---------------------------------------------------------------------- + d lsub "" * + * LSUB () "/" "INBOX" + d OK Lsub completed. + ---%<---------------------------------------------------------------------- + +Most IMAP clients have been tested with Dovecot and they work. + +Make a graceful exit +-------------------- + +To close the connection to Dovecot issue a logout: + +---%<------------------------------------------------------------------------- +e logout +* BYE Logging out +e OK Logout completed. +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/TestPop3Installation.txt b/doc/wiki/TestPop3Installation.txt new file mode 100644 index 0000000..dc38b40 --- /dev/null +++ b/doc/wiki/TestPop3Installation.txt @@ -0,0 +1,149 @@ +Contents + + + 1. Check that it's running + + 2. Check that it's listening + + 3. Check that it's allowing logins + + 4. Check that it's allowing remote logins + + 5. Check that it finds the mails + + 6. Check that real mail clients work + + 7. Make a graceful exit + +Check that it's running +----------------------- + +First check with 'ps' that 'dovecot' process is actually running. If it's not, +you had an error in 'dovecot.conf' and the error message was written to log. Go +back to <RunningDovecot.txt> and <Logging.txt> if you can't find it. + +Check that it's listening +------------------------- + +Next check that Dovecot is listening for connections: + +---%<------------------------------------------------------------------------- +# telnet localhost 110 +Trying 127.0.0.1... +Connected to localhost. +Escape character is '^]'. ++OK Dovecot ready. +---%<------------------------------------------------------------------------- + +If you got "connection refused", check that 'pop3' is included in 'protocols' +setting in 'dovecot.conf'. Also check that 'listen' setting is '*'. + +Next check that it also works from remote host: + +---%<------------------------------------------------------------------------- +# telnet imap.example.com 110 +Trying 1.2.3.4... +Connected to imap.example.com. +Escape character is '^]'. ++OK Dovecot ready. +---%<------------------------------------------------------------------------- + +If that didn't work, check all possible firewalls in between, and check that +'listen' setting is '*' in 'dovecot.conf'. + +Check that it's allowing logins +------------------------------- + +---%<------------------------------------------------------------------------- +# telnet localhost 110 +user username +pass password +---%<------------------------------------------------------------------------- + +Replace the username and password with the ones you added to 'passwd.dovecot' +in <BasicConfiguration.txt>. + +You should get an "+OK Logged in." reply. If you get "Authentication failed" +error, set 'auth_verbose = yes' and 'auth_debug = yes' in 'dovecot.conf', +restart Dovecot and try again. The log file should now show enough information +to help you fix the problem. + +Check that it's allowing remote logins +-------------------------------------- + +You'll need to try this from another computer, since all local IPs are treated +as secure: + +---%<------------------------------------------------------------------------- +# telnet imap.example.com 110 +user username +pass password +---%<------------------------------------------------------------------------- + +If the connection is hanging instead of giving '+OK Dovecot ready', you have a +firewall that's preventing the connections. + +Otherwise, the only difference here compared to step above is that you might +get: + +---%<------------------------------------------------------------------------- +-ERR Plaintext authentication disabled. +---%<------------------------------------------------------------------------- + +If this is the case, you didn't set 'disable_plaintext_auth = no'. You could +alternatively use OpenSSL to test that the server works with SSL (assuming +you've already set it up): + +---%<------------------------------------------------------------------------- +# openssl s_client -connect imap.example.com:995 ++OK Dovecot ready. +---%<------------------------------------------------------------------------- + +Check that it finds the mails +----------------------------- + +After logging in, check that it sees mails: + +---%<------------------------------------------------------------------------- +stat ++OK 1 1532 +retr 1 ++OK 1532 octets +<the mail's contents> +. +---%<------------------------------------------------------------------------- + +It should contain the mail that you sent to yourself in <FindMailLocation.txt> +step. + +If anything goes wrong, set 'mail_debug = yes' and try again. The log file +should now contain debugging information of where Dovecot is trying to find the +mails. Fix 'mail_location' setting and try again. + +Check that real mail clients work +--------------------------------- + +Since mail clients can be configured in various ways, please check first if the +problem is with Dovecot configuration or with the client's configuration. You +can rule out it being Dovecot's problem with the "telnet" methods described +above. + +If you can't log in, + + * Make sure SSL/TLS settings are correct. + * Make sure the client uses plaintext authentication method, unless you've + specifically configured Dovecot to accept others. + +Many POP3 clients have been tested with Dovecot and they work. + +Make a graceful exit +-------------------- + +To close the connection to Dovecot issue a logout: + +---%<------------------------------------------------------------------------- +quit ++OK Logging out. +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/TimeMovedBackwards.txt b/doc/wiki/TimeMovedBackwards.txt new file mode 100644 index 0000000..cb64a99 --- /dev/null +++ b/doc/wiki/TimeMovedBackwards.txt @@ -0,0 +1,87 @@ +Time moved backwards error +========================== + +Dovecot isn't very forgiving if your system's time moves backwards. There are +usually two possibilities why it's moving backwards: + + 1. You're running 'ntpdate' periodically. This isn't a good idea. + 2. You're using some kind of a virtual server and you haven't configured it + right (or it's buggy). + +Moving time backwards might cause various problems (see below), so Dovecot +versions older than v2.0 don't even try to handle the situation. + +Time synchronization +-------------------- + +There are two choices for synchronizing your clock: + + 1. Use ntpd [http://www.ntp.org/]. It periodically checks the current time + from NTP server and slows down or speeds up the clock if necessary. Unlike + ntpdate, it doesn't just move the time forwards or backwards (unless the + difference is large). + * If the time difference is too large for ntpd and it "steps", then use + "-x" as a command line option for ntpd or use "tinker step 0" in + '/etc/ntp.conf'. + * This shows up in logs as: 'ntpd[17697]: time reset -2.075483 s' + 2. If ntpd doesn't work well (e.g. a bad network connection), you can use + clockspeed [http://cr.yp.to/clockspeed.html] or chrony + [http://chrony.sunsite.dk/] as well. + +In some systems ntpd/ntpdate is run at boot, but only after Dovecot has +started. That can cause Dovecot to die immediately. If you have this problem, +fix your init scripts to run ntpd/ntpdate first, before starting Dovecot. +Also, seriously consider running ntp-wait before starting Dovecot. + +Bugs/Issues +----------- + + * With Xen you should run ntpd only in dom0. Other domains should synchronize + time automatically (see this Xen FAQ [http://xen.epiuse.com/xen-faq.txt] and + this thread [http://dovecot.org/list/dovecot/2009-October/043301.html]). + * With Vmware you follow the guidlines at the Vmware knowledge base + [http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427]. + * Time moved backwards by 4398 seconds + [http://www.dovecot.org/list/dovecot/2008-June/031548.html]? Buggy + kernel/hardware. + +What about Daylight Saving/Summer time? +--------------------------------------- + +On Unix-like systems, time is stored internally as the number of seconds since +January 1, 1970, 00:00:00 UTC (see Unix_time [WikiPedia:Unix_time] on +Wikipedia); concepts such as time zones and daylight saving time are applied in +user space by the C library, and will normally not have an impact on Dovecot's +behavior. + +Dovecot shouldn't just die! +--------------------------- + +Dovecot v2.0 finally tries to handle this a bit more gracefully. Its behavior +when time moves backwards is: + + * Existing imap and pop3 processes either sleep or die, just like with older + versions + * Master process stops creating new processes until either the original time + is reached, or after a maximum wait of 3 minutes. + * Other processes log a warning, but do nothing else. + * Timeouts are updated so that the timeout is executed approximately at the + original intended time. + +Dovecot v2.0 also notices when time unexpectedly jumps forwards. In that +situation it logs a warning and also updates timeouts. + +The reason why imap/pop3 processes get killed and new ones can't be created for +a while is to avoid problems related to timestamps. Some issues are: + + * Uniqueness of Maildir filenames and dbox global unique identifiers relies on + a growing timestamp + * Dotlock files' staleness is detected by looking at its mtime. + * Timestamps are stored internally all around in memory (as well as in index + files) and compared to current time. Those checks may or may not be buggy if + current time shrinks. + +While killing mail processes doesn't fully solve any of those issues, they're +at least less likely to happen then. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Timeouts.txt b/doc/wiki/Timeouts.txt new file mode 100644 index 0000000..8eaceba --- /dev/null +++ b/doc/wiki/Timeouts.txt @@ -0,0 +1,61 @@ +Timeouts +======== + +Dovecot has a lot of timeouts in various components. Most of them have +hardcoded values, because there's normally no need to change them. Here are +some of them listed: + +IMAP +---- + + * Before login: CLIENT_LOGIN_TIMEOUT_MSECS = MASTER_LOGIN_TIMEOUT_SECS*1000 = + 3 minutes (login-common/client-common.h and lib-master/master-interface.h) + * This may be shorter if all the available connections are in use (service + imap-login { client_limit * process_limit }). In that case the oldest + non-logged in connection is disconnected. + * After login: CLIENT_IDLE_TIMEOUT_MSECS = 30 minutes (minimum required by + IMAP RFC) + * If IDLE command is started, Dovecot never disconnects. Only if the + connection is lost there will be a disconnection. A dead connection is + detected by Dovecot periodically sending "I'm still here" notifications + to client (imap_idle_notify_interval setting - default every 2 minutes). + * IMAP clients are supposed to send something before 30 minutes are up, + but several clients don't do this. Some Outlook versions even stop + receiving new mails entirely until manual intervention if IMAP server + disconnects the client. + * Dovecot also disconnects an IMAP client that sends too many invalid + commands: + * Before login: Disconnect on 3rd invalid command (CLIENT_MAX_BAD_COMMANDS + in imap-login/client.c) + * After login: Disconnect on 20th invalid command (CLIENT_MAX_BAD_COMMANDS + in imap/imap-common.h) + +POP3 +---- + + * Before login: CLIENT_LOGIN_TIMEOUT_MSECS = MASTER_LOGIN_TIMEOUT_SECS*1000 = + 3 minutes (common with IMAP code) + * This may be shorter if all the available connections are in use (service + imap-login { client_limit * process_limit }). In that case the oldest + non-logged in connection is disconnected. + * After login: CLIENT_IDLE_TIMEOUT_MSECS = 10 minutes + * Dovecot also disconnects an POP3 client that sends too many invalid + commands: + * Before login: Disconnect on 3rd invalid command in v2.2.16+, 11th on + earlier versions (CLIENT_MAX_BAD_COMMANDS in pop3-login/client.c) + * After login: Disconnect on 20th invalid command (CLIENT_MAX_BAD_COMMANDS + in pop3/pop3-client.c) + +IMAP & POP3 proxies +------------------- + + * Dovecot proxy handles IMAP and POP3 pre-login timeouts and invalid error + command handling the same as Dovecot backend. After login the proxy will + continue proxying until the client or the backend disconnects. + * Connect timeout to backend is PROXY_DEFAULT_TIMEOUT_MSECS = 30 secs by + default. Can be overridden by proxy_timeout passdb extra field. + * After connection has been established, there's still a login timeout + CLIENT_LOGIN_TIMEOUT_MSECS = MASTER_LOGIN_TIMEOUT_SECS*1000 = 3 minutes + (login-common/client-common.h and lib-master/master-interface.h) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.1.0.txt b/doc/wiki/Upgrading.1.0.txt new file mode 100644 index 0000000..d61e159 --- /dev/null +++ b/doc/wiki/Upgrading.1.0.txt @@ -0,0 +1,137 @@ +Upgrading Dovecot v0.99.x to v1.0 +================================= + +Contents + + + 1. Upgrading Dovecot v0.99.x to v1.0 + + 1. Configuration file + + 2. PAM + + 3. SQL Configuration + + 4. Subscriptions + + 5. Keywords + + 6. POP3 UIDLs + + 7. mbox errors + + 8. Port Changes + + 9. Log Changes for POP before SMTP + +The upgrade from Dovecot v0.99.x to v1.0 isn't just a drop-in upgrade, but also +requires the administrator to adjust the configuration in several places, as +detailed in the chapters below. + +Configuration file +------------------ + +The configuration file has had a lot of changes, so it's better to just modify +the included 'dovecot-example.conf' file manually. The easiest way to remember +what you had changed in the old file is with grep: + +---%<------------------------------------------------------------------------- +egrep -v '^ *(#|$)'|less +---%<------------------------------------------------------------------------- + +PAM +--- + +Dovecot v0.99.x defaulted to using "imap" or "pop3" PAM service identifiers, +ie.'/etc/pam.d/imap' or '/etc/pam.d/pop3' files. With v1.0, the default is to +use "dovecot" service identifier, i.e.'/etc/pam.d/dovecot'. Either change the +passdb pam args to use the "*" parameter or set up the '/etc/pam.d/' files +properly. + +SQL Configuration +----------------- + +The SQL configuration has changed. 'password_query' in the new dovecot-sql.conf +must return a field named 'password'. + +Subscriptions +------------- + +*Maildir only*: The filename that stores user subscriptions has changed from +'.subscriptions' to 'subscriptions'. (It is still called '.subscriptions' for +mbox.). Since v1.0.rc2 the renaming is done automatically. + +Keywords +-------- + +*Maildir-only*: The former '.customflags' file has been renamed to +*dovecot-keywords*, which is incompatible with v0.99.x's format. Since v1.0.rc2 +Dovecot can convert the file automatically. (This conversion does not happen +when going directly from v0.99 to v1.1, though. The files must be renamed +manually.) + +POP3 UIDLs +---------- + +UIDLs are used by POP3 clients to keep track of what messages they've +downloaded, typically only if you've enabled "keep messages in server" option. +If the UIDL changes, the existing messages are re-downloaded as new messages, +which you should try to avoid. + +For compatibility with Dovecot v0.99.x, use: + +---%<------------------------------------------------------------------------- +pop3_uidl_format = %v.%u +---%<------------------------------------------------------------------------- + +However this doesn't work well for Outlook 2003 users (as it didn't with +v0.99.x either), which is the reason this isn't the default anymore. See +<POP3Server#uidl> [POP3Server.txt] for alternative (and better) UIDL formats. + +mbox errors +----------- + +The first time a user opens a mailbox after upgrading it may log some errors +such as: + +---%<------------------------------------------------------------------------- +mbox sync: UID inserted in the middle of mailbox /var/mail/**** (4409 > 4237) +---%<------------------------------------------------------------------------- + +The user may need to log out and log in again to see all of the messages in the +affected mailboxes. Note that this shouldn't happen unless the mbox already had +some errors. The new mbox parsing code just does better error checking. + +Port Changes +------------ + +The port settings are different. The new settings are: + +---%<------------------------------------------------------------------------- +# if you wish to just change IP address, change: +listen = 1.2.3.4 +ssl_listen = 1.2.3.4 + +# if you wish to change ports also, use instead: +protocol imap { + listen = *:143 + ssl_listen = *:993 +} +protocol pop3 { + listen = *:110 + ssl_listen = *:995 +} +---%<------------------------------------------------------------------------- + +Log Changes for POP before SMTP +------------------------------- + +If you used <POP-before-SMTP> [PopBSMTPAndDovecot.txt], the log strings are +different. This should work with new versions: + +---%<------------------------------------------------------------------------- + $s =~ s/^... .. ..:..:.. .* dovecot: (pop3|imap)-login: Login: .+ +rip=(\d+\.\d+\.\d+\.\d+),.*$/$2/i; +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.1.1.txt b/doc/wiki/Upgrading.1.1.txt new file mode 100644 index 0000000..892deb7 --- /dev/null +++ b/doc/wiki/Upgrading.1.1.txt @@ -0,0 +1,90 @@ +Upgrading Dovecot v1.0 to v1.1 +============================== + +You can use your old 'dovecot.conf' from v1.0. It should work without changes, +although a couple of deprecated settings have been removed. + +Upgrading from Dovecot versions older than v1.0.rc1 directly to v1.1 is not +recommended. v1.0 versions contain some backwards compatibility checks and file +format converters that have been removed from v1.1. + +If you want to downgrade back after running v1.1, don't downgrade to older +versions than v1.0.8 or you'll get crashes. Especially if you're using maildir, +the 'dovecot-uidlist' file format has changed and Dovecot versions older than +v1.0.2 can't read it. + +Connections +----------- + + * 'listen = [::]' listens only for IPv6 connections now in most operating + systems. If you want both IPv4 and IPv6 use 'listen = *, [::]' + * By default the number of connections per-user per-IP is limited to 10. You + can change this with the 'mail_max_userip_connections' setting. + +Authentication +-------------- + + * <passwd-file> [AuthDatabase.PasswdFile.txt]: If you use %d in args, it no + longer means that domain isn't looked up from the passwd-file. You'll need + to add 'username_format=%n' prefix to args (e.g. 'args = username_format=%n + /etc/virtual.%d'). + * Empty or NULL password no longer means "any password is valid". You'll also + have to return "nopassword" field. + * <PAM> [PasswordDatabase.PAM.txt]: There's no more 'blocking=yes' setting, + it's now always enabled. If you want to limit the number of lookups done by + a dovecot-auth worker, change 'auth_worker_max_request_count' setting. + Setting it to 1 makes it work basically the same as the old 'blocking=no'. + * <passwd> [AuthDatabase.Passwd.txt]: The problem with passwd lookups is that + temporary errors (e.g. LDAP server down) are returned as "user doesn't + exist" errors. You may want to try the new <NSS> [UserDatabase.NSS.txt] + userdb. + * <SQL> [AuthDatabase.SQL.txt] and <LDAP> [AuthDatabase.LDAP.txt]: + 'user_global_uid' and 'user_global_gid' fields have been removed from their + config files. Instead you can now use 'mail_uid' and 'mail_gid' settings in + 'dovecot.conf'. This also means that it's no longer a requirement to specify + a userdb at all (a dummy <static userdb> [UserDatabase.Static.txt] is used + internally). + +Mail handling +------------- + + * In v1.0 'mmap_disable=yes' might have worked faster. If you had changed this + only because of that, it's time to set it back to "no". + * <NFS.txt> users should now set 'mail_nfs_storage=yes' and + 'mail_nfs_index=yes'. Dovecot no longer requires attribute cache to be + disabled. + * <Quota.txt> plugin has completely new configuration. See <Quota.1.1.txt>. + * <Maildir> [MailboxFormat.Maildir.txt]: 'dovecot-uidlist' file is in a new + format. The old format is automatically converted to new one, but if you + plan to move back to v1.0 be sure to use at least v1.0.2 which will also + understand this new format. + * Index files have slightly changed as well. Upgrading to v1.1 should go + transparently, but moving back to v1.0 might again cause some errors. v1.0.8 + fixes some assert-crashes that were caused by reading v1.1-generated index + files. + * 'dotlock_use_excl=yes' is default nowadays. If you're still using an ancient + NFSv2 setup, you'll need to set this to "no". + * mbox: Delete existing dovecot.index.cache files from all mailboxes. + Otherwise you may see some errors in logs. + * 'default_mail_env' has been renamed to 'mail_location' (since v1.0.rc11 + already). + * Namespaces: + * deliver now supports namespaces. If you use namespace prefixes or a + non-default separator and you deliver to non-INBOXes, deliver will now + have to use the configured prefix and separators. + * This is especially important for Sieve scripts. For example if you + only have "INBOX." namespace prefix and you used to use 'fileinto + "box"', it now has to be instead: 'fileinto "INBOX.box"' + * 'hidden=yes' now hides the namespace only from IMAP NAMESPACE reply. + You'll also need to set 'list=no' to truly hide them from clients' + mailbox list. + +Removed settings +---------------- + + * mail_read_mmaped: Mails are never read mmaped anymore. There wasn't much + point. + * mmap_no_write: OpenBSD users will have to settle for mmap_disable=yes for + now. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.1.2.txt b/doc/wiki/Upgrading.1.2.txt new file mode 100644 index 0000000..f381e83 --- /dev/null +++ b/doc/wiki/Upgrading.1.2.txt @@ -0,0 +1,57 @@ +Upgrading Dovecot v1.1 to v1.2 +============================== + + * Relative home directory paths are giving errors now. They were never + supported, but earlier they just didn't usually cause problems. + * If you were using e.g. 'mail_location = maildir:/var/mail/%h', just + change it to 'mail_location = maildir:%h' and add '/var/mail/' prefix to + home dirs. + * To get absolute home dir from relative path in LDAP, use something like: + 'user_attrs = .., homeDirectory=home=/var/mail/%$' + * SQL dictionary (quota, expire plugin) configuration file is different than + in v1.1. See 'doc/dovecot-dict-sql-example.conf' or <Quota.Dict.txt> for the + new format. + * When creating files or directories to mailboxes, Dovecot now uses the + mailbox directory's permissions and GID for them. Previous versions simply + used 0600 mode always, so you should check the directories' permissions to + make sure they're strict enough. For backwards compatibility + 'dovecot-shared' file's permissions still override these with Maildir. + +Authentication: + + * system_user <userdb extra field> [UserDatabase.ExtraFields.txt] was renamed + to system_groups_user to better describe its functionality. + +Settings: + + * Renamed 'ssl_disable=yes' to 'ssl=no'. + * Renamed 'auth_ntlm_use_winbind' to 'auth_use_winbind', which also determines + if GSS-SPNEGO is handled by GSSAPI or winbind. + * Removed 'login_greeting_capability'. The capabilities are now always sent + (LEMONADE [http://www.lemonadeformobiles.com/] requires this and it's not + that much extra traffic). + * Removed 'auth_worker_max_request_count'. It was useful only with PAM, so it + can now be specified in 'passdb pam { args = max_requests=n } '. The default + is 100. + * Removed 'umask'. It wasn't really used anywhere anymore. + +ACL: + + * The global ACL file overrides per-mailbox ACL file. + +Sieve: + + * You should consider <migrating from CMU Sieve to Pigeonhole> + [Pigeonhole.Sieve.Configuration.txt] (see the link for instructions). + +<ManageSieve> [Pigeonhole.ManageSieve.txt]: + + * The 'sieve=' and 'sieve_storage=' settings need to be placed in the ' plugin + {}' section now and 'sieve_storage=' needs to be renamed to 'sieve_dir='. + This removes the duplication of these values with respect to the <Sieve + Plugin> [Pigeonhole.Sieve.txt] for <Deliver> [LDA.txt]. So, since you are + using the Sieve plugin, these settings should already be there and all that + needs to be done is remove the 'sieve=' and 'sieve_storage=' settings from + the ' protocol managesieve {} ' section. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.2.0.txt b/doc/wiki/Upgrading.2.0.txt new file mode 100644 index 0000000..1114ea1 --- /dev/null +++ b/doc/wiki/Upgrading.2.0.txt @@ -0,0 +1,92 @@ +Upgrading Dovecot v1.2 to v2.0 +============================== + +A lot of settings have changed. Dovecot v2.0 can still use most of the v1.x +configuration files, but it logs a lot of warnings at startup. A quick and easy +way to convert your old config file to v2.0 format is: + +---%<------------------------------------------------------------------------- +# convert old config to new temp config file +doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf + +# replace the old config file with the new generated file +mv dovecot-2.conf /etc/dovecot/dovecot.conf +---%<------------------------------------------------------------------------- + +This command logs a warning about each obsolete setting it converts to the new +format.*You can simply ignore all the warnings* in most cases. If you really +want to, you can modify your old config file using the instructions from the +warnings, but even that can be done more easily by looking at the generated +config file. Some of the warning messages aren't obvious. + +Once running v2.0, it's safe to downgrade to v1.2.5 or newer. Older versions +don't understand some of the changes to index files and will log errors. + +Permission related changes +-------------------------- + + * Dovecot uses two system users for internal purposes now by default: + "dovenull" and "dovecot". You need to create the "dovenull" user or change + 'default_login_user' setting. "dovenull" user is used by completely + untrustworthy processes, while "dovecot" user is used for slightly more + trusted processes. + * If you want to be using something else than "dovecot" as the other user, + you need to change 'default_internal_user' setting. + * Just like with "dovecot" user, "dovenull" doesn't need a password, home + directory or anything else (but it's good to give it its own private + "dovenull" group). + * "auth-master" socket related configuration should be replaced with + "auth-userdb" socket everywhere (auth-master should still work, but it gives + more permissions than necessary) + * If you get any kind of "permission denied" errors related to UNIX sockets, + you can change their permissions from 'service { unix_listener { ... } } ' + blocks. See 'example-config/conf.d/10-master.conf' for examples or 'doveconf + -a' output for their current values. + +Other major changes +------------------- + + * No more convert plugin, use <dsync> [Tools.Dsync.txt] instead + * No more expire-tool, use <doveadm expunge> [Plugins.Expire.txt] instead. + Also expire configuration is different. + * <Post-login scripts are configured differently> [PostLoginScripting.txt] and + need to be modified + * <Quota warnings are configured differently> [Quota.Configuration.txt] and + the script may need to be modified (most environment settings like $USER are + gone) + * Global ACL filenames now require namespace prefix (e.g. if you use "INBOX." + prefix,'/etc/acls/foo' needs to be renamed to '/etc/acls/INBOX.foo' + * Maildir: Permissions for newly created mail files are no longer copied from + dovecot-shared file, but instead from the mail directory (e.g. for "foo" + mailbox, they're taken from '~/Maildir/.foo' directory) + * dbox: v2.0 format is slightly different, but backwards compatible. The main + problem is that v2.0 no longer supports maildir-dbox hybrid resulting from + "fast Maildir migration". If you have any Maildir files in your dbox, you + need to convert them somehow (some examples + [http://dovecot.org/list/dovecot/2010-September/053012.html]). You might + also consider using <dsync> [Tools.Dsync.txt] to get rid of the old unused + metadata in your dbox files. + * Pre-login and post-login CAPABILITY reply is now different. Dovecot expects + clients to recognize new automatically sent capabilities. + [http://dovecot.org/list/dovecot/2010-April/048147.html] This should work + with all commonly used clients, but some rarely used clients might have + problems. Either get the client fixed, or set 'imap_capability' manually. + * ManageSieve protocol [http://tools.ietf.org/html/rfc5804] was assigned an + official port by IANA: 4190. This is used by <Pigeonhole.txt> by default + now. If you want to listen also on the old 2000 port, see the + <Pigeonhole.ManageSieve.Configuration.txt> example. + * 'dovecot --exec-mail imap' has been replaced by simply running "imap" + binary. You can also use "imap -u<username>" to access other users' mails + more easily. + +LDA +--- + + * deliver binary was renamed to dovecot-lda (but a symlink still exists for + now) + * -n parameter was replaced by lda_mailbox_autocreate setting. The default + also changed to "no". + * -s parameter was replaced by lda_mailbox_autosubscribe setting. The default + is "no", as before. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.2.1.txt b/doc/wiki/Upgrading.2.1.txt new file mode 100644 index 0000000..0dec9ff --- /dev/null +++ b/doc/wiki/Upgrading.2.1.txt @@ -0,0 +1,35 @@ +Upgrading Dovecot v2.0 to v2.1 +============================== + +v2.1 is mostly compatible with v2.0 configuration, except: + + * 15-mailboxes.conf included in the default configuration now specifies a few + default SPECIAL-USE [http://tools.ietf.org/html/rfc6154] mailboxes. This + file assumes that you already have 'namespace inbox { .. } ' specified (in + 10-mail.conf). If you don't, you'll get errors about namespaces. Note that + the namespace's name must be "inbox" (as well as usually include inbox=yes + setting). The solution is to either make sure that you have such a namespace + defined, or you can simply delete the 15-mailboxes.conf if you don't care + about SPECIAL-USE. + * <Plugins.txt> now use UTF-8 mailbox names rather than mUTF-7: acl, + autocreate, expire, trash, virtual + * Usernames in authentication are now lowercased by default. + * Non-lowercase usernames in password/user database result in "unknown + user" errors + * To allow mixed case usernames again, set 'auth_username_format=' (i.e. to + empty) + * <Solr> [Plugins.FTS.Solr.txt] full text search backend changed to use + mailbox GUIDs instead of mailbox names, requiring reindexing everything. + solr_old backend can be used with old indexes to avoid reindexing, but it + doesn't support some newer features. + * <Expire plugin> [Plugins.Expire.txt]: Only go through users listed by userdb + iteration. Delete dict rows for nonexistent users, unless + expire_keep_nonexistent_users=yes. + * <dsync> [Tools.Dsync.txt] was merged into doveadm. There is still "dsync" + symlink pointing to "doveadm", which you can use the old way for now. The + preferred ways to run dsync are "doveadm sync" (for old "dsync mirror") and + "doveadm backup". + * dsync protocol isn't compatible with v2.0's dsync, so you can't dsync + between v2.0 and v2.1 servers. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.2.2.txt b/doc/wiki/Upgrading.2.2.txt new file mode 100644 index 0000000..b8d6865 --- /dev/null +++ b/doc/wiki/Upgrading.2.2.txt @@ -0,0 +1,45 @@ +Upgrading Dovecot v2.1 to v2.2 +============================== + +v2.2 has a couple of changes to settings since v2.1: + + * doveadm_proxy_port setting renamed to doveadm_port (but the old exists still + as an alias) + * imapc_ssl_ca_dir and pop3c_ssl_ca_dir settings replaced by a common + ssl_client_ca_dir + +There are also some changes you should be aware of: + + * fts-solr no longer does "hard commits" to the Solr index for performance + reasons. <You must do this manually once in a while> [Plugins.FTS.Solr.txt]. + + * When creating home directories, the permissions are copied from the parent + directory if it has setgid-bit set. For full details, see + <SharedMailboxes.Permissions.txt>. + * "doveadm auth" command was renamed to "doveadm auth test" + * IMAP: ID command now advertises server name as Dovecot by default. It was + already trivial to guess this from command replies. + * LDA/LMTP: If saving a mail brings user from under quota to over quota, allow + it based on quota_grace setting (default: 10% above quota limit). + * pop3_lock_session=yes now uses a POP3-only dovecot-pop3-session.lock file + instead of actually locking the mailbox (and causing IMAP/LDA/LMTP to wait + for the POP3 session to close). + * mail_shared_explicit_inbox setting's default switched to "no". + * dsync isn't compatible with v2.1 protocol. (The new protocol will be + compatible with future Dovecot versions.) + * autocreate plugin is being deprecated and it will log warnings. Convert the + configuration to <MailboxSettings.txt> instead. + +Downgrading can be done fully safely to v2.1.16. + + * v2.1.16 adds support for "attribute changes", which are used by URLAUTH + command and dsync with ACLs and/or Sieve scripts. If none of these features + are used, you can downgrade safely to v2.1.11. + * The error message for these attribute changes is: 'Log synchronization + error at seq=..,offset=.. for .../dbox-Mails/dovecot.index: Unknown + transaction record type 0x0' + * v2.1.11 adds support for cache file changes. Older versions may think that + the 'dovecot.index.cache' files are corrupted and complain about "Invalid + magic in hole header". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.2.3.txt b/doc/wiki/Upgrading.2.3.txt new file mode 100644 index 0000000..8cda65f --- /dev/null +++ b/doc/wiki/Upgrading.2.3.txt @@ -0,0 +1,212 @@ +Upgrading Dovecot v2.2 to v2.3 +============================== + +Downgrading is possible to v2.2.27 and later. (v2.2.27 accidentally broke +dovecot.index* backwards compatibility a bit.) + +Settings Changes +---------------- + + * 'director_consistent_hashing' setting removed. It's always assumed to be + "yes" now. + * *WARNING*: You can't run a director ring with mixed + 'director_consistent_hashing' settings. If you already didn't have it set + to "yes", upgrading to v2.3 will require you to shutdown the entire + director ring. It may be safer to first do this setting change in v2.2 + before the upgrade. + * If you really don't wish to shutdown the ring, an alternative would be + to set up a whole new director ring. Then start moving users to the + new ring in the Dovecot proxy. To avoid the same user having + connections to both rings at the same time (-> two backends at the + same time), this would need to be done so that passdb moves the user + to the new ring and old connections are kicked. See + <PasswordDatabase/ExtraFields/Proxy#moving> + [PasswordDatabase.ExtraFields.Proxy.txt] + * 'director_doveadm_port' setting removed. Name the 'inet_listener doveadm { + .. }' instead. + * 'mdbox_purge_preserve_alt' setting removed. It's always assumed to be "yes" + now. + * 'recipient_delimiter' setting used to be treated as a separator string. Now + it's instead treated as a list of alternative delimiter characters. + * Time interval based settings no longer default to "seconds". All numbers + must explicitly be followed by the time unit (except 0). This is important, + because some settings now support milliseconds as well. + * fs-posix: prefix=path parameter no longer automatically appends '/' to the + path if it's not there. This allows using it properly as a prefix, instead + of only a directory prefix. Make sure you have the '/' appended to the + prefix, or the "dir/filename" will be accessed just as "dirnamename". + * 'ssl_protocols' setting was replaced by 'ssl_min_protocol'. Now you only + specify the minimum ssl protocol version Dovecot accepts, defaulting to + TLSv1. + * 'ssl_parameters' was replaced with 'ssl_dh'. See <Diffie-Hellman Parameters + for SSL> [Upgrading.2.3.txt]. + * 'SSLv2' is no longer supported in 'ssl_protocols'. + +Statistics Redesign +------------------- + +The statistics code was redesigned. + + * Statistics is no longer optional - it is always there. + * The old "stats" plugin was renamed to "old_stats". + * The "doveadm stats" command was renamed to "doveadm oldstats". + * There's a new "doveadm stats" command that isn't compatible with the old + one. + * The new stats code doesn't require a plugin, so make sure you remove 'stats' + from 'mail_plugin' setting. For more details see <Statistics.txt>. + +Config changes required to 2.2.x config to keep using the "old" stats: + + * 'mail_plugins = stats' -> 'mail_plugins = old_stats' + * 'mail_plugins = imap_stats' -> 'mail_plugins = imap_old_stats' + * 'service stats' -> 'service old-stats' + * 'executable = stats' -> 'executable = old-stats' + * 'fifo_listener stats-mail' -> 'fifo_listener old-stats-mail' + * 'fifo_listener stats-user' -> 'fifo_listener old-stats-user' + * 'unix_listener stats' -> 'unix_listener old-stats' + * 'plugin { stats_refresh }' -> 'plugin { old_stats_refresh }' + * 'plugin { stats_notify_path }' -> 'plugin { old_stats_notify_path }' + * 'plugin { stats_track_cmds }' -> 'plugin { old_stats_track_cmds }' + * 'auth_stats' -> keep as 'auth_stats' + * 'stats_*' settings -> 'old_stats_*' + +Submission Service (new) +------------------------ + +Dovecot can now act as a submission service. See <Submission.txt> for more +information. + +Localhost Auth Penalty +---------------------- + +Dovecot no longer disables auth penalty waits for clients connecting from +localhost (or 'login_trusted_networks' in general). The previous idea was that +it would likely be a webmail that would have its own delays, but there are no +guarantees about this. + +If the old behavior is still wanted, it's possible to do nowadays even more +generically with e.g. setting following as the first passdb: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = username_format=%{rip} /etc/dovecot/passdb + default_fields = noauthenticate=y +} +---%<------------------------------------------------------------------------- + +/etc/dovecot/passdb: + +---%<------------------------------------------------------------------------- +127.0.0.1:::::::nodelay=yes +192.168.10.124:::::::nodelay=yes +---%<------------------------------------------------------------------------- + +Changed Setting Defaults +------------------------ + ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| *Setting* | *Old Default Value* | *New Default Value* | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| 'mdbox_rotate_size' | 2M | 10M | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| 'mailbox_list_index' | no | yes | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| 'imap_logout_format' | n=%i out=%o | in=%i out=%o deleted=%{deleted} expunged=%{expunged} trashed=%{trashed} | +| | | hdr_count=%{fetch_hdr_count} hdr_bytes=%{fetch_hdr_bytes} body_count=%{fetch_body_count} | +| | | body_bytes=%{fetch_body_bytes} | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| 'ssl_cipher_list' | ALL:!LOW:!SSLv2:!EXP:!aNULL | ALL:!kRSA:!SRP:!kDHd:!DSS:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW@STRENGTH | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| 'mail_log_prefix' | "%s(%u): " | "%s(%u)<%{pid}><%{session}>: " | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| mysql: | no | yes | +| 'ssl_verify_server_cert' | | | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ +| 'ssl_options' | | no_compression is now the default, and a new compression option is introduced for enabling | +| | | compression | ++--------------------------+-----------------------------+----------------------------------------------------------------------------------------------+ + +Diffie-Hellman Parameters for SSL +--------------------------------- + + * ssl-parameters.dat file is now obsolete. You should use ssl_dh setting + instead:'ssl_dh=</etc/dovecot/dh.pem' + * You can convert an existing ssl-parameters.dat to dh.pem: + + ---%<------------------------------------------------------------------- + dd if=/var/lib/dovecot/ssl-parameters.dat bs=1 skip=88 | openssl dhparam + -inform der > /etc/dovecot/dh.pem + ---%<------------------------------------------------------------------- + + * ssl-params process has also been removed, as it is no longer used to + generate these parameters. + * You are encouraged to create at least 2048 bit parameters. 4096 is industry + recommendation. + * Note that it will take LONG TIME to generate the parameters, and it should + be done with a machine that has GOOD SOURCE OF ENTROPY. Running it on a + virtual machine is not recommended, unless there is some entropy + helper/driver installed. Running this on your production proxy can starve + connections due to lack of entropy. + * Since v2.3.3+ DH parameter usage is *optional* and can be omitted. You are + invited to amend ciphers to disallow non-ECC based DH algorithms, but if you + don't and someone does try to use them, error will be emitted. + * Example: + 'ssl_cipher_list=ALL:!kRSA:!SRP:!kDHd:!DSS:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW:!DH@STRENGTH' + +Other Changes +------------- + + * Invalid 'postmaster_address' now causes a failure early on with + sieve/imap_sieve plugin enabled. It still defaults to 'postmaster@%d', which + expands to invalid 'postmaster@' address if your usernames do not contain a + domain, or are converted into domainless usernames by passdb/userdb. See + <DomainLost.txt>. + * Linux: Dovecot no longer enables core dumping for "setuid processes", which + most of them are. + * To enable them with Linux kernel v3.6+: Make sure core dumps get written + to a globally shared directory and enable them with:'sysctl -w + fs.suid_dumpable=2' + * With older Linux kernel versions you can set it to 1, but that's not + good for security of your system. + * You can also revert to old behavior with: 'import_environment = + $import_environment PR_SET_DUMPABLE' + * However, this also may have some security implications depending on + the setup. Mainly if you have system users and you've enabled + chrooting or mail_access_groups, this could allow the system users to + gain unintentional access. + * userdb nss was removed. Use userdb passwd instead. + * doveadm: table formatter prints the header now to stdout, not stderr + * doveadm: Removed mount commands + * OpenSSL version is required to be at least 1.0.1 for Dovecot to build + * subscriptions file is written in a new version 2 format. Dovecot v2.2.17 and + newer can read this file. + * mail_log plugin: Headers are logged as UTF-8 (instead of MIME-encoded) + * auth: When iterating users in userdb passwd, skip users that aren't in the + first/last_valid_gid range + * auth protocol has changed some error fields: + * temp -> code=temp_fail + * authz -> code=authz_fail + * user_disabled -> code=user_disabled + * pass_expired -> code=pass_expired + * auth now supports bcrypt algorithm by default. + * Some API changes have been made, if you have your own plugins please be + aware that they might require change(s) to be compatible again. + * Due to the new stats environment, for now some environments may get harmless + errors about not being able to connect to stats-writer socket. To avoid + these errors, give enough permissions for the processes to connect to the + stats-writer, for example: + +---%<------------------------------------------------------------------------- +service stats { + client_limit = 10000 # make this large enough so all Dovecot processes +(especially imap, pop3, lmtp) can connect to it + unix_listener stats-writer { + user = vmail + #mode = 0666 # Use only if nothing else works. It's a bit insecure, since +it allows any user in the system to mess up with the statistics. + } +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Upgrading.txt b/doc/wiki/Upgrading.txt new file mode 100644 index 0000000..ae95ccb --- /dev/null +++ b/doc/wiki/Upgrading.txt @@ -0,0 +1,26 @@ +Upgrading between micro versions +================================ + +(For example v1.0.x -> v1.0.y or v1.1.x -> v1.1.y) + +The NEWS file [http://dovecot.org/doc/NEWS] contains all the important changes +marked with "*" character. Read them to see if there is anything that concerns +you. In general all the changes try to preserve backwards compatibility, but +some changes which are meant to improve the stability and correctness of the +configuration could mean breaking some existing installations. And that may be +a good thing, since it can expose problems which could otherwise show up as +random errors. + +Upgrading between other versions +================================ + + * <v0.99.x to v1.0> [Upgrading.1.0.txt] + * <v1.0 to v1.1> [Upgrading.1.1.txt] + * <v1.1 to v1.2> [Upgrading.1.2.txt] + * <v1.2 to v2.0> [Upgrading.2.0.txt] + * <v2.0 to v2.1> [Upgrading.2.1.txt] + * <v2.1 to v2.2> [Upgrading.2.2.txt] + * <v2.2 to v2.3> [Upgrading.2.3.txt] + * <v2.3 to v2.4> [Upgrading.2.4.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/UserDatabase.ExtraFields.txt b/doc/wiki/UserDatabase.ExtraFields.txt new file mode 100644 index 0000000..edf5c3f --- /dev/null +++ b/doc/wiki/UserDatabase.ExtraFields.txt @@ -0,0 +1,115 @@ +User database extra fields +========================== + +A user database lookup typically returns <uid, gid and home> [UserDatabase.txt] +fields, as per traditional /etc/passwd lookups. Other fields may also be stored +in the userdb, and these are called 'extra fields'. Possibilities are: + + * *mail*: <Mail location> [MailLocation.txt], overrides the global + 'mail_location' setting. + * *nice*: Set the mail process's priority to be the given value. + * *chroot*: Chroot to given directory. Overrides 'mail_chroot' setting in + 'dovecot.conf'. + * *system_groups_user*: Specifies the username whose groups are read from + '/etc/group' (or wherever NSS is configured to taken them from). The logged + in user has access to those groups. This may be useful for shared mailboxes. + * *userdb_import*: This allows returning multiple extra fields in one + TAB-separated field. It's useful for userdbs which are a bit less flexible + for returning a variable number of fields (e.g. SQL). + * *uidgid_file*: Get uid and gid for user based on the given filename. + * It's possible to override settings from 'dovecot.conf' (most commonly + quota_rule to set per-user quota limits or also plugin-settings). + * *user*: User can be overriden (normally set in <passdb> + [PasswordDatabase.txt]). + * *noreplicate*: User will not be replicated using replicator (see + <Replication.txt>) + * The extra fields are also passed to <post-login scripts> + [PostLoginScripting.txt]. + +The following suffixes added to a field name are handled specially: + + * *:protected*: Set this field only if it hasn't been set before. + * *:remove*: Remove this field entirely. + +These fields can be returned the exact same way as uid, gid, and home fields. +Below are examples for some user databases. + +Overriding settings +------------------- + +Most commonly settings are overridden from plugin section. For example if your +plugin section has 'quota_rule=*:storage=100M' value and the userdb lookup +returns 'quota_rule=*:storage=200M', the original quota setting gets +overridden. In fact if the lookup always returns a 'quota_rule' field, there's +no point in having the 'quota_rule' setting in the plugin section at all, +because it always gets overridden anyway. + +To understand how imap and pop3 processes see their settings, it may be helpful +to know how Dovecot internally passes them: + + 1. First all actual settings are first read into memory. + 2. Next all the extra fields returned by userdb lookup are used to override + the settings. Any unknown setting is placed into the plugin {} section + (e.g.'foo=bar' will be parsed as if it were 'plugin { foo=bar }'). + 3. Last, if <post-login scripting is used> [PostLoginScripting.txt], it may + modify the settings if wanted. + +If you want to override settings inside sections, you can separate the section +name and key with '/'. For example: + +---%<------------------------------------------------------------------------- +namespace default { + inbox = yes + separator = . + location = maildir:~/Maildir +} +---%<------------------------------------------------------------------------- + +The separator setting can be overridden by returning +'namespace/default/separator=.' extra field. + +Examples +-------- + +SQL +--- + +dovecot-sql.conf: + +---%<------------------------------------------------------------------------- +user_query = SELECT home, uid, gid, \ + CONCAT('*:bytes=', quota_bytes) AS quota_rule, \ + separator AS "namespace/default/separator" \ + FROM users WHERE username = '%n' and domain = '%d' +---%<------------------------------------------------------------------------- + +LDAP +---- + +dovecot-ldap.conf: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =home=%{ldap:homeDirectory}, \ + =uid=%{ldap:uidNumber}, + =gid=%{ldap:gidNumber}, + =quota_rule=*:bytes=%{ldap:quotaBytes}, + =namespace/default/separator=%{ldap:mailSeparator} +---%<------------------------------------------------------------------------- + +passwd-file +----------- + +Below are examples that show how to give two userdb extra fields ("mail" and +"quota"). Note that all userdb extra fields must be prefixed with "userdb_", +otherwise they're treated as <passdb extra fields> +[PasswordDatabase.ExtraFields.txt]. + +---%<------------------------------------------------------------------------- +user:{plain}pass:1000:1000::/home/user::userdb_mail=mbox:~/mail +userdb_quota_rule=*:storage=100M userdb_namespace/default/separator=/ +user2:{plain}pass2:1001:1001::/home/user2::userdb_mail=maildir:~/Maildir +userdb_quota_rule=*:storage=200M +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/UserDatabase.NSS.txt b/doc/wiki/UserDatabase.NSS.txt new file mode 100644 index 0000000..fac2a04 --- /dev/null +++ b/doc/wiki/UserDatabase.NSS.txt @@ -0,0 +1,34 @@ +NSS +=== + +NOTE: This userdb is probably useless with Dovecot v2.0.12+, since it uses +getpwnam_r(), which supports error reporting. + +Usually NSS [http://en.wikipedia.org/wiki/Name_Service_Switch] is used with +<passwd> [AuthDatabase.Passwd.txt] userdb, but it has one problem: It can't +distinguish between temporary and permanent errors. So if you're using e.g. +nss_ldap and your LDAP database is down, all userdb lookups may return "user +doesn't exist" errors. This is especially bad if you're using <LDA.txt>, which +causes the mails to be bounced back to sender. + +The NSS userdb works around this problem by loading the NSS modules and calling +them itself. This is a bit kludgy, and it probably works only with Linux. + +This userdb has two parameters: + + * *service=<name>*: This parameter is required. The name specifies what NSS + module to use, for example "ldap". + * *blocking=no* causes the lookups to be done in auth master processes instead + of in worker processes. + +Example +------- + +---%<------------------------------------------------------------------------- +userdb { + driver = nss + args = service=ldap +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/UserDatabase.Prefetch.txt b/doc/wiki/UserDatabase.Prefetch.txt new file mode 100644 index 0000000..715b8bc --- /dev/null +++ b/doc/wiki/UserDatabase.Prefetch.txt @@ -0,0 +1,87 @@ +Prefetch User Database +====================== + +Prefetch userdb can be used to combine passdb and userdb lookups into a single +lookup. It's usually used with <SQL> [AuthDatabase.SQL.txt], <LDAP> +[AuthDatabase.LDAP.txt] and <checkpassword> +[PasswordDatabase.CheckPassword.txt] passdbs. + +Prefetch basically works by requiring that the passdb returns the userdb +information in <extra fields> [PasswordDatabase.ExtraFields.txt] with 'userdb_' +prefixes. For example if a userdb typically returns 'uid', 'gid' and 'home' +fields, the passdb would have to return 'userdb_uid', 'userdb_gid' and +'userdb_home' fields. + +If you're using <LDA.txt> or <LMTP.txt>, you still need a valid userdb which +can be used to locate the users. You can do this by adding a normal SQL/LDAP +userdb *after the userdb prefetch*. The order of definitions is significant. +See below for examples. + +LDAP: 'auth_bind=yes' with 'auth_bind_userdn'-template is incompatible with +prefetch, because no passdb lookup is done then. If you want zero LDAP lookups, +you might want to use <static userdb> [UserDatabase.Static.txt] instead of +prefetch. + +SQL example +----------- + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +userdb { + driver = prefetch +} +# The userdb below is used only by lda. +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +'dovecot-sql.conf.ext': + +---%<------------------------------------------------------------------------- +password_query = SELECT userid AS user, password, \ + home AS userdb_home, uid AS userdb_uid, gid AS userdb_gid \ + FROM users \ + WHERE userid = '%u' + +# For LDA: +user_query = SELECT home, uid, gid FROM users WHERE userid = '%u' +---%<------------------------------------------------------------------------- + +LDAP example +------------ + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = ldap + args = /etc/dovecot/dovecot-ldap.conf.ext +} +userdb { + driver = prefetch +} +# The userdb below is used only by LDA. +userdb { + driver = ldap + args = /etc/dovecot/dovecot-ldap.conf.ext +} +---%<------------------------------------------------------------------------- + +'dovecot-ldap.conf.ext': + +---%<------------------------------------------------------------------------- +pass_attrs = uid=user, userPassword=password, \ + homeDirectory=userdb_home, uidNumber=userdb_uid, gidNumber=userdb_gid + +# For LDA: +user_attrs = homeDirectory=home, uidNumber=uid, gidNumber=gid +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/UserDatabase.Static.txt b/doc/wiki/UserDatabase.Static.txt new file mode 100644 index 0000000..53d9e30 --- /dev/null +++ b/doc/wiki/UserDatabase.Static.txt @@ -0,0 +1,41 @@ +Static User Database +==================== + +Static user database can be used when you want to use only single UID and GID +values for all users, and their home directories can be specified with a simple +template. The syntax is: + +---%<------------------------------------------------------------------------- +userdb { + driver = static + args = uid=<uid> gid=<gid> home=<dir template> +} +---%<------------------------------------------------------------------------- + +The home is optional. You can also return other <extra fields> +[UserDatabase.ExtraFields.txt]. You can use the standard <variables> +[Variables.txt] everywhere. + +LDA and passdb lookup for user verification +------------------------------------------- + +Unless your MTA already verifies that the user exists before calling +dovecot-lda, you'll most likely want dovecot-lda itself to verify the user's +existence. Since dovecot-lda looks up the user only from the userdb, it of +course doesn't work with static userdb because there is no list of users. +Normally static userdb handles this by doing a passdb lookup instead. This +works with most passdbs, with <PAM> [PasswordDatabase.PAM.txt] being the most +notable exception. If you want to avoid this user verification, you can add +'allow_all_users=yes' to the args in which case the passdb lookup is skipped. + +Example +------- + +---%<------------------------------------------------------------------------- +userdb { + driver = static + args = uid=500 gid=500 home=/home/%u +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/UserDatabase.txt b/doc/wiki/UserDatabase.txt new file mode 100644 index 0000000..31a745c --- /dev/null +++ b/doc/wiki/UserDatabase.txt @@ -0,0 +1,126 @@ +User Databases +============== + +After a user has been successfully authenticated, Dovecot looks up the user's +userdb information. The userdb lookup is also done by <LDA.txt> to find out how +to deliver mails for the user. + +The user database lookup can return these fields: + + * *uid*: User's <UID> [UserIds.txt] (UNIX user ID), overrides the global + 'mail_uid' setting. + * *gid*: User's <GID> [UserIds.txt] (UNIX group ID), overrides the global + 'mail_gid' setting. + * *home*: User's <home directory> [VirtualUsers.Home.txt], overrides the + global 'mail_home' setting. Although not required, it's <highly recommended + even for virtual users> [VirtualUsers.Home.txt]. + * Optional <extra fields> [UserDatabase.ExtraFields.txt] + * *user*: Changes the username (can also be done by the passdb lookup) + * Overwriting all mail-related settings, for example: + * *mail*: <Mail location> [MailLocation.txt], overrides the global + 'mail_location' setting. + * *quota_rule* to specify per-user quota limit + * The extra fields are also passed to <post-login scripts> + [PostLoginScripting.txt] + +The user and <password databases> [PasswordDatabase.txt] may be the same or +they may be different depending on your needs. You can also have <multiple +databases> [Authentication.MultipleDatabases.txt]. + +Currently supported user databases are: + + * <Passwd> [AuthDatabase.Passwd.txt]: System users (NSS, '/etc/passwd', or + similiar) + * <Passwd-file> [AuthDatabase.PasswdFile.txt]: '/etc/passwd'-like file in + specified location + * <NSS> [UserDatabase.NSS.txt]: Name Service Switch + * <LDAP> [AuthDatabase.LDAP.txt]: Lightweight Directory Access Protocol + * <SQL> [AuthDatabase.SQL.txt]: SQL database (PostgreSQL, MySQL, SQLite) + * <Dict> [AuthDatabase.Dict.txt]: Dict key-value database (Redis, memcached, + etc.) + * <Static> [UserDatabase.Static.txt]: Userdb information generated from a + given template + * <VPopMail> [AuthDatabase.VPopMail.txt]: External software used to handle + virtual domains + * <Prefetch> [UserDatabase.Prefetch.txt]: This assumes that the passdb already + returned also all the required user database information + * <Lua> [AuthDatabase.Lua.txt]: Lua script for authentication (v2.3.0+) + +Userdb settings +--------------- + +An example userdb entry might look like this: + +---%<------------------------------------------------------------------------- +userdb { + driver = passwd-file + args = username_format=%n /etc/dovecot/users + + default_fields = uid=vmail gid=vmail + override_fields = + + # v2.2.10+: + skip = never + result_failure = continue + result_internalfail = continue + result_success = return-ok + + # v2.2.24+: + auth_verbose = default +} +---%<------------------------------------------------------------------------- + +First we have the settings that provide content for the userdb lookup: + + * driver: The userdb backend name + * args: Arguments for the userdb backend. The format of this value depends on + the userdb driver. Each one uses different args. + * default_fields: Userdb fields (and <extra fields> + [UserDatabase.ExtraFields.txt]) that are used, unless overwritten by the + userdb backend. They are in format 'key=value key2=value2 ...'. The values + can contain <%variables> [Variables.txt]. + * override_fields: Same as default_fields, but instead of providing the + default values, these values override what the userdb backend returned. For + example useful with <userdb passwd> [AuthDatabase.Passwd.txt] for overriding + e.g. home directory or the uid/gid. + * auth_verbose: If this is explicitly set to yes or no, it overrides the + global auth_verbose setting. (However, auth_debug=yes overrides the + auth_verbose setting.) (v2.2.24+) + +Then we have the setting which specify when the userdb is used (v2.2.10+): + + * skip: Do we sometimes want to skip over this userdb? + * never + * found: Skip if an earlier userdb already found the user + * notfound: Skip if previous userdbs haven't yet found the user + +And finally we can control what happens when we're finished with this userdb +(v2.2.10+): + + * result_success: What to do if the user was found from the userdb (default: + return-ok) + * result_failure: What to do if the user wasn't found from the userdb + (default: continue) + * result_internalfail: What to do if the userdb lookup had an internal failure + (default: continue). If any of the userdbs had an internal failure and the + final userdb also returns "continue", the lookup will fail with "internal + error".*WARNING*: If multiple userdbs are required (results are merged), + it's important to set result_internalfail=return-fail to them, otherwise the + userdb lookup could still succeed but not all the intended extra fields are + set. + +The result values that can be used: + + * return-ok: Return success, don't continue to the next userdb. + * return-fail: Return "user doesn't exist", don't continue to the next userdb. + + * return: Return earlier userdb's success or failure, don't continue to the + next userdb. If this was the first userdb, return "user doesn't exist". + * continue-ok: Set the current user existence state to "found", and continue + to the next userdb. + * continue-fail: Set the current user existence state to "not found", and + continue to the next userdb. + * continue: Continue to the next userdb without changing the user existence + state. The initial state is "not found". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/UserIds.txt b/doc/wiki/UserIds.txt new file mode 100644 index 0000000..b71a7c7 --- /dev/null +++ b/doc/wiki/UserIds.txt @@ -0,0 +1,147 @@ +System users used by Dovecot +============================ + +Dovecot typically requires 3 or more system users: + + * /root/: Dovecot is started as root. + * /dovenull/: Dovecot uses an unprivileged /dovenull/ user for untrusted login + processes. + * /dovecot/: Dovecot uses an unprivileged /dovecot/ user for internal + processes. + * auth user: Password and user database lookups are done as auth user. + * mail user(s): Mails are accessed using yet another user. The mail user + should not be /dovecot/ user. + +Using multiple users allows privilege separation, which makes it harder for +attackers to compromise the whole system if a security hole is found from one +component. However, if you really want to run everything under a single user, +<it's possible> [HowTo.Rootless.txt]. + +Dovenull user +------------- + +/dovenull/ user is used internally for processing users' logins. It shouldn't +have access to any files, authentication databases or anything else either. It +should belong to its own private *dovenull* group where no one else belongs to, +and which doesn't have access to any files either (other than what Dovecot +internally creates). + +You can change the default /dovenull/ user to something else from +'default_login_user' setting. + +Dovecot user +------------ + +/dovecot/ user is used internally for unprivileged Dovecot processes. It should +belong to its own private /dovecot/ group. Mail files are not accessed as +/dovecot/ user, so you shouldn't give it access to mails. + +You can change the default /dovecot/ user to something else from +'default_internal_user' setting. + +Mail users +---------- + +You can use one or more system users for accessing users' mails. Most +configurations can be placed to two categories: + + 1. <System users> [SystemUsers.txt] where each Dovecot user has their own + system user in '/etc/passwd'. For system user setups you generally don't + have to worry about UIDs or GIDs, they are returned by the <userdb passwd> + [AuthDatabase.Passwd.txt] lookup. + 2. <Virtual users> [VirtualUsers.txt] where all Dovecot users run under a + single system user. Typically you'd set this with 'mail_uid' setting (e.g. + 'mail_uid=vmail'). Note that you most likely don't want the <userdb lookup> + [UserDatabase.txt] to return any UID/GID, as they override the 'mail_uid' + setting. + +However it's possible to use a setup that is anything between these two. For +example use a separate system user for each domain. See below for more +information about how UIDs can be used. + +UIDs +---- + +Dovecot's <user database> [UserDatabase.txt] configuration calls system users +UIDs. There are a few things you should know about them: + + * Although UID normally means a numeric ID (as specified by '/etc/passwd'), + it's anyway possible to use names as UID values and let Dovecot do the + lookup (eg. uid=vmail). However depending on where you used it, it may slow + down the authentication. + * The UIDs don't really have to exist in '/etc/passwd' (the kernel doesn't + care about that). For example you could decide to use UIDs 10000-59999 for + 50000 virtual Dovecot users. You'll then just have to be careful that the + UIDs aren't used unintentionally elsewhere. + * The important thing to consider with your UID allocation policy is that if + Dovecot has a security hole in its IMAP or POP3 implementation, the attacker + can read mails of other people who are using the same UID. So clearly the + most secure way is to allocate a different UID for each user. It can however + be a bit of a pain and OSes don't always support more than 65536 UIDs. + * By default Dovecot allows users to log in only with UID numbers 500 and + above. This check tries to make sure that no-one can ever log in as daemons + or other system users. If you're using an UID lower than 500, you'll need to + change the 'first_valid_uid' setting. + +GIDs +---- + +System groups (GIDs) work very much the same way as UIDs described above: You +can use names instead of numbers for GID values, and the used GIDs don't have +to exist in '/etc/group'. + +System groups are useful for sharing mailboxes between users that have +different UIDs but belong to a same group. Currently Dovecot doesn't try to do +anything special with the groups, so if you're not sure how you should create +them, you might as well place all the users into a single group or create a +separate group for each user. + +If you use multiple UIDs and you wish to create <shared mailboxes> +[SharedMailboxes.txt], setting up the groups properly may make your +configuration more secure. For example if you have two teams and their +mailboxes are shared only to their team members, you could create a group for +each team and set the shared mailbox's group to the team's group and +permissions to 0660, so neither team can even accidentally see each others' +shared mailboxes. + +Currently Dovecot supports specifying only the primary group, but if your +userdb returns 'system_user' <extra field> [UserDatabase.ExtraFields.txt], the +non-primary groups are taken from '/etc/group' for that user. In a future +version the whole GID list will be configurable without help from '/etc/group'. + +It's also possible to give all the users access to extra groups with +'mail_access_groups' setting. + +Authentication process user +--------------------------- + +Depending on passdb and userdb configuration, the lookups are done either by +auth process or auth worker process. They have different default users: + +---%<------------------------------------------------------------------------- +service auth { + user = $default_internal_user +} +service auth-worker { + user = root +} +---%<------------------------------------------------------------------------- + +The user must have access to your <password databases> [PasswordDatabase.txt] +and <user databases> [UserDatabase.txt]. It's not used for anything else. The +default is to use root, because it's guaranteed to have access to all the +password databases. If you don't need this, you should change it to +$default_internal_user. + +<PAM> [PasswordDatabase.PAM.txt] and <shadow> [PasswordDatabase.Shadow.txt] +passdbs are usually configured to read '/etc/shadow' file. Even this doesn't +need root access if the file is readable by shadow group: + +---%<------------------------------------------------------------------------- +service auth-worker { + user = $default_internal_user + group = shadow +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Variables.txt b/doc/wiki/Variables.txt new file mode 100644 index 0000000..9051220 --- /dev/null +++ b/doc/wiki/Variables.txt @@ -0,0 +1,355 @@ +Variables +========= + +You can use special variables in several places: + + * <mail_location> [MailLocation.txt] setting and <namespace> [Namespaces.txt] + locations + * <static userdb> [UserDatabase.Static.txt] and <passwd-file userdb> + [AuthDatabase.PasswdFile.txt] template strings + * <LDAP> [AuthDatabase.LDAP.txt] and <SQL> [AuthDatabase.SQL.txt] userdb query + strings + * log prefix for imap/pop3 process + * <Plugin> [Plugins.txt] settings + +The variables that work (almost) everywhere are: + + * ++------------+------------------------------+---------------------------------+ +| *Variable* | *Long name* | *Description* | ++------------+------------------------------+---------------------------------+ +| %% | | '%' character. See | +| | | <SharedMailboxes.Shared.txt> for| +| | | further information about %% | +| | | variables | ++------------+------------------------------+---------------------------------+ +| %u | user | full username (e.g. user@domain)| ++------------+------------------------------+---------------------------------+ +| %n | username | user part in user@domain, same | +| | | as %u if there's no domain | ++------------+------------------------------+---------------------------------+ +| %d | domain | domain part in user@domain, | +| | | empty if user with no domain | ++------------+------------------------------+---------------------------------+ +| %s | service | imap, pop3, smtp, lda (and | +| | | doveadm, dsync, etc.) | ++------------+------------------------------+---------------------------------+ +| %p | pid | PID of the current process | +| | | (login or imap/pop3 process) | ++------------+------------------------------+---------------------------------+ +| %l | lip | local IP address | ++------------+------------------------------+---------------------------------+ +| %r | rip | remote IP address | ++------------+------------------------------+---------------------------------+ +| | session | session ID for this client | +| | | connection (unique for 9 years) | +| | | (v2.1.6+) | ++------------+------------------------------+---------------------------------+ +| | auth_user | SASL authentication ID (e.g. if | +| | | master user login is done, this | +| | | contains the master username). | +| | | If username changes during | +| | | authentication, this value | +| | | contains the original username. | +| | | Otherwise the same as %{user}. | +| | | (v2.2.11+) | ++------------+------------------------------+---------------------------------+ +| | auth_username | user part in %{auth_user} | +| | | (v2.2.11+) | ++------------+------------------------------+---------------------------------+ +| | auth_domain | domain part in %{auth_user} | +| | | (v2.2.11+) | ++------------+------------------------------+---------------------------------+ +| | userdb:<name> | Expands to extra field "name" | +| | | returned by userdb (v2.2.19+) | ++------------+------------------------------+---------------------------------+ +| | encrypt;<parameters>:<field> | Encrypt field (v2.2.29+) (see | +| | | <Plugins.VarExpandCrypt.txt>) | ++------------+------------------------------+---------------------------------+ +| | decrypt;<parameters<:<field> | Decrypt field (v2.2.29+) (see | +| | | <Plugins.VarExpandCrypt.txt>) | ++------------+------------------------------+---------------------------------+ + +These variables work almost everywhere else except in Dovecot-auth (userdb +queries/templates): + + * ++------------+----------+-----------------------------------------------------+ +| *Variable* | *Long | *Description* | +| | name* | | ++------------+----------+-----------------------------------------------------+ +| %h | home | home directory. Use of ~/ is better whenever | +| | | possible. | ++------------+----------+-----------------------------------------------------+ +| %i | uid | UNIX UID of the user | ++------------+----------+-----------------------------------------------------+ +| | gid | UNIX group identifier of the user (v2.0.17+) | ++------------+----------+-----------------------------------------------------+ + +These variables work only in Dovecot-auth and 'login_log_format_elements' +setting: + + * ++----+--------------------+---------------------------------------------------+ +| %m | mech | <authentication mechanism> | +| | | [Authentication.Mechanisms.txt], e.g. PLAIN | ++----+--------------------+---------------------------------------------------+ +| %a | lport | Local port | ++----+--------------------+---------------------------------------------------+ +| %b | rport | Remote port | ++----+--------------------+---------------------------------------------------+ +| %c | secured | "secured" string with SSL, TLS and localhost | +| | | connections. Otherwise empty. | ++----+--------------------+---------------------------------------------------+ +| | real_rip | Same as %{rip}, except in proxy setups contains | +| | | the remote proxy's IP instead of the client's IP | +| | | (v2.1.10+) | ++----+--------------------+---------------------------------------------------+ +| | real_lip | Same as %{lip}, except in proxy setups contains | +| | | the local proxy's IP instead of the remote proxy's| +| | | IP (v2.2+) | ++----+--------------------+---------------------------------------------------+ +| | real_rport | Similar to %{real_rip} except for port instead of | +| | | IP (v2.2+) | ++----+--------------------+---------------------------------------------------+ +| | real_lport | Similar to %{real_lip} except for port instead of | +| | | IP (v2.2+) | ++----+--------------------+---------------------------------------------------+ +| | orig_user | Same as %{user}, except using the original | +| | | username the client sent before any changes by | +| | | auth process (v2.2.6+, v2.2.13+ for auth) | ++----+--------------------+---------------------------------------------------+ +| | orig_username | Same as %{username}, except using the original | +| | | username (v2.2.6+, v2.2.13+ for auth) | ++----+--------------------+---------------------------------------------------+ +| | orig_domain | Same as %{domain}, except using the original | +| | | username (v2.2.6+, v2.2.13+ for auth) | ++----+--------------------+---------------------------------------------------+ +| | local_name | Expands to TLS SNI hostname, if given (v2.2.26+) | ++----+--------------------+---------------------------------------------------+ +| | client_id | Expands to client ID request as IMAP arglist | +| | | (v2.2.29+/v2.3+). Needs imap_id_retain=yes | ++----+--------------------+---------------------------------------------------+ +| | passdb:<name> | Expands to extra field "name" returned by passdb | +| | | (v2.2.19+) | ++----+--------------------+---------------------------------------------------+ +| | forward_<variable> | Used by proxies to pass on values to next hop, see| +| | | <PasswordDatabase.ExtraFields.Proxy.txt> | +| | | (v2.2.29+/v2.3+) | ++----+--------------------+---------------------------------------------------+ + +These variables work only in Dovecot-auth: + + * ++------------+----------------+-----------------------------------------------+ +| *Variable* | *Long name* | *Description* | ++------------+----------------+-----------------------------------------------+ +| %w | password | plaintext password from plaintext | +| | | authentication mechanism | ++------------+----------------+-----------------------------------------------+ +| %k | cert | "valid" if client had sent a valid client | +| | | certificate, otherwise empty. | ++------------+----------------+-----------------------------------------------+ +| | login_user | For master user logins: Logged in user@domain | ++------------+----------------+-----------------------------------------------+ +| | login_username | For master user logins: Logged in user | ++------------+----------------+-----------------------------------------------+ +| | login_domain | For master user logins: Logged in domain | ++------------+----------------+-----------------------------------------------+ +| | domain_first | For "username@domain_first@domain_last" style | +| | | usernames (v2.2.6+) | ++------------+----------------+-----------------------------------------------+ +| | domain_last | For "username@domain_first@domain_last" style | +| | | usernames (v2.2.6+) | ++------------+----------------+-----------------------------------------------+ +| | master_user | For master user logins: The master username | +| | | (v2.2.7+) | ++------------+----------------+-----------------------------------------------+ +| | session_pid | For user logins: The PID of the IMAP/POP3 | +| | | process handling the session. (v2.2.7+) | ++------------+----------------+-----------------------------------------------+ +| | passdb:<name> | Return passdb extra field "name". | +| | | %{passdb:name:default} returns "default" if | +| | | "name" doesn't exist (not returned if name | +| | | exists but is empty) (v2.2.19+) | ++------------+----------------+-----------------------------------------------+ +| | userdb:<name> | Return userdb extra field "name". | +| | | %{userdb:name:default} returns "default" if | +| | | "name" doesn't exist (not returned if name | +| | | exists but is empty) (v2.2.19+) | ++------------+----------------+-----------------------------------------------+ + +These variables work only in 'login_log_format_elements' setting: + + * ++------------+--------------+-------------------------------------------------+ +| *Variable* | *Long name* | *Description* | ++------------+--------------+-------------------------------------------------+ +| %k | ssl_security | SSL protocol and cipher information, e.g. "TLSv1| +| | | with cipher DHE-RSA-AES256-SHA (256/256 bits)" | ++------------+--------------+-------------------------------------------------+ +| %e | mail_pid | Mail process (imap/pop3) PID that handles the | +| | | post-login connection | ++------------+--------------+-------------------------------------------------+ +| | listener | Expands to the socket listener name as specified| +| | | in config file (v2.2.19+) | ++------------+--------------+-------------------------------------------------+ + +These variables work only in 'deliver_log_format' setting: + + * ++------------+---------------+------------------------------------------------+ +| *Variable* | *Long name* | *Description* | ++------------+---------------+------------------------------------------------+ +| %$ | | Log entry | ++------------+---------------+------------------------------------------------+ +| %m | msgid | Message-ID | ++------------+---------------+------------------------------------------------+ +| %s | subject | Subject | ++------------+---------------+------------------------------------------------+ +| %f | from | From address | ++------------+---------------+------------------------------------------------+ +| %e | from_envelope | Envelope sender | ++------------+---------------+------------------------------------------------+ +| | to_envelope | Envelope recipient (v2.2.19+) | ++------------+---------------+------------------------------------------------+ +| %p | size | Message size | ++------------+---------------+------------------------------------------------+ +| %w | vsize | Virtual message size | ++------------+---------------+------------------------------------------------+ +| | delivery_time | How many milliseconds was spent actually | +| | | delivering the mail (v2.2.18+) | ++------------+---------------+------------------------------------------------+ +| | session_time | How many milliseconds the LMTP session took in | +| | | total, including network waits (v2.2.18+) | ++------------+---------------+------------------------------------------------+ + +These variables work only in 'auth_policy_request_attributes' setting: + + * ++------------+--------------------+-------------------------------------------+ +| *Variable* | *Long name* | *Description* | ++------------+--------------------+-------------------------------------------+ +| | hashed_password | Truncated auth policy hash of username and| +| | | password | ++------------+--------------------+-------------------------------------------+ +| | requested_username | Since v2.2.34, contains correct username | ++------------+--------------------+-------------------------------------------+ + + * Long variable names can be used like '%{long_name} ' or with L modifier: + '%L{long_name}'. + * Environment variables can be accessed with '%{env:ENVIRONMENT_VARIABLE} '. + * Additionally, the (self-explanatory) variables '%{pid} ' and '%{hostname} ' + are available. + +Modifiers +--------- + +You can apply a modifiers for each variable (e.g. %Us = POP3): + + * %L - lowercase + * %U - uppercase + * %E - escape '"', "'" and '\' characters by inserting '\' before them. Note + that variables in SQL queries are automatically escaped, you don't need to + use this modifier for them. + * %X - parse the variable as a base-10 number, and convert it to base-16 + (hexadecimal) + * %R - reverse the string + * %N - take a 32bit hash of the variable and return it as hex. You can also + limit the hash value. For example %256Nu gives values 0..ff. You might want + padding also, so %2.256Nu gives 00..ff. This can be useful for example in + dividing users automatically to multiple partitions. + * This is "New Hash", based on MD5 to give better distribution of values + (no need for any string reversing kludges either). (v2.2.3+) + * %H - Same as %N, but use "old hash" (not recommended anymore) + * %H hash function is a bit bad if all the strings end with the same + text, so if you're hashing usernames being in user@domain form, you + probably want to reverse the username to get better hash value + variety, e.g. %3RHu. + * %{<hash + algorithm>;rounds=<n>,truncate=<bits>,salt=s,format=<hex|hexuc|base64>:field} + - Generic hash function that outputs a hex (by default) or base64 value. + Hash algorithm is any of the supported ones, e.g. md5, sha1, sha256. Also + "pkcs5" is supported using SHA256. For example: %{sha256:user} or + %{md5;truncate=32:user}. (v2.2.27+) + * %M - return the string's MD5 sum as hex. + * %D - return "sub.domain.org" as "sub,dc=domain,dc=org" (for LDAP queries) + * %T - Trim trailing whitespace + +You can take a substring of the variable by giving optional offset followed by +'.' and width after the '%' character. For example %2u gives first two +characters of the username. %2.1u gives third character of the username. + +If the offset is negative, it counts from the end, for example %-2.2i gives the +UID mod 100 (last two characters of the UID printed in a string). If a positive +offset points outside the value, empty string is returned, if a negative offset +does then the string is taken from the start. + +If the width is prefixed with zero, the string isn't truncated, but only padded +with '0' character if the string is shorter. For example %04i may return +"0001", "1000" and "12345". %1.04i for the same string would return "001", +"000" and "2345". + +If the width is negative, it counts from the end, for example %0.-2u gives all +but the last two characters from the username. (v2.2.13+) + +The modifiers are applied from left-to-right order, except the substring is +always taken from the final string. + +Conditionals +------------ + +Since v2.2.33 it's possible to use conditionals in variable expansion. The +generic syntax is + +---%<------------------------------------------------------------------------- + %{if;value1;operator;value2;value-if-true;value-if-false} +---%<------------------------------------------------------------------------- + +Each field can contain another variable expansion, facilitating for nested ifs. +If some field refers to another field, it must use either %v or %{value} +syntax. + +Escaping is supported, so one can have values like \%{value} that will not get +expanded, or literal : and ; in the expression. + +Spaces and quotes are fully supported. + +Following operators are supported ++------------+--------------------------------------------------------------+ +| *operator* | *explanation* | ++------------+--------------------------------------------------------------+ +| == | NUMERIC equality | ++------------+--------------------------------------------------------------+ +| != | NUMERIC inequality | ++------------+--------------------------------------------------------------+ +| < | NUMERIC less than | ++------------+--------------------------------------------------------------+ +| <= | NUMERIC less or equal | ++------------+--------------------------------------------------------------+ +| > | NUMERIC greater than | ++------------+--------------------------------------------------------------+ +| >= | NUMERIC greater or equal | ++------------+--------------------------------------------------------------+ +| eq | String equality | ++------------+--------------------------------------------------------------+ +| ne | String inequality | ++------------+--------------------------------------------------------------+ +| lt | String inequality | ++------------+--------------------------------------------------------------+ +| le | String inequality | ++------------+--------------------------------------------------------------+ +| gt | String inequality | ++------------+--------------------------------------------------------------+ +| ge | String inequality | ++------------+--------------------------------------------------------------+ +| * | Wildcard match (mask on value2) | ++------------+--------------------------------------------------------------+ +| !* | Wildcard non-match (mask on value2) | ++------------+--------------------------------------------------------------+ +| ~ | Regular expression match (pattern on value2, extended POSIX) | ++------------+--------------------------------------------------------------+ +| !~ | String inequality (pattern on value2, extended POSIX) | ++------------+--------------------------------------------------------------+ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/VirtualUsers.Home.txt b/doc/wiki/VirtualUsers.Home.txt new file mode 100644 index 0000000..02b4f57 --- /dev/null +++ b/doc/wiki/VirtualUsers.Home.txt @@ -0,0 +1,89 @@ +Home Directories for Virtual Users +================================== + +Home directory is a per-user directory where *Dovecot can save user-specific +files*. + + * Dovecot's home directories have nothing to do with system users' home + directories. + * It's irrelevant if it's under '/home/' or '/var/mail/' or wherever. + * If you have trouble understanding this, mentally replace all occurrences of + "home directory" with "mail user's private state directory". + +And in particular: + + * Never configure your userdb to return the same home directory for multiple + users! + * Home directory must be an absolute path, don't even try to use relative + paths! + +Some uses for home directory are: + + * By default <Sieve> [Pigeonhole.Sieve.txt] scripts are in user's home + directory. + * Duplicate mail check database is in user's home directory. Suppression of + duplicate rejects/vacations won't work if home directory isn't specified. + * Debugging: If an imap or pop3 process crashes, the core file is written to + the user's home directory. + +Home vs. mail directory +----------------------- + +Home directory shouldn't be the same as mail directory with mbox or Maildir +formats (but with dbox/obox it's fine). It's possible to do that, but you might +run into trouble with it sooner or later. Some problems with this are: + + * Non-mailbox files may show up as mailboxes. + * If you see this with Maildir, 'maildir_stat_dirs=yes' hides them. + * Or user just might not be able to create mailbox with wanted name, because + there already exists a conflicting file + * e.g. with Maildir if you have '.dovecot.sieve' file, user can't create a + mailbox called "dovecot.sieve" (i.e. "dovecot" mailbox that has a "sieve" + child) + * And vice versa: If user creates "dovecot.sieve" mailbox, Dovecot will + probably start logging all kinds of errors because the mailbox directory + isn't a valid <Sieve> [Pigeonhole.Sieve.txt] script. + * If you ever intend to migrate to another mailbox format, it's much easier to + do if you can have both old and new mail directories under the user's home + directory. + +Ways to set up home directory +----------------------------- + +The directory layouts for home and mail directories could look like one of +these (in the preferred order): + + 1. Mail directory under home, for example: home='/var/vmail/domain/user/' + mail='/var/vmail/domain/user/mail/' + 2. Completely distinct home and mail directories: + home='/home/virtual/domain/user/' mail='/var/vmail/domain/user/' + 3. Home directory under mail, for example: + * Maildir: home='/var/vmail/domain/user/home/' + mail='/var/vmail/domain/user/' + * mbox: There's really no good and safe way to do it. + 4. The home directory is the same as the mail directory. + +If for example home='/var/vmail/domain/user/' +mail='/var/vmail/domain/user/mail/', set: + +---%<------------------------------------------------------------------------- +mail_home = /var/vmail/%d/%n +mail_location = maildir:~/mail +---%<------------------------------------------------------------------------- + +Examples +-------- + +LDAP with relative directory paths +---------------------------------- + +If your LDAP database uses e.g. 'mailDirectory = domain/user/', you can use it +as a base for home directory: + +---%<------------------------------------------------------------------------- +user_attrs = .., mailDirectory=home=/var/vmail/%$ +---%<------------------------------------------------------------------------- + +Then just use 'mail_location = maildir:~/Maildir'. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/VirtualUsers.txt b/doc/wiki/VirtualUsers.txt new file mode 100644 index 0000000..24a963d --- /dev/null +++ b/doc/wiki/VirtualUsers.txt @@ -0,0 +1,149 @@ +Virtual Users +============= + +There are many ways to configure Dovecot to use virtual users. If you have no +idea how you want your users to be configured, select some <HOWTO> [HowTo.txt] +and follow its instructions. + +Users are often categorized as being either system users (in '/etc/passwd') or +virtual users (not in '/etc/passwd'). However from Dovecot's point of view +there isn't much of a difference between them. If a <passwd> +[AuthDatabase.Passwd.txt] lookup and a <SQL> [AuthDatabase.SQL.txt] lookup +return the same <userdb> [UserDatabase.txt] information, Dovecot's behavior is +identical. + +Password and user databases +--------------------------- + +Dovecot supports many different <password databases> [PasswordDatabase.txt] and +<user databases> [UserDatabase.txt]. With virtual users the most commonly used +ones are <LDAP> [AuthDatabase.LDAP.txt], <SQL> [AuthDatabase.SQL.txt] and +<passwd-file> [AuthDatabase.PasswdFile.txt]. The databases usually contain the +following information: + + * Username + * Password + * UNIX User ID (UID) and primary UNIX Group ID (GID) + * Home directory and/or mail location + +Usernames and domains +--------------------- + +Dovecot doesn't care much about domains in usernames. IMAP and POP3 protocols +currently have no concept of "domain", so the username is just something that +shows up in your logs and maybe in some configuration, but they have no direct +functionality. + +So although Dovecot makes it easier to handle "user@domain" style usernames +(eg. %n and %d <variables> [Variables.txt]), nothing breaks if you use for +example "domain%user" style usernames instead. However some <authentication +mechanisms> [Authentication.Mechanisms.txt] do have an explicit support for +realms (pretty much the same as domains). If those mechanisms are used, the +username is changed to be "user@realm". + +And of course there's no need to have domains at all in the usernames. + +Passwords +--------- + +The password can be in <any format that Dovecot supports> +[Authentication.PasswordSchemes.txt], but you need to tell the format to +Dovecot because it won't try to guess it. The SQL and LDAP configuration files +have the 'default_pass_scheme' setting for this. If you have passwords in +multiple formats, or the passdb doesn't have such a setting, you'll need to +prefix each password with "{<scheme>}", for example "{PLAIN}plaintext-password" +or "{PLAIN-MD5}1a1dc91c907325c69271ddf0c944bc72". + +UNIX UIDs +--------- + +The most important thing you need to understand is that *Dovecot doesn't access +the users' mails as the /dovecot/ user*! So *don't* put /dovecot/ into the +/mail/ group, and don't make mails owned by the /dovecot/ user. That will only +make your Dovecot installation less secure. + +So, if not the /dovecot/ user, what then? You can decide that yourself. You can +create, for example, one /vmail/ user which owns all the mails, or you can +assign a separate UID for each user. See <UserIds#mailusers> [UserIds.txt] for +more information about different ways to allocate UIDs for users. + +UNIX GIDs +--------- + +Unless you're using <shared mailboxes> [SharedMailboxes.txt] and multiple UIDs, +it doesn't really matter what GIDs you use. You can, for example, use a single +GID for all users, or create a separate GID for each user. See <UserIds#gids> +[UserIds.txt] for more information. + +Home directories +---------------- + +Some people are opposed to the idea of virtual users having home directories, +but no matter what you call it, it's a good idea to have a directory where +user-specific configuration and other state is stored. See +<VirtualUsers.Home.txt> more information. + +Mail location +------------- + +The userdb can return the 'mail' <field> [UserDatabase.txt] to override the +default 'mail_location' setting. Usually you shouldn't need this. + +Examples +-------- + +Dynamic passwd-file locations +----------------------------- + +---%<------------------------------------------------------------------------- +mail_location = maildir:/home/%d/%n/Maildir +passdb { + driver = passwd-file + args = username_format=%n /home/%d/etc/shadow +} +userdb { + driver = passwd-file + args = username_format=%n /home/%d/etc/passwd +} +---%<------------------------------------------------------------------------- + +In the above examples users are expected to log in as "user@domain". Their mail +is kept in their home directory at '/home/<domain>/<username>/Maildir'. + +The usernames in the passwd and shadow files are expected to contain only the +user part, no domain. This is because the path itself already contained %d to +specify the domain. If you want the files to contain full user@domain names, +you can use 'username_format=%u'. + +static userdb +------------- + +Many people store only usernames and passwords in their database and don't want +to deal with UIDs or GIDs. In that case the easiest way to get Dovecot running +is to use the <static userdb> [UserDatabase.Static.txt]: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +passdb { + driver = pam +} +userdb { + driver = static + args = uid=vmail gid=vmail home=/var/mail/virtual/%d/%n +} +---%<------------------------------------------------------------------------- + +This makes Dovecot look up the mails from +'/var/mail/virtual/<domain>/<user>/Maildir/' directory, which should be owned +by vmail user and vmail group. + +Existing virtual user management software +----------------------------------------- + + * VPopMail [http://www.inter7.com/vpopmail/]: Dovecot <supports VPopMail + directly> [AuthDatabase.VPopMail.txt]. + * Linuxconf [http://www.solucorp.qc.ca/linuxconf/]: See + <Migration.Linuxconf.txt> + * Also see the <HowTo.txt> page + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/WhyDoesItNotWork.txt b/doc/wiki/WhyDoesItNotWork.txt new file mode 100644 index 0000000..54dd8e1 --- /dev/null +++ b/doc/wiki/WhyDoesItNotWork.txt @@ -0,0 +1,116 @@ +Why is Dovecot not working for me? +================================== + + * *Dovecot always logs an error message* if anything goes wrong, so make sure + you're looking at the correct log files. Debug messages may be written to a + different log file than error messages. See <Logging.txt>. + * Are you upgrading? Make sure you read the <upgrading documents> + [Upgrading.txt]. + * Make sure you're modifying the correct config file! Ubuntu creates two of + them. Other distributions may also have older config files lying around. For + example add "garbage=blah" to the config file and verify that Dovecot now + fails to start up. + * Try logging in manually by sending IMAP commands. If you're trying with an + IMAP client you can't be sure if the problem is with the client's + configuration or Dovecot's configuration. Many IMAP clients handle all + errors simply by showing you the login password dialog, even if the problem + has nothing to do with authentication. See <TestInstallation.txt> (or + <TestPop3Installation.txt>). + * Are you using an old version? If you have a problem, see if NEWS + [http://dovecot.org/doc/NEWS] file mentions anything related to it. There + are also <binary packages> [PrebuiltBinaries.txt] available for newer + versions. + +If you have upgraded OpenVZ to LXC (ProxMox 4) and you are getting permisssion +issues +------------------------------------------------------------------------------------- + +This is due to ACL option being added to mount options. Simple fix is to run + +---%<------------------------------------------------------------------------- +setfacl -k /var/run/dovecot +setfacl -b /var/run/dovecot/* +---%<------------------------------------------------------------------------- + +and the issue should go away. For more information about this, see +https://forum.proxmox.com/threads/permission-error-w-sockets-inside-ct-since-migration-to-pve-4-1.25244/ + +It says "Authentication failed" +------------------------------- + +First of all enable 'auth_debug_passwords=yes' and see if the logs show what +the problem is. For non-PAM setups it should contain all the information needed +to solve the problem. If it's trying to use a wrong password scheme, you can +change that. See <Authentication.PasswordSchemes.txt>. + +'Aborted login (no auth attempts)' means that the client isn't even attempting +to log in. Most likely you have 'disable_plaintext_auth=yes' (default) and the +client isn't configured to use SSL/TLS (or you've also set 'ssl=no'). + +If you're using <PAM> [PasswordDatabase.PAM.txt]: + + * Make sure that Dovecot's auth process is running as root (assuming it's + using '/etc/shadow'). + * PAM errors aren't written to Dovecot's own logs. Usually they go to + '/var/log/auth.log' or something similar. + * Unfortunately PAM's error messages aren't always all that helpful in + figuring out what exactly the problem is. + * You could (temporarily) try to use <passdb shadow> + [PasswordDatabase.Shadow.txt] instead to see if it logs something more + understandable. + * Usually the problem is that you don't have a correctly named file in + '/etc/pam.d/'. See <PasswordDatabase.PAM.txt>. + +Authenticated SMTP is hanging when authenticating, when configured with dovecot +authenticator +--------------------------------------------------------------------------------------------- + +It is possible that you SMTP daemon is configured with the wrong socket. Please +note that '/run/dovecot/auth-client' and '/run/dovecot/auth-userdb' do respond +do different protocols. They are however very similar and it is possible that +the difference may not be reported as an error and the SMTP server is waiting +for a response that will never come. + +It's not finding my emails +-------------------------- + +'mail_debug=yes' makes Dovecot log where it's really looking for mails. Also +'auth_debug=yes' may be helpful in debugging. See <MailLocation.txt> for how to +configure where the mails are looked up from. + +Permission errors accessing the mail storage +-------------------------------------------- + +---%<------------------------------------------------------------------------- +lda(user1): Error: chdir(/home/user1/) failed: Permission denied +(euid=1025(user1) egid=1026(user1) stat() failed: No such file or directory, +euid is not dir owner) +---%<------------------------------------------------------------------------- + +Check out the access permissions of the mentioned directory, check: + + 1. Unix permissions with the command 'ls -aln /home/user1', see Unix + permissions + [https://en.wikipedia.org/wiki/File_system_permissions#Traditional_Unix_permissions], + as well as make sure the user has "x" permission for "'/'" and all + directories, just "'/home'" in this case, test with: "'su - user1 ls + -aln /home/user1'" + 2. security tools, like SELinux: run "'sestatus'" and "'grep -i AVC + /var/log/audit/audit.log'" to identify SELinux caused denials, + 3. what file system the storage is located on, for instance AFS implements + different access permissions or POSIX eXtended attributes may change the + traditional permissions. + +---%<------------------------------------------------------------------------- +Couldn't create mailbox list lock +/data/mail/domain.com/username/mailboxes.lock: +file_create_locked(/data/mail/domain.com/username/mailboxes.lock) failed: +link(/data/mail/domain.com/username/mailboxes.lock6628a230290f9029, +/data/mail/domain.com/username/mailboxes.lock) failed: Operation not permitted +---%<------------------------------------------------------------------------- + + 1. Your filesystem is not supported + 2. You have SELinux/AppArmor/RBAC or some other security framework that + prevents this + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/maildrop.txt b/doc/wiki/maildrop.txt new file mode 100644 index 0000000..99d4883 --- /dev/null +++ b/doc/wiki/maildrop.txt @@ -0,0 +1,71 @@ +Maildrop +======== + +Maildrop is available at https://www.courier-mta.org/maildrop + +Although it is the default LDA for the courier mail server, it also works as a +standalone LDA. The install script can even build an rpm package if you prefer +to manage software that way. Maildrop can be used with sendmail as a drop-in +replacement for procmail. It has better maildir support than procmail and it +has filtering capability using syntax that looks a lot like shell script +language with a little bit of perl thrown in. + +To use maildrop with sendmail, make the following change in your sendmail.mc +file, then make etc. + +---%<------------------------------------------------------------------------- +dnl FEATURE(`local_procmail,', `procmail -t -Y -a $h -d $u')dnl +dnl # replace above line with below line to use maildrop instead of procmail +FEATURE(`local_procmail', `/usr/bin/maildrop', `maildrop -d $u')dnl +---%<------------------------------------------------------------------------- + +A default system wide configuration file for maildir style mailboxes might look +like the following: + +---%<------------------------------------------------------------------------- +# file: /etc/maildroprc +# system-wide settings for maildrop + +SHELL="/bin/bash" +SENDMAIL="/usr/sbin/sendmail -oi -t" +logfile "/var/log/maildrop.log" + +DEFAULT="$HOME/Maildir/" +---%<------------------------------------------------------------------------- + +In the above configuration, the users should be a member of the same group and +the group should have write privileges to the logfile. + +To use maildrop with Postfix, take a look at +https://www.postfix.org/MAILDROP_README.html + +Dovecot Authentication extension for maildrop +--------------------------------------------- + +Announced on the Dovecot's mailing list: +https://dovecot.org/list/dovecot/2009-April/039121.html + +A patch for maildrop that would allow it to perform user lookups directly +against Dovecot in a similar way how they are done by Dovecot's LDA deliver +depending on used version of maildrop is available at the following locations: ++--------------+---------------------------------------------------------------+ +| *maildrop* | *patch* | ++--------------+---------------------------------------------------------------+ +| 2.0.4 | https://www.max.rs/ozone/maildrop-2.0.4-dovecotauth.patch.txt | ++--------------+---------------------------------------------------------------+ +| 2.1.0 | https://www.max.rs/ozone/maildrop-2.1.0-dovecotauth.patch.txt | ++--------------+---------------------------------------------------------------+ +| 2.2.0 | https://www.max.rs/ozone/maildrop-2.2.0-dovecotauth.patch.txt | ++--------------+---------------------------------------------------------------+ +| 2.3.0, | https://www.max.rs/ozone/maildrop-2.3.0-dovecotauth.patch.txt | +| 2.4.0-2.4.3, | | +| 2.5.0-2.5.4 | | ++--------------+---------------------------------------------------------------+ + +When this patch is applied, maildrop will be extended with another command line +option "-t", which can be used to specify the location of Dovecot's master auth +socket that will be used when performing user lookups. There's a readme file +(README.dovecotauth) together with patch with a little bit more information +regarding the use of this extension. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/mutt.txt b/doc/wiki/mutt.txt new file mode 100644 index 0000000..6cebfc2 --- /dev/null +++ b/doc/wiki/mutt.txt @@ -0,0 +1,104 @@ +Contents + + + 1. Using mutt with IMAP + + 1. configuration + + 2. problems + + 1. Move read messages to mbox + + 2. New mail in a folder just left + + 3. Authentication realm + + 2. References + +Using mutt with IMAP +==================== + +configuration +------------- + +First, mutt needs to be told to use IMAP. To achieve this, edit '~/.muttrc' (or +a file it 'source's) to contain: + +---%<------------------------------------------------------------------------- +set spoolfile=imap://user@hostname/INBOX +set folder=imap://user@hostname/ +---%<------------------------------------------------------------------------- + +problems +-------- + +Move read messages to mbox +-------------------------- + +If mutt asks 'Move read messages to /home/$user/mbox? ([no]/yes):' + +Your alternatives (to set in '~/.muttrc') are (pick one and ignore the others): + + 1. don't ask about moving messages, just do it: + + ---%<--------------------------------------------------------------------- + set move=yes + ---%<--------------------------------------------------------------------- + + 2. don't ask about moving messages and _don't_ do it: + + ---%<--------------------------------------------------------------------- + set move=no + ---%<--------------------------------------------------------------------- + + 3. ask about moving message, default answer 'yes': + + ---%<--------------------------------------------------------------------- + set move=ask-yes + ---%<--------------------------------------------------------------------- + + 4. ask about moving message, default answer 'no': + + ---%<--------------------------------------------------------------------- + set move=ask-no + ---%<--------------------------------------------------------------------- + +New mail in a folder just left +------------------------------ + +There is a bug with dovecot in <=0.99. It can sometimes make mutt think there +is new email in a mailbox you just +'left'...https://dovecot.org/list/dovecot/2004-February/002950.html + +Authentication realm +-------------------- + +If you are using IMAP with SASL and get error messages like 'Invalid realm' in +your dovecot log files, try putting this in your '.muttrc' or '/etc/Muttrc]': + +---%<------------------------------------------------------------------------- +set imap_authenticators="plain" +---%<------------------------------------------------------------------------- + +It might be easier however to just put the following in your 'dovecot.conf': + +For Dovecot 1.0: + ---%<----------------------------------------------------------------------- + auth default { + mechanisms = plain + } + ---%<----------------------------------------------------------------------- + +For Dovecot 0.99: + ---%<----------------------------------------------------------------------- + auth_mechanisms = plain + ---%<----------------------------------------------------------------------- + +References +========== + + * Official mutt homepage [http://www.mutt.org/] + * mutt with IMAP documentation [http://mutt.sourceforge.net/imap/] + * http://jamespo.org.uk/blog/archives/000271.html + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/uw2dovecot.sh.txt b/doc/wiki/uw2dovecot.sh.txt new file mode 100644 index 0000000..7c63596 --- /dev/null +++ b/doc/wiki/uw2dovecot.sh.txt @@ -0,0 +1,13 @@ +---%<------------------------------------------------------------------------- +#!/bin/sh +# Written by Michal Grzedzicki - public domain +# handy script to rename uw-imapd style subscriptions to dovecot + +for hdir in `cat /etc/passwd|cut -d: -f6` ; do + if [ -f "$hdir/.mailboxlist" ]; then + cp -a "$hdir/.mailboxlist" "$hdir/.subscriptions" + fi +done +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) |