path: root/doc/man/doveadm-sync.1.in

.\" Copyright (c) 2014-2018 Dovecot authors, see the included COPYING file
.TH DOVEADM\-SYNC 1 "2015-08-31" "Dovecot v2.3" "Dovecot"
.SH NAME
doveadm\-sync \- Dovecot\(aqs two\-way mailbox synchronization utility
.br
doveadm\-backup \- Dovecot\(aqs one\-way mailbox synchronization utility
.\"------------------------------------------------------------------------
.SH SYNOPSIS
.BR doveadm " [" \-Dv "] " sync
[\fB\-u\fP \fIuser\fP|\fB\-A\fP|\fB\-F\fP \fIfile\fP]
[\fB\-S\fP \fIsocket_path\fP]
.RB [ \-1fPRU ]
[\fB\-l\fP \fIsecs\fP]
[\fB\-r\fP \fIrawlog path\fP]
[\fB\-m\fP \fImailbox\fP]
[\fB\-g\fP \fImailbox guid\fP]
[\fB\-n\fP \fInamespace\fP|\fB\-N\fP]
[\fB\-x\fP \fIexclude\fP]
[\fB\-a\fP \fIall mailbox\fP]
[\fB\-s\fP \fIstate\fP]
[\fB\-T\fP \fIsecs\fP]
[\fB\-t\fP \fIstart date\fP]
[\fB\-e\fP \fIend date\fP]
[\fB\-O\fP \fIsync flag\fP]
[\fB\-I\fP \fImax size\fP]
\fB\-d\fP|\fIdestination\fP
.\"-------------------------------------
.PP
.BR doveadm " [" \-Dv "] " backup
[\fB\-u\fP \fIuser\fP|\fB\-A\fP|\fB\-F\fP \fIfile\fP]
[\fB\-S\fP \fIsocket_path\fP]
.RB [ \-fPRU ]
[\fB\-l\fP \fIsecs\fP]
[\fB\-r\fP \fIrawlog path\fP]
[\fB\-m\fP \fImailbox\fP]
[\fB\-g\fP \fImailbox guid\fP]
[\fB\-n\fP \fInamespace\fP|\fB\-N\fP]
[\fB\-x\fP \fIexclude\fP]
[\fB\-a\fP \fIall mailbox\fP]
[\fB\-s\fP \fIstate\fP]
[\fB\-T\fP \fIsecs\fP]
[\fB\-t\fP \fIstart date\fP]
[\fB\-e\fP \fIend date\fP]
[\fB\-O\fP \fIsync flag\fP]
[\fB\-I\fP \fImax size\fP]
\fB\-d\fP|\fIdestination\fP
.\"------------------------------------------------------------------------
.SH DESCRIPTION
dsync is Dovecot\(aqs mailbox synchronization utility.
It can be used for several different use cases: Two\-way synchronization of
mailboxes, creating backups of mails, and convert mailboxes from/to
different mailbox formats.
All of these can be used within the same server or between different
servers (via
.BR ssh (1)
or tcp connections).
Remote mailboxes can be accessed also via IMAP protocol, which allows using
dsync for mailbox migration purposes.
.PP
You can run dsync in one of three modes:
.RS
.\"-------------------------------------
.IP \(bu
.B doveadm backup
performs one\-way synchronization.
If there are any changes in the destination they will be deleted, so the
destination will look exactly like the source.
.\"-------------------------------------
.IP \(bu
.B doveadm sync
performs two\-way synchronization.
It merges all changes without losing anything.
Both the mailboxes will end up looking identical after the synchronization
is finished.
.\"-------------------------------------
.IP \(bu
.B doveadm sync \-1
performs one\-way synchronization, but it merges the changes in destination
without deleting anything.
This doesn\(aqt currently work perfectly, so its use should be limited.
Its main purpose is that during mailbox migration you can run
.B doveadm backup
multiple times, then switch mails to be delivered to the new mailbox and
run
.B doveadm sync \-1
once more to transfer any last new mails from the old mailbox.
.IP
The one\-way algorithm is the same as two-way dsync algorithm except the
source account is not modified. It fetches the message's GUID (Global UID),
which is used to identify any conflicting UIDs in messages. As long as the
source and destination side has matching UID<\->GUID mapping, those emails
are assumed to be synced correctly. Only after the first mismatch will
changes begin.
.IP
Example: Source mailbox has messages UID 1..5; source mailbox is sync'd
using
.B doveadm backup
to the destination. Subsequently, UID 6 is delivered to the source mailbox
and UID 1 is expunged from the destination mailbox. In this example, UID 1
is kept removed (in destination) because UID 1..5 have identical
Date+Message\-ID headers. UID 6 is not seen in destination so it's copied.
.IP
If both source and destination have UID 6, but the messages are different,
the headers don't match and both the messages are kept in the destination but
they're given new UIDs 7 and 8 just to be sure any client didn't get confused
about what UID 6 actually was. Thus, one\-way sync begins to quickly diverge
from the source mailbox once changes start to occur on either side; one\-way
sync should therefore normally only be used within a short period of time
after a
.B doveadm backup
or
.B doveadm sync
command was used to synchronize the mailboxes.
.\"-------------------------------------
.RE
.PP
There are also three different synchronization algorithms:
.RS
.\"-------------------------------------
.IP \(bu
Full synchronization (\-f parameter) scans through all the messages in all
the mailboxes.
This guarantees that everything will be synchronized, but it\(aqs
unnecessarily slow for incremental synchronization.
.\"-------------------------------------
.IP \(bu
Fast synchronization (default) first attempts to find mailboxes that have
changed, and synchronize only those.
This is done by checking the mailboxes\(aq metadata (NEXTUID and
HIGHESTMODSEQ).
Usually this works fine, especially with one\-way synchronization, but if
both sides do exactly the same number of changes, the metadata may end up
containing the same values even if the changes were different.
.\"-------------------------------------
.IP \(bu
Stateful synchronization (\-s parameter) is the most efficient way to
synchronize mailboxes.
It relies on having the earlier dsync run\(aqs state saved somewhere and
being passed to the next dsync run.
Based on this state dsync can send only the changes that happened after the
previous dsync run.
As long as the state or the mailboxes aren\(aqt corrupted this algorithm
should work perfectly.
The replicator process uses this internally to perform most of the
synchronization.
.\"-------------------------------------
.RE
.PP
The syncing is done as perfectly as possible: an IMAP or a POP3 client
shouldn\(aqt be able to notice any differences between the two mailboxes.
Two\-way syncing means that it\(aqs safe to do any kind of modifications in
both sides, and dsync will merge the changes without losing any changes
done on either side.
This is possible because dsync can access Dovecot\(aqs index logs that keep
track of changes.
It\(aqs of course possible to have conflicts during merging, these are
resolved in a safe way.
See the
.I dsync design
document for more information.
.PP
dsync uses the same configuration files as the rest of Dovecot (via
.BR doveconf (1)
binary).
The entire configuration can be changed by giving \-c parameter to another
configuration file, or using \-o parameter to override specific settings.
When executing a remote dsync program it works the same way:
it uses its own local configuration.
.PP
dsync can be run completely standalone.
It doesn\(aqt require any Dovecot server processes to be running, except
when using \-u parameter to do a
.I userdb
lookup from auth process.
.PP
dsync can sync either one or multiple users using the \-u or \-A
parameters.
For continuous replication you can use the Dovecot replicator process,
which automatically runs dsync whenever messages have changed.
.\"------------------------------------------------------------------------
@INCLUDE:global-options@
.\" --- command specific options --- "/.
.PP
Command specific
.IR options :
@INCLUDE:option-A@
.\"-------------------------------------
@INCLUDE:option-F-file@
.\"-------------------------------------
.TP
.B \-1
Do one\-way synchronization instead of two\-way synchronization.
.\"-------------------------------------
.TP
.B \-f
Do full synchronization.
.\"-------------------------------------
.TP
.B \-N
Synchronize all the available namespaces.
By default only namespaces that don\(aqt have explicit location setting
are synchronized.
.\"-------------------------------------
.TP
.B \-P
Run a
.BR doveadm\-purge (1)
for the destination (remote) storage after synchronization.
.\"-------------------------------------
.TP
.B \-R
Do a reverse sync. Normally, messages would be pushed from the local
system to the destination (remote). This option reverses the flow, and
will instead pull messages from the remote to the local storage.
.\"-------------------------------------
@INCLUDE:option-S-socket@
.\"-------------------------------------
.TP
.BI \-T \ secs
Specify the time in seconds, how long
.BR doveadm (1)
should wait for stalled I/O operations.
The default timeout is 600 seconds.
.\"-------------------------------------
.TP
.B \-U
This is used internally by replicator to have dsync notify it when the
synchronization is finished.
.\"-------------------------------------
.TP
.B \-d
Use the default destination, which is looked up from the
.I mail_replica userdb
extra field.
.\"-------------------------------------
.TP
.BI \-g \ mailbox_guid
Same as \-m, but find the mailbox to be synchronized by its GUID instead
of by name.
.\"-------------------------------------
.TP
.BI \-l \ secs
Lock the dsync for this user.
Wait for maximum
.I secs
before giving up.
This parameter should be used to avoid broken synchronization if it\(aqs
possible that dsync is being run concurrently for the same user.
.\"-------------------------------------
.TP
.BI \-m \ mailbox
Synchronize only this mailbox name.
.\"-------------------------------------
.TP
.BI \-n \ namespace
Synchronize only the specified namespace.
This parameter can be used multiple times.
.\"-------------------------------------
.TP
.BI \-a \ all\ mailbox
Name for the "All mails" virtual mailbox. If specified, mails are attempted to
be copied from this mailbox instead of being saved separately. This may
reduce the total disk space usage as well as disk IO.
.\"-------------------------------------
.TP
.BI \-t \ start\ date
Skip any mails whose received-timestamp is older than the specified time.
.\"-------------------------------------
.TP
.BI \-e \ end\ date
Skip any mails whose received-timestamp is newer than the specified time.
.\"-------------------------------------
.TP
.BI \-O \ sync\ flag
Sync only mails that have the specified flag.
If the flag name begins with \(dq\fB\-\fP\(dq, sync all mails except the ones
with the specified flag.
.\"-------------------------------------
.TP
.BI \-I \ max\ size
Skip any mails larger than the specified size.
.\"-------------------------------------
.TP
.BI \-r \ rawlog_path
Running dsync remotely, write the remote input/output traffic to the
specified log file.
.\"-------------------------------------
.TP
.BI \-s \ previous_state
Use stateful synchronization.
If the previous state is unknown, use an empty string.
The new state is always printed to standard output.
.\"-------------------------------------
@INCLUDE:option-u-user@
.\"-------------------------------------
.TP
.BI \-x \ mailbox_mask
Exclude the specified mailbox name/mask.
The mask may contain \(dq\fB?\fP\(dq and \(dq\fB*\fP\(dq wildcards.
The mask can also be a special-use name (e.g. \\Trash).
This parameter can be used multiple times.
.\"------------------------------------------------------------------------
.SH ARGUMENTS
.TP
.I destination
This argument specifies the synchronized destination.
It can be one of:
.RS
.TP
location
Same as
.I mail_location
setting, e.g. maildir:\(ti/Maildir
.TP
.BI remote: login@host
Uses
.I dsync_remote_cmd
setting to connect to the remote host (usually via ssh)
.TP
.I remoteprefix:login@host
This is the same as remote, except \(dquser@domain\(rsn\(dq is sent before
dsync protocol starts.
This allows implementing a trusted wrapper script that runs doveadm
dsync\-server by reading the username from the first line.
.TP
.BI tcp: host[:port]
Connects to remote doveadm server via TCP.
The default port is specified by
.IR doveadm_port " setting."
.TP
.BI tcps: host[:port]
This is the same as tcp, but with SSL.
.TP
.BI command\ [arg1\ [,\ arg2,\ ...]]
Runs a local command that connects its standard input & output
to a dsync server.
.RE
.\"------------------------------------------------------------------------
.SH "EXIT STATUS"
.B dsync
will exit with one of the following values:
.TP 4
.B 0
Synchronization was done perfectly.
.TP
.B 2
Synchronization was done without errors, but some changes couldn\(aqt be done,
so the mailboxes aren\(aqt perfectly synchronized. Running dsync again
usually fixes this. Typically this occurs for message modification
sequences with newly created mailboxes. It can also occur if one of the
mailboxes change during the syncing.
.TP
.B 1, >2
Synchronization failed.
.\"------------------------------------------------------------------------
.SH EXAMPLE
.SS SYNCHRONIZATION
Synchronize mailboxes with a remote server.
Any errors are written to stderr.
.PP
.RS
.nf
.ft B
doveadm sync \-u username@example.com remote:server\-replica.example.com
.ft P
.fi
.RE
.PP
If you need more complex parameters to ssh, you can use e.g.:
.PP
.RS
.nf
.ft B
doveadm sync \-u username@example.com ssh \-i id_dsa.dovecot \(rs
mailuser@example.com doveadm dsync\-server \-u username@example.com
.ft P
.fi
.RE
.\"------------------------------------------------------------------------
.SS CONVERTING
Assuming that the
.I mail_location
setting in
.I @pkgsysconfdir@/conf.d/10\-mail.conf
is set to:
.BR "mail_location = mdbox:\(ti/mdbox" ,
a logged in system user may convert her/his mails from its Maildir in
her/his home directory to the mdbox mailbox format.
The user has to execute the command:
.PP
.RS
.nf
.ft B
doveadm sync maildir:\(ti/Maildir
.ft P
.fi
.RE
.PP
If you want to do this without any downtime, you can do the conversion one
user at a time.
Initially:
.RS 4
.IP \(bu 4
Configuration uses
.B mail_location = maildir:\(ti/Maildir
.IP \(bu
Set up the possibility of doing per\-user mail location using
.I userdb
extra fields.
.RE
.PP
Then for each user:
.RS 4
.IP 1. 4
Run
.I doveadm sync
once to do the initial conversion.
.IP 2.
Run
.I doveadm sync
again, because the initial conversion could have taken a while and new
changes could have occurred during it.
This second time only applies changes, so it should be fast.
.IP 3.
Update mail extra field in userdb to
.BR mdbox:\(ti/mdbox .
If you\(aqre using auth cache, you need to flush it, e.g.
.BR "doveadm auth cache flush" .
.IP 4.
Wait for a few seconds and then kill (doveadm kick) the user\(aqs all
existing imap and pop3 sessions (that are still using maildir).
.IP 5.
Run
.I doveadm sync
once more to apply final changes that were possibly done.
After this there should be no changes to Maildir, because the user\(aqs
mail location has been changed and all existing processes using it have
been killed.
.RE
.PP
Once all users have been converted, you can set the default
.I mail_location
to mdbox and remove the per\-user mail locations from
.IR userdb .
.\"------------------------------------------------------------------------
@INCLUDE:reporting-bugs@
.\"------------------------------------------------------------------------
.SH SEE ALSO
.BR doveadm (1),
.BR doveadm\-auth (1),
.BR doveadm\-kick (1),
.BR doveadm\-purge (1),
.BR doveconf (1)
.\"-------------------------------------
.PP
Additional resources:
.IP "dsync design"
http://wiki2.dovecot.org/Design/Dsync