Migrating from any IMAP/POP3 server to Dovecot via dsync
========================================================

You need Dovecot v2.1.4+ for this.

This page describes how to migrate mails to Dovecot from a remote IMAP/POP3
server, preserving the IMAP UIDs, POP3 UIDLs and everything else.

We'll assume that you can log in with master user "master" that has a password
"masteruser-secret". If you can't use master users, you'll need the users'
plaintext passwords and change the appropriate configuration to use them.

See also server-specific details:

 * <Gmail> [Migration.Gmail.txt]

Generic configuration
---------------------

dsync can be configured with features and workarounds. This is done by using
dsync_features setting in config file, such as

---%<-------------------------------------------------------------------------
dsync_features = empty-header-workaround
---%<-------------------------------------------------------------------------

 * empty-header-workaround - enables dsync to cope with broken (Zimbra) servers
   that sometimes send FETCH replies that are missing all the headers, even
   though the mail actually has headers and another FETCH might return them. In
   this situation dsync simply assumes for existing mails that the mails are a
   match. (v2.2.25.3+)

In v2.2.30+ dsync can avoid creating a single huge transaction that can fail
all at once. In case of failures the sync can be then done incrementally.

---%<-------------------------------------------------------------------------
dsync_commit_msgs_interval = 100 # default in v2.2.30+
---%<-------------------------------------------------------------------------

In v2.2.33+ you can specify which email headers are used in incremental syncing
for checking whether the local email matches the remote email. This should only
include headers that can be efficiently downloaded from the remote server. The
default is:

---%<-------------------------------------------------------------------------
dsync_hashed_headers = Date Message-ID
---%<-------------------------------------------------------------------------

IMAP migration configuration
----------------------------

Set up configuration for the IMAP server you wish to migrate from:

---%<-------------------------------------------------------------------------
imapc_host = imap.example.com

# Authenticate as masteruser / masteruser-secret, but use a separate login
user.
# If you don't have a master user, remove the imapc_master_user setting.
imapc_user = %u
imapc_master_user = masteruser
imapc_password = masteruser-secret

imapc_features = rfc822.size
# If you have Dovecot v2.2.8+ you may get a significant performance improvement
with fetch-headers:
imapc_features = $imapc_features fetch-headers
# Read multiple mails in parallel, improves performance
mail_prefetch_count = 20

# If the old IMAP server uses INBOX. namespace prefix, set:
#imapc_list_prefix = INBOX

# for SSL:
#imapc_port = 993
#imapc_ssl = imaps
#ssl_client_ca_dir = /etc/ssl
#imapc_ssl_verify = yes
# for <2.2.0: change ssl_client_ca_dir=/etc/ssl to imapc_ssl_ca_dir=/etc/ssl
---%<-------------------------------------------------------------------------

POP3 migration configuration
----------------------------

Set up configuration for the POP3 server you wish to migrate from:

---%<-------------------------------------------------------------------------
pop3c_host = pop3.example.com

# Authenticate as masteruser / masteruser-secret, but use a separate login
user.
# If you don't have a master user, remove the pop3c_master_user setting.
pop3c_user = %u
pop3c_master_user = masteruser
pop3c_password = masteruser-secret

# for SSL:
#pop3c_port = 995
#pop3c_ssl = pop3s
# for <2.2.0: change ssl_client_ca_dir=/etc/ssl to pop3c_ssl_ca_dir=/etc/ssl
#ssl_client_ca_dir = /etc/ssl
#pop3c_ssl_verify = yes

# make sure you have also the regular inbox namespace defined, e.g.:
#namespace inbox {
#  inbox = yes
#}
namespace {
  prefix = POP3-MIGRATION-NS/
  location = pop3c:
  list = no
  hidden = yes
}
protocol doveadm {
  mail_plugins = $mail_plugins pop3_migration
}
plugin {
  pop3_migration_mailbox = POP3-MIGRATION-NS/INBOX
}
---%<-------------------------------------------------------------------------

The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is
handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3
server and figures out which IMAP messages match which POP3 messages and then
returns the appropriate POP3 UIDL.

Running
-------

Make sure destination is exactly as source, deleting/reverting any changes in
destination if necessary:

---%<-------------------------------------------------------------------------
doveadm -o mail_fsync=never backup -R -u user@domain imapc:
---%<-------------------------------------------------------------------------

or incremental one-way merge (it's ok to do changes on both sides):

---%<-------------------------------------------------------------------------
doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc:
---%<-------------------------------------------------------------------------

(Fsyncing is disabled just for migration efficiency.)

For per-user user/passwords use:

---%<-------------------------------------------------------------------------
doveadm -o imapc_user=foo -o pop3c_user=foo -o imapc_password=bar -o
pop3c_password=bar backup -R -u user@domain imapc:
---%<-------------------------------------------------------------------------

Once the users are migrated, remember to remove the pop3_migration plugin from
settings.

Problems
--------

 * POP3 message order (when it's different from IMAP message order) is
   currently preserved only when destination is Maildir, sdbox or mdbox.
 * If source POP3 server merges multiple IMAP mailboxes into one POP3 INBOX,
   the migration won't be transparent.
 * If source IMAP and POP3 servers return messages somehow differently,
   pop3-migration plugin might not be able to match the messages

(This file was created from the wiki on 2019-06-19 12:42)