Adding upstream version 1:2.3.19.1+dfsg1.upstream/1%2.3.19.1+dfsg1upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 128 insertions, 0 deletions

diff --git a/doc/wiki/Design.Storage.Mailbox.Sync.txt b/doc/wiki/Design.Storage.Mailbox.Sync.txt
new file mode 100644
index 0000000..a7d295a
--- /dev/null
+++ b/doc/wiki/Design.Storage.Mailbox.Sync.txt
@@ -0,0 +1,128 @@
+Mailbox Synchronization
+=======================
+
+The idea behind synchronization is to find out what changes other sessions have
+done to the mailbox and to finalize our own changes to the mailbox.
+
+For example if you expunge a message in a transaction and commit it, the commit
+will only write a "please expunge UID n" record to Dovecot's transaction log
+file. The message still exists on the disk. The next time Dovecot syncs the
+mailbox (either the session that wrote the record or another one), it goes
+through all the non-synchronized records in transaction log and applies the
+requested changes to the backend mailbox. Syncing can be a bit heavyweight
+operation, so it's possible to commit multiple transactions and perform a
+single sync for all of them. Dovecot attempts to do this with IMAP protocol
+when pipelining commands.
+
+The other important job of syncing is to refresh mailbox's state:
+
+ * Finding out about external modifications to mailbox (e.g. a new mail
+   delivered to Maildir/new/).
+ * Updating in-memory view of what messages exist, what their flags are, etc.
+
+When a mailbox is opened, its state starts with what index files contain at the
+time. Since the backend mailbox may have already changed, and syncing an
+up-to-date mailbox is usually really cheap, there isn't much point in not
+syncing mailbox immediately after opening. The mailbox state stays the same
+until you synchronize the mailbox again, before that no new messages show up
+and no messages get expunged.
+
+Typically you would sync the mailbox
+
+ * after committing a transaction that modifies backend mailbox in any way
+   (instead of just internal index data), such as after changing message flags
+   or expunging a message.
+ * whenever you want to find out if there are any changes. With IMAP protocol
+   this is done every time after running a command.
+
+Initializing
+------------
+
+'mailbox_sync_init()' initializes syncing.
+
+There are some flags that control how much effort is spent on syncing:
+
+ * 'MAILBOX_SYNC_FLAG_FAST' can be given when you're ready for mailbox to be
+   refreshed, but don't care much if it actually is or not. When this flag is
+   set, Dovecot still notices all internal changes, but external changes are
+   checked only once every few seconds or so.
+ * 'MAILBOX_SYNC_FLAG_FULL_READ' is mainly useful with mboxes. If
+   'mbox_dirty_syncs=yes' and a new mail gets appended to mbox by an external
+   program, Dovecot assumes that the only change was the added mail, even
+   though the program may have also modified existing messages' flags by
+   rewriting Status: headers. If 'mbox_very_dirty_syncs=no', these changes are
+   noticed after the next time mailbox is opened. So when this flag is enabled,
+   it means Dovecot should try harder to find out if there were any external
+   unexpected changes. It's currently used only with IMAP SELECT and CHECK
+   commands and POP3 startup. Probably unnecessary elsewhere.
+ * 'MAILBOX_SYNC_FLAG_FULL_WRITE' is again mainly useful with mboxes. If
+   'mbox_lazy_writes=no', Dovecot delays writing flag changes to mbox file
+   until mailbox is closed or IMAP CHECK command is issued. Using this
+   elsewhere is probably unnecessary, except as an optimization if mailbox is
+   in any case synced just before closing it, you might as well give this flag
+   to it to avoid double-syncing with mbox.
+ * 'MAILBOX_SYNC_FLAG_FORCE_RESYNC' is used to force resyncing indexes. The
+   only time this should be done is when manually triggered by administrator.
+
+Then there are also other syncing flags:
+
+ * 'MAILBOX_SYNC_FLAG_NO_EXPUNGES': No expunged messages are removed from the
+   in-memory mailbox view. Their removal is delayed until syncing is done
+   without this flag. Attempting to access the expunged messages may or may not
+   work, depending on what information is accessed and what storage backend is
+   used.
+ * 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT': Normally when the internal mailbox
+   state can't be consistently updated (typically due to index file
+   corruption), the syncing fails. When this flag is set, it means that the
+   caller doesn't care about mailbox's previous state and just wants to get it
+   accessible again. Typically this is used when the mailbox is being opened,
+   but not afterwards.
+ * 'MAILBOX_SYNC_FLAG_EXPUNGE' is mainly intended for virtual plugin with IMAP
+   protocol. You probably shouldn't use it.
+
+Reading changes
+---------------
+
+While 'mailbox_sync_next()' returns TRUE, it fills out sync record:
+
+ * seq1, seq2: Message sequence numbers that were affected
+ * type: expunge, flag change or modseq change.
+
+Expunge records don't immediately change the view's sequence numbers. After
+seeing an expunge record you can still fetch the expunged messages' flags and
+possibly other information. Only after syncing is deinitialized, the sequences
+change.
+
+Message flag change records don't actually show what the changes were. You can
+find the new flags just by fetching them ('mail_get_flags()', etc.), they're
+available immediately. You'll need to create a <transaction>
+[Design.Storage.Mailbox.Transaction.txt] and a <mail> [Design.Storage.Mail.txt]
+for that. For example:
+
+---%<-------------------------------------------------------------------------
+sync_ctx = mailbox_sync_init(box, flags);
+trans = mailbox_transaction_begin(box, 0);
+mail = mail_alloc(trans, MAIL_FETCH_FLAGS, 0);
+---%<-------------------------------------------------------------------------
+
+If you don't actually care about sync records, you don't necessarily have to
+even call 'mailbox_sync_next()'. In that case it's actually easiest to perform
+the whole sync using a one-step 'mailbox_sync()' function. This function also
+sets 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT' flag automatically.
+
+Deinitializing
+--------------
+
+'mailbox_sync_deinit()' finalizes the syncing. If any errors occurred during
+sync, it'll return -1.
+
+If 'MAILBOX_SYNC_FLAG_NO_EXPUNGES' was used and some expunges were actually
+delayed,'status_r->sync_delayed_expunges' is set to TRUE.
+
+Implementing sync for a storage backend
+---------------------------------------
+
+FIXME: talk about mail_index_sync_*() and how to change stuff and how to update
+internal state.
+
+(This file was created from the wiki on 2019-06-19 12:42)