Adding upstream version 1:2.3.21+dfsg1.upstream/1%2.3.21+dfsg1

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 370 insertions, 0 deletions

diff --git a/TODO b/TODO
new file mode 100644
index 0000000..d5f01ad
--- /dev/null
+++ b/TODO
@@ -0,0 +1,370 @@
+ - remove mail_deliver_session after all, do all the stuff transparently
+   by hooking into mailbox_copy().
+     - use this hook also to do the mail deduplication: 1) sort all destination
+       users, 2) create mail_user only once for each user, 3) remember in
+       src_mail the previously copied mail, 4) use that for mailbox_copy()ing
+       to following recipients
+     - make sure this removes duplicate dbox mails when sieve saves mail to
+       multiple mailboxes
+ - auth: user iterations shouldn't be able to use up all the workers
+ - indexer: if workers are stuck, we keep adding more and more stuff to them
+   which causes the ostream size to become huge. 
+ - quota: maybe check quota once more at commit time to make sure the whole
+   transaction fits. avoids multiple parallel slow COPY commands from being
+   able to go over quota
+ - METADATA: quota, NOTIFY interaction, METADATA-SERVER capability
+ - fts: if SEARCH X-MAILBOX is used on virtual/all folder, it doesn't update
+   any indexes. (and it should skip those physical mailboxes that don't
+   match the X-MAILBOX)
+ - fts: if indexer has request queued, SEARCH won't return anything until
+   it's done.
+     - maybe abort entirely after X time and return NO
+     - prioritize small quick indexing before slow large indexing?
+     - in virtual mailbox searches don't wait for indexing to finish to
+       large unindexed mailboxes, just show what you got
+ - figure out some way to avoid a million error messages getting logged
+   when service imap/pop3 reaches process_limit (some kind of notification
+   to login process that the post-login process is full?)
+ - lda: mail sending (bounce? forward?) is sending mixed CRLF+LFs
+ - auth: remove protocol !flop {} requirement. try again remote {} and local {}
+   support for auth. where do we go stuck? at least need to be able to share
+   identical passdb/userdbs
+ - doveadm sync -l: lock also when syncing public mailboxes? per-mailbox locks?
+ - dsync: dsync_mailbox_export_init() can be very slow and not send anything
+   to remote dsync for a long time, which thinks the other side is dead and
+   kills it. need to send some kind of keepalive-notifications.
+ - dsync: rename + re-subscribe isn't handled right in first sync, because
+   dsync moves the subscribed-flag when it renames the node
+ - "/asdf" in subscriptions -> LSUB lists -> dsync assert-crashes
+ - replicator: automatically remove users who don't exist
+ - imapc: sync_uid_next handling doesn't seem to be correct, especially with
+   Courier that doesn't send UIDNEXT on SELECT
+ - sdbox: dbox_file_fix() should assume there is only one message..
+ - pop3: if we can't fetch "order" field for UIDL (but could fetch it
+   initially), the order will be wrong and error is logged. probably just
+   need to read all the UIDLs into memory at startup?..
+
+ - fs_list_get_mailbox_flags() is unnecessarily stat()ing files/dirs
+ - doveadm-server: dsync doesn't work through proxying, because the data isn't
+   actually being proxied but handled via doveadm_print()
+ - CATENATE: Allow ~{binary} data but fail if there are any c-t-e: binary parts?
+   or simply silently save it?
+
+ - master-settings.c warnings aren't logged to log file at startup
+ - dsync: delete foo, rename bar foo -> foo, foo-temp-1
+ - dsync+imapc:
+     - mailbox list could be synced pretty optimally by ignoring
+       (name, uidvalidity) matches. for the left if uidvalidities are unique
+       and can be matched -> rename mailbox.
+     - GUID-less sync could optionally use just rfc822.size [and internaldate]
+       to match messages.
+
+ - virtual plugin doesn't verify the index file's data, crashes if broken.
+ - libsasl: use it in pop3c, managesieve-login, doveadm auth
+ - per-msg checksums? per-cache-msg checksums? per-log record checksums?
+ - if transaction log file corruption is noticed, make sure new dovecot.index
+   snapshot gets written and don't mark the whole file corrupted.. rather maybe
+   just rotate and truncate it
+ - mdbox: purging in alt storage could create files back to alt storage
+ - LAYOUT=index:
+    - after doing a lot of changes the list's memory pool keeps growing.
+      do an occasional re-parsing to clear the pool
+    - quota recalc + dict-file [+acl?] assert-crashes in !indexing->syncing
+ - imaptest: add condstore, qresync tests
+
+ - Track highestmodseq always, just don't keep per-message modseqs unless
+   they're enabled. Then don't return [NOMODSEQ] on select.
+ - URLAUTH: if client tries to access nonexistent user, do a delay in
+   imap-urlauth-client.c (AFTER destroying the worker)
+     - special response in the control connection to make the imap-urlauth
+       master wait before starting a new worker
+ - shared user should get settings from userdb extra fields, especially
+   plugin/quota_rule to get different quota limits for shared mailboxes.
+   the problem is that user doesn't currently have set_parser available,
+   and adding it would probably waste memory..
+ - auth_debug[_passwords]=yes ability for specific users via doveadm. for
+   both login-common and auth
+ - settings parsing is horribly bloaty
+ - doveadm: if running via doveadm-server and it fails, say something about
+   error being in the log
+ - indexer-worker and maybe others (doveadm?) could support dropping privileges
+   permanently when service_count=1. Note that LMTP can't with multiple RCPT
+   TOs.
+ - after reading whole message text, update has_nul-state to cache
+ - FIFOs maybe should be counted as connections, but unlisten should
+   unlink+reopen it in master?
+ - recreate mailbox -> existing sessions log "indexid changed" error
+ - add message/mime limits
+ - imapc:
+     - prefetching to THREAD and SORT
+     - check all imap extensions and see if some don't work (condstore)
+ - per-namespace imapc_* settings? create a way to "copy" a settings struct,
+   so mail_storage_settings are copied to mail_namespace_settings. use the
+   change tracking to figure out what settings are namespace-specific.
+
+ - doveadm import: add -d parameter to deduplicate mails based on their GUID
+   (or perhaps do it by default?)
+ - sdbox: altmoving is done with mailbox locked. that's not necessary, it could
+   do the copying while unlocked and delete the primary files while locked
+ - passdb, userdb { username_format } that doesn't permanently change
+   the username
+ - mdbox/sdbox index rebuild -> quota rebuild?
+ - solr separate attachments (patch)
+ - sql connection pooling: Count lookup latencies, avoid servers with
+   significantly higher latencies. optionally use the secondary server only
+   as fallback
+ - maildir_storage_sync_force() shouldn't do anything except find the new
+   file, don't go expunging any more stuff or we could get recursively back to
+   where we started, and stuff would break
+ - fuzzy: be fuzzy about date/size
+ - mailbox list index:
+    - with in-memory indexes be sure to refresh it more often
+    - refreshing could refresh only the parts that are actually requested,
+      e.g. %
+ - notify_sync() could have "what changed" struct with old/new flags
+ - maildir: copy dovecot-shared file from parent mailbox, not root.
+
+ - master passdb preserves userdb_* extra fields. should it preserve
+   non-userdb_* extra fields too?
+ - imap, pop3: if client init fails, wait a second or two before disconnecting
+   client.
+ - doveadm search savedbefore 7d could be optimized in large mailboxes..
+ - mdbox: storage rebuilding could log about changes it does
+ - mdbox: broken extrefs header keeps causing index rebuilds
+ - sent, drafts: .Sent/dovecot.index: modseq_hdr.log_offset too large
+ - mail_max_lock_timeout error could be reported more nicely, also ones coming
+   from lib-index
+ - sql pool: if async query is pending and sync query is sent and there
+   are no more empty connections, it should flush the async query first
+ - NTLMv1 and LM should be disabled if disable_plaintext_auth=yes
+ - SEARCH SENT*/HEADER/etc. doesn't seem optimized when using with TEXT/BODY
+ - dict sql: support ignoring some search key hierarchies (e.g. acl "anyone")
+ - dsync: avoid sending email when it could be copied from another mailbox.
+   probably requires storage to have guid => { instances } map? that's
+   rather annoying to add.
+
+ - mdbox
+    - dotlocking: cleanup should delete stale *.lock files
+    - purging seems to be inefficient. run imaptest for a while, get >500
+      files, start purging, it's slow until there are about 100 files left,
+      then the rest is suddenly fast.
+    - make sure that when reading mdbox mails sequentially the data is being
+      read from disk in n kB blocks and reads cross mail boundaries and when
+      reading the next mail it uses the previously read data in buffer
+    - Add some kind of checksum about data+metadata and use it when checking
+      consistency
+    - figure out a way to efficiently trigger purging when user has too much
+      mail expunged (e.g. keep track of total storage size, trigger purging
+      when it's 2*quota limit)
+    - keep track of total bytes in dbox storage in map header. also if
+      possible keep track of refcount=0 bytes. use these to optimize checks.
+    - save some stuff to map index header so we don't need to keep retrying
+      it. like when saving the lowest file_id which to bother checking.
+    - test crash-fixing
+    - optimize away reading file header?
+ - maildir: out-of-disk-space failures apparently cause all kinds of
+   problems, e.g. "Expunged message reappeared", "Duplicate file entry"?
+ - deliver -r <address> used as autoreplies' From-address?
+ - istream-seekable is inefficient. it shouldn't be reading the temp file
+   immediately after writing to it
+ - config process is handling requests too slowly. maybe add some caching.
+    - maybe config should return all of the protocol/local/remote overrides
+      when requested? then the caller could do a single lookup at start and
+      merge them later internally. this would really help login processes.
+ - ipv6: auth penalty should begin from /64 and gradually grow to /48 if
+   necessary. and the same could be done for ipv4 as well..
+
+ - ldap: fix multiple-gid support somehow
+ - search: use mail_get_parts() only when it's already cached. if it's not,
+   add it to cache afterwards.
+
+	/* currently non-external transactions can be applied multiple times,
+	   causing multiple increments. */
+	//FIXME:i_assert((t->flags & MAIL_INDEX_TRANSACTION_FLAG_EXTERNAL) != 0);
+  ^ appears to work now though, probably because of the added syncing stuff..
+
+ - use backup index in mail_index_fsck()
+ - proxying: support fallbacking to local (or other?) server if the first
+   one is down
+ - virtual: If last message matching INTHREAD rule gets expunged, the rest of
+   the thread doesn't go away
+ - how do shared mailboxes work with plugins?
+    - lazy-expunge, fts, etc.?
+    - listescape+acl can't handle shared mailboxes with escape chars
+ - dovecot-acl-list:
+    - how does it work with global acls?
+    - update immediately after SETACL: add/remove entries, update timestamps
+    - read the entire file to memory only once and keep it there, stat() later
+      to see if it has changed. if not, perhaps don't even bother stat()ing
+      dovecot-acl files? at least not that often..
+ - fs quota: getquotaroot inbox vs. other-box should return different quotas
+   if two quotas are defined
+ - auth_log_prefix setting similar to mail_log_prefix
+
+ - thread indexes: if we expunge a duplicate message-id: and we have a sibling
+   with identical message-id:, we can probably just move the children?
+   (unless there are non-sibling duplicates)
+ - SEARCH INTHREAD requires no thread sorting by date - don't do it
+ - CONDSTORE: use per-flag/per-keyword conflict checking
+ - QRESYNC: Drop expunges from the middle of given seq sets if possible
+ - use universal hash functions?
+
+ - UIDVALIDITY changed while saving -> sync errors
+   - mbox: copy to Trash, manually delete copied msg, change uidvalidity,
+     set nextuid=1, copy again -> error
+   - recent_uids assert at least with mbox
+ - quota fs: Should values returned by quota be divided by the actual
+   filesystem block size instead of hardcoded DEV_BSIZE? not with AIX..
+ - squat:
+   - wrong indexid
+   - fts_build_init() assertion failed: (last_uid < last_uid_locked)
+   - is locking done right? it reads header without file being locked?
+   - split after ~8 bytes?
+   - expunges are delayed until more mails are added
+ - test replacement chars (SEARCH / SORT / Squat)
+
+ - DEBUG: buffer overflow checking code probably doesn't handle a successful
+   t_try_realloc() or pool_alloconly_realloc() properly
+ - ldap:
+   - multiple ldap values could be joined into one field with specified
+     separator (e.g. mail_access_groups=%{ldap:gidNumber:,})
+
+ - maildir+pop3 fast updates:
+   - don't update dovecot-uidlist if dovecot.index.cache doesn't exist /
+     there's nothing to cache
+   - if all messages are expunged and there are no unknown extensions in index,
+     unlink dovecot.index and rotate log and add some initial useful info to
+     the log (uidvalidity, nextuid)
+
+ - maildir
+   - don't allow more than 26 keywords
+
+ - file_cache: we're growing the mmap in page size blocks, which is horribly
+   slow if mremap() doesn't exist.
+
+ - keywords:
+    - add some limits to how many there can be
+       - don't return \* in PERMANENTFLAGS when we're full
+    - remove unused keywords?
+
+ - mail caching
+    - force bits should be used only for nonregistered fields
+    - change envelope parsing not to use get_headers() so imap.envelope can
+      actually be cached without all the headers..
+    - if there's no other pressure for compression, we should do it when
+      enough temp fields are ready to be dropped
+    - we could try compressing same field values into a single
+      location in cache file.
+    - place some maximum limit of fields to cache file? maybe some soft and
+      hard limits, so when soft limit is reached drop fields that have
+      been used only once. when hard limit is reached drop any fields to get
+      more space. all this to avoid cache file growing infinitely.
+
+ - mbox
+    - UID renumbering doesn't really work after all?
+    - still problems with CRLF mboxes.. especially with broken Content-Length
+      headers (pointing between CR-LF?)
+    - syncing existing indexes takes 4x longer than creating new one, why?
+    - how well does dirty sync + status work? it reads the last mail every
+      time? not very good..
+    - always add empty line. make the parser require it too? syncing should
+      make sure there always exists two LFs at end of file. raw-mbox-stream
+      should make sure the last message ends with LF even if it doesn't exist
+      in the file
+    - Quote "From ", unquote ">From "
+    - COPY doesn't work to itself (lock assert crash, for now just disallowed)
+
+ - index
+    - index file format changes:
+        - split to "old" and "new" indexes and try to avoid loading "old" into
+	  memory until needed
+	- pack UIDs to beginning of file with UID ranges
+	- use squat-like compressed uid ranges everywhere
+        - write first extension intros in dovecot.index.log always with names
+	   - or better yet, drop the intro concept completely as it is now
+
+ - login
+    - Digest-MD5: support integrity protection, and maybe crypting. Do it
+      through login process like SSL is done?
+
+ - auth
+    - with blocking passdb we're not caching lookups if the password was wrong
+    - non-plaintext authentication doesn't support all features:
+        - multiple passdbs don't work, only the first one is used
+	- auth cache's last_success password change check doesn't exist
+	- auth_cache_negative_ttl doesn't check password mismatches
+    - dovecot-auth should limit how fast authentication requests are allowed
+      from login processes. especially if there's one login/connection the speed
+      should be something like once/sec. also limit how fast to accept new
+      connections.
+    - support read-only logins. user could with alternative password get only
+      read-access to mails so mails could be read relatively safely with
+      untrusted computers. Maybe always send [ALERT] about the previous
+      read-only login time with IP?
+
+ - ssl
+    - add setting: ssl_options = bitmask. by default we enable all openssl
+      workarounds, this could be used to disable some of them
+    - gnutls support isn't working
+
+ - search
+    - message header search: we should ignore LWSP between two MIME blocks(?)
+    - message_search_init() could accept multiple search keywords so we
+      wouldn't need to call it separately for each one (so we wouldn't need
+      to parse the message multiple times).
+    - Create our own extension: When searching with TEXT/BODY, return
+      the message text surrounding the keywords just like web search engines
+      do. like: SEARCH X-PRINT-MATCHES TEXT "hello" -> * SEARCH 1 "He said:
+      Hello world!" 2 "Hello, I'm ...". This would be especially useful with
+      the above attachment scanning.
+
+ - general
+    - things break if next_uid gets to 2^32
+
+ - lib-http:
+    - Client:
+        - Handle HTTP/1.0 servers properly:
+            -> Transfer-Encoding is not allowed
+        - Implement support for priority/deadline-based scheduling.
+          Much like: https://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html
+        - Allow handling non-idempotent requests specially
+          (no automatic retry, block pipeline)
+        - Implement support for `Range:' requests.
+        - Implement optional round-robin request scheduling for when
+          host has multiple IPs.
+    - Server:
+        - Implement API structure for virtual hosts and resources. This way,
+          multiple services can coexist independently on the same HTTP server.
+        - Implement support for `Range:' requests.
+    - Review compliance with RFC 7230 and RFC 7231
+
+ - lmtp:
+    - Implement parallel pipelined RCPT TO: verification (requires auth API
+      changes).
+    - Improve efficiency and security by splitting lmtp up into a protocol
+      handler and a one-user local delivery service.
+    - Fully support DSN extension (especially ORCPT)
+    - Calculate incoming mail's hash, forward it via proxying, have the
+      final delivery code verify that it's correct
+
+ - submission
+    - Implement support for Postfix XFORWARD (analogous to XCLIENT)
+    - Implement a re-connect attempt to the relay server if the connection is
+      lost at some point. We now terminate the whole client with a 421, which
+      is a waste of resources.
+    - Implement running submission service without access to mail storage.
+    - Implement auto-save-to-sent feature.
+    - Implement proxy support for various (sometimes bizarre) SMTP extensions.
+      We only announce the support if available on the relay and forward
+      commands and command options. We likely don't need to implement much
+      ourselves:
+       -> RFC 2852: Deliver By
+       -> RFC 3885: Message Tracking
+       -> RFC 4865: Future Message Release
+       -> RFC 6710: Message Transfer Priorities
+    - Implement Internationalized Email (RFC 6531) support
+    - Low priority:
+       - Investigate relevance of RFC 4405 (Responsible Submitter)
+       - Add RFC5451 Authentication-Results header
+       - Implement downgrading to always support BINARYMIME
+       - Implement downgrading to always support 8BITMIME