Adding upstream version 3.7.10.upstream/3.7.10

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

3 files changed, 345 insertions, 0 deletions

diff --git a/implementation-notes/DSN b/implementation-notes/DSN
new file mode 100644
index 0000000..0d81585
--- /dev/null
+++ b/implementation-notes/DSN
@@ -0,0 +1,33 @@
+Postfix DSN support implementation notes
+========================================
+
+In delivery status reports, Postfix now properly reports remote
+LMTP/SMTP server replies with Diagnostic-Type: SMTP, with the
+Diagnostic-Code: equal to the server reply, and with Remote-MTA:
+equal to the name of the remote MTA.
+
+Of course Postfix still produces the same "informal" error descriptions
+that it produced before (for example, the error text that appears
+in the first section of a bounce report).
+
+Other error reports are not in the form of SMTP-style replies.
+
+- The Postfix LMTP/SMTP client generates Diagnostic-Type: X-Postfix
+for locally generated errors (host not found, connection timed out
+etc.).  It generates Diagnostic-Type: SMTP only for replies from
+an SMTP server.
+
+- The queue manager generates Diagnostic-Type: X-Postfix for errors
+that it detects. It also receives error information from delivery
+agents and reports that information unmodified when it decides to
+"temporarily suspend" a delivery channel.
+
+- The "pipe to command" code in local(8) and pipe(8) produces
+Diagnostic-Type: X-UNIX, and Diagnostic-Code: text that is taken
+from /usr/include/sysexits.h or from the command output.
+
+- The code that delivers to mailbox produces Diagnostic-Type:
+X-Postfix and Diagnostic-Code: text that is the same good old
+Postfix error message that we are already familiar with. Typically
+these are errno-style reports about locking a file or appending a
+file.
diff --git a/implementation-notes/ENHANCED_STATUS_CODES b/implementation-notes/ENHANCED_STATUS_CODES
new file mode 100644
index 0000000..2b4c2af
--- /dev/null
+++ b/implementation-notes/ENHANCED_STATUS_CODES
@@ -0,0 +1,58 @@
+Postfix enhanced status code implementation notes
+=================================================
+
+RFC 3463 Enhanced status code support is implemented in stages. In
+the first stage, the goal is to minimize code changes (it's several
+hundred pages of context diffs already). For this reason, the
+pre-existing status variables (success, defer, etc.) are still
+updated separately from the diagnostic text and the RFC 3463 enhanced
+status code.  All this means that one has to be careful when updating
+the code, to keep things in sync.
+
+Specific issues that one should be aware of:
+
+- In the SMTP client, update the enhanced status code and text
+whenever smtp_errno or resp->code are updated, or place an explicit
+comment that says no update is needed.
+
+- In the SMTP client, don't worry about the initial enhanced status
+digit when reporting failure to look up or connect to a host. For
+convenience, the SMTP client top-level code automatically changes
+the initial digit into '4' or '5' as appropriate.
+
+- In the SMTP server, don't worry about the initial enhanced status
+code digit when an smtpd_mumble_restriction rejects access. For
+convenience, the smtpd_check_reject() routine automatically changes
+the initial digit into '4' or '5' as appropriate.
+
+- Some low-level support routines update the diagnostic text but
+not the enhanced status code. To identify these, search for functions
+that are called with why->vstring as output parameter, and make
+sure that the caller updates the enhanced status code in all
+appropriate cases.
+
+- By design, the pipe, local and virtual delivery agent code never
+update the diagnostic text separately from the enhanced status code.
+
+- Don't rely on the system errno value after calling a routine that
+performs or prepares for mail delivery. Instead, have that routine
+update the enhanced status code (and text) when the error happens.
+This was an issue with mailbox, maildir and file delivery.  Currently
+there remains one exception to this errno usage rule; the maildir
+delivery routines log a helpful warning when delivery fails with
+EACCES.  The latter happens to work because mbox_open() does not
+need to unlock the output file, so it won't clobber the errno value.
+
+- Avoid passing around strings that combine enhanced status code
+and diagnostic text. Instead, use separate variables for status
+code and text, so that the compiler can enforce that everything has
+a status code.  Currently there are two exceptions to this rule:
+the cleanup server status reply, and the delivery agent status
+reply.  Once these protocols are updated we can remove the dns_prepend()
+routine. The third exception, enhanced status codes in external
+command output, is a feature.
+
+- The bounce/defer/sent library modules will catch the cases where
+an enhanced status code does not match the reject/defer/success
+status.  They log a warning message, and replace the incorrect
+enhanced status code by a generic one.
diff --git a/implementation-notes/MILTER b/implementation-notes/MILTER
new file mode 100644
index 0000000..f67fb90
--- /dev/null
+++ b/implementation-notes/MILTER
@@ -0,0 +1,254 @@
+Distribution of Milter responsibility
+=====================================
+
+Milters look at the SMTP commands as well as the message content.
+In Postfix these are handled by different processes:
+
+- smtpd(8) (the SMTP server) focuses on the SMTP commands, strips
+  the SMTP encapsulation, and passes envelope information and message
+  content to the cleanup server.
+
+- the cleanup(8) server parses the message content (it understands
+  headers, body, and MIME structure), and creates a queue file with
+  envelope and content information. The cleanup server adds additional
+  envelope records, such as when to send a "delayed mail" notice.
+
+If we want to support message modifications (add/delete recipient,
+add/delete/replace header, replace body) then it pretty much has
+to be implemented in the cleanup server, if we want to avoid extra
+temporary files.
+
+Network versus local submission
+===============================
+
+As of Sendmail 8.12, all mail is received via SMTP, so all mail is
+subject to Miltering (local submissions are queued in a submission
+queue and then delivered via SMTP to the main MTA, or appended to
+$HOME/dead.letter). In Postfix, local submissions are received by
+the pickup server, which feeds the mail into the cleanup server
+after doing basic sanity checks.
+
+How do we set up the Milters with SMTP mail versus local submissions?
+
+- SMTP mail: smtpd creates Milter contexts, and sends them, including
+  their sockets, to the cleanup server. The smtpd is responsible
+  for sending the Milter abort and close messages. Both smtpd and
+  cleanup are responsible for closing their Milter socket. Since
+  smtpd and cleanup inspect mail at different times, there is no
+  conflict with access to the Milter socket.
+
+- Local submission: the cleanup server creates Milter contexts.
+  The cleanup server provides dummy connect and helo information,
+  or perhaps none at all, and provides sender and recipient events.
+  The cleanup server is responsible for sending the Milter abort
+  and close messages, and for closing the Milter socket.
+
+A special case of local submission is "sendmail -t". This creates
+a record stream in which recipients appear after content. However,
+Milters expect to receive envelope information before content, not
+after.  This is not a problem: just like a queue manager, the
+cleanup-side Milter client can jump around through the queue file
+and send the information to the Milter in the expected order.
+
+Interaction with XCLIENT, "postsuper -r", and external content filters
+======================================================================
+
+Milter applications expect that the MTA supplies context information
+in the form of Sendmail-like macros (j=hostname, {client_name}=the
+SMTP client hostname, etc.). Not all these macros have a Postfix
+equivalent. Postfix 2.3 makes a subset available.
+
+If Postfix does not implement a specific macro, people can usually
+work around it. But we should avoid inconsistency. If Postfix can
+make macro X available at Milter protocol stage Y, then it must
+also be able to make that macro available at all later Milter
+protocol stages, even when some of those stages are handled by a
+different Postfix process.
+
+Thus, when adding Milter support for a specific Sendmail-like macro    
+to the SMTP server:
+
+- We may have to update the XCLIENT protocol, so that Milter
+  applications can be tested with XCLIENT. If not, then we must
+  prominently document everywhere that XCLIENT does not provide
+  100% accurate simulation for Milters. An additional complication
+  is that the SMTP command length is limited, and that each XCLIENT
+  command resets the SMTP server to the 220 stage and generates
+  "connect" events for anvil(8) and for Milters.
+
+- The SMTP server has to send the corresponding attribute to the
+  cleanup server.  The cleanup server then stores the attribute in
+  the queue file, so that Milters produce consistent results when
+  mail is re-queued with "postsuper -r".
+
+But wait, there is more. If mail is filtered by an external content
+filter, then it needs to preserve all the Milter attributes so that
+after "postsuper -r", Milters produce the exact same result as when
+mail was received originally by Postfix. Specifically, after
+"postsuper -r" a signing Milter must not sign mail that it did not
+sign on the first pass through Postfix, and it must not reject mail
+that it accepted on the first pass through Postfix.
+
+Instead of trying to re-create the Milter execution environment
+after "postsuper -r" we simply disable Milter processing. The
+rationale for this is: if mail was Miltered before it was written
+to queue file, then there is no need to Milter it again.
+
+We might want to take a similar approach with external (signing or
+blocking) content filters: don't filter mail that has already been
+filtered, and don't filter mail that didn't need to be filtered.
+Such mail can be recognized by the absence of a "content_filter"
+record. To make the implementation efficient, the cleanup server
+would have to record the presence of a "content_filter" record in
+the queue file header.
+
+Message envelope or content modifications
+=========================================
+
+Milters can send modification requests after receiving the end of
+the message body.  If we can implement all the header/body-related
+Milter operations in the cleanup server, then we can try to edit
+the queue file in place, without ever having to make a temporary
+copy. Once a Milter is done editing, the queue file can be used as
+input for the next Milter, and so on. Finally, the cleanup server
+calls fsync() and waits for successful return.
+
+To implement in-place queue file edits, we need to introduce
+surprisingly little change to the existing Postfix queue file
+structure.  All we need is a way to specify a jump from one place
+in the file to another.
+
+Postfix does not store queue files as plain text files. Instead all
+information is stored in records with an explicit type and length
+for sender, recipient, arrival time, and so on.  Even the content
+that makes up the message header and body is stored as records with
+an explicit type and length.  This organization makes it very easy
+to introduce pointer records, which is what we will use to jump
+from one place in a queue file to another place.
+
+- Deleting a recipient or header record is easy - just mark the
+  record as killed.  When deleting a recipient, we must kill all
+  recipient records that result from virtual alias expansion of the
+  original recipient address. When deleting a very long header or
+  body line, multiple queue file records may need to be killed. We
+  won't try to reuse the deleted space for other purposes.
+
+- Replacing header or body records involves pointer records.
+  Basically, a record is replaced by overwriting it with a forward
+  pointer to space after the end of the queue file, putting the new
+  record there, followed by a reverse pointer to the record that
+  follows the replaced information. If the replaced record is shorter
+  than a pointer record, we relocate the records that follow it to
+  the new area, until we have enough space for the forward pointer
+  record. See below for a discussion on what it takes to make this
+  safe.
+
+  Postfix queue files are segmented. The first segment is for
+  envelope records, the second for message header and body content,
+  and the third segment is for information that was extracted or
+  generated from the message header and body content.  Each segment
+  is terminated by a marker record. For now we don't want to change
+  their location. In particular, we want to avoid moving the start
+  of a segment.
+
+  To ensure that we can always replace a header or body record by
+  a pointer record, without having to relocate a marker record, the
+  cleanup server always places a dummy pointer record at the end
+  of the headers and at the end of the body.
+
+  When a Milter wants to replace an entire body, we have the option
+  to overwrite existing body records until we run out of space, and
+  then writing a pointer to space at the end of the queue file,
+  followed by the remainder of the body, and a pointer to the marker
+  that ends the message content segment.
+
+- Appending a recipient or header record involves pointer records
+  as well. This requires that the queue file already contains a
+  dummy pointer record at the place where we want to append recipient
+  or header content (Milters currently do not replace individual
+  body records, but we could add this if need be).  To append,
+  change the dummy pointer into a forward pointer to space after
+  the end of a message, put the new record there, followed by a
+  reverse pointer to the record that follows the forward pointer.
+
+  To append another record, replace the reverse pointer by a forward
+  pointer to space after the end of a message, put the new record
+  there, followed by the value of the reverse pointer that we
+  replace. Thus, there is no one-to-one correspondence between
+  forward and backward pointers! In fact, there can be multiple
+  forward pointers for one reverse pointer.
+
+When relocating a record we must not relocate the target of a jump
+==================================================================
+
+As discussed above, when replacing an existing record, we overwrite
+it with a forward pointer to the new information. If the old record
+is too small we relocate one or more records that follow the record
+that's being replaced, until we have enough space for the forward
+pointer record.
+
+Now we have to become really careful. Could we end up relocating a
+record that is the target of a forward or reverse pointer, and thus
+corrupt the queue file? The answer is NO.
+
+- We never relocate end-of-segment marker records. Instead, the
+  cleanup server writes dummy pointer records to guarantee that
+  there is always space for a pointer.
+
+- When a record is the target of a forward pointer, it is "edited"
+  information that is preceded either by the end-of-queue-file
+  marker record, or it is preceded by the reverse pointer at the
+  end of earlier written "edited" information. Thus, the target of
+  a forward pointer will not be relocated to make space for a pointer
+  record.
+
+- When a record is the target of a reverse pointer, it is always
+  preceded by a forward pointer record (or by a forward pointer
+  record followed by some unused space). Thus, the target of a
+  reverse pointer will not be relocated to make space for a pointer
+  record.
+
+Could we end up relocating a pointer record?  Yes, but that is OK,
+as long as pointers contain absolute offsets.
+
+Pointer records introduce the possibility of loops
+==================================================
+
+When a queue file is damaged, a bogus pointer value may send Postfix
+into a loop. This must not happen.
+
+Detecting loops is not trivial:
+
+- A sequence of multiple forward pointers may be followed by one
+  legitimate reverse pointer to the location after the first forward
+  pointer. See above for a discussion of how to append a record to
+  an appended record.
+
+- We do know, however, that there will not be more reverse pointers
+  than forward pointers. But this does not help much.
+
+Perhaps we can include a record count at the start of the queue
+file, so that the record walking code knows that it's looking at
+some records more than once, and return an error indication.
+
+How many bytes do we need for a pointer record?
+===============================================
+
+A pointer record would look like this:
+
+    type (1 byte)
+    offset (see below)
+
+Postfix uses long for queue file size/offset information, and stores
+them as %15ld in the SIZE record at the start of the queue file.
+This is somewhat less than a 64-bit long, but it is enough for a
+some time to come, and it is easily changed without breaking forward
+or backward compatibility.
+
+It does mean, however, that a pointer record can easily exceed the
+length of a header record. This is why we go through the trouble
+of record relocation and dummy records. 
+
+In Postfix 2.4 we fixed this by adding padding to short message
+header records so that we can always write a pointer record over a
+message header.  This immensly simplifies the code.