Adding upstream version 3.5.24.upstream/3.5.24upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 254 insertions, 0 deletions

diff --git a/implementation-notes/MILTER b/implementation-notes/MILTER
new file mode 100644
index 0000000..f67fb90
--- /dev/null
+++ b/implementation-notes/MILTER
@@ -0,0 +1,254 @@
+Distribution of Milter responsibility
+=====================================
+
+Milters look at the SMTP commands as well as the message content.
+In Postfix these are handled by different processes:
+
+- smtpd(8) (the SMTP server) focuses on the SMTP commands, strips
+  the SMTP encapsulation, and passes envelope information and message
+  content to the cleanup server.
+
+- the cleanup(8) server parses the message content (it understands
+  headers, body, and MIME structure), and creates a queue file with
+  envelope and content information. The cleanup server adds additional
+  envelope records, such as when to send a "delayed mail" notice.
+
+If we want to support message modifications (add/delete recipient,
+add/delete/replace header, replace body) then it pretty much has
+to be implemented in the cleanup server, if we want to avoid extra
+temporary files.
+
+Network versus local submission
+===============================
+
+As of Sendmail 8.12, all mail is received via SMTP, so all mail is
+subject to Miltering (local submissions are queued in a submission
+queue and then delivered via SMTP to the main MTA, or appended to
+$HOME/dead.letter). In Postfix, local submissions are received by
+the pickup server, which feeds the mail into the cleanup server
+after doing basic sanity checks.
+
+How do we set up the Milters with SMTP mail versus local submissions?
+
+- SMTP mail: smtpd creates Milter contexts, and sends them, including
+  their sockets, to the cleanup server. The smtpd is responsible
+  for sending the Milter abort and close messages. Both smtpd and
+  cleanup are responsible for closing their Milter socket. Since
+  smtpd and cleanup inspect mail at different times, there is no
+  conflict with access to the Milter socket.
+
+- Local submission: the cleanup server creates Milter contexts.
+  The cleanup server provides dummy connect and helo information,
+  or perhaps none at all, and provides sender and recipient events.
+  The cleanup server is responsible for sending the Milter abort
+  and close messages, and for closing the Milter socket.
+
+A special case of local submission is "sendmail -t". This creates
+a record stream in which recipients appear after content. However,
+Milters expect to receive envelope information before content, not
+after.  This is not a problem: just like a queue manager, the
+cleanup-side Milter client can jump around through the queue file
+and send the information to the Milter in the expected order.
+
+Interaction with XCLIENT, "postsuper -r", and external content filters
+======================================================================
+
+Milter applications expect that the MTA supplies context information
+in the form of Sendmail-like macros (j=hostname, {client_name}=the
+SMTP client hostname, etc.). Not all these macros have a Postfix
+equivalent. Postfix 2.3 makes a subset available.
+
+If Postfix does not implement a specific macro, people can usually
+work around it. But we should avoid inconsistency. If Postfix can
+make macro X available at Milter protocol stage Y, then it must
+also be able to make that macro available at all later Milter
+protocol stages, even when some of those stages are handled by a
+different Postfix process.
+
+Thus, when adding Milter support for a specific Sendmail-like macro    
+to the SMTP server:
+
+- We may have to update the XCLIENT protocol, so that Milter
+  applications can be tested with XCLIENT. If not, then we must
+  prominently document everywhere that XCLIENT does not provide
+  100% accurate simulation for Milters. An additional complication
+  is that the SMTP command length is limited, and that each XCLIENT
+  command resets the SMTP server to the 220 stage and generates
+  "connect" events for anvil(8) and for Milters.
+
+- The SMTP server has to send the corresponding attribute to the
+  cleanup server.  The cleanup server then stores the attribute in
+  the queue file, so that Milters produce consistent results when
+  mail is re-queued with "postsuper -r".
+
+But wait, there is more. If mail is filtered by an external content
+filter, then it needs to preserve all the Milter attributes so that
+after "postsuper -r", Milters produce the exact same result as when
+mail was received originally by Postfix. Specifically, after
+"postsuper -r" a signing Milter must not sign mail that it did not
+sign on the first pass through Postfix, and it must not reject mail
+that it accepted on the first pass through Postfix.
+
+Instead of trying to re-create the Milter execution environment
+after "postsuper -r" we simply disable Milter processing. The
+rationale for this is: if mail was Miltered before it was written
+to queue file, then there is no need to Milter it again.
+
+We might want to take a similar approach with external (signing or
+blocking) content filters: don't filter mail that has already been
+filtered, and don't filter mail that didn't need to be filtered.
+Such mail can be recognized by the absence of a "content_filter"
+record. To make the implementation efficient, the cleanup server
+would have to record the presence of a "content_filter" record in
+the queue file header.
+
+Message envelope or content modifications
+=========================================
+
+Milters can send modification requests after receiving the end of
+the message body.  If we can implement all the header/body-related
+Milter operations in the cleanup server, then we can try to edit
+the queue file in place, without ever having to make a temporary
+copy. Once a Milter is done editing, the queue file can be used as
+input for the next Milter, and so on. Finally, the cleanup server
+calls fsync() and waits for successful return.
+
+To implement in-place queue file edits, we need to introduce
+surprisingly little change to the existing Postfix queue file
+structure.  All we need is a way to specify a jump from one place
+in the file to another.
+
+Postfix does not store queue files as plain text files. Instead all
+information is stored in records with an explicit type and length
+for sender, recipient, arrival time, and so on.  Even the content
+that makes up the message header and body is stored as records with
+an explicit type and length.  This organization makes it very easy
+to introduce pointer records, which is what we will use to jump
+from one place in a queue file to another place.
+
+- Deleting a recipient or header record is easy - just mark the
+  record as killed.  When deleting a recipient, we must kill all
+  recipient records that result from virtual alias expansion of the
+  original recipient address. When deleting a very long header or
+  body line, multiple queue file records may need to be killed. We
+  won't try to reuse the deleted space for other purposes.
+
+- Replacing header or body records involves pointer records.
+  Basically, a record is replaced by overwriting it with a forward
+  pointer to space after the end of the queue file, putting the new
+  record there, followed by a reverse pointer to the record that
+  follows the replaced information. If the replaced record is shorter
+  than a pointer record, we relocate the records that follow it to
+  the new area, until we have enough space for the forward pointer
+  record. See below for a discussion on what it takes to make this
+  safe.
+
+  Postfix queue files are segmented. The first segment is for
+  envelope records, the second for message header and body content,
+  and the third segment is for information that was extracted or
+  generated from the message header and body content.  Each segment
+  is terminated by a marker record. For now we don't want to change
+  their location. In particular, we want to avoid moving the start
+  of a segment.
+
+  To ensure that we can always replace a header or body record by
+  a pointer record, without having to relocate a marker record, the
+  cleanup server always places a dummy pointer record at the end
+  of the headers and at the end of the body.
+
+  When a Milter wants to replace an entire body, we have the option
+  to overwrite existing body records until we run out of space, and
+  then writing a pointer to space at the end of the queue file,
+  followed by the remainder of the body, and a pointer to the marker
+  that ends the message content segment.
+
+- Appending a recipient or header record involves pointer records
+  as well. This requires that the queue file already contains a
+  dummy pointer record at the place where we want to append recipient
+  or header content (Milters currently do not replace individual
+  body records, but we could add this if need be).  To append,
+  change the dummy pointer into a forward pointer to space after
+  the end of a message, put the new record there, followed by a
+  reverse pointer to the record that follows the forward pointer.
+
+  To append another record, replace the reverse pointer by a forward
+  pointer to space after the end of a message, put the new record
+  there, followed by the value of the reverse pointer that we
+  replace. Thus, there is no one-to-one correspondence between
+  forward and backward pointers! In fact, there can be multiple
+  forward pointers for one reverse pointer.
+
+When relocating a record we must not relocate the target of a jump
+==================================================================
+
+As discussed above, when replacing an existing record, we overwrite
+it with a forward pointer to the new information. If the old record
+is too small we relocate one or more records that follow the record
+that's being replaced, until we have enough space for the forward
+pointer record.
+
+Now we have to become really careful. Could we end up relocating a
+record that is the target of a forward or reverse pointer, and thus
+corrupt the queue file? The answer is NO.
+
+- We never relocate end-of-segment marker records. Instead, the
+  cleanup server writes dummy pointer records to guarantee that
+  there is always space for a pointer.
+
+- When a record is the target of a forward pointer, it is "edited"
+  information that is preceded either by the end-of-queue-file
+  marker record, or it is preceded by the reverse pointer at the
+  end of earlier written "edited" information. Thus, the target of
+  a forward pointer will not be relocated to make space for a pointer
+  record.
+
+- When a record is the target of a reverse pointer, it is always
+  preceded by a forward pointer record (or by a forward pointer
+  record followed by some unused space). Thus, the target of a
+  reverse pointer will not be relocated to make space for a pointer
+  record.
+
+Could we end up relocating a pointer record?  Yes, but that is OK,
+as long as pointers contain absolute offsets.
+
+Pointer records introduce the possibility of loops
+==================================================
+
+When a queue file is damaged, a bogus pointer value may send Postfix
+into a loop. This must not happen.
+
+Detecting loops is not trivial:
+
+- A sequence of multiple forward pointers may be followed by one
+  legitimate reverse pointer to the location after the first forward
+  pointer. See above for a discussion of how to append a record to
+  an appended record.
+
+- We do know, however, that there will not be more reverse pointers
+  than forward pointers. But this does not help much.
+
+Perhaps we can include a record count at the start of the queue
+file, so that the record walking code knows that it's looking at
+some records more than once, and return an error indication.
+
+How many bytes do we need for a pointer record?
+===============================================
+
+A pointer record would look like this:
+
+    type (1 byte)
+    offset (see below)
+
+Postfix uses long for queue file size/offset information, and stores
+them as %15ld in the SIZE record at the start of the queue file.
+This is somewhat less than a 64-bit long, but it is enough for a
+some time to come, and it is easily changed without breaking forward
+or backward compatibility.
+
+It does mean, however, that a pointer record can easily exceed the
+length of a header record. This is why we go through the trouble
+of record relocation and dummy records. 
+
+In Postfix 2.4 we fixed this by adding padding to short message
+header records so that we can always write a pointer record over a
+message header.  This immensly simplifies the code.