summaryrefslogtreecommitdiffstats
path: root/implementation-notes/MILTER
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--implementation-notes/MILTER254
1 files changed, 254 insertions, 0 deletions
diff --git a/implementation-notes/MILTER b/implementation-notes/MILTER
new file mode 100644
index 0000000..f67fb90
--- /dev/null
+++ b/implementation-notes/MILTER
@@ -0,0 +1,254 @@
+Distribution of Milter responsibility
+=====================================
+
+Milters look at the SMTP commands as well as the message content.
+In Postfix these are handled by different processes:
+
+- smtpd(8) (the SMTP server) focuses on the SMTP commands, strips
+ the SMTP encapsulation, and passes envelope information and message
+ content to the cleanup server.
+
+- the cleanup(8) server parses the message content (it understands
+ headers, body, and MIME structure), and creates a queue file with
+ envelope and content information. The cleanup server adds additional
+ envelope records, such as when to send a "delayed mail" notice.
+
+If we want to support message modifications (add/delete recipient,
+add/delete/replace header, replace body) then it pretty much has
+to be implemented in the cleanup server, if we want to avoid extra
+temporary files.
+
+Network versus local submission
+===============================
+
+As of Sendmail 8.12, all mail is received via SMTP, so all mail is
+subject to Miltering (local submissions are queued in a submission
+queue and then delivered via SMTP to the main MTA, or appended to
+$HOME/dead.letter). In Postfix, local submissions are received by
+the pickup server, which feeds the mail into the cleanup server
+after doing basic sanity checks.
+
+How do we set up the Milters with SMTP mail versus local submissions?
+
+- SMTP mail: smtpd creates Milter contexts, and sends them, including
+ their sockets, to the cleanup server. The smtpd is responsible
+ for sending the Milter abort and close messages. Both smtpd and
+ cleanup are responsible for closing their Milter socket. Since
+ smtpd and cleanup inspect mail at different times, there is no
+ conflict with access to the Milter socket.
+
+- Local submission: the cleanup server creates Milter contexts.
+ The cleanup server provides dummy connect and helo information,
+ or perhaps none at all, and provides sender and recipient events.
+ The cleanup server is responsible for sending the Milter abort
+ and close messages, and for closing the Milter socket.
+
+A special case of local submission is "sendmail -t". This creates
+a record stream in which recipients appear after content. However,
+Milters expect to receive envelope information before content, not
+after. This is not a problem: just like a queue manager, the
+cleanup-side Milter client can jump around through the queue file
+and send the information to the Milter in the expected order.
+
+Interaction with XCLIENT, "postsuper -r", and external content filters
+======================================================================
+
+Milter applications expect that the MTA supplies context information
+in the form of Sendmail-like macros (j=hostname, {client_name}=the
+SMTP client hostname, etc.). Not all these macros have a Postfix
+equivalent. Postfix 2.3 makes a subset available.
+
+If Postfix does not implement a specific macro, people can usually
+work around it. But we should avoid inconsistency. If Postfix can
+make macro X available at Milter protocol stage Y, then it must
+also be able to make that macro available at all later Milter
+protocol stages, even when some of those stages are handled by a
+different Postfix process.
+
+Thus, when adding Milter support for a specific Sendmail-like macro
+to the SMTP server:
+
+- We may have to update the XCLIENT protocol, so that Milter
+ applications can be tested with XCLIENT. If not, then we must
+ prominently document everywhere that XCLIENT does not provide
+ 100% accurate simulation for Milters. An additional complication
+ is that the SMTP command length is limited, and that each XCLIENT
+ command resets the SMTP server to the 220 stage and generates
+ "connect" events for anvil(8) and for Milters.
+
+- The SMTP server has to send the corresponding attribute to the
+ cleanup server. The cleanup server then stores the attribute in
+ the queue file, so that Milters produce consistent results when
+ mail is re-queued with "postsuper -r".
+
+But wait, there is more. If mail is filtered by an external content
+filter, then it needs to preserve all the Milter attributes so that
+after "postsuper -r", Milters produce the exact same result as when
+mail was received originally by Postfix. Specifically, after
+"postsuper -r" a signing Milter must not sign mail that it did not
+sign on the first pass through Postfix, and it must not reject mail
+that it accepted on the first pass through Postfix.
+
+Instead of trying to re-create the Milter execution environment
+after "postsuper -r" we simply disable Milter processing. The
+rationale for this is: if mail was Miltered before it was written
+to queue file, then there is no need to Milter it again.
+
+We might want to take a similar approach with external (signing or
+blocking) content filters: don't filter mail that has already been
+filtered, and don't filter mail that didn't need to be filtered.
+Such mail can be recognized by the absence of a "content_filter"
+record. To make the implementation efficient, the cleanup server
+would have to record the presence of a "content_filter" record in
+the queue file header.
+
+Message envelope or content modifications
+=========================================
+
+Milters can send modification requests after receiving the end of
+the message body. If we can implement all the header/body-related
+Milter operations in the cleanup server, then we can try to edit
+the queue file in place, without ever having to make a temporary
+copy. Once a Milter is done editing, the queue file can be used as
+input for the next Milter, and so on. Finally, the cleanup server
+calls fsync() and waits for successful return.
+
+To implement in-place queue file edits, we need to introduce
+surprisingly little change to the existing Postfix queue file
+structure. All we need is a way to specify a jump from one place
+in the file to another.
+
+Postfix does not store queue files as plain text files. Instead all
+information is stored in records with an explicit type and length
+for sender, recipient, arrival time, and so on. Even the content
+that makes up the message header and body is stored as records with
+an explicit type and length. This organization makes it very easy
+to introduce pointer records, which is what we will use to jump
+from one place in a queue file to another place.
+
+- Deleting a recipient or header record is easy - just mark the
+ record as killed. When deleting a recipient, we must kill all
+ recipient records that result from virtual alias expansion of the
+ original recipient address. When deleting a very long header or
+ body line, multiple queue file records may need to be killed. We
+ won't try to reuse the deleted space for other purposes.
+
+- Replacing header or body records involves pointer records.
+ Basically, a record is replaced by overwriting it with a forward
+ pointer to space after the end of the queue file, putting the new
+ record there, followed by a reverse pointer to the record that
+ follows the replaced information. If the replaced record is shorter
+ than a pointer record, we relocate the records that follow it to
+ the new area, until we have enough space for the forward pointer
+ record. See below for a discussion on what it takes to make this
+ safe.
+
+ Postfix queue files are segmented. The first segment is for
+ envelope records, the second for message header and body content,
+ and the third segment is for information that was extracted or
+ generated from the message header and body content. Each segment
+ is terminated by a marker record. For now we don't want to change
+ their location. In particular, we want to avoid moving the start
+ of a segment.
+
+ To ensure that we can always replace a header or body record by
+ a pointer record, without having to relocate a marker record, the
+ cleanup server always places a dummy pointer record at the end
+ of the headers and at the end of the body.
+
+ When a Milter wants to replace an entire body, we have the option
+ to overwrite existing body records until we run out of space, and
+ then writing a pointer to space at the end of the queue file,
+ followed by the remainder of the body, and a pointer to the marker
+ that ends the message content segment.
+
+- Appending a recipient or header record involves pointer records
+ as well. This requires that the queue file already contains a
+ dummy pointer record at the place where we want to append recipient
+ or header content (Milters currently do not replace individual
+ body records, but we could add this if need be). To append,
+ change the dummy pointer into a forward pointer to space after
+ the end of a message, put the new record there, followed by a
+ reverse pointer to the record that follows the forward pointer.
+
+ To append another record, replace the reverse pointer by a forward
+ pointer to space after the end of a message, put the new record
+ there, followed by the value of the reverse pointer that we
+ replace. Thus, there is no one-to-one correspondence between
+ forward and backward pointers! In fact, there can be multiple
+ forward pointers for one reverse pointer.
+
+When relocating a record we must not relocate the target of a jump
+==================================================================
+
+As discussed above, when replacing an existing record, we overwrite
+it with a forward pointer to the new information. If the old record
+is too small we relocate one or more records that follow the record
+that's being replaced, until we have enough space for the forward
+pointer record.
+
+Now we have to become really careful. Could we end up relocating a
+record that is the target of a forward or reverse pointer, and thus
+corrupt the queue file? The answer is NO.
+
+- We never relocate end-of-segment marker records. Instead, the
+ cleanup server writes dummy pointer records to guarantee that
+ there is always space for a pointer.
+
+- When a record is the target of a forward pointer, it is "edited"
+ information that is preceded either by the end-of-queue-file
+ marker record, or it is preceded by the reverse pointer at the
+ end of earlier written "edited" information. Thus, the target of
+ a forward pointer will not be relocated to make space for a pointer
+ record.
+
+- When a record is the target of a reverse pointer, it is always
+ preceded by a forward pointer record (or by a forward pointer
+ record followed by some unused space). Thus, the target of a
+ reverse pointer will not be relocated to make space for a pointer
+ record.
+
+Could we end up relocating a pointer record? Yes, but that is OK,
+as long as pointers contain absolute offsets.
+
+Pointer records introduce the possibility of loops
+==================================================
+
+When a queue file is damaged, a bogus pointer value may send Postfix
+into a loop. This must not happen.
+
+Detecting loops is not trivial:
+
+- A sequence of multiple forward pointers may be followed by one
+ legitimate reverse pointer to the location after the first forward
+ pointer. See above for a discussion of how to append a record to
+ an appended record.
+
+- We do know, however, that there will not be more reverse pointers
+ than forward pointers. But this does not help much.
+
+Perhaps we can include a record count at the start of the queue
+file, so that the record walking code knows that it's looking at
+some records more than once, and return an error indication.
+
+How many bytes do we need for a pointer record?
+===============================================
+
+A pointer record would look like this:
+
+ type (1 byte)
+ offset (see below)
+
+Postfix uses long for queue file size/offset information, and stores
+them as %15ld in the SIZE record at the start of the queue file.
+This is somewhat less than a 64-bit long, but it is enough for a
+some time to come, and it is easily changed without breaking forward
+or backward compatibility.
+
+It does mean, however, that a pointer record can easily exceed the
+length of a header record. This is why we go through the trouble
+of record relocation and dummy records.
+
+In Postfix 2.4 we fixed this by adding padding to short message
+header records so that we can always write a pointer record over a
+message header. This immensly simplifies the code.