postfix/implementation-notes/MILTER

Distribution of Milter responsibility
=====================================

Milters look at the SMTP commands as well as the message content.
In Postfix these are handled by different processes:

- smtpd(8) (the SMTP server) focuses on the SMTP commands, strips
  the SMTP encapsulation, and passes envelope information and message
  content to the cleanup server.

- the cleanup(8) server parses the message content (it understands
  headers, body, and MIME structure), and creates a queue file with
  envelope and content information. The cleanup server adds additional
  envelope records, such as when to send a "delayed mail" notice.

If we want to support message modifications (add/delete recipient,
add/delete/replace header, replace body) then it pretty much has
to be implemented in the cleanup server, if we want to avoid extra
temporary files.

Network versus local submission
===============================

As of Sendmail 8.12, all mail is received via SMTP, so all mail is
subject to Miltering (local submissions are queued in a submission
queue and then delivered via SMTP to the main MTA, or appended to
$HOME/dead.letter). In Postfix, local submissions are received by
the pickup server, which feeds the mail into the cleanup server
after doing basic sanity checks.

How do we set up the Milters with SMTP mail versus local submissions?

- SMTP mail: smtpd creates Milter contexts, and sends them, including
  their sockets, to the cleanup server. The smtpd is responsible
  for sending the Milter abort and close messages. Both smtpd and
  cleanup are responsible for closing their Milter socket. Since
  smtpd and cleanup inspect mail at different times, there is no
  conflict with access to the Milter socket.

- Local submission: the cleanup server creates Milter contexts.
  The cleanup server provides dummy connect and helo information,
  or perhaps none at all, and provides sender and recipient events.
  The cleanup server is responsible for sending the Milter abort
  and close messages, and for closing the Milter socket.

A special case of local submission is "sendmail -t". This creates
a record stream in which recipients appear after content. However,
Milters expect to receive envelope information before content, not
after.  This is not a problem: just like a queue manager, the
cleanup-side Milter client can jump around through the queue file
and send the information to the Milter in the expected order.

Interaction with XCLIENT, "postsuper -r", and external content filters
======================================================================

Milter applications expect that the MTA supplies context information
in the form of Sendmail-like macros (j=hostname, {client_name}=the
SMTP client hostname, etc.). Not all these macros have a Postfix
equivalent. Postfix 2.3 makes a subset available.

If Postfix does not implement a specific macro, people can usually
work around it. But we should avoid inconsistency. If Postfix can
make macro X available at Milter protocol stage Y, then it must
also be able to make that macro available at all later Milter
protocol stages, even when some of those stages are handled by a
different Postfix process.

Thus, when adding Milter support for a specific Sendmail-like macro    
to the SMTP server:

- We may have to update the XCLIENT protocol, so that Milter
  applications can be tested with XCLIENT. If not, then we must
  prominently document everywhere that XCLIENT does not provide
  100% accurate simulation for Milters. An additional complication
  is that the SMTP command length is limited, and that each XCLIENT
  command resets the SMTP server to the 220 stage and generates
  "connect" events for anvil(8) and for Milters.

- The SMTP server has to send the corresponding attribute to the
  cleanup server.  The cleanup server then stores the attribute in
  the queue file, so that Milters produce consistent results when
  mail is re-queued with "postsuper -r".

But wait, there is more. If mail is filtered by an external content
filter, then it needs to preserve all the Milter attributes so that
after "postsuper -r", Milters produce the exact same result as when
mail was received originally by Postfix. Specifically, after
"postsuper -r" a signing Milter must not sign mail that it did not
sign on the first pass through Postfix, and it must not reject mail
that it accepted on the first pass through Postfix.

Instead of trying to re-create the Milter execution environment
after "postsuper -r" we simply disable Milter processing. The
rationale for this is: if mail was Miltered before it was written
to queue file, then there is no need to Milter it again.

We might want to take a similar approach with external (signing or
blocking) content filters: don't filter mail that has already been
filtered, and don't filter mail that didn't need to be filtered.
Such mail can be recognized by the absence of a "content_filter"
record. To make the implementation efficient, the cleanup server
would have to record the presence of a "content_filter" record in
the queue file header.

Message envelope or content modifications
=========================================

Milters can send modification requests after receiving the end of
the message body.  If we can implement all the header/body-related
Milter operations in the cleanup server, then we can try to edit
the queue file in place, without ever having to make a temporary
copy. Once a Milter is done editing, the queue file can be used as
input for the next Milter, and so on. Finally, the cleanup server
calls fsync() and waits for successful return.

To implement in-place queue file edits, we need to introduce
surprisingly little change to the existing Postfix queue file
structure.  All we need is a way to specify a jump from one place
in the file to another.

Postfix does not store queue files as plain text files. Instead all
information is stored in records with an explicit type and length
for sender, recipient, arrival time, and so on.  Even the content
that makes up the message header and body is stored as records with
an explicit type and length.  This organization makes it very easy
to introduce pointer records, which is what we will use to jump
from one place in a queue file to another place.

- Deleting a recipient or header record is easy - just mark the
  record as killed.  When deleting a recipient, we must kill all
  recipient records that result from virtual alias expansion of the
  original recipient address. When deleting a very long header or
  body line, multiple queue file records may need to be killed. We
  won't try to reuse the deleted space for other purposes.

- Replacing header or body records involves pointer records.
  Basically, a record is replaced by overwriting it with a forward
  pointer to space after the end of the queue file, putting the new
  record there, followed by a reverse pointer to the record that
  follows the replaced information. If the replaced record is shorter
  than a pointer record, we relocate the records that follow it to
  the new area, until we have enough space for the forward pointer
  record. See below for a discussion on what it takes to make this
  safe.

  Postfix queue files are segmented. The first segment is for
  envelope records, the second for message header and body content,
  and the third segment is for information that was extracted or
  generated from the message header and body content.  Each segment
  is terminated by a marker record. For now we don't want to change
  their location. In particular, we want to avoid moving the start
  of a segment.

  To ensure that we can always replace a header or body record by
  a pointer record, without having to relocate a marker record, the
  cleanup server always places a dummy pointer record at the end
  of the headers and at the end of the body.

  When a Milter wants to replace an entire body, we have the option
  to overwrite existing body records until we run out of space, and
  then writing a pointer to space at the end of the queue file,
  followed by the remainder of the body, and a pointer to the marker
  that ends the message content segment.

- Appending a recipient or header record involves pointer records
  as well. This requires that the queue file already contains a
  dummy pointer record at the place where we want to append recipient
  or header content (Milters currently do not replace individual
  body records, but we could add this if need be).  To append,
  change the dummy pointer into a forward pointer to space after
  the end of a message, put the new record there, followed by a
  reverse pointer to the record that follows the forward pointer.

  To append another record, replace the reverse pointer by a forward
  pointer to space after the end of a message, put the new record
  there, followed by the value of the reverse pointer that we
  replace. Thus, there is no one-to-one correspondence between
  forward and backward pointers! In fact, there can be multiple
  forward pointers for one reverse pointer.

When relocating a record we must not relocate the target of a jump
==================================================================

As discussed above, when replacing an existing record, we overwrite
it with a forward pointer to the new information. If the old record
is too small we relocate one or more records that follow the record
that's being replaced, until we have enough space for the forward
pointer record.

Now we have to become really careful. Could we end up relocating a
record that is the target of a forward or reverse pointer, and thus
corrupt the queue file? The answer is NO.

- We never relocate end-of-segment marker records. Instead, the
  cleanup server writes dummy pointer records to guarantee that
  there is always space for a pointer.

- When a record is the target of a forward pointer, it is "edited"
  information that is preceded either by the end-of-queue-file
  marker record, or it is preceded by the reverse pointer at the
  end of earlier written "edited" information. Thus, the target of
  a forward pointer will not be relocated to make space for a pointer
  record.

- When a record is the target of a reverse pointer, it is always
  preceded by a forward pointer record (or by a forward pointer
  record followed by some unused space). Thus, the target of a
  reverse pointer will not be relocated to make space for a pointer
  record.

Could we end up relocating a pointer record?  Yes, but that is OK,
as long as pointers contain absolute offsets.

Pointer records introduce the possibility of loops
==================================================

When a queue file is damaged, a bogus pointer value may send Postfix
into a loop. This must not happen.

Detecting loops is not trivial:

- A sequence of multiple forward pointers may be followed by one
  legitimate reverse pointer to the location after the first forward
  pointer. See above for a discussion of how to append a record to
  an appended record.

- We do know, however, that there will not be more reverse pointers
  than forward pointers. But this does not help much.

Perhaps we can include a record count at the start of the queue
file, so that the record walking code knows that it's looking at
some records more than once, and return an error indication.

How many bytes do we need for a pointer record?
===============================================

A pointer record would look like this:

    type (1 byte)
    offset (see below)

Postfix uses long for queue file size/offset information, and stores
them as %15ld in the SIZE record at the start of the queue file.
This is somewhat less than a 64-bit long, but it is enough for a
some time to come, and it is easily changed without breaking forward
or backward compatibility.

It does mean, however, that a pointer record can easily exceed the
length of a header record. This is why we go through the trouble
of record relocation and dummy records. 

In Postfix 2.4 we fixed this by adding padding to short message
header records so that we can always write a pointer record over a
message header.  This immensly simplifies the code.