path: root/README_FILES/FILTER_README

PPoossttffiixx AAfftteerr--QQuueeuuee CCoonntteenntt FFiilltteerr

-------------------------------------------------------------------------------

IInnttrroodduuccttiioonn

This document requires Postfix version 2.1 or later.

Normally, Postfix receives mail, stores it in the mail queue and then delivers
it. With the external content filter described here, mail is filtered AFTER it
is queued. This approach decouples mail receiving processes from mail filtering
processes, and gives you maximal control over how many filtering processes you
are willing to run in parallel.

The after-queue content filter is meant to be used as follows:

    Network or  -> Postfix -> CCoonntteenntt -> Postfix ->  Network or
    local users     queue     ffiilltteerr      queue     local mailbox

This document describes implementations that use a single Postfix instance for
everything: receiving, filtering and delivering mail. Applications that use two
separate Postfix instances will be covered by a later version of this document.

The after-queue content filter is not to be confused with the approaches
described in the SMTPD_PROXY_README or MILTER_README documents, where incoming
SMTP mail is filtered BEFORE it is stored into the Postfix queue.

This document describes two approaches to content filter all email, as well as
several options to filter mail selectively:

  * Principles of operation
  * Simple content filter

      o Simple content filter example
      o Simple content filter performance
      o Simple content filter limitations
      o Turning off the simple content filter

  * Advanced content filter

      o Advanced content filter example
      o Advanced content filter performance
      o Turning off the advanced content filter

  * Selective content filtering

      o Filtering mail from outside users only
      o Different filters for different domains
      o FILTER actions in access or header/body tables

PPrriinncciipplleess ooff ooppeerraattiioonn

An after-queue content filter receives unfiltered mail from Postfix (as
described further below) and can do one of the following:

 1. Re-inject the mail back into Postfix, perhaps after changing content and/or
    destination.

 2. Discard or quarantine the mail.

 3. Reject the mail (by sending a suitable status code back to Postfix).
    Postfix will send the mail back to the sender address.

NOTE: in this time of mail worms and forged spam, it is a VERY BAD IDEA to send
viruses back to the sender address, because the sender address is almost
certainly not the originator. It is better to discard known viruses, and to
quarantine material that is suspect so that a human can decide what to do with
it.

SSiimmppllee ccoonntteenntt ffiilltteerr eexxaammppllee

The first example is simple to set up, but has major limitations that will be
addressed in a second example. Postfix receives unfiltered mail from the
network with the smtpd(8) server, and delivers unfiltered mail to a content
filter with the Postfix pipe(8) delivery agent. The content filter injects
filtered mail back into Postfix with the Postfix sendmail(1) command, so that
Postfix can deliver it to the final destination.

This means that mail submitted via the Postfix sendmail(1) command cannot be
content filtered.

In the figure below, names followed by a number represent Postfix commands or
daemon programs. See the OVERVIEW document for an introduction to the Postfix
architecture.

    Unfiltered -> smtpd(8)                   qmgr(8)     local(8) -> Filtered
                            >- cleanup(8) -> Postfix  -< smtp(8)  -> Filtered
                  pickup(8)                   queue      pipe(8)

                        ^                                     |
                        |                                     v

                  maildrop      Postfix      Postfix     Content
                    queue   <-  postdrop  <- sendmail <-  filter
                                  (1)          (1)

The content filter can be a simple shell script like this:

     1 #!/bin/sh
     2
     3 # Simple shell-based filter. It is meant to be invoked as follows:
     4 #       /path/to/script -f sender recipients...
     5
     6 # Localize these. The -G option does nothing before Postfix 2.3.
     7 INSPECT_DIR=/var/spool/filter
     8 SENDMAIL="/usr/sbin/sendmail -G -i" # NEVER NEVER NEVER use "-t" here.
     9
    10 # Exit codes from <sysexits.h>
    11 EX_TEMPFAIL=75
    12 EX_UNAVAILABLE=69
    13
    14 # Clean up when done or when aborting.
    15 trap "rm -f in.$$" 0 1 2 3 15
    16
    17 # Start processing.
    18 cd $INSPECT_DIR || {
    19     echo $INSPECT_DIR does not exist; exit $EX_TEMPFAIL; }
    20
    21 cat >in.$$ || {
    22     echo Cannot save mail to file; exit $EX_TEMPFAIL; }
    23
    24 # Specify your content filter here.
    25 # filter <in.$$ || {
    26 #   echo Message content rejected; exit $EX_UNAVAILABLE; }
    27
    28 $SENDMAIL "$@" <in.$$
    29
    30 exit $?

Notes:

  * Line 8: The -G option says the filter output is not a local mail
    submission: don't do silly things like appending the local domain name to
    addresses in message headers. This option does nothing before Postfix
    version 2.3.

  * Line 8: The -i option says don't stop reading input when a line contains
    "." only.

  * Line 8: NEVER NEVER NEVER use the "-t" command-line option here. It will
    mis-deliver mail, like sending messages from a mailing list back to the
    mailing list.

  * Line 21: The idea is to first capture the message to file and then run the
    content through a third-party content filter program.

  * Line 22: If the message cannot be captured to file, mail delivery is
    deferred by terminating with exit status 75 (EX_TEMPFAIL). Postfix places
    the message in the deferred mail queue and tries again later.

  * Line 25: You will need to specify a real content filter program here that
    receives the content on standard input.

  * Line 26: If the content filter program finds a problem, the mail is bounced
    by terminating with exit status 69 (EX_UNAVAILABLE). Postfix will send the
    message back to the sender as undeliverable mail.

  * NOTE: in this time of mail worms and spam, it is a BAD IDEA to send known
    viruses or spam back to the sender, because that address is likely to be
    forged. It is safer to discard known viruses and to quarantine suspicious
    content so that it can be inspected by a human being.

  * Line 28: If the content is OK, it is given as input to the Postfix sendmail
    command, and the exit status of the filter command is whatever exit status
    the Postfix sendmail command produces. Postfix will deliver the message as
    usual.

  * Line 30: Postfix returns the exit status of the Postfix sendmail command.

I suggest that you first run this script by hand until you are satisfied with
the results. Run it with a real message (headers+body) as input:

    % /path/to/script -f sender -- recipient... <message-file

Once you're satisfied with the content filtering script:

  * Create a dedicated local user account called "filter". This user handles
    all potentially dangerous mail content - that is why it should be a
    separate account. Do not use "nobody", and most certainly do not use "root"
    or "postfix".

  * Create a directory /var/spool/filter that is accessible only to the
    "filter" user. This is where the content filtering script is supposed to
    store its temporary files.

  * Configure Postfix to deliver mail to the content filter with the pipe(8)
    delivery agent (see the pipe(8) manpage for a description of the command
    syntax below).

    /etc/postfix/master.cf:
      # =============================================================
      # service type  private unpriv  chroot  wakeup  maxproc command
      #               (yes)   (yes)   (yes)   (never) (100)
      # =============================================================
      filter    unix  -       n       n       -       10      pipe
        flags=Rq user=filter null_sender=
        argv=/path/to/script -f ${sender} -- ${recipient}

    This runs up to 10 content filters in parallel. Instead of a limit of 10
    concurrent processes, use whatever process limit is feasible for your
    machine. Content inspection software can gobble up a lot of system
    resources, so you don't want to have too much of it running at the same
    time. The empty null_sender setting is required with Postfix 2.3 and later.

  * To turn on content filtering for mail arriving via SMTP only, append "-
    o content_filter=filter:dummy" to the master.cf entry that defines the
    Postfix SMTP server:

    /etc/postfix/master.cf:
      # =============================================================
      # service type  private unpriv  chroot  wakeup  maxproc command
      #               (yes)   (yes)   (yes)   (never) (100)
      # =============================================================
      smtp      inet  ...other stuff here, do not change...   smtpd
            -o content_filter=filter:dummy

    The "-o content_filter" line causes Postfix to add one content filter
    request record to each incoming mail message, with content "filter:dummy".
    This record overrides the normal mail routing and causes mail to be given
    to the content filter instead.

    The content_filter configuration parameter expects a value of the form
    transport:destination. The transport name specifies the first field of a
    mail delivery agent definition in master.cf; the syntax of the next-hop
    destination is described in the manual page of the corresponding delivery
    agent.

    The meaning of an empty next-hop filter destination is version dependent.
    Postfix 2.7 and later will use the recipient domain; earlier versions will
    use $myhostname. Specify "default_filter_nexthop = $myhostname" for
    compatibility with Postfix 2.6 or earlier, or specify a non-empty next-hop
    filter destination.

    The content_filter setting has lower precedence than a FILTER action that
    is specified in an access(5), header_checks(5) or body_checks(5) table.

  * Execute "ppoossttffiixx rreellooaadd" to complete the change.

SSiimmppllee ccoonntteenntt ffiilltteerr ppeerrffoorrmmaannccee

With the shell script as shown above you will lose a factor of four in Postfix
performance for transit mail that arrives and leaves via SMTP. You will lose
another factor in transit performance for each additional temporary file that
is created and deleted in the process of content filtering. The performance
impact is less for mail that is submitted or delivered locally, because such
deliveries are already slower than SMTP transit mail.

SSiimmppllee ccoonntteenntt ffiilltteerr lliimmiittaattiioonnss

The problem with content filters like the one above is that they are not very
robust. The reason is that the software does not talk a well-defined protocol
with Postfix. If the filter shell script aborts because the shell runs into
some memory allocation problem, the script will not produce a nice exit status
as defined in the file /usr/include/sysexits.h. Instead of going to the
deferred queue, mail will bounce. The same lack of robustness can happen when
the content filtering software itself runs into a resource problem.

The simple content filter method is not suitable for content filter actions
that are invoked via header_checks or body_checks patterns. These patterns will
be applied again after mail is re-injected with the Postfix sendmail command,
resulting in a mail filtering loop. The advanced content filtering method (see
below) makes it possible to turn off header_checks or body_checks patterns for
filtered mail.

TTuurrnniinngg ooffff tthhee ssiimmppllee ccoonntteenntt ffiilltteerr

To turn off "simple" content filtering:

  * Edit the master.cf file, remove the "-o content_filter=filter:dummy" text
    from the entry that defines the Postfix SMTP server.

  * Execute "ppoossttssuuppeerr --rr AALLLL" to remove content filter request records from
    existing queue files.

  * Execute another "ppoossttffiixx rreellooaadd".

AAddvvaanncceedd ccoonntteenntt ffiilltteerr eexxaammppllee

The second example is more complex, but can give better performance, and is
less likely to bounce mail when the machine runs into some resource problem.
This content filter receives unfiltered mail with SMTP on localhost port 10025,
and sends filtered mail back into Postfix with SMTP on localhost port 10026.

For non-SMTP capable content filtering software, Bennett Todd's SMTP proxy
implements a nice PERL/SMTP content filtering framework. See: https://
web.archive.org/web/20151022025756/http://bent.latency.net/smtpprox/.

In the figure below, names followed by a number represent Postfix commands or
daemon programs. See the OVERVIEW document for an introduction to the Postfix
architecture.

    Unfiltered -> smtpd(8)                   qmgr(8)     smtp(8)  -> Filtered
                            >- cleanup(8) -> Postfix  -<
    Unfiltered -> pickup(8)                   queue      local(8) -> Filtered

                                     ^            |
                                     |            v

                                smtpd(8)     smtp(8)
                                 10026

                                     ^            |
                                     |            v

                                content filter 10025

The example given here filters all mail, including mail that arrives via SMTP
and mail that is locally submitted via the Postfix sendmail command (local
submissions enter Postfix via the pickup(8) server; to keep the figure simple
we omit local submission details). See examples near the end of this document
for how to exclude local users from filtering, or how to configure a
destination dependent content filter.

You can expect to lose about a factor of two in Postfix performance for mail
that arrives and leaves via SMTP, provided that the content filter creates no
temporary files. Each temporary file created by the content filter adds another
factor to the performance loss.

AAddvvaanncceedd ccoonntteenntt ffiilltteerr:: rreeqquueessttiinngg tthhaatt aallll mmaaiill iiss ffiilltteerreedd

To enable the advanced content filter method for all mail, specify in main.cf:

    /etc/postfix/main.cf:
        content_filter = scan:localhost:10025
        receive_override_options = no_address_mappings

  * The "receive_override_options" line disables address manipulation before
    the content filter, so that the content filter sees the original mail
    addresses instead of the result of virtual alias expansion, canonical
    mapping, automatic bcc, address masquerading, etc.

  * The "content_filter" line causes Postfix to add one content filter request
    record to each incoming mail message, with content "scan:localhost:10025".
    The content filter request records are added by the smtpd(8) and pickup(8)
    servers (and qmqpd(8) if you decide to enable this service).

  * Content filter requests are stored in queue files; this is how Postfix
    keeps track of what mail needs filtering. When a queue file contains a
    content filter request, the queue manager will deliver the mail to the
    specified content filter regardless of its final destination.

  * The content_filter configuration parameter expects a value of the form
    transport:destination. The transport name specifies the first field of a
    mail delivery agent definition in master.cf; the syntax of the next-hop
    destination is described in the manual page of the corresponding delivery
    agent.

  * The meaning of an empty next-hop filter destination is version dependent.
    Postfix 2.7 and later will use the recipient domain; earlier versions will
    use $myhostname. Specify "default_filter_nexthop = $myhostname" for
    compatibility with Postfix 2.6 or earlier, or specify a non-empty next-hop
    filter destination.

  * The content_filter setting has lower precedence than a FILTER action that
    is specified in an access(5), header_checks(5) or body_checks(5) table.

AAddvvaanncceedd ccoonntteenntt ffiilltteerr:: sseennddiinngg uunnffiilltteerreedd mmaaiill ttoo tthhee ccoonntteenntt ffiilltteerr

In this example, "scan" is an instance of the Postfix SMTP client with slightly
different configuration parameters. This is how one would set up the service in
the Postfix master.cf file:

    /etc/postfix/master.cf:
        # =============================================================
        # service type  private unpriv  chroot  wakeup  maxproc command
        #               (yes)   (yes)   (yes)   (never) (100)
        # =============================================================
        scan      unix  -       -       n       -       10      smtp
            -o smtp_send_xforward_command=yes
            -o disable_mime_output_conversion=yes
            -o smtp_generic_maps=

  * This runs up to 10 content filters in parallel. Instead of a limit of 10
    concurrent processes, use whatever process limit is feasible for your
    machine. Content inspection software can gobble up a lot of system
    resources, so you don't want to have too much of it running at the same
    time.

  * With "-o smtp_send_xforward_command=yes", the scan transport will try to
    forward the original client name and IP address through the content filter
    to the after-filter smtpd process, so that filtered mail is logged with the
    real client name IP address. See smtp(8) and XFORWARD_README for more
    information.

  * The "-o disable_mime_output_conversion=yes" is a workaround that prevents
    the breaking of domainkeys and other digital signatures. This is needed
    because some SMTP-based content filters don't announce 8BITMIME support,
    even though they can handle 8-bit mail.

  * The "-o smtp_generic_maps=" is a workaround that prevents local address
    rewriting with generic(5) maps. Such rewriting should happen only when mail
    is sent out to the Internet.

AAddvvaanncceedd ccoonntteenntt ffiilltteerr:: rruunnnniinngg tthhee ccoonntteenntt ffiilltteerr

The content filter can be set up with the Postfix spawn service, which is the
Postfix equivalent of inetd. For example, to instantiate up to 10 content
filtering processes on localhost port 10025:

    /etc/postfix/master.cf:
        # ===================================================================
        # service       type  private unpriv  chroot  wakeup  maxproc command
        #                     (yes)   (yes)   (yes)   (never) (100)
        # ===================================================================
        localhost:10025 inet  n       n       n       -       10      spawn
            user=filter argv=/path/to/filter localhost 10026

  * "filter" is a dedicated local user account. The user will never log in, and
    can be given a "*" password and non-existent shell and home directory. This
    user handles all potentially dangerous mail content - that is why it should
    be a separate account.

  * By default, Postfix will terminate a command that runs longer than
    command_time_limit seconds (default: 1000s). This is a safety measure that
    prevents filters from running forever.

If you want to have your filter listening on port localhost:10025 instead of
Postfix, then you must run your filter as a stand-alone program, and must not
use the Postfix spawn service.

AAddvvaanncceedd ffiilltteerr:: iinnjjeeccttiinngg mmaaiill bbaacckk iinnttoo PPoossttffiixx

The job of the content filter is to either bounce mail with a suitable
diagnostic, or to feed the mail back into Postfix through a dedicated listener
on port localhost 10026.

The simplest content filter just copies SMTP commands and data between its
inputs and outputs. If it has a problem, all it has to do is to reply to an
input of `.' from Postfix with `550 content rejected', and to disconnect
without sending `.' on the connection that injects mail back into Postfix.

    /etc/postfix/master.cf:
        # ===================================================================
        # service       type  private unpriv  chroot  wakeup  maxproc command
        #                     (yes)   (yes)   (yes)   (never) (100)
        # ===================================================================
        localhost:10026 inet  n       -       n       -       10      smtpd
            -o content_filter=
            -
    o
    receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
            -o smtpd_helo_restrictions=
            -o smtpd_client_restrictions=
            -o smtpd_sender_restrictions=
            # Postfix 2.10 and later: specify empty smtpd_relay_restrictions.
            -o smtpd_relay_restrictions=
            -o smtpd_recipient_restrictions=permit_mynetworks,reject
            -o mynetworks=127.0.0.0/8
            -o smtpd_authorized_xforward_hosts=127.0.0.0/8

  * NOTE: do not use spaces around the "=" or "," characters.

  * NOTE: the SMTP server must not have a smaller process limit than the
    "filter" master.cf entry.

  * The "-o content_filter=" overrides main.cf settings, and requests no
    content filtering for mail from the content filter. This is required or
    else mail will loop.

  * The "-o receive_override_options" overrides main.cf settings to avoid
    duplicating work that was already done before the content filter. These
    options are complementary to the options that are specified in main.cf:

      o We specify "no_unknown_recipient_checks" to disable attempts to find
        out if a recipient is unknown.

      o We specify "no_header_body_checks" to disable header/body checks.

      o We specify "no_milters" to disable Milter applications (this option is
        available only in Postfix 2.3 and later).

      o We don't specify "no_address_mappings" here. This enables virtual alias
        expansion, canonical mappings, address masquerading, and other address
        mappings after the content filter. The main.cf setting of
        "receive_override_options" disables these mappings before the content
        filter.

    These receive override options are either implemented by the SMTP server
    itself, or they are passed on to the cleanup server.

  * The "-o smtpd_xxx_restrictions" and "-o mynetworks=127.0.0.0/8" override
    main.cf settings. They turn off junk mail controls that would only waste
    time here.

  * With "-o smtpd_authorized_xforward_hosts=127.0.0.0/8", the scan transport
    will try to forward the original client name and IP address to the after-
    filter smtpd process, so that filtered mail is logged with the real client
    name and IP address. See XFORWARD_README and smtpd(8).

AAddvvaanncceedd ccoonntteenntt ffiilltteerr ppeerrffoorrmmaannccee

With the "sandwich" approach to content filtering described here, it is
important to match the filter concurrency to the available CPU, memory and I/
O resources. Too few content filter processes and mail accumulates in the
active queue even with low traffic volume; too much concurrency and Postfix
ends up deferring mail destined for the content filter because processes fail
due to insufficient resources.

Currently, content filter performance tuning is a process of trial and error;
analysis is handicapped because filtered and unfiltered messages share the same
queue. As mentioned in the introduction of this document, content filtering
with multiple Postfix instances will be covered in a future version.

TTuurrnniinngg ooffff tthhee aaddvvaanncceedd ccoonntteenntt ffiilltteerr

To turn off "advanced" content filtering:

  * Delete or comment out the two following main.cf lines. The other changes
    made for advanced content filtering have no effect when content filtering
    is turned off.

        /etc/postfix/main.cf:
            content_filter = scan:localhost:10025
            receive_override_options = no_address_mappings

  * Execute "ppoossttssuuppeerr --rr AALLLL" to remove content filter request records from
    existing queue files.

  * Execute another "ppoossttffiixx rreellooaadd".

FFiilltteerriinngg mmaaiill ffrroomm oouuttssiiddee uusseerrss oonnllyy

The easiest approach is to configure ONE Postfix instance with multiple SMTP
server IP addresses in master.cf:

  * Two SMTP server IP addresses for mail from inside users only, with content
    filtering turned off.

    /etc/postfix.master.cf:
        # ==================================================================
        # service      type  private unpriv  chroot  wakeup  maxproc command
        #                    (yes)   (yes)   (yes)   (never) (100)
        # ==================================================================
        1.2.3.4:smtp   inet  n       -       n       -       -       smtpd
            -o smtpd_client_restrictions=permit_mynetworks,reject
        127.0.0.1:smtp inet  n       -       n       -       -       smtpd
            -o smtpd_client_restrictions=permit_mynetworks,reject

  * One SMTP server address for mail from outside users with content filtering
    turned on.

    /etc/postfix.master.cf:
        # =================================================================
        # service     type  private unpriv  chroot  wakeup  maxproc command
        #                   (yes)   (yes)   (yes)   (never) (100)
        # =================================================================
        1.2.3.5:smtp  inet  n       -       n       -       -       smtpd
            -o content_filter=filter-service:filter-destination
            -o receive_override_options=no_address_mappings

After this, you can follow the same procedure as outlined in the "advanced" or
"simple" content filtering examples above, except that you must not specify
"content_filter" or "receive_override_options" in the main.cf file.

DDiiffffeerreenntt ffiilltteerrss ffoorr ddiiffffeerreenntt ddoommaaiinnss

If you are an MX service provider and want to apply different content filters
for different domains, you can configure ONE Postfix instance with multiple
SMTP server IP addresses in master.cf. Each address provides a different
content filter service.

    /etc/postfix.master.cf:
        # =================================================================
        # service     type  private unpriv  chroot  wakeup  maxproc command
        #                   (yes)   (yes)   (yes)   (never) (100)
        # =================================================================
        # SMTP service for domains that are filtered with service1:dest1
        1.2.3.4:smtp  inet  n       -       n       -       -       smtpd
            -o content_filter=service1:dest1
            -o receive_override_options=no_address_mappings

        # SMTP service for domains that are filtered with service2:dest2
        1.2.3.5:smtp  inet  n       -       n       -       -       smtpd
            -o content_filter=service2:dest2
            -o receive_override_options=no_address_mappings

After this, you can follow the same procedure as outlined in the "advanced" or
"simple" content filtering examples above, except that you must not specify
"content_filter" or "receive_override_options" in the main.cf file.

Set up MX records in the DNS that route each domain to the proper SMTP server
instance.

FFIILLTTEERR aaccttiioonnss iinn aacccceessss oorr hheeaaddeerr//bbooddyy ttaabblleess

The above filtering configurations are static. Mail that follows a given path
is either always filtered or it is never filtered. As of Postfix 2.0 you can
also turn on content filtering on the fly.

To turn on content filtering with an access(5) table rule:

    /etc/postfix/access:
        whatever       FILTER foo:bar

To turn on content filtering with a header_checks(5) or body_checks(5) table
pattern:

    /etc/postfix/header_checks:
        /whatever/     FILTER foo:bar

You can do this in smtpd access maps as well as the cleanup server's header/
body_checks. This feature must be used with great care: you must disable all
the UCE features in the after-filter smtpd and cleanup daemons or else you will
have a content filtering loop.

Limitations:

  * FILTER actions from smtpd access maps and header/body_checks take
    precedence over filters specified with the main.cf content_filter
    parameter.

  * If a message triggers more than one filter action, only the last one takes
    effect.

  * The same content filter is applied to all the recipients of a given
    message.