path: root/README_FILES/BUILTIN_FILTER_README

 PPoossttffiixx BBuuiilltt--iinn CCoonntteenntt IInnssppeeccttiioonn

-------------------------------------------------------------------------------

BBuuiilltt--iinn ccoonntteenntt iinnssppeeccttiioonn iinnttrroodduuccttiioonn

Postfix supports a built-in filter mechanism that examines message header and
message body content, one line at a time, before it is stored in the Postfix
queue. The filter is usually implemented with POSIX or PCRE regular
expressions, as described in the header_checks(5) manual page.

The original purpose of the built-in filter is to stop an outbreak of specific
email worms or viruses, and it does this job well. The filter has also helped
to block bounced junk email, bounced email from worms or viruses, and
notifications from virus detection systems. Information about this secondary
application is given in the BACKSCATTER_README document.

Because the built-in filter is optimized for stopping specific worms and virus
outbreaks, it has limitations that make it NOT suitable for general junk email
and virus detection. For that, you should use one of the external content
inspection methods that are described in the FILTER_README, SMTPD_PROXY_README
and MILTER_README documents.

The following diagram gives an over-all picture of how Postfix built-in content
inspection works:

                                Postmaster
                               notifications

                                        |
                                        v

    Network or  -> BBuuiilltt--iinn ->     Postfix   -> Delivery ->  Network or
    local users     ffiilltteerr          queue        agents     local mailbox

                                        ^            |
                                        |            v

                                    Undeliverable mail
                                      Forwarded mail

The picture makes clear that the filter works while Postfix is receiving new
mail. This means that Postfix can reject mail from the network without having
to return undeliverable mail to the originator address (which is often spoofed
anyway). However, this ability comes at a price: if mail inspection takes too
much time, then the remote client will time out, and the client may send the
same message repeatedly.

Topics covered by this document:

  * What mail is subjected to header/body checks
  * Limitations of Postfix header/body checks
  * Preventing daily mail status reports from being blocked
  * Configuring header/body checks for mail from outside users only
  * Configuring different header/body checks for MX service and submission
    service
  * Configuring header/body checks for mail to some domains only

WWhhaatt mmaaiill iiss ssuubbjjeecctteedd ttoo hheeaaddeerr//bbooddyy cchheecckkss

Postfix header/body checks are implemented by the cleanup(8) server before it
injects mail into the incoming queue. The diagram below zooms in on the cleanup
(8) server, and shows that this server handles mail from many different
sources. In order to keep the diagram readable, the sources of postmaster
notifications are not shown, because they can be produced by many Postfix
daemon processes.

                    bounce(8)
                 (undeliverable)

    ssmmttppdd((88))               |
    ((nneettwwoorrkk)) \            v

    qqmmqqppdd((88))  -\    cleanup(8)   -> incoming
    ((nneettwwoorrkk)) -/                     queue

    ppiicckkuupp((88)) /            ^
     ((llooccaall))               |

                     local(8)
                    (forwarded)

For efficiency reasons, only mail that enters from outside of Postfix is
inspected with header/body checks. It would be inefficient to filter already
filtered mail again, and it would be undesirable to block postmaster
notifications. The table below summarizes what mail is and is not subject to
header/body checks.

     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    |MMeessssaaggee ttyyppee      |SSoouurrccee   |HHeeaaddeerr//bbooddyy cchheecckkss??|
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
    |Undeliverable mail|bounce(8)|No                 |
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
    |Network mail      |smtpd(8) |Configurable       |
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
    |Network mail      |qmqpd(8) |Configurable       |
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
    |Local submission  |pickup(8)|Configurable       |
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
    |Local forwarding  |local(8) |No                 |
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
    |Postmaster notice |many     |No                 |
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |

How does Postfix decide what mail needs to be filtered? It would be clumsy to
make the decision in the cleanup(8) server, as this program receives mail from
so many different sources. Instead, header/body checks are requested by the
source. Examples of how to turn off header/body checks for mail received with
smtpd(8), qmqpd(8) or pickup(8) are given below under "Configuring header/body
checks for mail from outside users only", "Configuring different header/body
checks for MX service and submission service", and "Configuring header/body
checks for mail to some domains only".

LLiimmiittaattiioonnss ooff PPoossttffiixx hheeaaddeerr//bbooddyy cchheecckkss

  * Header/body checks do not decode message headers or message body content.
    For example, if text in the message body is BASE64 encoded (RFC 2045) then
    your regular expressions will have to match the BASE64 encoded form.
    Likewise, message headers with encoded non-ASCII characters (RFC 2047) need
    to be matched in their encoded form.

  * Header/body checks cannot filter on a combination of message headers or
    body lines. Header/body checks examine content one message header at a
    time, or one message body line at a time, and cannot carry a decision over
    to the next message header or body line.

  * Header/body checks cannot depend on the recipient of a message.

      o One message can have multiple recipients, and all recipients of a
        message receive the same treatment. Workarounds have been proposed that
        involve selectively deferring some recipients of multi-recipient mail,
        but that results in poor SMTP performance and does not work for non-
        SMTP mail.

      o Some sources of mail send the headers and content ahead of the
        recipient information. It would be inefficient to buffer up an entire
        message before deciding if it needs to be filtered, and it would be
        clumsy to filter mail and to buffer up all the actions until it is
        known whether those actions need to be executed.

  * Despite warnings, some people try to use the built-in filter feature for
    general junk email and/or virus blocking, using hundreds or even thousands
    of regular expressions. This can result in catastrophic performance
    failure. The symptoms are as follows:

      o The cleanup(8) processes use up all available CPU time in order to
        process the regular expressions, and/or they use up all available
        memory so that the system begins to swap. This slows down all incoming
        mail deliveries.

      o As Postfix needs more and more time to receive an email message, the
        number of simultaneous SMTP sessions increases to the point that the
        SMTP server process limit is reached.

      o While all SMTP server processes are waiting for the cleanup(8) servers
        to finish, new SMTP clients have to wait until an SMTP server process
        becomes available. This causes mail deliveries to time out before they
        have even begun.

    The remedy for this type of performance problem is simple: don't use
    header/body checks for general junk email and/or virus blocking, and don't
    filter mail before it is queued. When performance is a concern, use an
    external content filter that runs after mail is queued, as described in the
    FILTER_README document.

PPrreevveennttiinngg ddaaiillyy mmaaiill ssttaattuuss rreeppoorrttss ffrroomm bbeeiinngg bblloocckkeedd

The following is quoted from Jim Seymour's Pflogsumm FAQ at http://
jimsun.linxnet.com/downloads/pflogsumm-faq.txt. Pflogsumm is a program that
analyzes Postfix logs, including the logging from rejected mail. If these logs
contain text that was rejected by Postfix body_checks patterns, then the
logging is also likely to be rejected by those same body_checks patterns. This
problem does not exist with header_checks patterns, because those are not
applied to the text that is part of the mail status report.

    You configure Postfix to do body checks, Postfix does its thing, Pflogsumm
    reports it and Postfix catches the same string in the Pflogsumm report.
    There are several solutions to this.

    Wolfgang Zeikat contributed this:

        #!/usr/bin/perl
        use MIME::Lite;

        ### Create a new message:
        $msg = MIME::Lite->new(
            From     => 'your@send.er',
            To       => 'your@recipie.nt',
            # Cc     => 'some@other.com, some@more.com',
            Subject  => 'pflogsumm',
            Date     => `date`,
            Type     => 'text/plain',
            Encoding => 'base64',
            Path     => '/tmp/pflogg',
        );

        $msg->send;

    Where "/tmp/pflogg" is the output of Pflogsumm. This puts Pflogsumm's
    output in a base64 MIME attachment.

Note by Wietse: if you run this on a machine that is accessible by untrusted
users, it is safer to store the Pflogsumm report in a directory that is not
world writable.

    In a follow-up to a thread in the postfix-users mailing list, Ralf
    Hildebrandt noted:

        "mpack does the same thing."

And it does. Which tool one should use is a matter of preference.

Other solutions involve additional body_checks rules that make exceptions for
daily mail status reports, but this is not recommended. Such rules slow down
all mail and complicate Postfix maintenance.

CCoonnffiigguurriinngg hheeaaddeerr//bbooddyy cchheecckkss ffoorr mmaaiill ffrroomm oouuttssiiddee uusseerrss oonnllyy

The following information applies to Postfix 2.1 and later. Earlier Postfix
versions do not support the receive_override_options feature.

The easiest approach is to configure ONE Postfix instance with multiple SMTP
server IP addresses in master.cf:

  * Two SMTP server IP addresses for mail from inside users only, with header/
    body filtering turned off, and a local mail pickup service with header/body
    filtering turned off.

    /etc/postfix.master.cf:
        # ==================================================================
        # service      type  private unpriv  chroot  wakeup  maxproc command
        #                    (yes)   (yes)   (yes)   (never) (100)
        # ==================================================================
        1.2.3.4:smtp   inet  n       -       n       -       -       smtpd
            -o receive_override_options=no_header_body_checks
        127.0.0.1:smtp inet  n       -       n       -       -       smtpd
            -o receive_override_options=no_header_body_checks
        pickup         fifo  n       -       n       60      1       pickup
            -o receive_override_options=no_header_body_checks

  * Add some firewall rule to prevent access to 1.2.3.4:smtp from the outside
    world.

  * One SMTP server address for mail from outside users with header/body
    filtering turned on via main.cf.

    /etc/postfix.master.cf:
        # =================================================================
        # service     type  private unpriv  chroot  wakeup  maxproc command
        #                   (yes)   (yes)   (yes)   (never) (100)
        # =================================================================
        1.2.3.5:smtp  inet  n       -       n       -       -       smtpd

CCoonnffiigguurriinngg ddiiffffeerreenntt hheeaaddeerr//bbooddyy cchheecckkss ffoorr MMXX sseerrvviiccee aanndd ssuubbmmiissssiioonn sseerrvviiccee

If authorized user submissions require different header/body checks than mail
from remote MTAs, then this is possible as long as you have separate mail
streams for authorized users and for MX service.

The example below assumes that authorized users connect to TCP port 587
(submission) or 465 (smtps), and that remote MTAs connect to TCP port 25
(smtp).

First, we define a few "user-defined" parameters that will override settings
for the submission and smtps services.

    /etc/postfix/main.cf:
        msa_cleanup_service_name = msa_cleanup
        msa_header_checks = pcre:/etc/postfix/msa_header_checks
        msa_body_checks = pcre:/etc/postfix/msa_body_checks

Next, we define msa_cleanup as a dedicated cleanup service that will be used
only by the submission and smtps services. This service uses the header_checks
and body_checks overrides that were defined above.

    /etc/postfix.master.cf:
        # =================================================================
        # service     type  private unpriv  chroot  wakeup  maxproc command
        #                   (yes)   (yes)   (yes)   (never) (100)
        # =================================================================
        smtp          inet  n       -       n       -       -       smtpd
        msa_cleanup   unix  n       -       n       -       0       cleanup
            -o header_checks=$msa_header_checks
            -o body_checks=$msa_body_checks
        submission    inet  n       -       n       -       -       smtpd
            -o cleanup_service_name=$msa_cleanup_service_name
            -o syslog_name=postfix/submission
            ...[see sample master.cf file for more]...
        smtps         inet  n       -       n       -       -       smtpd
            -o cleanup_service_name=$msa_cleanup_service_name
            -o syslog_name=postfix/smtps
            -o smtpd_tls_wrappermode=yes
            ...[see sample master.cf file for more]...

By keeping the "msa_xxx" parameter settings in main.cf, you keep your master.cf
file simple, and you minimize the amount of duplication.

CCoonnffiigguurriinngg hheeaaddeerr//bbooddyy cchheecckkss ffoorr mmaaiill ttoo ssoommee ddoommaaiinnss oonnllyy

The following information applies to Postfix 2.1. Earlier Postfix versions do
not support the receive_override_options feature.

If you are an MX service provider and want to enable header/body checks only
for some domains, you can configure ONE Postfix instance with multiple SMTP
server IP addresses in master.cf. Each address provides a different service.

    /etc/postfix.master.cf:
        # =================================================================
        # service     type  private unpriv  chroot  wakeup  maxproc command
        #                   (yes)   (yes)   (yes)   (never) (100)
        # =================================================================
        # SMTP service for domains with header/body checks turned on.
        1.2.3.4:smtp  inet  n       -       n       -       -       smtpd

        # SMTP service for domains with header/body checks turned off.
        1.2.3.5:smtp  inet  n       -       n       -       -       smtpd
            -o receive_override_options=no_header_body_checks

Once this is set up you can configure MX records in the DNS that route each
domain to the proper SMTP server instance.