diff options
Diffstat (limited to 'README_FILES/FILTER_README')
| -rw-r--r-- | README_FILES/FILTER_README | 617 |
1 files changed, 617 insertions, 0 deletions
diff --git a/README_FILES/FILTER_README b/README_FILES/FILTER_README new file mode 100644 index 0000000..4a76bb9 --- /dev/null +++ b/README_FILES/FILTER_README @@ -0,0 +1,617 @@ +PPoossttffiixx AAfftteerr--QQuueeuuee CCoonntteenntt FFiilltteerr + +------------------------------------------------------------------------------- + +IInnttrroodduuccttiioonn + +This document requires Postfix version 2.1 or later. + +Normally, Postfix receives mail, stores it in the mail queue and then delivers +it. With the external content filter described here, mail is filtered AFTER it +is queued. This approach decouples mail receiving processes from mail filtering +processes, and gives you maximal control over how many filtering processes you +are willing to run in parallel. + +The after-queue content filter is meant to be used as follows: + + Network or -> Postfix -> CCoonntteenntt -> Postfix -> Network or + local users queue ffiilltteerr queue local mailbox + +This document describes implementations that use a single Postfix instance for +everything: receiving, filtering and delivering mail. Applications that use two +separate Postfix instances will be covered by a later version of this document. + +The after-queue content filter is not to be confused with the approaches +described in the SMTPD_PROXY_README or MILTER_README documents, where incoming +SMTP mail is filtered BEFORE it is stored into the Postfix queue. + +This document describes two approaches to content filter all email, as well as +several options to filter mail selectively: + + * Principles of operation + * Simple content filter + + o Simple content filter example + o Simple content filter performance + o Simple content filter limitations + o Turning off the simple content filter + + * Advanced content filter + + o Advanced content filter example + o Advanced content filter performance + o Turning off the advanced content filter + + * Selective content filtering + + o Filtering mail from outside users only + o Different filters for different domains + o FILTER actions in access or header/body tables + +PPrriinncciipplleess ooff ooppeerraattiioonn + +An after-queue content filter receives unfiltered mail from Postfix (as +described further below) and can do one of the following: + + 1. Re-inject the mail back into Postfix, perhaps after changing content and/or + destination. + + 2. Discard or quarantine the mail. + + 3. Reject the mail (by sending a suitable status code back to Postfix). + Postfix will send the mail back to the sender address. + +NOTE: in this time of mail worms and forged spam, it is a VERY BAD IDEA to send +viruses back to the sender address, because the sender address is almost +certainly not the originator. It is better to discard known viruses, and to +quarantine material that is suspect so that a human can decide what to do with +it. + +SSiimmppllee ccoonntteenntt ffiilltteerr eexxaammppllee + +The first example is simple to set up, but has major limitations that will be +addressed in a second example. Postfix receives unfiltered mail from the +network with the smtpd(8) server, and delivers unfiltered mail to a content +filter with the Postfix pipe(8) delivery agent. The content filter injects +filtered mail back into Postfix with the Postfix sendmail(1) command, so that +Postfix can deliver it to the final destination. + +This means that mail submitted via the Postfix sendmail(1) command cannot be +content filtered. + +In the figure below, names followed by a number represent Postfix commands or +daemon programs. See the OVERVIEW document for an introduction to the Postfix +architecture. + + Unfiltered -> smtpd(8) qmgr(8) local(8) -> Filtered + >- cleanup(8) -> Postfix -< smtp(8) -> Filtered + pickup(8) queue pipe(8) + + ^ | + | v + + maildrop Postfix Postfix Content + queue <- postdrop <- sendmail <- filter + (1) (1) + +The content filter can be a simple shell script like this: + + 1 #!/bin/sh + 2 + 3 # Simple shell-based filter. It is meant to be invoked as follows: + 4 # /path/to/script -f sender recipients... + 5 + 6 # Localize these. The -G option does nothing before Postfix 2.3. + 7 INSPECT_DIR=/var/spool/filter + 8 SENDMAIL="/usr/sbin/sendmail -G -i" # NEVER NEVER NEVER use "-t" here. + 9 + 10 # Exit codes from <sysexits.h> + 11 EX_TEMPFAIL=75 + 12 EX_UNAVAILABLE=69 + 13 + 14 # Clean up when done or when aborting. + 15 trap "rm -f in.$$" 0 1 2 3 15 + 16 + 17 # Start processing. + 18 cd $INSPECT_DIR || { + 19 echo $INSPECT_DIR does not exist; exit $EX_TEMPFAIL; } + 20 + 21 cat >in.$$ || { + 22 echo Cannot save mail to file; exit $EX_TEMPFAIL; } + 23 + 24 # Specify your content filter here. + 25 # filter <in.$$ || { + 26 # echo Message content rejected; exit $EX_UNAVAILABLE; } + 27 + 28 $SENDMAIL "$@" <in.$$ + 29 + 30 exit $? + +Notes: + + * Line 8: The -G option says the filter output is not a local mail + submission: don't do silly things like appending the local domain name to + addresses in message headers. This option does nothing before Postfix + version 2.3. + + * Line 8: The -i option says don't stop reading input when a line contains + "." only. + + * Line 8: NEVER NEVER NEVER use the "-t" command-line option here. It will + mis-deliver mail, like sending messages from a mailing list back to the + mailing list. + + * Line 21: The idea is to first capture the message to file and then run the + content through a third-party content filter program. + + * Line 22: If the message cannot be captured to file, mail delivery is + deferred by terminating with exit status 75 (EX_TEMPFAIL). Postfix places + the message in the deferred mail queue and tries again later. + + * Line 25: You will need to specify a real content filter program here that + receives the content on standard input. + + * Line 26: If the content filter program finds a problem, the mail is bounced + by terminating with exit status 69 (EX_UNAVAILABLE). Postfix will send the + message back to the sender as undeliverable mail. + + * NOTE: in this time of mail worms and spam, it is a BAD IDEA to send known + viruses or spam back to the sender, because that address is likely to be + forged. It is safer to discard known viruses and to quarantine suspicious + content so that it can be inspected by a human being. + + * Line 28: If the content is OK, it is given as input to the Postfix sendmail + command, and the exit status of the filter command is whatever exit status + the Postfix sendmail command produces. Postfix will deliver the message as + usual. + + * Line 30: Postfix returns the exit status of the Postfix sendmail command. + +I suggest that you first run this script by hand until you are satisfied with +the results. Run it with a real message (headers+body) as input: + + % /path/to/script -f sender -- recipient... <message-file + +Once you're satisfied with the content filtering script: + + * Create a dedicated local user account called "filter". This user handles + all potentially dangerous mail content - that is why it should be a + separate account. Do not use "nobody", and most certainly do not use "root" + or "postfix". + + * Create a directory /var/spool/filter that is accessible only to the + "filter" user. This is where the content filtering script is supposed to + store its temporary files. + + * Configure Postfix to deliver mail to the content filter with the pipe(8) + delivery agent (see the pipe(8) manpage for a description of the command + syntax below). + + /etc/postfix/master.cf: + # ============================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ============================================================= + filter unix - n n - 10 pipe + flags=Rq user=filter null_sender= + argv=/path/to/script -f ${sender} -- ${recipient} + + This runs up to 10 content filters in parallel. Instead of a limit of 10 + concurrent processes, use whatever process limit is feasible for your + machine. Content inspection software can gobble up a lot of system + resources, so you don't want to have too much of it running at the same + time. The empty null_sender setting is required with Postfix 2.3 and later. + + * To turn on content filtering for mail arriving via SMTP only, append "- + o content_filter=filter:dummy" to the master.cf entry that defines the + Postfix SMTP server: + + /etc/postfix/master.cf: + # ============================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ============================================================= + smtp inet ...other stuff here, do not change... smtpd + -o content_filter=filter:dummy + + The "-o content_filter" line causes Postfix to add one content filter + request record to each incoming mail message, with content "filter:dummy". + This record overrides the normal mail routing and causes mail to be given + to the content filter instead. + + The content_filter configuration parameter expects a value of the form + transport:destination. The transport name specifies the first field of a + mail delivery agent definition in master.cf; the syntax of the next-hop + destination is described in the manual page of the corresponding delivery + agent. + + The meaning of an empty next-hop filter destination is version dependent. + Postfix 2.7 and later will use the recipient domain; earlier versions will + use $myhostname. Specify "default_filter_nexthop = $myhostname" for + compatibility with Postfix 2.6 or earlier, or specify a non-empty next-hop + filter destination. + + The content_filter setting has lower precedence than a FILTER action that + is specified in an access(5), header_checks(5) or body_checks(5) table. + + * Execute "ppoossttffiixx rreellooaadd" to complete the change. + +SSiimmppllee ccoonntteenntt ffiilltteerr ppeerrffoorrmmaannccee + +With the shell script as shown above you will lose a factor of four in Postfix +performance for transit mail that arrives and leaves via SMTP. You will lose +another factor in transit performance for each additional temporary file that +is created and deleted in the process of content filtering. The performance +impact is less for mail that is submitted or delivered locally, because such +deliveries are already slower than SMTP transit mail. + +SSiimmppllee ccoonntteenntt ffiilltteerr lliimmiittaattiioonnss + +The problem with content filters like the one above is that they are not very +robust. The reason is that the software does not talk a well-defined protocol +with Postfix. If the filter shell script aborts because the shell runs into +some memory allocation problem, the script will not produce a nice exit status +as defined in the file /usr/include/sysexits.h. Instead of going to the +deferred queue, mail will bounce. The same lack of robustness can happen when +the content filtering software itself runs into a resource problem. + +The simple content filter method is not suitable for content filter actions +that are invoked via header_checks or body_checks patterns. These patterns will +be applied again after mail is re-injected with the Postfix sendmail command, +resulting in a mail filtering loop. The advanced content filtering method (see +below) makes it possible to turn off header_checks or body_checks patterns for +filtered mail. + +TTuurrnniinngg ooffff tthhee ssiimmppllee ccoonntteenntt ffiilltteerr + +To turn off "simple" content filtering: + + * Edit the master.cf file, remove the "-o content_filter=filter:dummy" text + from the entry that defines the Postfix SMTP server. + + * Execute "ppoossttssuuppeerr --rr AALLLL" to remove content filter request records from + existing queue files. + + * Execute another "ppoossttffiixx rreellooaadd". + +AAddvvaanncceedd ccoonntteenntt ffiilltteerr eexxaammppllee + +The second example is more complex, but can give better performance, and is +less likely to bounce mail when the machine runs into some resource problem. +This content filter receives unfiltered mail with SMTP on localhost port 10025, +and sends filtered mail back into Postfix with SMTP on localhost port 10026. + +For non-SMTP capable content filtering software, Bennett Todd's SMTP proxy +implements a nice PERL/SMTP content filtering framework. See: https:// +web.archive.org/web/20151022025756/http://bent.latency.net/smtpprox/. + +In the figure below, names followed by a number represent Postfix commands or +daemon programs. See the OVERVIEW document for an introduction to the Postfix +architecture. + + Unfiltered -> smtpd(8) qmgr(8) smtp(8) -> Filtered + >- cleanup(8) -> Postfix -< + Unfiltered -> pickup(8) queue local(8) -> Filtered + + ^ | + | v + + smtpd(8) smtp(8) + 10026 + + ^ | + | v + + content filter 10025 + +The example given here filters all mail, including mail that arrives via SMTP +and mail that is locally submitted via the Postfix sendmail command (local +submissions enter Postfix via the pickup(8) server; to keep the figure simple +we omit local submission details). See examples near the end of this document +for how to exclude local users from filtering, or how to configure a +destination dependent content filter. + +You can expect to lose about a factor of two in Postfix performance for mail +that arrives and leaves via SMTP, provided that the content filter creates no +temporary files. Each temporary file created by the content filter adds another +factor to the performance loss. + +AAddvvaanncceedd ccoonntteenntt ffiilltteerr:: rreeqquueessttiinngg tthhaatt aallll mmaaiill iiss ffiilltteerreedd + +To enable the advanced content filter method for all mail, specify in main.cf: + + /etc/postfix/main.cf: + content_filter = scan:localhost:10025 + receive_override_options = no_address_mappings + + * The "receive_override_options" line disables address manipulation before + the content filter, so that the content filter sees the original mail + addresses instead of the result of virtual alias expansion, canonical + mapping, automatic bcc, address masquerading, etc. + + * The "content_filter" line causes Postfix to add one content filter request + record to each incoming mail message, with content "scan:localhost:10025". + The content filter request records are added by the smtpd(8) and pickup(8) + servers (and qmqpd(8) if you decide to enable this service). + + * Content filter requests are stored in queue files; this is how Postfix + keeps track of what mail needs filtering. When a queue file contains a + content filter request, the queue manager will deliver the mail to the + specified content filter regardless of its final destination. + + * The content_filter configuration parameter expects a value of the form + transport:destination. The transport name specifies the first field of a + mail delivery agent definition in master.cf; the syntax of the next-hop + destination is described in the manual page of the corresponding delivery + agent. + + * The meaning of an empty next-hop filter destination is version dependent. + Postfix 2.7 and later will use the recipient domain; earlier versions will + use $myhostname. Specify "default_filter_nexthop = $myhostname" for + compatibility with Postfix 2.6 or earlier, or specify a non-empty next-hop + filter destination. + + * The content_filter setting has lower precedence than a FILTER action that + is specified in an access(5), header_checks(5) or body_checks(5) table. + +AAddvvaanncceedd ccoonntteenntt ffiilltteerr:: sseennddiinngg uunnffiilltteerreedd mmaaiill ttoo tthhee ccoonntteenntt ffiilltteerr + +In this example, "scan" is an instance of the Postfix SMTP client with slightly +different configuration parameters. This is how one would set up the service in +the Postfix master.cf file: + + /etc/postfix/master.cf: + # ============================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ============================================================= + scan unix - - n - 10 smtp + -o smtp_send_xforward_command=yes + -o disable_mime_output_conversion=yes + -o smtp_generic_maps= + + * This runs up to 10 content filters in parallel. Instead of a limit of 10 + concurrent processes, use whatever process limit is feasible for your + machine. Content inspection software can gobble up a lot of system + resources, so you don't want to have too much of it running at the same + time. + + * With "-o smtp_send_xforward_command=yes", the scan transport will try to + forward the original client name and IP address through the content filter + to the after-filter smtpd process, so that filtered mail is logged with the + real client name IP address. See smtp(8) and XFORWARD_README for more + information. + + * The "-o disable_mime_output_conversion=yes" is a workaround that prevents + the breaking of domainkeys and other digital signatures. This is needed + because some SMTP-based content filters don't announce 8BITMIME support, + even though they can handle 8-bit mail. + + * The "-o smtp_generic_maps=" is a workaround that prevents local address + rewriting with generic(5) maps. Such rewriting should happen only when mail + is sent out to the Internet. + +AAddvvaanncceedd ccoonntteenntt ffiilltteerr:: rruunnnniinngg tthhee ccoonntteenntt ffiilltteerr + +The content filter can be set up with the Postfix spawn service, which is the +Postfix equivalent of inetd. For example, to instantiate up to 10 content +filtering processes on localhost port 10025: + + /etc/postfix/master.cf: + # =================================================================== + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # =================================================================== + localhost:10025 inet n n n - 10 spawn + user=filter argv=/path/to/filter localhost 10026 + + * "filter" is a dedicated local user account. The user will never log in, and + can be given a "*" password and non-existent shell and home directory. This + user handles all potentially dangerous mail content - that is why it should + be a separate account. + + * By default, Postfix will terminate a command that runs longer than + command_time_limit seconds (default: 1000s). This is a safety measure that + prevents filters from running forever. + +If you want to have your filter listening on port localhost:10025 instead of +Postfix, then you must run your filter as a stand-alone program, and must not +use the Postfix spawn service. + +AAddvvaanncceedd ffiilltteerr:: iinnjjeeccttiinngg mmaaiill bbaacckk iinnttoo PPoossttffiixx + +The job of the content filter is to either bounce mail with a suitable +diagnostic, or to feed the mail back into Postfix through a dedicated listener +on port localhost 10026. + +The simplest content filter just copies SMTP commands and data between its +inputs and outputs. If it has a problem, all it has to do is to reply to an +input of `.' from Postfix with `550 content rejected', and to disconnect +without sending `.' on the connection that injects mail back into Postfix. + + /etc/postfix/master.cf: + # =================================================================== + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # =================================================================== + localhost:10026 inet n - n - 10 smtpd + -o content_filter= + - + o + receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters + -o smtpd_helo_restrictions= + -o smtpd_client_restrictions= + -o smtpd_sender_restrictions= + # Postfix 2.10 and later: specify empty smtpd_relay_restrictions. + -o smtpd_relay_restrictions= + -o smtpd_recipient_restrictions=permit_mynetworks,reject + -o mynetworks=127.0.0.0/8 + -o smtpd_authorized_xforward_hosts=127.0.0.0/8 + + * NOTE: do not use spaces around the "=" or "," characters. + + * NOTE: the SMTP server must not have a smaller process limit than the + "filter" master.cf entry. + + * The "-o content_filter=" overrides main.cf settings, and requests no + content filtering for mail from the content filter. This is required or + else mail will loop. + + * The "-o receive_override_options" overrides main.cf settings to avoid + duplicating work that was already done before the content filter. These + options are complementary to the options that are specified in main.cf: + + o We specify "no_unknown_recipient_checks" to disable attempts to find + out if a recipient is unknown. + + o We specify "no_header_body_checks" to disable header/body checks. + + o We specify "no_milters" to disable Milter applications (this option is + available only in Postfix 2.3 and later). + + o We don't specify "no_address_mappings" here. This enables virtual alias + expansion, canonical mappings, address masquerading, and other address + mappings after the content filter. The main.cf setting of + "receive_override_options" disables these mappings before the content + filter. + + These receive override options are either implemented by the SMTP server + itself, or they are passed on to the cleanup server. + + * The "-o smtpd_xxx_restrictions" and "-o mynetworks=127.0.0.0/8" override + main.cf settings. They turn off junk mail controls that would only waste + time here. + + * With "-o smtpd_authorized_xforward_hosts=127.0.0.0/8", the scan transport + will try to forward the original client name and IP address to the after- + filter smtpd process, so that filtered mail is logged with the real client + name and IP address. See XFORWARD_README and smtpd(8). + +AAddvvaanncceedd ccoonntteenntt ffiilltteerr ppeerrffoorrmmaannccee + +With the "sandwich" approach to content filtering described here, it is +important to match the filter concurrency to the available CPU, memory and I/ +O resources. Too few content filter processes and mail accumulates in the +active queue even with low traffic volume; too much concurrency and Postfix +ends up deferring mail destined for the content filter because processes fail +due to insufficient resources. + +Currently, content filter performance tuning is a process of trial and error; +analysis is handicapped because filtered and unfiltered messages share the same +queue. As mentioned in the introduction of this document, content filtering +with multiple Postfix instances will be covered in a future version. + +TTuurrnniinngg ooffff tthhee aaddvvaanncceedd ccoonntteenntt ffiilltteerr + +To turn off "advanced" content filtering: + + * Delete or comment out the two following main.cf lines. The other changes + made for advanced content filtering have no effect when content filtering + is turned off. + + /etc/postfix/main.cf: + content_filter = scan:localhost:10025 + receive_override_options = no_address_mappings + + * Execute "ppoossttssuuppeerr --rr AALLLL" to remove content filter request records from + existing queue files. + + * Execute another "ppoossttffiixx rreellooaadd". + +FFiilltteerriinngg mmaaiill ffrroomm oouuttssiiddee uusseerrss oonnllyy + +The easiest approach is to configure ONE Postfix instance with multiple SMTP +server IP addresses in master.cf: + + * Two SMTP server IP addresses for mail from inside users only, with content + filtering turned off. + + /etc/postfix.master.cf: + # ================================================================== + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ================================================================== + 1.2.3.4:smtp inet n - n - - smtpd + -o smtpd_client_restrictions=permit_mynetworks,reject + 127.0.0.1:smtp inet n - n - - smtpd + -o smtpd_client_restrictions=permit_mynetworks,reject + + * One SMTP server address for mail from outside users with content filtering + turned on. + + /etc/postfix.master.cf: + # ================================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ================================================================= + 1.2.3.5:smtp inet n - n - - smtpd + -o content_filter=filter-service:filter-destination + -o receive_override_options=no_address_mappings + +After this, you can follow the same procedure as outlined in the "advanced" or +"simple" content filtering examples above, except that you must not specify +"content_filter" or "receive_override_options" in the main.cf file. + +DDiiffffeerreenntt ffiilltteerrss ffoorr ddiiffffeerreenntt ddoommaaiinnss + +If you are an MX service provider and want to apply different content filters +for different domains, you can configure ONE Postfix instance with multiple +SMTP server IP addresses in master.cf. Each address provides a different +content filter service. + + /etc/postfix.master.cf: + # ================================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ================================================================= + # SMTP service for domains that are filtered with service1:dest1 + 1.2.3.4:smtp inet n - n - - smtpd + -o content_filter=service1:dest1 + -o receive_override_options=no_address_mappings + + # SMTP service for domains that are filtered with service2:dest2 + 1.2.3.5:smtp inet n - n - - smtpd + -o content_filter=service2:dest2 + -o receive_override_options=no_address_mappings + +After this, you can follow the same procedure as outlined in the "advanced" or +"simple" content filtering examples above, except that you must not specify +"content_filter" or "receive_override_options" in the main.cf file. + +Set up MX records in the DNS that route each domain to the proper SMTP server +instance. + +FFIILLTTEERR aaccttiioonnss iinn aacccceessss oorr hheeaaddeerr//bbooddyy ttaabblleess + +The above filtering configurations are static. Mail that follows a given path +is either always filtered or it is never filtered. As of Postfix 2.0 you can +also turn on content filtering on the fly. + +To turn on content filtering with an access(5) table rule: + + /etc/postfix/access: + whatever FILTER foo:bar + +To turn on content filtering with a header_checks(5) or body_checks(5) table +pattern: + + /etc/postfix/header_checks: + /whatever/ FILTER foo:bar + +You can do this in smtpd access maps as well as the cleanup server's header/ +body_checks. This feature must be used with great care: you must disable all +the UCE features in the after-filter smtpd and cleanup daemons or else you will +have a content filtering loop. + +Limitations: + + * FILTER actions from smtpd access maps and header/body_checks take + precedence over filters specified with the main.cf content_filter + parameter. + + * If a message triggers more than one filter action, only the last one takes + effect. + + * The same content filter is applied to all the recipients of a given + message. + |