diff options
Diffstat (limited to 'html/BUILTIN_FILTER_README.html')
| -rw-r--r-- | html/BUILTIN_FILTER_README.html | 488 |
1 files changed, 488 insertions, 0 deletions
diff --git a/html/BUILTIN_FILTER_README.html b/html/BUILTIN_FILTER_README.html new file mode 100644 index 0000000..3cd3dec --- /dev/null +++ b/html/BUILTIN_FILTER_README.html @@ -0,0 +1,488 @@ +<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" + "http://www.w3.org/TR/html4/loose.dtd"> + +<html> + +<head> + +<title>Postfix Built-in Content Inspection</title> + +<meta http-equiv="Content-Type" content="text/html; charset=utf-8"> + +</head> + +<body> + +<h1><img src="postfix-logo.jpg" width="203" height="98" alt=""> +Postfix Built-in Content Inspection</h1> + +<hr> + +<h2>Built-in content inspection introduction </h2> + +<p> Postfix supports a built-in filter mechanism that examines +message header and message body content, one line at a time, before +it is stored in the Postfix queue. The filter is usually implemented +with POSIX or PCRE regular expressions, as described in the +<a href="header_checks.5.html">header_checks(5)</a> manual page. </p> + +<p> The original purpose of the built-in filter is to stop an +outbreak of specific email worms or viruses, and it does this job +well. The filter has also helped to block bounced junk email, +bounced email from worms or viruses, and notifications from virus +detection systems. Information about this secondary application +is given in the <a href="BACKSCATTER_README.html">BACKSCATTER_README</a> document. </p> + +<p> Because the built-in filter is optimized for stopping specific +worms and virus outbreaks, it has <a href="#limitations">limitations</a> +that make it NOT suitable for general junk email and virus detection. +For that, you should use one of the external content inspection +methods that are described in the <a href="FILTER_README.html">FILTER_README</a>, <a href="SMTPD_PROXY_README.html">SMTPD_PROXY_README</a> +and <a href="MILTER_README.html">MILTER_README</a> documents. </p> + +<p> The following diagram gives an over-all picture of how Postfix +built-in content inspection works: </p> + +<blockquote> + +<table> + +<tr> + + <td colspan="4"> <td bgcolor="#f0f0ff" align="center" + valign="middle"> Postmaster<br> notifications </td> + +</tr> + +<tr> + + <td colspan="4"> <td align="center"> <tt> |<br>v </tt></td> + +</tr> + +<tr> + + <td bgcolor="#f0f0ff" align="center" valign="middle"> + Network or<br> local users </td> + + <td align="center" valign="middle"> <tt> -> </tt> </td> + + <td bgcolor="#f0f0ff" align="center" valign="middle"> + + <b> Built-in<br> filter</b> </td> + + <td align="center" valign="middle"> <tt> -> </tt> </td> + + <td bgcolor="#f0f0ff" align="center" valign="middle"> + Postfix<br> queue </td> + + <td align="center" valign="middle"> <tt> -> </tt> </td> + + <td bgcolor="#f0f0ff" align="center" valign="middle"> + Delivery<br> agents </td> + + <td align="center" valign="middle"> <tt> -> </tt> </td> + + <td bgcolor="#f0f0ff" align="center" valign="middle"> + Network or<br> local mailbox </td> + +</tr> + +<tr> + + <td colspan="4"> <td align="center"> ^<br> <tt> | </tt> </td> + <td> </td> <td align="center"> <tt> |<br>v </tt> </td> + +</tr> + +<tr> + + <td colspan="4"> <td colspan="3" bgcolor="#f0f0ff" align="center" + valign="middle"> Undeliverable mail<br> Forwarded mail</td> + +</tr> + +</table> + +</blockquote> + +<p> The picture makes clear that the filter works while Postfix is +receiving new mail. This means that Postfix can reject mail from +the network without having to return undeliverable mail to the +originator address (which is often spoofed anyway). However, this +ability comes at a price: if mail inspection takes too much time, +then the remote client will time out, and the client may send the +same message repeatedly. </p> + +<p>Topics covered by this document: </p> + +<ul> + +<li><a href="#what">What mail is subjected to header/body checks </a> + +<li><a href="#limitations">Limitations of Postfix header/body checks </a> + +<li><a href="#daily">Preventing daily mail status reports from being blocked </a> + +<li><a href="#remote_only">Configuring header/body checks for mail from outside users only</a> + +<li><a href="#mx_submission">Configuring different header/body checks for MX service and submission service</a> + +<li><a href="#domain_except">Configuring header/body checks for mail to some domains only</a> + +</ul> + +<h2><a name="what">What mail is subjected to header/body checks </a></h2> + +<p> Postfix header/body checks are implemented by the <a href="cleanup.8.html">cleanup(8)</a> +server before it injects mail into the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a>. The diagram +below zooms in on the <a href="cleanup.8.html">cleanup(8)</a> server, and shows that this server +handles mail from many different sources. In order to keep the +diagram readable, the sources of postmaster notifications are not +shown, because they can be produced by many Postfix daemon processes. +</p> + +<blockquote> + +<table> + +<tr> <td colspan="2"> </td> <td bgcolor="#f0f0ff" align="center" +valign="middle"> <a href="bounce.8.html">bounce(8)</a><br> (undeliverable) </td> </tr> + +<tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b> +<a href="smtpd.8.html">smtpd(8)</a><br> (network)</b> </td> <td align="left" valign="bottom"> +<tt> \ </tt> </td> <td align="center" valign="middle"> <tt> |<br>v +</tt> </td> </tr> + +<tr> <td> </td> <td> </td> </tr> + +<tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b> +<a href="qmqpd.8.html">qmqpd(8)</a><br> (network)</b> </td> <td align="center" valign="middle"> +<tt> -\<br>-/ </tt> </td> <td bgcolor="#f0f0ff" align="center" +valign="middle"> <a href="cleanup.8.html">cleanup(8)</a> </td> <td align="center" valign="middle"> +<tt> -> </tt> </td> <td bgcolor="#f0f0ff" align="center" +valign="middle"> <a href="QSHAPE_README.html#incoming_queue"> +incoming<br> queue </a> </td> </tr> + +<tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b> +<a href="pickup.8.html">pickup(8)</a><br> (local)</b> </td> <td align="left" valign="top"> <tt> +/ </tt> </td> <td align="center" valign="middle"> ^<br> <tt> | +</tt> </td> </tr> + +<tr> <td colspan="2"> </td> <td bgcolor="#f0f0ff" align="center" +valign="middle"> <a href="local.8.html">local(8)</a><br> (forwarded) </td> </tr> + +</table> + +</blockquote> + +<p> For efficiency reasons, only mail that enters from outside of +Postfix is inspected with header/body checks. It would be inefficient +to filter already filtered mail again, and it would be undesirable +to block postmaster notifications. The table below summarizes what +mail is and is not subject to header/body checks. </p> + +<blockquote> + +<table border="1"> + +<tr> <th> Message type </th> <th> Source </th> <th> Header/body checks? </th> </tr> + +<tr> <td> Undeliverable mail </td> <td> <a href="bounce.8.html">bounce(8)</a> </td> <td> No </td> </tr> + +<tr> <td> Network mail </td> <td> <a href="smtpd.8.html">smtpd(8)</a> </td> <td> Configurable </td> </tr> + +<tr> <td> Network mail </td> <td> <a href="qmqpd.8.html">qmqpd(8)</a> </td> <td> Configurable </td> </tr> + +<tr> <td> Local submission </td> <td> <a href="pickup.8.html">pickup(8)</a> </td> <td> Configurable </td> </tr> + +<tr> <td> Local forwarding </td> <td> <a href="local.8.html">local(8)</a> </td> <td> No </td> </tr> + +<tr> <td> Postmaster notice </td> <td> many </td> <td> No </td> </tr> + +</table> + +</blockquote> + +<p> How does Postfix decide what mail needs to be filtered? It +would be clumsy to make the decision in the <a href="cleanup.8.html">cleanup(8)</a> server, as +this program receives mail from so many different sources. Instead, +header/body checks are requested by the source. Examples of how +to turn off header/body checks for mail received with <a href="smtpd.8.html">smtpd(8)</a>, +<a href="qmqpd.8.html">qmqpd(8)</a> or <a href="pickup.8.html">pickup(8)</a> are given below under "<a +href="#remote_only">Configuring header/body checks for mail from +outside users only</a>", "<a href="#mx_submission">Configuring +different header/body checks for MX service and submission +service</a>", and "<a href="#domain_except">Configuring +header/body checks for mail to some domains only</a>". </p> + +<h2><a name="limitations">Limitations of Postfix header/body checks </a></h2> + +<ul> + +<li> <p> Header/body checks do not decode message headers or message +body content. For example, if text in the message body is BASE64 +encoded (<a href="https://tools.ietf.org/html/rfc2045">RFC 2045</a>) then your regular expressions will have to match +the BASE64 encoded form. Likewise, message headers with encoded +non-ASCII characters (<a href="https://tools.ietf.org/html/rfc2047">RFC 2047</a>) need to be matched in their encoded +form. </p> + +<li> <p> Header/body checks cannot filter on a combination of +message headers or body lines. Header/body checks examine content +one message header at a time, or one message body line at a time, +and cannot carry a decision over to the next message header or body +line. </p> + +<li> <p> Header/body checks cannot depend on the recipient of a +message. </p> + +<ul> + +<li> <p> One message can have multiple recipients, and all recipients +of a message receive the same treatment. Workarounds have been +proposed that involve selectively deferring some recipients of +multi-recipient mail, but that results in poor SMTP performance +and does not work for non-SMTP mail. </p> + +<li> <p> Some sources of mail send the headers and content ahead +of the recipient information. It would be inefficient to buffer up +an entire message before deciding if it needs to be filtered, and +it would be clumsy to filter mail and to buffer up all the actions +until it is known whether those actions need to be executed. </p> + +</ul> + +<li> <p> Despite warnings, some people try to use the built-in +filter feature for general junk email and/or virus blocking, using +hundreds or even thousands of regular expressions. This can result +in catastrophic performance failure. The symptoms are as follows: +</p> + +<ul> + +<li> <p> The <a href="cleanup.8.html">cleanup(8)</a> processes use up all available CPU time in +order to process the regular expressions, and/or they use up all +available memory so that the system begins to swap. This slows down +all incoming mail deliveries. </p> + +<li> <p> As Postfix needs more and more time to receive an email +message, the number of simultaneous SMTP sessions increases to the +point that the SMTP server process limit is reached. </p> + +<li> <p> While all SMTP server processes are waiting for the +<a href="cleanup.8.html">cleanup(8)</a> servers to finish, new SMTP clients have to wait until +an SMTP server process becomes available. This causes mail deliveries +to time out before they have even begun. </p> + +</ul> + +<p> The remedy for this type of performance problem is simple: +don't use header/body checks for general junk email and/or virus +blocking, and don't filter mail before it is queued. When performance +is a concern, use an external content filter that runs after mail +is queued, as described in the <a href="FILTER_README.html">FILTER_README</a> document. </p> + +</ul> + +<h2><a name="daily">Preventing daily mail status reports from being blocked </a></h2> + +<p>The following is quoted from Jim Seymour's Pflogsumm FAQ at +<a href="http://jimsun.linxnet.com/downloads/pflogsumm-faq.txt">http://jimsun.linxnet.com/downloads/pflogsumm-faq.txt</a>. Pflogsumm +is a program that analyzes Postfix logs, including the logging from +rejected mail. If these logs contain text that was rejected by +Postfix <a href="postconf.5.html#body_checks">body_checks</a> patterns, then the logging is also likely to +be rejected by those same <a href="postconf.5.html#body_checks">body_checks</a> patterns. This problem does +not exist with <a href="postconf.5.html#header_checks">header_checks</a> patterns, because those are not applied +to the text that is part of the mail status report. </p> + +<blockquote> + +<p>You configure Postfix to do body checks, Postfix does its thing, +Pflogsumm reports it and Postfix catches the same string in the +Pflogsumm report. There are several solutions to this. </p> + +<p> Wolfgang Zeikat contributed this: </p> + +<blockquote> +<pre> +#!/usr/bin/perl +use MIME::Lite; + +### Create a new message: +$msg = MIME::Lite->new( + From => 'your@send.er', + To => 'your@recipie.nt', + # Cc => 'some@other.com, some@more.com', + Subject => 'pflogsumm', + Date => `date`, + Type => 'text/plain', + Encoding => 'base64', + Path => '/tmp/pflogg', +); + +$msg->send; +</pre> +</blockquote> + +<p> Where "/tmp/pflogg" is the output of Pflogsumm. This puts Pflogsumm's +output in a base64 MIME attachment. </p> + +</blockquote> + +<p> Note by Wietse: if you run this on a machine that is accessible +by untrusted users, it is safer to store the Pflogsumm report in +a directory that is not world writable. </p> + +<blockquote> + +<p> In a follow-up to a thread in the postfix-users mailing list, Ralf +Hildebrandt noted: </p> + +<blockquote> <p> "mpack does the same thing." </p> </blockquote> + +</blockquote> + +<p> And it does. Which tool one should use is a matter of preference. +</p> + +<p> Other solutions involve additional <a href="postconf.5.html#body_checks">body_checks</a> rules that make +exceptions for daily mail status reports, but this is not recommended. +Such rules slow down all mail and complicate Postfix maintenance. +</p> + +<h2><a name="remote_only">Configuring header/body checks for mail from outside users only</a></h2> + +<p> The following information applies to Postfix 2.1 and later. +Earlier +Postfix versions do not support the <a href="postconf.5.html#receive_override_options">receive_override_options</a> feature. +</p> + +<p> The easiest approach is to configure ONE Postfix instance with +multiple SMTP server IP addresses in <a href="master.5.html">master.cf</a>: </p> + +<ul> + +<li> <p> Two SMTP server IP addresses for mail from inside users +only, with header/body filtering turned off, and a local mail pickup +service with header/body filtering turned off. </p> + +<pre> +/etc/postfix.<a href="master.5.html">master.cf</a>: + # ================================================================== + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ================================================================== + 1.2.3.4:smtp inet n - n - - smtpd + -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> + 127.0.0.1:smtp inet n - n - - smtpd + -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> + pickup fifo n - n 60 1 pickup + -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> +</pre> + +<li> <p> Add some firewall rule to prevent access to 1.2.3.4:smtp +from the outside world. </p> + +<li> <p> One SMTP server address for mail from outside users with +header/body filtering turned on via <a href="postconf.5.html">main.cf</a>. </p> + +<pre> +/etc/postfix.<a href="master.5.html">master.cf</a>: + # ================================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ================================================================= + 1.2.3.5:smtp inet n - n - - smtpd +</pre> + +</ul> + +<h2><a name="mx_submission">Configuring different header/body checks for MX service and submission service</a></h2> + +<p> If authorized user submissions require different header/body +checks than mail from remote MTAs, then this is possible as long +as you have separate mail streams for authorized users and for MX +service. </p> + +<p> The example below assumes that authorized users connect to TCP +port 587 (submission) or 465 (smtps), and that remote MTAs connect +to TCP port 25 (smtp). </p> + +<p> First, we define a few "user-defined" parameters that will +override settings for the submission and smtps services. </p> + +<blockquote> +<pre> +/etc/postfix/<a href="postconf.5.html">main.cf</a>: + msa_cleanup_service_name = msa_cleanup + msa_header_checks = <a href="pcre_table.5.html">pcre</a>:/etc/postfix/msa_header_checks + msa_body_checks = <a href="pcre_table.5.html">pcre</a>:/etc/postfix/msa_body_checks +</pre> +</blockquote> + +<p> Next, we define msa_cleanup as a dedicated cleanup service that +will be used only by the submission and smtps services. This service +uses the <a href="postconf.5.html#header_checks">header_checks</a> and <a href="postconf.5.html#body_checks">body_checks</a> overrides that were defined +above. </p> + +<blockquote> +<pre> +/etc/postfix.<a href="master.5.html">master.cf</a>: + # ================================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ================================================================= + smtp inet n - n - - smtpd + msa_cleanup unix n - n - 0 cleanup + -o <a href="postconf.5.html#header_checks">header_checks</a>=$msa_header_checks + -o <a href="postconf.5.html#body_checks">body_checks</a>=$msa_body_checks + submission inet n - n - - smtpd + -o <a href="postconf.5.html#cleanup_service_name">cleanup_service_name</a>=$msa_cleanup_service_name + -o <a href="postconf.5.html#syslog_name">syslog_name</a>=postfix/submission + <i>...[see sample <a href="master.5.html">master.cf</a> file for more]...</i> + smtps inet n - n - - smtpd + -o <a href="postconf.5.html#cleanup_service_name">cleanup_service_name</a>=$msa_cleanup_service_name + -o <a href="postconf.5.html#syslog_name">syslog_name</a>=postfix/smtps + -o <a href="postconf.5.html#smtpd_tls_wrappermode">smtpd_tls_wrappermode</a>=yes + <i>...[see sample <a href="master.5.html">master.cf</a> file for more]...</i> +</pre> +</blockquote> + +<p> By keeping the "msa_xxx" parameter settings in <a href="postconf.5.html">main.cf</a>, you +keep your <a href="master.5.html">master.cf</a> file simple, and you minimize the amount +of duplication. </p> + +<h2><a name="domain_except">Configuring header/body checks for mail to some domains only</a></h2> + +<p> The following information applies to Postfix 2.1. Earlier +Postfix versions do not support the <a href="postconf.5.html#receive_override_options">receive_override_options</a> feature. +</p> + +<p> If you are an MX service provider and want to enable header/body +checks only for some domains, you can configure ONE Postfix +instance with multiple SMTP server IP addresses in <a href="master.5.html">master.cf</a>. Each +address provides a different service. </p> + +<blockquote> + +<pre> +/etc/postfix.<a href="master.5.html">master.cf</a>: + # ================================================================= + # service type private unpriv chroot wakeup maxproc command + # (yes) (yes) (yes) (never) (100) + # ================================================================= + # SMTP service for domains with header/body checks turned on. + 1.2.3.4:smtp inet n - n - - smtpd + + # SMTP service for domains with header/body checks turned off. + 1.2.3.5:smtp inet n - n - - smtpd + -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> +</pre> +</blockquote> + +<p> Once this is set up you can configure MX records in the DNS +that route each domain to the proper SMTP server instance. </p> + +</body> + +</html> |