path: root/README_FILES/QSHAPE_README

PPoossttffiixx BBoottttlleenneecckk AAnnaallyyssiiss

-------------------------------------------------------------------------------

PPuurrppoossee ooff tthhiiss ddooccuummeenntt

This document is an introduction to Postfix queue congestion analysis. It
explains how the qshape(1) program can help to track down the reason for queue
congestion. qshape(1) is bundled with Postfix 2.1 and later source code, under
the "auxiliary" directory. This document describes qshape(1) as bundled with
Postfix 2.4.

This document covers the following topics:

  * Introducing the qshape tool
  * Trouble shooting with qshape
  * Example 1: Healthy queue
  * Example 2: Deferred queue full of dictionary attack bounces
  * Example 3: Congestion in the active queue
  * Example 4: High volume destination backlog
  * Postfix queue directories

      o The "maildrop" queue
      o The "hold" queue
      o The "incoming" queue
      o The "active" queue
      o The "deferred" queue

  * Credits

IInnttrroodduucciinngg tthhee qqsshhaappee ttooooll

When mail is draining slowly or the queue is unexpectedly large, run qshape(1)
as the super-user (root) to help zero in on the problem. The qshape(1) program
displays a tabular view of the Postfix queue contents.

  * On the horizontal axis, it displays the queue age with fine granularity for
    recent messages and (geometrically) less fine granularity for older
    messages.

  * The vertical axis displays the destination (or with the "-s" switch the
    sender) domain. Domains with the most messages are listed first.

For example, in the output below we see the top 10 lines of the (mostly forged)
sender domain distribution for captured spam in the "hold" queue:

    $ qshape -s hold | head
                             T  5 10 20 40 80 160 320 640 1280 1280+
                     TOTAL 486  0  0  1  0  0   2   4  20   40   419
                 yahoo.com  14  0  0  1  0  0   0   0   1    0    12
      extremepricecuts.net  13  0  0  0  0  0   0   0   2    0    11
            ms35.hinet.net  12  0  0  0  0  0   0   0   0    1    11
          winnersdaily.net  12  0  0  0  0  0   0   0   2    0    10
               hotmail.com  11  0  0  0  0  0   0   0   0    1    10
               worldnet.fr   6  0  0  0  0  0   0   0   0    0     6
            ms41.hinet.net   6  0  0  0  0  0   0   0   0    0     6
                    osn.de   5  0  0  0  0  0   1   0   0    0     4

  * The "T" column shows the total (in this case sender) count for each domain.
    The columns with numbers above them, show counts for messages aged fewer
    than that many minutes, but not younger than the age limit for the previous
    column. The row labeled "TOTAL" shows the total count for all domains.

  * In this example, there are 14 messages allegedly from yahoo.com, 1 between
    10 and 20 minutes old, 1 between 320 and 640 minutes old and 12 older than
    1280 minutes (1440 minutes in a day).

When the output is a terminal intermediate results showing the top 20 domains
(-n option) are displayed after every 1000 messages (-N option) and the final
output also shows only the top 20 domains. This makes qshape useful even when
the "deferred" queue is very large and it may otherwise take prohibitively long
to read the entire "deferred" queue.

By default, qshape shows statistics for the union of both the "incoming" and
"active" queues which are the most relevant queues to look at when analyzing
performance.

One can request an alternate list of queues:

    $ qshape deferred
    $ qshape incoming active deferred

this will show the age distribution of the "deferred" queue or the union of the
"incoming", "active" and "deferred" queues.

Command line options control the number of display "buckets", the age limit for
the smallest bucket, display of parent domain counts and so on. The "-h" option
outputs a summary of the available switches.

TTrroouubbllee sshhoooottiinngg wwiitthh qqsshhaappee

Large numbers in the qshape output represent a large number of messages that
are destined to (or alleged to come from) a particular domain. It should be
possible to tell at a glance which domains dominate the queue sender or
recipient counts, approximately when a burst of mail started, and when it
stopped.

The problem destinations or sender domains appear near the top left corner of
the output table. Remember that the "active" queue can accommodate up to 20000
($qmgr_message_active_limit) messages. To check whether this limit has been
reached, use:

    $ qshape -s active       (show sender statistics)

If the total sender count is below 20000 the "active" queue is not yet
saturated, any high volume sender domains show near the top of the output.

With oqmgr(8) the "active" queue is also limited to at most 20000 recipient
addresses ($qmgr_message_recipient_limit). To check for exhaustion of this
limit use:

    $ qshape active          (show recipient statistics)

Having found the high volume domains, it is often useful to search the logs for
recent messages pertaining to the domains in question.

    # Find deliveries to example.com
    #
    $ tail -10000 /var/log/maillog |
            grep -E -i ': to=<.*@example\.com>,' |
            less

    # Find messages from example.com
    #
    $ tail -10000 /var/log/maillog |
            grep -E -i ': from=<.*@example\.com>,' |
            less

You may want to drill in on some specific queue ids:

    # Find all messages for a specific queue id.
    #
    $ tail -10000 /var/log/maillog | grep -E ': 2B2173FF68: '

Also look for queue manager warning messages in the log. These warnings can
suggest strategies to reduce congestion.

    $ grep -E 'qmgr.*(panic|fatal|error|warning):' /var/log/maillog

When all else fails try the Postfix mailing list for help, but please don't
forget to include the top 10 or 20 lines of qshape(1) output.

EExxaammppllee 11:: HHeeaalltthhyy qquueeuuee

When looking at just the "incoming" and "active" queues, under normal
conditions (no congestion) the "incoming" and "active" queues are nearly empty.
Mail leaves the system almost as quickly as it comes in or is deferred without
congestion in the "active" queue.

    $ qshape        (show "incoming" and "active" queue status)

                     T  5 10 20 40 80 160 320 640 1280 1280+
              TOTAL  5  0  0  0  1  0   0   0   1    1     2
      meri.uwasa.fi  5  0  0  0  1  0   0   0   1    1     2

If one looks at the two queues separately, the "incoming" queue is empty or
perhaps briefly has one or two messages, while the "active" queue holds more
messages and for a somewhat longer time:

    $ qshape incoming

                     T  5 10 20 40 80 160 320 640 1280 1280+
              TOTAL  0  0  0  0  0  0   0   0   0    0     0

    $ qshape active

                     T  5 10 20 40 80 160 320 640 1280 1280+
              TOTAL  5  0  0  0  1  0   0   0   1    1     2
      meri.uwasa.fi  5  0  0  0  1  0   0   0   1    1     2

EExxaammppllee 22:: DDeeffeerrrreedd qquueeuuee ffuullll ooff ddiiccttiioonnaarryy aattttaacckk bboouunncceess

This is from a server where recipient validation is not yet available for some
of the hosted domains. Dictionary attacks on the unvalidated domains result in
bounce backscatter. The bounces dominate the queue, but with proper tuning they
do not saturate the "incoming" or "active" queues. The high volume of deferred
mail is not a direct cause for alarm.

    $ qshape deferred | head

                             T  5 10 20 40 80 160 320 640 1280 1280+
                    TOTAL 2234  4  2  5  9 31  57 108 201  464  1353
      heyhihellothere.com  207  0  0  1  1  6   6   8  25   68    92
      pleazerzoneprod.com  105  0  0  0  0  0   0   0   5   44    56
           groups.msn.com   63  2  1  2  4  4  14  14  14    8     0
        orion.toppoint.de   49  0  0  0  1  0   2   4   3   16    23
              kali.com.cn   46  0  0  0  0  1   0   2   6   12    25
            meri.uwasa.fi   44  0  0  0  0  1   0   2   8   11    22
        gjr.paknet.com.pk   43  1  0  0  1  1   3   3   6   12    16
     aristotle.algonet.se   41  0  0  0  0  0   1   2  11   12    15

The domains shown are mostly bulk-mailers and all the volume is the tail end of
the time distribution, showing that short term arrival rates are moderate.
Larger numbers and lower message ages are more indicative of current trouble.
Old mail still going nowhere is largely harmless so long as the "active" and
"incoming" queues are short. We can also see that the groups.msn.com
undeliverables are low rate steady stream rather than a concentrated dictionary
attack that is now over.

    $ qshape -s deferred | head

                         T  5 10 20 40 80 160 320 640 1280 1280+
                TOTAL 2193  4  4  5  8 33  56 104 205  465  1309
        MAILER-DAEMON 1709  4  4  5  8 33  55 101 198  452   849
          example.com  263  0  0  0  0  0   0   0   0    2   261
          example.org  209  0  0  0  0  0   1   3   6   11   188
          example.net    6  0  0  0  0  0   0   0   0    0     6
          example.edu    3  0  0  0  0  0   0   0   0    0     3
          example.gov    2  0  0  0  0  0   0   0   1    0     1
          example.mil    1  0  0  0  0  0   0   0   0    0     1

Looking at the sender distribution, we see that as expected most of the
messages are bounces.

EExxaammppllee 33:: CCoonnggeessttiioonn iinn tthhee aaccttiivvee qquueeuuee

This example is taken from a Feb 2004 discussion on the Postfix Users list.
Congestion was reported with the "active" and "incoming" queues large and not
shrinking despite very large delivery agent process limits. The thread is
archived at: http://groups.google.com/
groups?threadm=c0b7js$2r65$1@FreeBSD.csie.NCTU.edu.tw and http://
archives.neohapsis.com/archives/postfix/2004-02/thread.html#1371

Using an older version of qshape(1) it was quickly determined that all the
messages were for just a few destinations:

    $ qshape        (show "incoming" and "active" queue status)

                               T   A   5  10  20  40  80 160 320 320+
                     TOTAL 11775 9996  0   0   1   1  42  94 221 1420
      user.sourceforge.net  7678 7678  0   0   0   0   0   0   0    0
     lists.sourceforge.net  2313 2313  0   0   0   0   0   0   0    0
            gzd.gotdns.com   102    0  0   0   0   0   0   0   2  100

The "A" column showed the count of messages in the "active" queue, and the
numbered columns showed totals for the "deferred" queue. At 10000 messages
(Postfix 1.x "active" queue size limit) the "active" queue is full. The
"incoming" queue was growing rapidly.

With the trouble destinations clearly identified, the administrator quickly
found and fixed the problem. It is substantially harder to glean the same
information from the logs. While a careful reading of mailq(1) output should
yield similar results, it is much harder to gauge the magnitude of the problem
by looking at the queue one message at a time.

EExxaammppllee 44:: HHiigghh vvoolluummee ddeessttiinnaattiioonn bbaacckklloogg

When a site you send a lot of email to is down or slow, mail messages will
rapidly build up in the "deferred" queue, or worse, in the "active" queue. The
qshape output will show large numbers for the destination domain in all age
buckets that overlap the starting time of the problem:

    $ qshape deferred | head

                        T   5  10  20  40   80  160 320 640 1280 1280+
               TOTAL 5000 200 200 400 800 1600 1000 200 200  200   200
      highvolume.com 4000 160 160 320 640 1280 1440   0   0    0     0
                 ...

Here the "highvolume.com" destination is continuing to accumulate deferred
mail. The "incoming" and "active" queues are fine, but the "deferred" queue
started growing some time between 1 and 2 hours ago and continues to grow.

If the high volume destination is not down, but is instead slow, one might see
similar congestion in the "active" queue. "Active" queue congestion is a
greater cause for alarm; one might need to take measures to ensure that the
mail is deferred instead or even add an access(5) rule asking the sender to try
again later.

If a high volume destination exhibits frequent bursts of consecutive
connections refused by all MX hosts or "421 Server busy errors", it is possible
for the queue manager to mark the destination as "dead" despite the transient
nature of the errors. The destination will be retried again after the
expiration of a $minimal_backoff_time timer. If the error bursts are frequent
enough it may be that only a small quantity of email is delivered before the
destination is again marked "dead". In some cases enabling static (not on
demand) connection caching by listing the appropriate nexthop domain in a table
included in "smtp_connection_cache_destinations" may help to reduce the error
rate, because most messages will re-use existing connections.

The MTA that has been observed most frequently to exhibit such bursts of errors
is Microsoft Exchange, which refuses connections under load. Some proxy virus
scanners in front of the Exchange server propagate the refused connection to
the client as a "421" error.

Note that it is now possible to configure Postfix to exhibit similarly erratic
behavior by misconfiguring the anvil(8) service. Do not use anvil(8) for
steady-state rate limiting, its purpose is (unintentional) DoS prevention and
the rate limits set should be very generous!

If one finds oneself needing to deliver a high volume of mail to a destination
that exhibits frequent brief bursts of errors and connection caching does not
solve the problem, there is a subtle workaround.

  * Postfix version 2.5 and later:

      o In master.cf set up a dedicated clone of the "smtp" transport for the
        destination in question. In the example below we will call it
        "fragile".

      o In master.cf configure a reasonable process limit for the cloned smtp
        transport (a number in the 10-20 range is typical).

      o IMPORTANT!!! In main.cf configure a large per-destination pseudo-cohort
        failure limit for the cloned smtp transport.

        /etc/postfix/main.cf:
            transport_maps = hash:/etc/postfix/transport
            fragile_destination_concurrency_failed_cohort_limit = 100
            fragile_destination_concurrency_limit = 20

        /etc/postfix/transport:
            example.com  fragile:

        /etc/postfix/master.cf:
            # service type  private unpriv  chroot  wakeup  maxproc command
            fragile   unix     -       -       n       -      20    smtp

        See also the documentation for
        default_destination_concurrency_failed_cohort_limit and
        default_destination_concurrency_limit.

  * Earlier Postfix versions:

      o In master.cf set up a dedicated clone of the "smtp" transport for the
        destination in question. In the example below we will call it
        "fragile".

      o In master.cf configure a reasonable process limit for the transport (a
        number in the 10-20 range is typical).

      o IMPORTANT!!! In main.cf configure a very large initial and destination
        concurrency limit for this transport (say 2000).

        /etc/postfix/main.cf:
            transport_maps = hash:/etc/postfix/transport
            initial_destination_concurrency = 2000
            fragile_destination_concurrency_limit = 2000

        /etc/postfix/transport:
            example.com  fragile:

        /etc/postfix/master.cf:
            # service type  private unpriv  chroot  wakeup  maxproc command
            fragile   unix     -       -       n       -      20    smtp

        See also the documentation for default_destination_concurrency_limit.

The effect of this configuration is that up to 2000 consecutive errors are
tolerated without marking the destination dead, while the total concurrency
remains reasonable (10-20 processes). This trick is only for a very specialized
situation: high volume delivery into a channel with multi-error bursts that is
capable of high throughput, but is repeatedly throttled by the bursts of
errors.

When a destination is unable to handle the load even after the Postfix process
limit is reduced to 1, a desperate measure is to insert brief delays between
delivery attempts.

  * Postfix version 2.5 and later:

      o In master.cf set up a dedicated clone of the "smtp" transport for the
        problem destination. In the example below we call it "slow".

      o In main.cf configure a short delay between deliveries to the same
        destination.

        /etc/postfix/main.cf:
            transport_maps = hash:/etc/postfix/transport
            slow_destination_rate_delay = 1
            slow_destination_concurrency_failed_cohort_limit = 100

        /etc/postfix/transport:
            example.com  slow:

        /etc/postfix/master.cf:
            # service type  private unpriv  chroot  wakeup  maxproc command
            slow      unix     -       -       n       -       -    smtp

    See also the documentation for default_destination_rate_delay.

    This solution forces the Postfix smtp(8) client to wait for
    $slow_destination_rate_delay seconds between deliveries to the same
    destination.

    IMPORTANT!! The large slow_destination_concurrency_failed_cohort_limit
    value is needed. This prevents Postfix from deferring all mail for the same
    destination after only one connection or handshake error (the reason for
    this is that non-zero slow_destination_rate_delay forces a per-destination
    concurrency of 1).

  * Earlier Postfix versions:

      o In the transport map entry for the problem destination, specify a dead
        host as the primary nexthop.

      o In the master.cf entry for the transport specify the problem
        destination as the fallback_relay and specify a small
        smtp_connect_timeout value.

        /etc/postfix/main.cf:
            transport_maps = hash:/etc/postfix/transport

        /etc/postfix/transport:
            example.com  slow:[dead.host]

        /etc/postfix/master.cf:
            # service type  private unpriv  chroot  wakeup  maxproc command
            slow      unix     -       -       n       -       1    smtp
                -o fallback_relay=problem.example.com
                -o smtp_connect_timeout=1
                -o smtp_connection_cache_on_demand=no

    This solution forces the Postfix smtp(8) client to wait for
    $smtp_connect_timeout seconds between deliveries. The connection caching
    feature is disabled to prevent the client from skipping over the dead host.

PPoossttffiixx qquueeuuee ddiirreeccttoorriieess

The following sections describe Postfix queues: their purpose, what normal
behavior looks like, and how to diagnose abnormal behavior.

TThhee ""mmaaiillddrroopp"" qquueeuuee

Messages that have been submitted via the Postfix sendmail(1) command, but not
yet brought into the main Postfix queue by the pickup(8) service, await
processing in the "maildrop" queue. Messages can be added to the "maildrop"
queue even when the Postfix system is not running. They will begin to be
processed once Postfix is started.

The "maildrop" queue is drained by the single threaded pickup(8) service
scanning the queue directory periodically or when notified of new message
arrival by the postdrop(1) program. The postdrop(1) program is a setgid helper
that allows the unprivileged Postfix sendmail(1) program to inject mail into
the "maildrop" queue and to notify the pickup(8) service of its arrival.

All mail that enters the main Postfix queue does so via the cleanup(8) service.
The cleanup service is responsible for envelope and header rewriting, header
and body regular expression checks, automatic bcc recipient processing, milter
content processing, and reliable insertion of the message into the Postfix
"incoming" queue.

In the absence of excessive CPU consumption in cleanup(8) header or body
regular expression checks or other software consuming all available CPU
resources, Postfix performance is disk I/O bound. The rate at which the pickup
(8) service can inject messages into the queue is largely determined by disk
access times, since the cleanup(8) service must commit the message to stable
storage before returning success. The same is true of the postdrop(1) program
writing the message to the "maildrop" directory.

As the pickup service is single threaded, it can only deliver one message at a
time at a rate that does not exceed the reciprocal disk I/O latency (+ CPU if
not negligible) of the cleanup service.

Congestion in this queue is indicative of an excessive local message submission
rate or perhaps excessive CPU consumption in the cleanup(8) service due to
excessive body_checks, or (Postfix >= 2.3) high latency milters.

Note, that once the "active" queue is full, the cleanup service will attempt to
slow down message injection by pausing $in_flow_delay for each message. In this
case "maildrop" queue congestion may be a consequence of congestion downstream,
rather than a problem in its own right.

Note, you should not attempt to deliver large volumes of mail via the pickup(8)
service. High volume sites should avoid using "simple" content filters that re-
inject scanned mail via Postfix sendmail(1) and postdrop(1).

A high arrival rate of locally submitted mail may be an indication of an
uncaught forwarding loop, or a run-away notification program. Try to keep the
volume of local mail injection to a moderate level.

The "postsuper -r" command can place selected messages into the "maildrop"
queue for reprocessing. This is most useful for resetting any stale
content_filter settings. Requeuing a large number of messages using "postsuper
-r" can clearly cause a spike in the size of the "maildrop" queue.

TThhee ""hhoolldd"" qquueeuuee

The administrator can define "smtpd" access(5) policies, or cleanup(8) header/
body checks that cause messages to be automatically diverted from normal
processing and placed indefinitely in the "hold" queue. Messages placed in the
"hold" queue stay there until the administrator intervenes. No periodic
delivery attempts are made for messages in the "hold" queue. The postsuper(1)
command can be used to manually release messages into the "deferred" queue.

Messages can potentially stay in the "hold" queue longer than
$maximal_queue_lifetime. If such "old" messages need to be released from the
"hold" queue, they should typically be moved into the "maildrop" queue using
"postsuper -r", so that the message gets a new timestamp and is given more than
one opportunity to be delivered. Messages that are "young" can be moved
directly into the "deferred" queue using "postsuper -H".

The "hold" queue plays little role in Postfix performance, and monitoring of
the "hold" queue is typically more closely motivated by tracking spam and
malware, than by performance issues.

TThhee ""iinnccoommiinngg"" qquueeuuee

All new mail entering the Postfix queue is written by the cleanup(8) service
into the "incoming" queue. New queue files are created owned by the "postfix"
user with an access bitmask (or mode) of 0600. Once a queue file is ready for
further processing the cleanup(8) service changes the queue file mode to 0700
and notifies the queue manager of new mail arrival. The queue manager ignores
incomplete queue files whose mode is 0600, as these are still being written by
cleanup.

The queue manager scans the "incoming" queue bringing any new mail into the
"active" queue if the "active" queue resource limits have not been exceeded. By
default, the "active" queue accommodates at most 20000 messages. Once the
"active" queue message limit is reached, the queue manager stops scanning the
"incoming" queue (and the "deferred" queue, see below).

Under normal conditions the "incoming" queue is nearly empty (has only mode
0600 files), with the queue manager able to import new messages into the
"active" queue as soon as they become available.

The "incoming" queue grows when the message input rate spikes above the rate at
which the queue manager can import messages into the "active" queue. The main
factors slowing down the queue manager are disk I/O and lookup queries to the
trivial-rewrite service. If the queue manager is routinely not keeping up,
consider not using "slow" lookup services (MySQL, LDAP, ...) for transport
lookups or speeding up the hosts that provide the lookup service. If the
problem is I/O starvation, consider striping the queue over more disks, faster
controllers with a battery write cache, or other hardware improvements. At the
very least, make sure that the queue directory is mounted with the "noatime"
option if applicable to the underlying filesystem.

The in_flow_delay parameter is used to clamp the input rate when the queue
manager starts to fall behind. The cleanup(8) service will pause for
$in_flow_delay seconds before creating a new queue file if it cannot obtain a
"token" from the queue manager.

Since the number of cleanup(8) processes is limited in most cases by the SMTP
server concurrency, the input rate can exceed the output rate by at most "SMTP
connection count" / $in_flow_delay messages per second.

With a default process limit of 100, and an in_flow_delay of 1s, the coupling
is strong enough to limit a single run-away injector to 1 message per second,
but is not strong enough to deflect an excessive input rate from many sources
at the same time.

If a server is being hammered from multiple directions, consider raising the
in_flow_delay to 10 seconds, but only if the "incoming" queue is growing even
while the "active" queue is not full and the trivial-rewrite service is using a
fast transport lookup mechanism.

TThhee ""aaccttiivvee"" qquueeuuee

The queue manager is a delivery agent scheduler; it works to ensure fast and
fair delivery of mail to all destinations within designated resource limits.

The "active" queue is somewhat analogous to an operating system's process run
queue. Messages in the "active" queue are ready to be sent (runnable), but are
not necessarily in the process of being sent (running).

While most Postfix administrators think of the "active" queue as a directory on
disk, the real "active" queue is a set of data structures in the memory of the
queue manager process.

Messages in the "maildrop", "hold", "incoming" and "deferred" queues (see
below) do not occupy memory; they are safely stored on disk waiting for their
turn to be processed. The envelope information for messages in the "active"
queue is managed in memory, allowing the queue manager to do global scheduling,
allocating available delivery agent processes to an appropriate message in the
"active" queue.

Within the "active" queue, (multi-recipient) messages are broken up into groups
of recipients that share the same transport/nexthop combination; the group size
is capped by the transport's recipient concurrency limit.

Multiple recipient groups (from one or more messages) are queued for delivery
grouped by transport/nexthop combination. The ddeessttiinnaattiioonn concurrency limit for
the transports caps the number of simultaneous delivery attempts for each
nexthop. Transports with a rreecciippiieenntt concurrency limit of 1 are special: these
are grouped by the actual recipient address rather than the nexthop, yielding
per-recipient concurrency limits rather than per-domain concurrency limits.
Per-recipient limits are appropriate when performing final delivery to
mailboxes rather than when relaying to a remote server.

Congestion occurs in the "active" queue when one or more destinations drain
slower than the corresponding message input rate.

Input into the "active" queue comes both from new mail in the "incoming" queue,
and retries of mail in the "deferred" queue. Should the "deferred" queue get
really large, retries of old mail can dominate the arrival rate of new mail.
Systems with more CPU, faster disks and more network bandwidth can deal with
larger "deferred" queues, but as a rule of thumb the "deferred" queue scales to
somewhere between 100,000 and 1,000,000 messages with good performance unlikely
above that "limit". Systems with queues this large should typically stop
accepting new mail, or put the backlog "on hold" until the underlying issue is
fixed (provided that there is enough capacity to handle just the new mail).

When a destination is down for some time, the queue manager will mark it dead,
and immediately defer all mail for the destination without trying to assign it
to a delivery agent. In this case the messages will quickly leave the "active"
queue and end up in the "deferred" queue (with Postfix < 2.4, this is done
directly by the queue manager, with Postfix >= 2.4 this is done via the "retry"
delivery agent).

When the destination is instead simply slow, or there is a problem causing an
excessive arrival rate the "active" queue will grow and will become dominated
by mail to the congested destination.

The only way to reduce congestion is to either reduce the input rate or
increase the throughput. Increasing the throughput requires either increasing
the concurrency or reducing the latency of deliveries.

For high volume sites a key tuning parameter is the number of "smtp" delivery
agents allocated to the "smtp" and "relay" transports. High volume sites tend
to send to many different destinations, many of which may be down or slow, so a
good fraction of the available delivery agents will be blocked waiting for slow
sites. Also mail destined across the globe will incur large SMTP command-
response latencies, so high message throughput can only be achieved with more
concurrent delivery agents.

The default "smtp" process limit of 100 is good enough for most sites, and may
even need to be lowered for sites with low bandwidth connections (no use
increasing concurrency once the network pipe is full). When one finds that the
queue is growing on an "idle" system (CPU, disk I/O and network not exhausted)
the remaining reason for congestion is insufficient concurrency in the face of
a high average latency. If the number of outbound SMTP connections (either
ESTABLISHED or SYN_SENT) reaches the process limit, mail is draining slowly and
the system and network are not loaded, raise the "smtp" and/or "relay" process
limits!

When a high volume destination is served by multiple MX hosts with typically
low delivery latency, performance can suffer dramatically when one of the MX
hosts is unresponsive and SMTP connections to that host timeout. For example,
if there are 2 equal weight MX hosts, the SMTP connection timeout is 30 seconds
and one of the MX hosts is down, the average SMTP connection will take
approximately 15 seconds to complete. With a default per-destination
concurrency limit of 20 connections, throughput falls to just over 1 message
per second.

The best way to avoid bottlenecks when one or more MX hosts is non-responsive
is to use connection caching. Connection caching was introduced with Postfix
2.2 and is by default enabled on demand for destinations with a backlog of mail
in the "active" queue. When connection caching is in effect for a particular
destination, established connections are re-used to send additional messages,
this reduces the number of connections made per message delivery and maintains
good throughput even in the face of partial unavailability of the destination's
MX hosts.

If connection caching is not available (Postfix < 2.2) or does not provide a
sufficient latency reduction, especially for the "relay" transport used to
forward mail to "your own" domains, consider setting lower than default SMTP
connection timeouts (1-5 seconds) and higher than default destination
concurrency limits. This will further reduce latency and provide more
concurrency to maintain throughput should latency rise.

Setting high concurrency limits to domains that are not your own may be viewed
as hostile by the receiving system, and steps may be taken to prevent you from
monopolizing the destination system's resources. The defensive measures may
substantially reduce your throughput or block access entirely. Do not set
aggressive concurrency limits to remote domains without coordinating with the
administrators of the target domain.

If necessary, dedicate and tune custom transports for selected high volume
destinations. The "relay" transport is provided for forwarding mail to domains
for which your server is a primary or backup MX host. These can make up a
substantial fraction of your email traffic. Use the "relay" and not the "smtp"
transport to send email to these domains. Using the "relay" transport allocates
a separate delivery agent pool to these destinations and allows separate tuning
of timeouts and concurrency limits.

Another common cause of congestion is unwarranted flushing of the entire
"deferred" queue. The "deferred" queue holds messages that are likely to fail
to be delivered and are also likely to be slow to fail delivery (time out). As
a result the most common reaction to a large "deferred" queue (flush it!) is
more than likely counter-productive, and typically makes the congestion worse.
Do not flush the "deferred" queue unless you expect that most of its content
has recently become deliverable (e.g. relayhost back up after an outage)!

Note that whenever the queue manager is restarted, there may already be
messages in the "active" queue directory, but the "real" "active" queue in
memory is empty. In order to recover the in-memory state, the queue manager
moves all the "active" queue messages back into the "incoming" queue, and then
uses its normal "incoming" queue scan to refill the "active" queue. The process
of moving all the messages back and forth, redoing transport table (trivial-
rewrite(8) resolve service) lookups, and re-importing the messages back into
memory is expensive. At all costs, avoid frequent restarts of the queue manager
(e.g. via frequent execution of "postfix reload").

TThhee ""ddeeffeerrrreedd"" qquueeuuee

When all the deliverable recipients for a message are delivered, and for some
recipients delivery failed for a transient reason (it might succeed later), the
message is placed in the "deferred" queue.

The queue manager scans the "deferred" queue periodically. The scan interval is
controlled by the queue_run_delay parameter. While a "deferred" queue scan is
in progress, if an "incoming" queue scan is also in progress (ideally these are
brief since the "incoming" queue should be short), the queue manager alternates
between looking for messages in the "incoming" queue and in the "deferred"
queue. This "round-robin" strategy prevents starvation of either the "incoming"
or the "deferred" queues.

Each "deferred" queue scan only brings a fraction of the "deferred" queue back
into the "active" queue for a retry. This is because each message in the
"deferred" queue is assigned a "cool-off" time when it is deferred. This is
done by time-warping the modification time of the queue file into the future.
The queue file is not eligible for a retry if its modification time is not yet
reached.

The "cool-off" time is at least $minimal_backoff_time and at most
$maximal_backoff_time. The next retry time is set by doubling the message's age
in the queue, and adjusting up or down to lie within the limits. This means
that young messages are initially retried more often than old messages.

If a high volume site routinely has large "deferred" queues, it may be useful
to adjust the queue_run_delay, minimal_backoff_time and maximal_backoff_time to
provide short enough delays on first failure (Postfix >= 2.4 has a sensibly low
minimal backoff time by default), with perhaps longer delays after multiple
failures, to reduce the retransmission rate of old messages and thereby reduce
the quantity of previously deferred mail in the "active" queue. If you want a
really low minimal_backoff_time, you may also want to lower queue_run_delay,
but understand that more frequent scans will increase the demand for disk I/O.

One common cause of large "deferred" queues is failure to validate recipients
at the SMTP input stage. Since spammers routinely launch dictionary attacks
from unrepliable sender addresses, the bounces for invalid recipient addresses
clog the "deferred" queue (and at high volumes proportionally clog the "active"
queue). Recipient validation is strongly recommended through use of the
local_recipient_maps and relay_recipient_maps parameters. Even when bounces
drain quickly they inundate innocent victims of forgery with unwanted email. To
avoid this, do not accept mail for invalid recipients.

When a host with lots of deferred mail is down for some time, it is possible
for the entire "deferred" queue to reach its retry time simultaneously. This
can lead to a very full "active" queue once the host comes back up. The
phenomenon can repeat approximately every maximal_backoff_time seconds if the
messages are again deferred after a brief burst of congestion. Perhaps, a
future Postfix release will add a random offset to the retry time (or use a
combination of strategies) to reduce the odds of repeated complete "deferred"
queue flushes.

CCrreeddiittss

The qshape(1) program was developed by Victor Duchovni of Morgan Stanley, who
also wrote the initial version of this document.