Adding upstream version 3.4.23.upstream/3.4.23upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 1160 insertions, 0 deletions

diff --git a/README_FILES/SCHEDULER_README b/README_FILES/SCHEDULER_README
new file mode 100644
index 0000000..a6f7702
--- /dev/null
+++ b/README_FILES/SCHEDULER_README
@@ -0,0 +1,1160 @@
+PPoossttffiixx QQuueeuuee SScchheedduulleerr
+
+-------------------------------------------------------------------------------
+
+DDiissccllaaiimmeerr
+
+Many of the transport-specific configuration parameters discussed in this
+document will not show up in "postconf" command output before Postfix version
+2.9. This limitation applies to many parameters whose name is a combination of
+a master.cf service name such as "relay" and a built-in suffix such as
+"_destination_concurrency_limit".
+
+OOvveerrvviieeww
+
+The queue manager is by far the most complex part of the Postfix mail system.
+It schedules delivery of new mail, retries failed deliveries at specific times,
+and removes mail from the queue after the last delivery attempt. There are two
+major classes of mechanisms that control the operation of the queue manager.
+
+Topics covered by this document:
+
+  * Concurrency scheduling, concerned with the number of concurrent deliveries
+    to a specific destination, including decisions on when to suspend
+    deliveries after persistent failures.
+  * Preemptive scheduling, concerned with the selection of email messages and
+    recipients for a given destination.
+  * Credits, something this document would not be complete without.
+
+CCoonnccuurrrreennccyy sscchheedduulliinngg
+
+The following sections document the Postfix 2.5 concurrency scheduler, after a
+discussion of the limitations of the earlier concurrency scheduler. This is
+followed by results of medium-concurrency experiments, and a discussion of
+trade-offs between performance and robustness.
+
+The material is organized as follows:
+
+  * Drawbacks of the existing concurrency scheduler
+  * Summary of the Postfix 2.5 concurrency feedback algorithm
+  * Summary of the Postfix 2.5 "dead destination" detection algorithm
+  * Pseudocode for the Postfix 2.5 concurrency scheduler
+  * Results for delivery to concurrency limited servers
+  * Discussion of concurrency limited server results
+  * Limitations of less-than-1 per delivery feedback
+  * Concurrency configuration parameters
+
+DDrraawwbbaacckkss ooff tthhee eexxiissttiinngg ccoonnccuurrrreennccyy sscchheedduulleerr
+
+From the start, Postfix has used a simple but robust algorithm where the per-
+destination delivery concurrency is decremented by 1 after delivery failed due
+to connection or handshake failure, and incremented by 1 otherwise. Of course
+the concurrency is never allowed to exceed the maximum per-destination
+concurrency limit. And when a destination's concurrency level drops to zero,
+the destination is declared "dead" and delivery is suspended.
+
+Drawbacks of +/-1 concurrency feedback per delivery are:
+
+  * Overshoot due to exponential delivery concurrency growth with each pseudo-
+    cohort(*). This can be an issue with high-concurrency channels. For
+    example, with the default initial concurrency of 5, concurrency would
+    proceed over time as (5-10-20).
+
+  * Throttling down to zero concurrency after a single pseudo-cohort(*)
+    failure. This was especially an issue with low-concurrency channels where a
+    single failure could be sufficient to mark a destination as "dead", causing
+    the suspension of further deliveries to the affected destination.
+
+(*) A pseudo-cohort is a number of delivery requests equal to a destination's
+delivery concurrency.
+
+The revised concurrency scheduler has a highly modular structure. It uses
+separate mechanisms for per-destination concurrency control and for "dead
+destination" detection. The concurrency control in turn is built from two
+separate mechanisms: it supports less-than-1 feedback per delivery to allow for
+more gradual concurrency adjustments, and it uses feedback hysteresis to
+suppress concurrency oscillations. And instead of waiting for delivery
+concurrency to throttle down to zero, a destination is declared "dead" after a
+configurable number of pseudo-cohorts reports connection or handshake failure.
+
+SSuummmmaarryy ooff tthhee PPoossttffiixx 22..55 ccoonnccuurrrreennccyy ffeeeeddbbaacckk aallggoorriitthhmm
+
+We want to increment a destination's delivery concurrency when some (not
+necessarily consecutive) number of deliveries complete without connection or
+handshake failure. This is implemented with positive feedback g(N) where N is
+the destination's delivery concurrency. With g(N)=1 feedback per delivery,
+concurrency increases by 1 after each positive feedback event; this gives us
+the old scheduler's exponential growth in time. With g(N)=1/N feedback per
+delivery, concurrency increases by 1 after an entire pseudo-cohort N of
+positive feedback reports; this gives us linear growth in time. Less-than-
+1 feedback per delivery and integer truncation naturally give us hysteresis, so
+that transitions to larger concurrency happen every 1/g(N) positive feedback
+events.
+
+We want to decrement a destination's delivery concurrency when some (not
+necessarily consecutive) number of deliveries complete after connection or
+handshake failure. This is implemented with negative feedback f(N) where N is
+the destination's delivery concurrency. With f(N)=1 feedback per delivery,
+concurrency decreases by 1 after each negative feedback event; this gives us
+the old scheduler's behavior where concurrency is throttled down dramatically
+after a single pseudo-cohort failure. With f(N)=1/N feedback per delivery,
+concurrency backs off more gently. Again, less-than-1 feedback per delivery and
+integer truncation naturally give us hysteresis, so that transitions to lower
+concurrency happen every 1/f(N) negative feedback events.
+
+However, with negative feedback we introduce a subtle twist. We "reverse" the
+negative hysteresis cycle so that the transition to lower concurrency happens
+at the bbeeggiinnnniinngg of a sequence of 1/f(N) negative feedback events. Otherwise, a
+correction for overload would be made too late. This makes the choice of f(N)
+relatively unimportant, as borne out by measurements later in this document.
+
+In summary, the main ingredients for the Postfix 2.5 concurrency feedback
+algorithm are a) the option of less-than-1 positive feedback per delivery to
+avoid overwhelming servers, b) the option of less-than-1 negative feedback per
+delivery to avoid giving up too fast, c) feedback hysteresis to avoid rapid
+oscillation, and d) a "reverse" hysteresis cycle for negative feedback, so that
+it can correct for overload quickly.
+
+SSuummmmaarryy ooff tthhee PPoossttffiixx 22..55 ""ddeeaadd ddeessttiinnaattiioonn"" ddeetteeccttiioonn aallggoorriitthhmm
+
+We want to suspend deliveries to a specific destination after some number of
+deliveries suffers connection or handshake failure. The old scheduler declares
+a destination "dead" when negative (-1) feedback throttles the delivery
+concurrency down to zero. With less-than-1 feedback per delivery, this
+throttling down would obviously take too long. We therefore have to separate
+"dead destination" detection from concurrency feedback. This is implemented by
+introducing the concept of pseudo-cohort failure. The Postfix 2.5 concurrency
+scheduler declares a destination "dead" after a configurable number of pseudo-
+cohorts suffers from connection or handshake failures. The old scheduler
+corresponds to the special case where the pseudo-cohort failure limit is equal
+to 1.
+
+PPsseeuuddooccooddee ffoorr tthhee PPoossttffiixx 22..55 ccoonnccuurrrreennccyy sscchheedduulleerr
+
+The pseudo code shows how the ideas behind new concurrency scheduler are
+implemented as of November 2007. The actual code can be found in the module
+qmgr/qmgr_queue.c.
+
+Types:
+        Each destination has one set of the following variables
+        int concurrency
+        double success
+        double failure
+        double fail_cohorts
+
+Feedback functions:
+        N is concurrency; x, y are arbitrary numbers in [0..1] inclusive
+        positive feedback: g(N) = x/N | x/sqrt(N) | x
+        negative feedback: f(N) = y/N | y/sqrt(N) | y
+
+Initialization:
+        concurrency = initial_concurrency
+        success = 0
+        failure = 0
+        fail_cohorts = 0
+
+After success:
+        fail_cohorts = 0
+        Be prepared for feedback > hysteresis, or rounding error
+        success += g(concurrency)
+        while (success >= 1)            Hysteresis 1
+            concurrency += 1            Hysteresis 1
+            failure = 0
+            success -= 1                Hysteresis 1
+        Be prepared for overshoot
+        if (concurrency > concurrency limit)
+            concurrency = concurrency limit
+
+Safety:
+        Don't apply positive feedback unless
+            concurrency < busy_refcount + init_dest_concurrency
+        otherwise negative feedback effect could be delayed
+
+After failure:
+        if (concurrency > 0)
+            fail_cohorts += 1.0 / concurrency
+            if (fail_cohorts > cohort_failure_limit)
+                concurrency = 0
+        if (concurrency > 0)
+            Be prepared for feedback > hysteresis, rounding errors
+            failure -= f(concurrency)
+            while (failure < 0)
+                concurrency -= 1        Hysteresis 1
+                failure += 1            Hysteresis 1
+                success = 0
+            Be prepared for overshoot
+            if (concurrency < 1)
+                concurrency = 1
+
+RReessuullttss ffoorr ddeelliivveerryy ttoo ccoonnccuurrrreennccyy--lliimmiitteedd sseerrvveerrss
+
+Discussions about the concurrency scheduler redesign started early 2004, when
+the primary goal was to find alternatives that did not exhibit exponential
+growth or rapid concurrency throttling. No code was implemented until late
+2007, when the primary concern had shifted towards better handling of server
+concurrency limits. For this reason we measure how well the new scheduler does
+this job. The table below compares mail delivery performance of the old +/-
+1 feedback per delivery with several less-than-1 feedback functions, for
+different limited-concurrency server scenarios. Measurements were done with a
+FreeBSD 6.2 client and with FreeBSD 6.2 and various Linux servers.
+
+Server configuration:
+
+  * The mail flow was slowed down with 1 second latency per recipient
+    ("smtpd_client_restrictions = sleep 1"). The purpose was to make results
+    less dependent on hardware details, by avoiding slow-downs by queue file I/
+    O, logging I/O, and network I/O.
+  * Concurrency was limited by the server process limit ("default_process_limit
+    = 5" and "smtpd_client_event_limit_exceptions = static:all"). Postfix was
+    stopped and started after changing the process limit, because the same
+    number is also used as the backlog argument to the listen(2) system call,
+    and "postfix reload" does not re-issue this call.
+  * Mail was discarded with "local_recipient_maps = static:all" and
+    "local_transport = discard". The discard action in access maps or header/
+    body checks could not be used as it fails to update the in_flow_delay
+    counters.
+
+Client configuration:
+
+  * Queue file overhead was minimized by sending one message to a virtual alias
+    that expanded into 2000 different remote recipients. All recipients were
+    accounted for according to the maillog file. The
+    virtual_alias_expansion_limit setting was increased to avoid complaints
+    from the cleanup(8) server.
+  * The number of deliveries was maximized with
+    "smtp_destination_recipient_limit = 2". A smaller limit would cause Postfix
+    to schedule the concurrency per recipient instead of domain, which is not
+    what we want.
+  * Maximum concurrency was limited with "smtp_destination_concurrency_limit =
+    20", and initial_destination_concurrency was set to the same value.
+  * The positive and negative concurrency feedback hysteresis was 1.
+    Concurrency was incremented by 1 at the END of 1/feedback steps of positive
+    feedback, and was decremented by 1 at the START of 1/feedback steps of
+    negative feedback.
+  * The SMTP client used the default 30s SMTP connect timeout and 300s SMTP
+    greeting timeout.
+
+IImmppaacctt ooff tthhee 3300ss SSMMTTPP ccoonnnneecctt ttiimmeeoouutt
+
+The first results are for a FreeBSD 6.2 server, where our artificially low
+listen(2) backlog results in a very short kernel queue for established
+connections. The table shows that all deferred deliveries failed due to a 30s
+connection timeout, and none failed due to a server greeting timeout. This
+measurement simulates what happens when the server's connection queue is
+completely full under load, and the TCP engine drops new connections.
+
+    cclliieenntt sseerrvveerr ffeeeeddbbaacckk  ccoonnnneeccttiioonn ppeerrcceennttaaggee cclliieenntt         ttiimmeedd--oouutt iinn
+    lliimmiitt  lliimmiitt  ssttyyllee     ccaacchhiinngg    ddeeffeerrrreedd   ccoonnccuurrrreennccyy    ccoonnnneecctt//
+                                                  aavveerraaggee//ssttddddeevv ggrreeeettiinngg
+
+    -------------------------------------------------------------------------
+       20     5       1/N         no        9.9   19.4    0.49   198      -
+
+       20     5       1/N        yes       10.3   19.4    0.49   206      -
+
+       20     5   1/sqrt(N)       no       10.4   19.6    0.59   208      -
+
+       20     5   1/sqrt(N)      yes       10.6   19.6    0.61   212      -
+
+       20     5         1         no       10.1   19.5    1.29   202      -
+
+       20     5         1        yes       10.8   19.3    1.57   216      -
+
+    -------------------------------------------------------------------------
+
+    A busy server with a completely full connection queue. N is the client
+    delivery concurrency. Failed deliveries time out after 30s without
+    completing the TCP handshake. See text for a discussion of results.
+
+IImmppaacctt ooff tthhee 330000ss SSMMTTPP ggrreeeettiinngg ttiimmeeoouutt
+
+The next table shows results for a Fedora Core 8 server (results for RedHat 7.3
+are identical). In this case, the artificially small listen(2) backlog argument
+does not impact our measurement. The table shows that practically all deferred
+deliveries fail after the 300s SMTP greeting timeout. As these timeouts were
+10x longer than with the first measurement, we increased the recipient count
+(and thus the running time) by a factor of 10 to keep the results comparable.
+The deferred mail percentages are a factor 10 lower than with the first
+measurement, because the 1s per-recipient delay was 1/300th of the greeting
+timeout instead of 1/30th of the connection timeout.
+
+    cclliieenntt sseerrvveerr ffeeeeddbbaacckk  ccoonnnneeccttiioonn ppeerrcceennttaaggee cclliieenntt         ttiimmeedd--oouutt iinn
+    lliimmiitt  lliimmiitt  ssttyyllee     ccaacchhiinngg    ddeeffeerrrreedd   ccoonnccuurrrreennccyy    ccoonnnneecctt//
+                                                  aavveerraaggee//ssttddddeevv ggrreeeettiinngg
+
+    -------------------------------------------------------------------------
+       20     5       1/N         no       1.16   19.8    0.37   -      230
+
+       20     5       1/N        yes       1.36   19.8    0.36   -      272
+
+       20     5   1/sqrt(N)       no       1.21   19.9    0.23   4      238
+
+       20     5   1/sqrt(N)      yes       1.36   20.0    0.23   -      272
+
+       20     5         1         no       1.18   20.0    0.16   -      236
+
+       20     5         1        yes       1.39   20.0    0.16   -      278
+
+    -------------------------------------------------------------------------
+
+    A busy server with a non-full connection queue. N is the client delivery
+    concurrency. Failed deliveries complete at the TCP level, but time out
+    after 300s while waiting for the SMTP greeting. See text for a discussion
+    of results.
+
+IImmppaacctt ooff aaccttiivvee sseerrvveerr ccoonnccuurrrreennccyy lliimmiitteerr
+
+The final concurrency-limited result shows what happens when SMTP connections
+don't time out, but are rejected immediately with the Postfix server's
+smtpd_client_connection_count_limit feature (the server replies with a 421
+status and disconnects immediately). Similar results can be expected with
+concurrency limiting features built into other MTAs or firewalls. For this
+measurement we specified a server concurrency limit and a client initial
+destination concurrency of 5, and a server process limit of 10; all other
+conditions were the same as with the first measurement. The same result would
+be obtained with a FreeBSD or Linux server, because the "pushing back" is done
+entirely by the receiving side.
+
+    cclliieenntt sseerrvveerr ffeeeeddbbaacckk  ccoonnnneeccttiioonn ppeerrcceennttaaggee cclliieenntt         tthheeoorreettiiccaall
+    lliimmiitt  lliimmiitt  ssttyyllee     ccaacchhiinngg    ddeeffeerrrreedd   ccoonnccuurrrreennccyy    ddeeffeerr rraattee
+                                                  aavveerraaggee//ssttddddeevv
+
+    -------------------------------------------------------------------------
+       20     5       1/N         no       16.5   5.17    0.38         1/6
+
+       20     5       1/N        yes       16.5   5.17    0.38         1/6
+
+       20     5   1/sqrt(N)       no       24.5   5.28    0.45         1/4
+
+       20     5   1/sqrt(N)      yes       24.3   5.28    0.46         1/4
+
+       20     5         1         no       49.7   5.63    0.67         1/2
+
+       20     5         1        yes       49.7   5.68    0.70         1/2
+
+    -------------------------------------------------------------------------
+
+    A server with active per-client concurrency limiter that replies with 421
+    and disconnects. N is the client delivery concurrency. The theoretical
+    defer rate is 1/(1+roundup(1/feedback)). This is always 1/2 with the fixed
+    +/-1 feedback per delivery; with the concurrency-dependent feedback
+    variants, the defer rate decreases with increasing concurrency. See text
+    for a discussion of results.
+
+DDiissccuussssiioonn ooff ccoonnccuurrrreennccyy--lliimmiitteedd sseerrvveerr rreessuullttss
+
+All results in the previous sections are based on the first delivery runs only;
+they do not include any second etc. delivery attempts. It's also worth noting
+that the measurements look at steady-state behavior only. They don't show what
+happens when the client starts sending at a much higher or lower concurrency.
+
+The first two examples show that the effect of feedback is negligible when
+concurrency is limited due to congestion. This is because the initial
+concurrency is already at the client's concurrency maximum, and because there
+is 10-100 times more positive than negative feedback. Under these conditions,
+it is no surprise that the contribution from SMTP connection caching is also
+negligible.
+
+In the last example, the old +/-1 feedback per delivery will defer 50% of the
+mail when confronted with an active (anvil-style) server concurrency limit,
+where the server hangs up immediately with a 421 status (a TCP-level RST would
+have the same result). Less aggressive feedback mechanisms fare better than
+more aggressive ones. Concurrency-dependent feedback fares even better at
+higher concurrencies than shown here, but has limitations as discussed in the
+next section.
+
+LLiimmiittaattiioonnss ooff lleessss--tthhaann--11 ppeerr ddeelliivveerryy ffeeeeddbbaacckk
+
+Less-than-1 feedback is of interest primarily when sending large amounts of
+mail to destinations with active concurrency limiters (servers that reply with
+421, or firewalls that send RST). When sending small amounts of mail per
+destination, less-than-1 per-delivery feedback won't have a noticeable effect
+on the per-destination concurrency, because the number of deliveries to the
+same destination is too small. You might just as well use zero per-delivery
+feedback and stay with the initial per-destination concurrency. And when mail
+deliveries fail due to congestion instead of active concurrency limiters, the
+measurements above show that per-delivery feedback has no effect. With large
+amounts of mail you might just as well use zero per-delivery feedback and start
+with the maximal per-destination concurrency.
+
+The scheduler with less-than-1 concurrency feedback per delivery solves a
+problem with servers that have active concurrency limiters. This works only
+because feedback is handled in a peculiar manner: positive feedback will
+increment the concurrency by 1 at the eenndd of a sequence of events of length 1/
+feedback, while negative feedback will decrement concurrency by 1 at the
+bbeeggiinnnniinngg of such a sequence. This is how Postfix adjusts quickly for overshoot
+without causing lots of mail to be deferred. Without this difference in
+feedback treatment, less-than-1 feedback per delivery would defer 50% of the
+mail, and would be no better in this respect than the old +/-1 feedback per
+delivery.
+
+Unfortunately, the same feature that corrects quickly for concurrency overshoot
+also makes the scheduler more sensitive for noisy negative feedback. The reason
+is that one lonely negative feedback event has the same effect as a complete
+sequence of length 1/feedback: in both cases delivery concurrency is dropped by
+1 immediately. As a worst-case scenario, consider multiple servers behind a
+load balancer on a single IP address, and no backup MX address. When 1 out of K
+servers fails to complete the SMTP handshake or drops the connection, a
+scheduler with 1/N (N = concurrency) feedback stops increasing its concurrency
+once it reaches a concurrency level of about K, even though the good servers
+behind the load balancer are perfectly capable of handling more traffic.
+
+This noise problem gets worse as the amount of positive feedback per delivery
+gets smaller. A compromise is to use fixed less-than-1 positive feedback values
+instead of concurrency-dependent positive feedback. For example, to tolerate 1
+of 4 bad servers in the above load balancer scenario, use positive feedback of
+1/4 per "good" delivery (no connect or handshake error), and use an equal or
+smaller amount of negative feedback per "bad" delivery. The downside of using
+concurrency-independent feedback is that some of the old +/-1 feedback problems
+will return at large concurrencies. Sites that must deliver mail at non-trivial
+per-destination concurrencies will require special configuration.
+
+CCoonnccuurrrreennccyy ccoonnffiigguurraattiioonn ppaarraammeetteerrss
+
+The Postfix 2.5 concurrency scheduler is controlled with the following
+configuration parameters, where "transport_foo" provides a transport-specific
+parameter override. All parameter default settings are compatible with earlier
+Postfix versions.
+
+    PPaarraammeetteerr nnaammee                                        PPoossttffiixx DDeessccrriippttiioonn
+                                                          vveerrssiioonn
+
+    ---------------------------------------------------------------------------
+                                                                  Initial per-
+    initial_destination_concurrency                          all  destination
+    transport_initial_destination_concurrency                2.5  delivery
+                                                                  concurrency
+
+                                                                  Maximum per-
+    default_destination_concurrency_limit                    all  destination
+    transport_destination_concurrency_limit                  all  delivery
+                                                                  concurrency
+
+                                                                  Per-
+                                                                  destination
+                                                                  positive
+                                                                  feedback
+    default_destination_concurrency_positive_feedback        2.5  amount, per
+    transport_destination_concurrency_positive_feedback      2.5  delivery that
+                                                                  does not fail
+                                                                  with
+                                                                  connection or
+                                                                  handshake
+                                                                  failure
+
+                                                                  Per-
+                                                                  destination
+                                                                  negative
+                                                                  feedback
+    default_destination_concurrency_negative_feedback        2.5  amount, per
+    transport_destination_concurrency_negative_feedback      2.5  delivery that
+                                                                  fails with
+                                                                  connection or
+                                                                  handshake
+                                                                  failure
+
+                                                                  Number of
+                                                                  failed
+                                                                  pseudo-
+                                                                  cohorts after
+    default_destination_concurrency_failed_cohort_limit      2.5  which a
+    transport_destination_concurrency_failed_cohort_limit    2.5  destination
+                                                                  is declared
+                                                                  "dead" and
+                                                                  delivery is
+                                                                  suspended
+
+                                                                  Enable
+                                                                  verbose
+    destination_concurrency_feedback_debug                   2.5  logging of
+                                                                  concurrency
+                                                                  scheduler
+                                                                  activity
+
+    ---------------------------------------------------------------------------
+
+PPrreeeemmppttiivvee sscchheedduulliinngg
+
+The following sections describe the new queue manager and its preemptive
+scheduler algorithm. Note that the document was originally written to describe
+the changes between the new queue manager (in this text referred to as nqmgr,
+the name it was known by before it became the default queue manager) and the
+old queue manager (referred to as oqmgr). This is why it refers to oqmgr every
+so often.
+
+This document is divided into sections as follows:
+
+  * The structures used by nqmgr
+  * What happens when nqmgr picks up the message - how it is assigned to
+    transports, jobs, peers, entries
+  * How the entry selection works
+  * How the preemption works - what messages may be preempted and how and what
+    messages are chosen to preempt them
+  * How destination concurrency limits affect the scheduling algorithm
+  * Dealing with memory resource limits
+
+TThhee ssttrruuccttuurreess uusseedd bbyy nnqqmmggrr
+
+Let's start by recapitulating the structures and terms used when referring to
+queue manager and how it operates. Many of these are partially described
+elsewhere, but it is nice to have a coherent overview in one place:
+
+  * Each message structure represents one mail message which Postfix is to
+    deliver. The message recipients specify to what destinations is the message
+    to be delivered and what transports are going to be used for the delivery.
+
+  * Each recipient entry groups a batch of recipients of one message which are
+    all going to be delivered to the same destination (and over the same
+    transport).
+
+  * Each transport structure groups everything what is going to be delivered by
+    delivery agents dedicated for that transport. Each transport maintains a
+    set of queues (describing the destinations it shall talk to) and jobs
+    (referencing the messages it shall deliver).
+
+  * Each transport queue (not to be confused with the on-disk active queue or
+    incoming queue) groups everything what is going be delivered to given
+    destination (aka nexthop) by its transport. Each queue belongs to one
+    transport, so each destination may be referred to by several queues, one
+    for each transport. Each queue maintains a list of all recipient entries
+    (batches of message recipients) which shall be delivered to given
+    destination (the todo list), and a list of recipient entries already being
+    delivered by the delivery agents (the busy list).
+
+  * Each queue corresponds to multiple peer structures. Each peer structure is
+    like the queue structure, belonging to one transport and referencing one
+    destination. The difference is that it lists only the recipient entries
+    which all originate from the same message, unlike the queue structure,
+    whose entries may originate from various messages. For messages with few
+    recipients, there is usually just one recipient entry for each destination,
+    resulting in one recipient entry per peer. But for large mailing list
+    messages the recipients may need to be split to multiple recipient entries,
+    in which case the peer structure may list many entries for single
+    destination.
+
+  * Each transport job groups everything it takes to deliver one message via
+    its transport. Each job represents one message within the context of the
+    transport. The job belongs to one transport and message, so each message
+    may have multiple jobs, one for each transport. The job groups all the peer
+    structures, which describe the destinations the job's message has to be
+    delivered to.
+
+The first four structures are common to both nqmgr and oqmgr, the latter two
+were introduced by nqmgr.
+
+These terms are used extensively in the text below, feel free to look up the
+description above anytime you'll feel you have lost a sense what is what.
+
+WWhhaatt hhaappppeennss wwhheenn nnqqmmggrr ppiicckkss uupp tthhee mmeessssaaggee
+
+Whenever nqmgr moves a queue file into the active queue, the following happens:
+It reads all necessary information from the queue file as oqmgr does, and also
+reads as many recipients as possible - more on that later, for now let's just
+pretend it always reads all recipients.
+
+Then it resolves the recipients as oqmgr does, which means obtaining (address,
+nexthop, transport) triple for each recipient. For each triple, it finds the
+transport; if it does not exist yet, it instantiates it (unless it's dead).
+Within the transport, it finds the destination queue for given nexthop; if it
+does not exist yet, it instantiates it (unless it's dead). The triple is then
+bound to given destination queue. This happens in qmgr_resolve() and is
+basically the same as in oqmgr.
+
+Then for each triple which was bound to some queue (and thus transport), the
+program finds the job which represents the message within that transport's
+context; if it does not exist yet, it instantiates it. Within the job, it finds
+the peer which represents the bound destination queue within this jobs context;
+if it does not exist yet, it instantiates it. Finally, it stores the address
+from the resolved triple to the recipient entry which is appended to both the
+queue entry list and the peer entry list. The addresses for same nexthop are
+batched in the entries up to recipient_concurrency limit for that transport.
+This happens in qmgr_assign() and apart from that it operates with job and peer
+structures it is basically the same as in oqmgr.
+
+When the job is instantiated, it is enqueued on the transport's job list based
+on the time its message was picked up by nqmgr. For first batch of recipients
+this means it is appended to the end of the job list, but the ordering of the
+job list by the enqueue time is important as we will see shortly.
+
+[Now you should have pretty good idea what is the state of the nqmgr after
+couple of messages was picked up, what is the relation between all those job,
+peer, queue and entry structures.]
+
+HHooww tthhee eennttrryy sseelleeccttiioonn wwoorrkkss
+
+Having prepared all those above mentioned structures, the task of the nqmgr's
+scheduler is to choose the recipient entries one at a time and pass them to the
+delivery agent for corresponding transport. Now how does this work?
+
+The first approximation of the new scheduling algorithm is like this:
+
+    foreach transport (round-robin-by-transport)
+    do
+        if transport busy continue
+        if transport process limit reached continue
+        foreach transport's job (in the order of the transport's job list)
+        do
+            foreach job's peer (round-robin-by-destination)
+                 if peer->queue->concurrency < peer->queue->window
+                     return next peer entry.
+            done
+        done
+    done
+
+Now what is the "order of the transport's job list"? As we know already, the
+job list is by default kept in the order the message was picked up by the
+nqmgr. So by default we get the top-level round-robin transport, and within
+each transport we get the FIFO message delivery. The round-robin of the peers
+by the destination is perhaps of little importance in most real-life cases
+(unless the recipient_concurrency limit is reached, in one job there is only
+one peer structure for each destination), but theoretically it makes sure that
+even within single jobs, destinations are treated fairly.
+
+[By now you should have a feeling you really know how the scheduler works,
+except for the preemption, under ideal conditions - that is, no recipient
+resource limits and no destination concurrency problems.]
+
+HHooww tthhee pprreeeemmppttiioonn wwoorrkkss
+
+As you might perhaps expect by now, the transport's job list does not remain
+sorted by the job's message enqueue time all the time. The most cool thing
+about nqmgr is not the simple FIFO delivery, but that it is able to slip mail
+with little recipients past the mailing-list bulk mail. This is what the job
+preemption is about - shuffling the jobs on the transport's job list to get the
+best message delivery rates. Now how is it achieved?
+
+First I have to tell you that there are in fact two job lists in each
+transport. One is the scheduler's job list, which the scheduler is free to play
+with, while the other one keeps the jobs always listed in the order of the
+enqueue time and is used for recipient pool management we will discuss later.
+For now, we will deal with the scheduler's job list only.
+
+So, we have the job list, which is first ordered by the time the jobs' messages
+were enqueued, oldest messages first, the most recently picked one at the end.
+For now, let's assume that there are no destination concurrency problems.
+Without preemption, we pick some entry of the first (oldest) job on the queue,
+assign it to delivery agent, pick another one from the same job, assign it
+again, and so on, until all the entries are used and the job is delivered. We
+would then move onto the next job and so on and on. Now how do we manage to
+sneak in some entries from the recently added jobs when the first job on the
+job list belongs to a message going to the mailing-list and has thousands of
+recipient entries?
+
+The nqmgr's answer is that we can artificially "inflate" the delivery time of
+that first job by some constant for free - it is basically the same trick you
+might remember as "accumulation of potential" from the amortized complexity
+lessons. For example, instead of delivering the entries of the first job on the
+job list every time a delivery agent becomes available, we can do it only every
+second time. If you view the moments the delivery agent becomes available on a
+timeline as "delivery slots", then instead of using every delivery slot for the
+first job, we can use only every other slot, and still the overall delivery
+efficiency of the first job remains the same. So the delivery 11112222 becomes
+1.1.1.1.2.2.2.2 (1 and 2 are the imaginary job numbers, . denotes the free
+slot). Now what do we do with free slots?
+
+As you might have guessed, we will use them for sneaking the mail with little
+recipients in. For example, if we have one four-recipient mail followed by four
+one recipients mail, the delivery sequence (that is, the sequence in which the
+jobs are assigned to the delivery slots) might look like this: 12131415. Hmm,
+fine for sneaking in the single recipient mail, but how do we sneak in the mail
+with more than one recipient? Say if we have one four-recipient mail followed
+by two two-recipient mails?
+
+The simple answer would be to use delivery sequence 12121313. But the problem
+is that this does not scale well. Imagine you have mail with thousand
+recipients followed by mail with hundred recipients. It is tempting to suggest
+the delivery sequence like 121212...., but alas! Imagine there arrives another
+mail with say ten recipients. But there are no free slots anymore, so it can't
+slip by, not even if it had just only one recipients. It will be stuck until
+the hundred-recipient mail is delivered, which really sucks.
+
+So, it becomes obvious that while inflating the message to get free slots is
+great idea, one has to be really careful of how the free slots are assigned,
+otherwise one might corner himself. So, how does nqmgr really use the free
+slots?
+
+The key idea is that one does not have to generate the free slots in a uniform
+way. The delivery sequence 111...1 is no worse than 1.1.1.1, in fact, it is
+even better as some entries are in the first case selected earlier than in the
+second case, and none is selected later! So it is possible to first
+"accumulate" the free delivery slots and then use them all at once. It is even
+possible to accumulate some, then use them, then accumulate some more and use
+them again, as in 11..1.1 .
+
+Let's get back to the one hundred recipient example. We now know that we could
+first accumulate one hundred free slots, and only after then to preempt the
+first job and sneak the one hundred recipient mail in. Applying the algorithm
+recursively, we see the hundred recipient job can accumulate ten free delivery
+slots, and then we could preempt it and sneak in the ten-recipient mail... Wait
+wait wait! Could we? Aren't we overinflating the original one thousand
+recipient mail?
+
+Well, despite it looks so at the first glance, another trick will allow us to
+answer "no, we are not!". If we had said that we will inflate the delivery time
+twice at maximum, and then we consider every other slot as a free slot, then we
+would overinflate in case of the recursive preemption. BUT! The trick is that
+if we use only every n-th slot as a free slot for n>2, there is always some
+worst inflation factor which we can guarantee not to be breached, even if we
+apply the algorithm recursively. To be precise, if for every k>1 normally used
+slots we accumulate one free delivery slot, than the inflation factor is not
+worse than k/(k-1) no matter how many recursive preemptions happen. And it's
+not worse than (k+1)/k if only non-recursive preemption happens. Now, having
+got through the theory and the related math, let's see how nqmgr implements
+this.
+
+Each job has so called "available delivery slot" counter. Each transport has a
+transport_delivery_slot_cost parameter, which defaults to
+default_delivery_slot_cost parameter which is set to 5 by default. This is the
+k from the paragraph above. Each time k entries of the job are selected for
+delivery, this counter is incremented by one. Once there are some slots
+accumulated, job which requires no more than that number of slots to be fully
+delivered can preempt this job.
+
+[Well, the truth is, the counter is incremented every time an entry is selected
+and it is divided by k when it is used. But for the understanding it's good
+enough to use the above approximation of the truth.]
+
+OK, so now we know the conditions which must be satisfied so one job can
+preempt another one. But what job gets preempted, how do we choose what job
+preempts it if there are several valid candidates, and when does all this
+exactly happen?
+
+The answer for the first part is simple. The job whose entry was selected the
+last time is so called current job. Normally, it is the first job on the
+scheduler's job list, but destination concurrency limits may change this as we
+will see later. It is always only the current job which may get preempted.
+
+Now for the second part. The current job has certain amount of recipient
+entries, and as such may accumulate at maximum some amount of available
+delivery slots. It might have already accumulated some, and perhaps even
+already used some when it was preempted before (remember a job can be preempted
+several times). In either case, we know how many are accumulated and how many
+are left to deliver, so we know how many it may yet accumulate at maximum.
+Every other job which may be delivered by less than that number of slots is a
+valid candidate for preemption. How do we choose among them?
+
+The answer is - the one with maximum enqueue_time/recipient_entry_count. That
+is, the older the job is, the more we should try to deliver it in order to get
+best message delivery rates. These rates are of course subject to how many
+recipients the message has, therefore the division by the recipient (entry)
+count. No one shall be surprised that message with n recipients takes n times
+longer to deliver than message with one recipient.
+
+Now let's recap the previous two paragraphs. Isn't it too complicated? Why
+don't the candidates come only among the jobs which can be delivered within the
+number of slots the current job already accumulated? Why do we need to estimate
+how much it has yet to accumulate? If you found out the answer, congratulate
+yourself. If we did it this simple way, we would always choose the candidate
+with least recipient entries. If there were enough single recipient mails
+coming in, they would always slip by the bulk mail as soon as possible, and the
+two and more recipients mail would never get a chance, no matter how long they
+have been sitting around in the job list.
+
+This candidate selection has interesting implication - that when we choose the
+best candidate for preemption (this is done in qmgr_choose_candidate()), it may
+happen that we may not use it for preemption immediately. This leads to an
+answer to the last part of the original question - when does the preemption
+happen?
+
+The preemption attempt happens every time next transport's recipient entry is
+to be chosen for delivery. To avoid needless overhead, the preemption is not
+attempted if the current job could never accumulate more than
+transport_minimum_delivery_slots (defaults to default_minimum_delivery_slots
+which defaults to 3). If there is already enough accumulated slots to preempt
+the current job by the chosen best candidate, it is done immediately. This
+basically means that the candidate is moved in front of the current job on the
+scheduler's job list and decreasing the accumulated slot counter by the amount
+used by the candidate. If there is not enough slots... well, I could say that
+nothing happens and the another preemption is attempted the next time. But
+that's not the complete truth.
+
+The truth is that it turns out that it is not really necessary to wait until
+the jobs counter accumulates all the delivery slots in advance. Say we have
+ten-recipient mail followed by two two-recipient mails. If the preemption
+happened when enough delivery slot accumulate (assuming slot cost 2), the
+delivery sequence becomes 11112211113311. Now what would we get if we would
+wait only for 50% of the necessary slots to accumulate and we promise we would
+wait for the remaining 50% later, after we get back to the preempted job? If we
+use such slot loan, the delivery sequence becomes 11221111331111. As we can
+see, it makes it no considerably worse for the delivery of the ten-recipient
+mail, but it allows the small messages to be delivered sooner.
+
+The concept of these slot loans is where the transport_delivery_slot_discount
+and transport_delivery_slot_loan come from (they default to
+default_delivery_slot_discount and default_delivery_slot_loan, whose values are
+by default 50 and 3, respectively). The discount (resp. loan) specifies how
+many percent (resp. how many slots) one "gets in advance", when the number of
+slots required to deliver the best candidate is compared with the number of
+slots the current slot had accumulated so far.
+
+And it pretty much concludes this chapter.
+
+[Now you should have a feeling that you pretty much understand the scheduler
+and the preemption, or at least that you will have it after you read the last
+chapter couple more times. You shall clearly see the job list and the
+preemption happening at its head, in ideal delivery conditions. The feeling of
+understanding shall last until you start wondering what happens if some of the
+jobs are blocked, which you might eventually figure out correctly from what had
+been said already. But I would be surprised if your mental image of the
+scheduler's functionality is not completely shattered once you start wondering
+how it works when not all recipients may be read in-core. More on that later.]
+
+HHooww ddeessttiinnaattiioonn ccoonnccuurrrreennccyy lliimmiittss aaffffeecctt tthhee sscchheedduulliinngg aallggoorriitthhmm
+
+The nqmgr uses the same algorithm for destination concurrency control as oqmgr.
+Now what happens when the destination limits are reached and no more entries
+for that destination may be selected by the scheduler?
+
+From user's point of view it is all simple. If some of the peers of a job can't
+be selected, those peers are simply skipped by the entry selection algorithm
+(the pseudo-code described before) and only the selectable ones are used. If
+none of the peers may be selected, the job is declared a "blocker job". Blocker
+jobs are skipped by the entry selection algorithm and they are also excluded
+from the candidates for preemption of current job. Thus the scheduler
+effectively behaves as if the blocker jobs didn't exist on the job list at all.
+As soon as at least one of the peers of a blocker job becomes unblocked (that
+is, the delivery agent handling the delivery of the recipient entry for given
+destination successfully finishes), the job's blocker status is removed and the
+job again participates in all further scheduler actions normally.
+
+So the summary is that the users don't really have to be concerned about the
+interaction of the destination limits and scheduling algorithm. It works well
+on its own and there are no knobs they would need to control it.
+
+From a programmer's point of view, the blocker jobs complicate the scheduler
+quite a lot. Without them, the jobs on the job list would be normally delivered
+in strict FIFO order. If the current job is preempted, the job preempting it is
+completely delivered unless it is preempted itself. Without blockers, the
+current job is thus always either the first job on the job list, or the top of
+the stack of jobs preempting the first job on the job list.
+
+The visualization of the job list and the preemption stack without blockers
+would be like this:
+
+    first job->    1--2--3--5--6--8--...    <- job list
+    on job list    |
+                   4    <- preemption stack
+                   |
+    current job->  7
+
+In the example above we see that job 1 was preempted by job 4 and then job 4
+was preempted by job 7. After job 7 is completed, remaining entries of job 4
+are selected, and once they are all selected, job 1 continues.
+
+As we see, it's all very clean and straightforward. Now how does this change
+because of blockers?
+
+The answer is: a lot. Any job may become blocker job at any time, and also
+become normal job again at any time. This has several important implications:
+
+ 1. The jobs may be completed in arbitrary order. For example, in the example
+    above, if the current job 7 becomes blocked, the next job 4 may complete
+    before the job 7 becomes unblocked again. Or if both 7 and 4 are blocked,
+    then 1 is completed, then 7 becomes unblocked and is completed, then 2 is
+    completed and only after that 4 becomes unblocked and is completed... You
+    get the idea.
+
+    [Interesting side note: even when jobs are delivered out of order, from
+    single destination's point of view the jobs are still delivered in the
+    expected order (that is, FIFO unless there was some preemption involved).
+    This is because whenever a destination queue becomes unblocked (the
+    destination limit allows selection of more recipient entries for that
+    destination), all jobs which have peers for that destination are unblocked
+    at once.]
+
+ 2. The idea of the preemption stack at the head of the job list is gone. That
+    is, it must be possible to preempt any job on the job list. For example, if
+    the jobs 7, 4, 1 and 2 in the example above become all blocked, job 3
+    becomes the current job. And of course we do not want the preemption to be
+    affected by the fact that there are some blocked jobs or not. Therefore, if
+    it turns out that job 3 might be preempted by job 6, the implementation
+    shall make it possible.
+
+ 3. The idea of the linear preemption stack itself is gone. It's no longer true
+    that one job is always preempted by only one job at one time (that is
+    directly preempted, not counting the recursively nested jobs). For example,
+    in the example above, job 1 is directly preempted by only job 4, and job 4
+    by job 7. Now assume job 7 becomes blocked, and job 4 is being delivered.
+    If it accumulates enough delivery slots, it is natural that it might be
+    preempted for example by job 8. Now job 4 is preempted by both job 7 AND
+    job 8 at the same time.
+
+Now combine the points 2) and 3) with point 1) again and you realize that the
+relations on the once linear job list became pretty complicated. If we extend
+the point 3) example: jobs 7 and 8 preempt job 4, now job 8 becomes blocked
+too, then job 4 completes. Tricky, huh?
+
+If I illustrate the relations after the above mentioned examples (but those in
+point 1)), the situation would look like this:
+
+                                v- parent
+
+    adoptive parent ->    1--2--3--5--...      <- "stack" level 0
+                          |     |
+    parent gone ->        ?     6              <- "stack" level 1
+                         / \
+    children ->         7   8   ^- child       <- "stack" level 2
+
+                          ^- siblings
+
+Now how does nqmgr deal with all these complicated relations?
+
+Well, it maintains them all as described, but fortunately, all these relations
+are necessary only for purposes of proper counting of available delivery slots.
+For purposes of ordering the jobs for entry selection, the original rule still
+applies: "the job preempting the current job is moved in front of the current
+job on the job list". So for entry selection purposes, the job relations remain
+as simple as this:
+
+    7--8--1--2--6--3--5--..   <- scheduler's job list order
+
+The job list order and the preemption parent/child/siblings relations are
+maintained separately. And because the selection works only with the job list,
+you can happily forget about those complicated relations unless you want to
+study the nqmgr sources. In that case the text above might provide some helpful
+introduction to the problem domain. Otherwise I suggest you just forget about
+all this and stick with the user's point of view: the blocker jobs are simply
+ignored.
+
+[By now, you should have a feeling that there is more things going under the
+hood than you ever wanted to know. You decide that forgetting about this
+chapter is the best you can do for the sake of your mind's health and you
+basically stick with the idea how the scheduler works in ideal conditions, when
+there are no blockers, which is good enough.]
+
+DDeeaalliinngg wwiitthh mmeemmoorryy rreessoouurrccee lliimmiittss
+
+When discussing the nqmgr scheduler, we have so far assumed that all recipients
+of all messages in the active queue are completely read into the memory. This
+is simply not true. There is an upper bound on the amount of memory the nqmgr
+may use, and therefore it must impose some limits on the information it may
+store in the memory at any given time.
+
+First of all, not all messages may be read in-core at once. At any time, only
+qmgr_message_active_limit messages may be read in-core at maximum. When read
+into memory, the messages are picked from the incoming and deferred message
+queues and moved to the active queue (incoming having priority), so if there is
+more than qmgr_message_active_limit messages queued in the active queue, the
+rest will have to wait until (some of) the messages in the active queue are
+completely delivered (or deferred).
+
+Even with the limited amount of in-core messages, there is another limit which
+must be imposed in order to avoid memory exhaustion. Each message may contain
+huge amount of recipients (tens or hundreds of thousands are not uncommon), so
+if nqmgr read all recipients of all messages in the active queue, it may easily
+run out of memory. Therefore there must be some upper bound on the amount of
+message recipients which are read into the memory at the same time.
+
+Before discussing how exactly nqmgr implements the recipient limits, let's see
+how the sole existence of the limits themselves affects the nqmgr and its
+scheduler.
+
+The message limit is straightforward - it just limits the size of the lookahead
+the nqmgr's scheduler has when choosing which message can preempt the current
+one. Messages not in the active queue simply are not considered at all.
+
+The recipient limit complicates more things. First of all, the message reading
+code must support reading the recipients in batches, which among other things
+means accessing the queue file several times and continuing where the last
+recipient batch ended. This is invoked by the scheduler whenever the current
+job has space for more recipients, subject to transport's refill_limit and
+refill_delay parameters. It is also done any time when all in-core recipients
+of the message are dealt with (which may also mean they were deferred) but
+there are still more in the queue file.
+
+The second complication is that with some recipients left unread in the queue
+file, the scheduler can't operate with exact counts of recipient entries. With
+unread recipients, it is not clear how many recipient entries there will be, as
+they are subject to per-destination grouping. It is not even clear to what
+transports (and thus jobs) the recipients will be assigned. And with messages
+coming from the deferred queue, it is not even clear how many unread recipients
+are still to be delivered. This all means that the scheduler must use only
+estimates of how many recipients entries there will be. Fortunately, it is
+possible to estimate the minimum and maximum correctly, so the scheduler can
+always err on the safe side. Obviously, the better the estimates, the better
+results, so it is best when we are able to read all recipients in-core and turn
+the estimates into exact counts, or at least try to read as many as possible to
+make the estimates as accurate as possible.
+
+The third complication is that it is no longer true that the scheduler is done
+with a job once all of its in-core recipients are delivered. It is possible
+that the job will be revived later, when another batch of recipients is read in
+core. It is also possible that some jobs will be created for the first time
+long after the first batch of recipients was read in core. The nqmgr code must
+be ready to handle all such situations.
+
+And finally, the fourth complication is that the nqmgr code must somehow impose
+the recipient limit itself. Now how does it achieve it?
+
+Perhaps the easiest solution would be to say that each message may have at
+maximum X recipients stored in-core, but such solution would be poor for
+several reasons. With reasonable qmgr_message_active_limit values, the X would
+have to be quite low to maintain reasonable memory footprint. And with low X
+lots of things would not work well. The nqmgr would have problems to use the
+transport_destination_recipient_limit efficiently. The scheduler's preemption
+would be suboptimal as the recipient count estimates would be inaccurate. The
+message queue file would have to be accessed many times to read in more
+recipients again and again.
+
+Therefore it seems reasonable to have a solution which does not use a limit
+imposed on per-message basis, but which maintains a pool of available recipient
+slots, which can be shared among all messages in the most efficient manner. And
+as we do not want separate transports to compete for resources whenever
+possible, it seems appropriate to maintain such recipient pool for each
+transport separately. This is the general idea, now how does it work in
+practice?
+
+First we have to solve little chicken-and-egg problem. If we want to use the
+per-transport recipient pools, we first need to know to what transport(s) is
+the message assigned. But we will find that out only after we read in the
+recipients first. So it is obvious that we first have to read in some
+recipients, use them to find out to what transports is the message to be
+assigned, and only after that we can use the per-transport recipient pools.
+
+Now how many recipients shall we read for the first time? This is what
+qmgr_message_recipient_minimum and qmgr_message_recipient_limit values control.
+The qmgr_message_recipient_minimum value specifies how many recipients of each
+message we will read for the first time, no matter what. It is necessary to
+read at least one recipient before we can assign the message to a transport and
+create the first job. However, reading only qmgr_message_recipient_minimum
+recipients even if there are only few messages with few recipients in-core
+would be wasteful. Therefore if there is less than qmgr_message_recipient_limit
+recipients in-core so far, the first batch of recipients may be larger than
+qmgr_message_recipient_minimum - as large as is required to reach the
+qmgr_message_recipient_limit limit.
+
+Once the first batch of recipients was read in core and the message jobs were
+created, the size of the subsequent recipient batches (if any - of course it's
+best when all recipients are read in one batch) is based solely on the position
+of the message jobs on their corresponding transports' job lists. Each
+transport has a pool of transport_recipient_limit recipient slots which it can
+distribute among its jobs (how this is done is described later). The subsequent
+recipient batch may be as large as the sum of all recipient slots of all jobs
+of the message permits (plus the qmgr_message_recipient_minimum amount which
+always applies).
+
+For example, if a message has three jobs, first with 1 recipient still in-core
+and 4 recipient slots, second with 5 recipient in-core and 5 recipient slots,
+and third with 2 recipients in-core and 0 recipient slots, it has 1+5+2=7
+recipients in-core and 4+5+0=9 jobs' recipients slots in total. This means that
+we could immediately read 2+qmgr_message_recipient_minimum more recipients of
+that message in core.
+
+The above example illustrates several things which might be worth mentioning
+explicitly: first, note that although the per-transport slots are assigned to
+particular jobs, we can't guarantee that once the next batch of recipients is
+read in core, that the corresponding amounts of recipients will be assigned to
+those jobs. The jobs lend its slots to the message as a whole, so it is
+possible that some jobs end up sponsoring other jobs of their message. For
+example, if in the example above the 2 newly read recipients were assigned to
+the second job, the first job sponsored the second job with 2 slots. The second
+notable thing is the third job, which has more recipients in-core than it has
+slots. Apart from sponsoring by other job we just saw it can be result of the
+first recipient batch, which is sponsored from global recipient pool of
+qmgr_message_recipient_limit recipients. It can be also sponsored from the
+message recipient pool of qmgr_message_recipient_minimum recipients.
+
+Now how does each transport distribute the recipient slots among its jobs? The
+strategy is quite simple. As most scheduler activity happens on the head of the
+job list, it is our intention to make sure that the scheduler has the best
+estimates of the recipient counts for those jobs. As we mentioned above, this
+means that we want to try to make sure that the messages of those jobs have all
+recipients read in-core. Therefore the transport distributes the slots "along"
+the job list from start to end. In this case the job list sorted by message
+enqueue time is used, because it doesn't change over time as the scheduler's
+job list does.
+
+More specifically, each time a job is created and appended to the job list, it
+gets all unused recipient slots from its transport's pool. It keeps them until
+all recipients of its message are read. When this happens, all unused recipient
+slots are transferred to the next job (which is now in fact now first such job)
+on the job list which still has some recipients unread, or eventually back to
+the transport pool if there is no such job. Such transfer then also happens
+whenever a recipient entry of that job is delivered.
+
+There is also a scenario when a job is not appended to the end of the job list
+(for example it was created as a result of second or later recipient batch).
+Then it works exactly as above, except that if it was put in front of the first
+unread job (that is, the job of a message which still has some unread
+recipients in queue file), that job is first forced to return all of its unused
+recipient slots to the transport pool.
+
+The algorithm just described leads to the following state: The first unread job
+on the job list always gets all the remaining recipient slots of that transport
+(if there are any). The jobs queued before this job are completely read (that
+is, all recipients of their message were already read in core) and have at
+maximum as many slots as they still have recipients in-core (the maximum is
+there because of the sponsoring mentioned before) and the jobs after this job
+get nothing from the transport recipient pool (unless they got something before
+and then the first unread job was created and enqueued in front of them later -
+in such case the also get at maximum as many slots as they have recipients in-
+core).
+
+Things work fine in such state for most of the time, because the current job is
+either completely read in-core or has as much recipient slots as there are, but
+there is one situation which we still have to take care of specially. Imagine
+if the current job is preempted by some unread job from the job list and there
+are no more recipient slots available, so this new current job could read only
+batches of qmgr_message_recipient_minimum recipients at a time. This would
+really degrade performance. For this reason, each transport has extra pool of
+transport_extra_recipient_limit recipient slots, dedicated exactly for this
+situation. Each time an unread job preempts the current job, it gets half of
+the remaining recipient slots from the normal pool and this extra pool.
+
+And that's it. It sure does sound pretty complicated, but fortunately most
+people don't really have to care how exactly it works as long as it works.
+Perhaps the only important things to know for most people are the following
+upper bound formulas:
+
+Each transport has at maximum
+
+    max(
+    qmgr_message_recipient_minimum * qmgr_message_active_limit
+    + *_recipient_limit + *_extra_recipient_limit,
+    qmgr_message_recipient_limit
+    )
+
+recipients in core.
+
+The total amount of recipients in core is
+
+    max(
+    qmgr_message_recipient_minimum * qmgr_message_active_limit
+    + sum( *_recipient_limit + *_extra_recipient_limit ),
+    qmgr_message_recipient_limit
+    )
+
+where the sum is over all used transports.
+
+And this terribly complicated chapter concludes the documentation of nqmgr
+scheduler.
+
+[By now you should theoretically know the nqmgr scheduler inside out. In
+practice, you still hope that you will never have to really understand the last
+or last two chapters completely, and fortunately most people really won't.
+Understanding how the scheduler works in ideal conditions is more than good
+enough for vast majority of users.]
+
+CCrreeddiittss
+
+  * Wietse Venema designed and implemented the initial queue manager with per-
+    domain FIFO scheduling, and per-delivery +/-1 concurrency feedback.
+  * Patrik Rak designed and implemented preemption where mail with fewer
+    recipients can slip past mail with more recipients in a controlled manner,
+    and wrote up its documentation.
+  * Wietse Venema initiated a discussion with Patrik Rak and Victor Duchovni on
+    alternatives for the +/-1 feedback scheduler's aggressive behavior. This is
+    when K/N feedback was reviewed (N = concurrency). The discussion ended
+    without a good solution for both negative feedback and dead site detection.
+  * Victor Duchovni resumed work on concurrency feedback in the context of
+    concurrency-limited servers.
+  * Wietse Venema then re-designed the concurrency scheduler in terms of the
+    simplest possible concepts: less-than-1 concurrency feedback per delivery,
+    forward and reverse concurrency feedback hysteresis, and pseudo-cohort
+    failure. At this same time, concurrency feedback was separated from dead
+    site detection.
+  * These simplifications, and their modular implementation, helped to develop
+    further insights into the different roles that positive and negative
+    concurrency feedback play, and helped to identify some worst-case
+    scenarios.
+