Adding upstream version 2.9.5.upstream/2.9.5

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 1700 insertions, 0 deletions

diff --git a/doc/intro.txt b/doc/intro.txt
new file mode 100644
index 0000000..f4133a1
--- /dev/null
+++ b/doc/intro.txt
@@ -0,0 +1,1700 @@
+                             -----------------------
+                              HAProxy Starter Guide
+                             -----------------------
+                                   version 2.9
+
+
+This document is an introduction to HAProxy for all those who don't know it, as
+well as for those who want to re-discover it when they know older versions. Its
+primary focus is to provide users with all the elements to decide if HAProxy is
+the product they're looking for or not. Advanced users may find here some parts
+of solutions to some ideas they had just because they were not aware of a given
+new feature. Some sizing information is also provided, the product's lifecycle
+is explained, and comparisons with partially overlapping products are provided.
+
+This document doesn't provide any configuration help or hints, but it explains
+where to find the relevant documents. The summary below is meant to help you
+search sections by name and navigate through the document.
+
+Note to documentation contributors :
+    This document is formatted with 80 columns per line, with even number of
+    spaces for indentation and without tabs. Please follow these rules strictly
+    so that it remains easily printable everywhere. If you add sections, please
+    update the summary below for easier searching.
+
+
+Summary
+-------
+
+1. Available documentation
+
+2. Quick introduction to load balancing and load balancers
+
+3.    Introduction to HAProxy
+3.1.      What HAProxy is and is not
+3.2.      How HAProxy works
+3.3.      Basic features
+3.3.1.        Proxying
+3.3.2.        SSL
+3.3.3.        Monitoring
+3.3.4.        High availability
+3.3.5.        Load balancing
+3.3.6.        Stickiness
+3.3.7.        Logging
+3.3.8.        Statistics
+3.4.      Standard features
+3.4.1.        Sampling and converting information
+3.4.2.        Maps
+3.4.3.        ACLs and conditions
+3.4.4.        Content switching
+3.4.5.        Stick-tables
+3.4.6.        Formatted strings
+3.4.7.        HTTP rewriting and redirection
+3.4.8.        Server protection
+3.5.      Advanced features
+3.5.1.        Management
+3.5.2.        System-specific capabilities
+3.5.3.        Scripting
+3.6.      Sizing
+3.7.      How to get HAProxy
+
+4.    Companion products and alternatives
+4.1.      Apache HTTP server
+4.2.      NGINX
+4.3.      Varnish
+4.4.      Alternatives
+
+5. Contacts
+
+
+1. Available documentation
+--------------------------
+
+The complete HAProxy documentation is contained in the following documents.
+Please ensure to consult the relevant documentation to save time and to get the
+most accurate response to your needs. Also please refrain from sending questions
+to the mailing list whose responses are present in these documents.
+
+  - intro.txt (this document) : it presents the basics of load balancing,
+    HAProxy as a product, what it does, what it doesn't do, some known traps to
+    avoid, some OS-specific limitations, how to get it, how it evolves, how to
+    ensure you're running with all known fixes, how to update it, complements
+    and alternatives.
+
+  - management.txt : it explains how to start haproxy, how to manage it at
+    runtime, how to manage it on multiple nodes, and how to proceed with
+    seamless upgrades.
+
+  - configuration.txt : the reference manual details all configuration keywords
+    and their options. It is used when a configuration change is needed.
+
+  - coding-style.txt : this is for developers who want to propose some code to
+    the project. It explains the style to adopt for the code. It is not very
+    strict and not all the code base completely respects it, but contributions
+    which diverge too much from it will be rejected.
+
+  - proxy-protocol.txt : this is the de-facto specification of the PROXY
+    protocol which is implemented by HAProxy and a number of third party
+    products.
+
+  - README : how to build HAProxy from sources
+
+
+2. Quick introduction to load balancing and load balancers
+----------------------------------------------------------
+
+Load balancing consists in aggregating multiple components in order to achieve
+a total processing capacity above each component's individual capacity, without
+any intervention from the end user and in a scalable way. This results in more
+operations being performed simultaneously by the time it takes a component to
+perform only one. A single operation however will still be performed on a single
+component at a time and will not get faster than without load balancing. It
+always requires at least as many operations as available components and an
+efficient load balancing mechanism to make use of all components and to fully
+benefit from the load balancing. A good example of this is the number of lanes
+on a highway which allows as many cars to pass during the same time frame
+without increasing their individual speed.
+
+Examples of load balancing :
+
+  - Process scheduling in multi-processor systems
+  - Link load balancing (e.g. EtherChannel, Bonding)
+  - IP address load balancing (e.g. ECMP, DNS round-robin)
+  - Server load balancing (via load balancers)
+
+The mechanism or component which performs the load balancing operation is
+called a load balancer. In web environments these components are called a
+"network load balancer", and more commonly a "load balancer" given that this
+activity is by far the best known case of load balancing.
+
+A load balancer may act :
+
+  - at the link level : this is called link load balancing, and it consists in
+    choosing what network link to send a packet to;
+
+  - at the network level : this is called network load balancing, and it
+    consists in choosing what route a series of packets will follow;
+
+  - at the server level : this is called server load balancing and it consists
+    in deciding what server will process a connection or request.
+
+Two distinct technologies exist and address different needs, though with some
+overlapping. In each case it is important to keep in mind that load balancing
+consists in diverting the traffic from its natural flow and that doing so always
+requires a minimum of care to maintain the required level of consistency between
+all routing decisions.
+
+The first one acts at the packet level and processes packets more or less
+individually. There is a 1-to-1 relation between input and output packets, so
+it is possible to follow the traffic on both sides of the load balancer using a
+regular network sniffer. This technology can be very cheap and extremely fast.
+It is usually implemented in hardware (ASICs) allowing to reach line rate, such
+as switches doing ECMP. Usually stateless, it can also be stateful (consider
+the session a packet belongs to and called layer4-LB or L4), may support DSR
+(direct server return, without passing through the LB again) if the packets
+were not modified, but provides almost no content awareness. This technology is
+very well suited to network-level load balancing, though it is sometimes used
+for very basic server load balancing at high speed.
+
+The second one acts on session contents. It requires that the input streams is
+reassembled and processed as a whole. The contents may be modified, and the
+output stream is segmented into new packets. For this reason it is generally
+performed by proxies and they're often called layer 7 load balancers or L7.
+This implies that there are two distinct connections on each side, and that
+there is no relation between input and output packets sizes nor counts. Clients
+and servers are not required to use the same protocol (for example IPv4 vs
+IPv6, clear vs SSL). The operations are always stateful, and the return traffic
+must pass through the load balancer. The extra processing comes with a cost so
+it's not always possible to achieve line rate, especially with small packets.
+On the other hand, it offers wide possibilities and is generally achieved by
+pure software, even if embedded into hardware appliances. This technology is
+very well suited for server load balancing.
+
+Packet-based load balancers are generally deployed in cut-through mode, so they
+are installed on the normal path of the traffic and divert it according to the
+configuration. The return traffic doesn't necessarily pass through the load
+balancer. Some modifications may be applied to the network destination address
+in order to direct the traffic to the proper destination. In this case, it is
+mandatory that the return traffic passes through the load balancer. If the
+routes doesn't make this possible, the load balancer may also replace the
+packets' source address with its own in order to force the return traffic to
+pass through it.
+
+Proxy-based load balancers are deployed as a server with their own IP addresses
+and ports, without architecture changes. Sometimes this requires to perform some
+adaptations to the applications so that clients are properly directed to the
+load balancer's IP address and not directly to the server's. Some load balancers
+may have to adjust some servers' responses to make this possible (e.g. the HTTP
+Location header field used in HTTP redirects). Some proxy-based load balancers
+may intercept traffic for an address they don't own, and spoof the client's
+address when connecting to the server. This allows them to be deployed as if
+they were a regular router or firewall, in a cut-through mode very similar to
+the packet based load balancers. This is particularly appreciated for products
+which combine both packet mode and proxy mode. In this case DSR is obviously
+still not possible and the return traffic still has to be routed back to the
+load balancer.
+
+A very scalable layered approach would consist in having a front router which
+receives traffic from multiple load balanced links, and uses ECMP to distribute
+this traffic to a first layer of multiple stateful packet-based load balancers
+(L4). These L4 load balancers in turn pass the traffic to an even larger number
+of proxy-based load balancers (L7), which have to parse the contents to decide
+what server will ultimately receive the traffic.
+
+The number of components and possible paths for the traffic increases the risk
+of failure; in very large environments, it is even normal to permanently have
+a few faulty components being fixed or replaced. Load balancing done without
+awareness of the whole stack's health significantly degrades availability. For
+this reason, any sane load balancer will verify that the components it intends
+to deliver the traffic to are still alive and reachable, and it will stop
+delivering traffic to faulty ones. This can be achieved using various methods.
+
+The most common one consists in periodically sending probes to ensure the
+component is still operational. These probes are called "health checks". They
+must be representative of the type of failure to address. For example a ping-
+based check will not detect that a web server has crashed and doesn't listen to
+a port anymore, while a connection to the port will verify this, and a more
+advanced request may even validate that the server still works and that the
+database it relies on is still accessible. Health checks often involve a few
+retries to cover for occasional measuring errors. The period between checks
+must be small enough to ensure the faulty component is not used for too long
+after an error occurs.
+
+Other methods consist in sampling the production traffic sent to a destination
+to observe if it is processed correctly or not, and to evict the components
+which return inappropriate responses. However this requires to sacrifice a part
+of the production traffic and this is not always acceptable. A combination of
+these two mechanisms provides the best of both worlds, with both of them being
+used to detect a fault, and only health checks to detect the end of the fault.
+A last method involves centralized reporting : a central monitoring agent
+periodically updates all load balancers about all components' state. This gives
+a global view of the infrastructure to all components, though sometimes with
+less accuracy or responsiveness. It's best suited for environments with many
+load balancers and many servers.
+
+Layer 7 load balancers also face another challenge known as stickiness or
+persistence. The principle is that they generally have to direct multiple
+subsequent requests or connections from a same origin (such as an end user) to
+the same target. The best known example is the shopping cart on an online
+store. If each click leads to a new connection, the user must always be sent
+to the server which holds his shopping cart. Content-awareness makes it easier
+to spot some elements in the request to identify the server to deliver it to,
+but that's not always enough. For example if the source address is used as a
+key to pick a server, it can be decided that a hash-based algorithm will be
+used and that a given IP address will always be sent to the same server based
+on a divide of the address by the number of available servers. But if one
+server fails, the result changes and all users are suddenly sent to a different
+server and lose their shopping cart. The solution against this issue consists
+in memorizing the chosen target so that each time the same visitor is seen,
+he's directed to the same server regardless of the number of available servers.
+The information may be stored in the load balancer's memory, in which case it
+may have to be replicated to other load balancers if it's not alone, or it may
+be stored in the client's memory using various methods provided that the client
+is able to present this information back with every request (cookie insertion,
+redirection to a sub-domain, etc). This mechanism provides the extra benefit of
+not having to rely on unstable or unevenly distributed information (such as the
+source IP address). This is in fact the strongest reason to adopt a layer 7
+load balancer instead of a layer 4 one.
+
+In order to extract information such as a cookie, a host header field, a URL
+or whatever, a load balancer may need to decrypt SSL/TLS traffic and even
+possibly to re-encrypt it when passing it to the server. This expensive task
+explains why in some high-traffic infrastructures, sometimes there may be a
+lot of load balancers.
+
+Since a layer 7 load balancer may perform a number of complex operations on the
+traffic (decrypt, parse, modify, match cookies, decide what server to send to,
+etc), it can definitely cause some trouble and will very commonly be accused of
+being responsible for a lot of trouble that it only revealed. Often it will be
+discovered that servers are unstable and periodically go up and down, or for
+web servers, that they deliver pages with some hard-coded links forcing the
+clients to connect directly to one specific server without passing via the load
+balancer, or that they take ages to respond under high load causing timeouts.
+That's why logging is an extremely important aspect of layer 7 load balancing.
+Once a trouble is reported, it is important to figure if the load balancer took
+a wrong decision and if so why so that it doesn't happen anymore.
+
+
+3. Introduction to HAProxy
+--------------------------
+
+HAProxy is written as "HAProxy" to designate the product, and as "haproxy" to
+designate the executable program, software package or a process. However, both
+are commonly used for both purposes, and are pronounced H-A-Proxy. Very early,
+"haproxy" used to stand for "high availability proxy" and the name was written
+in two separate words, though by now it means nothing else than "HAProxy".
+
+
+3.1. What HAProxy is and isn't
+------------------------------
+
+HAProxy is :
+
+  - a TCP proxy : it can accept a TCP connection from a listening socket,
+    connect to a server and attach these sockets together allowing traffic to
+    flow in both directions; IPv4, IPv6 and even UNIX sockets are supported on
+    either side, so this can provide an easy way to translate addresses between
+    different families.
+
+  - an HTTP reverse-proxy (called a "gateway" in HTTP terminology) : it presents
+    itself as a server, receives HTTP requests over connections accepted on a
+    listening TCP socket, and passes the requests from these connections to
+    servers using different connections. It may use any combination of HTTP/1.x
+    or HTTP/2 on any side and will even automatically detect the protocol
+    spoken on each side when ALPN is used over TLS.
+
+  - an SSL terminator / initiator / offloader : SSL/TLS may be used on the
+    connection coming from the client, on the connection going to the server,
+    or even on both connections. A lot of settings can be applied per name
+    (SNI), and may be updated at runtime without restarting. Such setups are
+    extremely scalable and deployments involving tens to hundreds of thousands
+    of certificates were reported.
+
+  - a TCP normalizer : since connections are locally terminated by the operating
+    system, there is no relation between both sides, so abnormal traffic such as
+    invalid packets, flag combinations, window advertisements, sequence numbers,
+    incomplete connections (SYN floods), or so will not be passed to the other
+    side. This protects fragile TCP stacks from protocol attacks, and also
+    allows to optimize the connection parameters with the client without having
+    to modify the servers' TCP stack settings.
+
+  - an HTTP normalizer : when configured to process HTTP traffic, only valid
+    complete requests are passed. This protects against a lot of protocol-based
+    attacks. Additionally, protocol deviations for which there is a tolerance
+    in the specification are fixed so that they don't cause problem on the
+    servers (e.g. multiple-line headers).
+
+  - an HTTP fixing tool : it can modify / fix / add / remove / rewrite the URL
+    or any request or response header. This helps fixing interoperability issues
+    in complex environments.
+
+  - a content-based switch : it can consider any element from the request to
+    decide what server to pass the request or connection to. Thus it is possible
+    to handle multiple protocols over a same port (e.g. HTTP, HTTPS, SSH).
+
+  - a server load balancer : it can load balance TCP connections and HTTP
+    requests. In TCP mode, load balancing decisions are taken for the whole
+    connection. In HTTP mode, decisions are taken per request.
+
+  - a traffic regulator : it can apply some rate limiting at various points,
+    protect the servers against overloading, adjust traffic priorities based on
+    the contents, and even pass such information to lower layers and outer
+    network components by marking packets.
+
+  - a protection against DDoS and service abuse : it can maintain a wide number
+    of statistics per IP address, URL, cookie, etc and detect when an abuse is
+    happening, then take action (slow down the offenders, block them, send them
+    to outdated contents, etc).
+
+  - an observation point for network troubleshooting : due to the precision of
+    the information reported in logs, it is often used to narrow down some
+    network-related issues.
+
+  - an HTTP compression offloader : it can compress responses which were not
+    compressed by the server, thus reducing the page load time for clients with
+    poor connectivity or using high-latency, mobile networks.
+
+  - a caching proxy : it may cache responses in RAM so that subsequent requests
+    for the same object avoid the cost of another network transfer from the
+    server as long as the object remains present and valid. It will however not
+    store objects to any persistent storage. Please note that this caching
+    feature is designed to be maintenance free and focuses solely on saving
+    haproxy's precious resources and not on save the server's resources. Caches
+    designed to optimize servers require much more tuning and flexibility. If
+    you instead need such an advanced cache, please use Varnish Cache, which
+    integrates perfectly with haproxy, especially when SSL/TLS is needed on any
+    side.
+
+  - a FastCGI gateway : FastCGI can be seen as a different representation of
+    HTTP, and as such, HAProxy can directly load-balance a farm comprising any
+    combination of FastCGI application servers without requiring to insert
+    another level of gateway between them. This results in resource savings and
+    a reduction of maintenance costs.
+
+HAProxy is not :
+
+  - an explicit HTTP proxy, i.e. the proxy that browsers use to reach the
+    internet. There are excellent open-source software dedicated for this task,
+    such as Squid. However HAProxy can be installed in front of such a proxy to
+    provide load balancing and high availability.
+
+  - a data scrubber : it will not modify the body of requests nor responses.
+
+  - a static web server : during startup, it isolates itself inside a chroot
+    jail and drops its privileges, so that it will not perform any single file-
+    system access once started. As such it cannot be turned into a static web
+    server (dynamic servers are supported through FastCGI however). There are
+    excellent open-source software for this such as Apache or Nginx, and
+    HAProxy can be easily installed in front of them to provide load balancing,
+    high availability and acceleration.
+
+  - a packet-based load balancer : it will not see IP packets nor UDP datagrams,
+    will not perform NAT or even less DSR. These are tasks for lower layers.
+    Some kernel-based components such as IPVS (Linux Virtual Server) already do
+    this pretty well and complement perfectly with HAProxy.
+
+
+3.2. How HAProxy works
+----------------------
+
+HAProxy is an event-driven, non-blocking engine combining a very fast I/O layer
+with a priority-based, multi-threaded scheduler. As it is designed with a data
+forwarding goal in mind, its architecture is optimized to move data as fast as
+possible with the least possible operations. It focuses on optimizing the CPU
+cache's efficiency by sticking connections to the same CPU as long as possible.
+As such it implements a layered model offering bypass mechanisms at each level
+ensuring data doesn't reach higher levels unless needed. Most of the processing
+is performed in the kernel, and HAProxy does its best to help the kernel do the
+work as fast as possible by giving some hints or by avoiding certain operation
+when it guesses they could be grouped later. As a result, typical figures show
+15% of the processing time spent in HAProxy versus 85% in the kernel in TCP or
+HTTP close mode, and about 30% for HAProxy versus 70% for the kernel in HTTP
+keep-alive mode.
+
+A single process can run many proxy instances; configurations as large as
+300000 distinct proxies in a single process were reported to run fine. A single
+core, single CPU setup is far more than enough for more than 99% users, and as
+such, users of containers and virtual machines are encouraged to use the
+absolute smallest images they can get to save on operational costs and simplify
+troubleshooting. However the machine HAProxy runs on must never ever swap, and
+its CPU must not be artificially throttled (sub-CPU allocation in hypervisors)
+nor be shared with compute-intensive processes which would induce a very high
+context-switch latency.
+
+Threading allows to exploit all available processing capacity by using one
+thread per CPU core. This is mostly useful for SSL or when data forwarding
+rates above 40 Gbps are needed. In such cases it is critically important to
+avoid communications between multiple physical CPUs, which can cause strong
+bottlenecks in the network stack and in HAProxy itself. While counter-intuitive
+to some, the first thing to do when facing some performance issues is often to
+reduce the number of CPUs HAProxy runs on.
+
+HAProxy only requires the haproxy executable and a configuration file to run.
+For logging it is highly recommended to have a properly configured syslog daemon
+and log rotations in place. Logs may also be sent to stdout/stderr, which can be
+useful inside containers. The configuration files are parsed before starting,
+then HAProxy tries to bind all listening sockets, and refuses to start if
+anything fails. Past this point it cannot fail anymore. This means that there
+are no runtime failures and that if it accepts to start, it will work until it
+is stopped.
+
+Once HAProxy is started, it does exactly 3 things :
+
+  - process incoming connections;
+
+  - periodically check the servers' status (known as health checks);
+
+  - exchange information with other haproxy nodes.
+
+Processing incoming connections is by far the most complex task as it depends
+on a lot of configuration possibilities, but it can be summarized as the 9 steps
+below :
+
+  - accept incoming connections from listening sockets that belong to a
+    configuration entity known as a "frontend", which references one or multiple
+    listening addresses;
+
+  - apply the frontend-specific processing rules to these connections that may
+    result in blocking them, modifying some headers, or intercepting them to
+    execute some internal applets such as the statistics page or the CLI;
+
+  - pass these incoming connections to another configuration entity representing
+    a server farm known as a "backend", which contains the list of servers and
+    the load balancing strategy for this server farm;
+
+  - apply the backend-specific processing rules to these connections;
+
+  - decide which server to forward the connection to according to the load
+    balancing strategy;
+
+  - apply the backend-specific processing rules to the response data;
+
+  - apply the frontend-specific processing rules to the response data;
+
+  - emit a log to report what happened in fine details;
+
+  - in HTTP, loop back to the second step to wait for a new request, otherwise
+    close the connection.
+
+Frontends and backends are sometimes considered as half-proxies, since they only
+look at one side of an end-to-end connection; the frontend only cares about the
+clients while the backend only cares about the servers. HAProxy also supports
+full proxies which are exactly the union of a frontend and a backend. When HTTP
+processing is desired, the configuration will generally be split into frontends
+and backends as they open a lot of possibilities since any frontend may pass a
+connection to any backend. With TCP-only proxies, using frontends and backends
+rarely provides a benefit and the configuration can be more readable with full
+proxies.
+
+
+3.3. Basic features
+-------------------
+
+This section will enumerate a number of features that HAProxy implements, some
+of which are generally expected from any modern load balancer, and some of
+which are a direct benefit of HAProxy's architecture. More advanced features
+will be detailed in the next section.
+
+
+3.3.1. Basic features : Proxying
+--------------------------------
+
+Proxying is the action of transferring data between a client and a server over
+two independent connections. The following basic features are supported by
+HAProxy regarding proxying and connection management :
+
+  - Provide the server with a clean connection to protect them against any
+    client-side defect or attack;
+
+  - Listen to multiple IP addresses and/or ports, even port ranges;
+
+  - Transparent accept : intercept traffic targeting any arbitrary IP address
+    that doesn't even belong to the local system;
+
+  - Server port doesn't need to be related to listening port, and may even be
+    translated by a fixed offset (useful with ranges);
+
+  - Transparent connect : spoof the client's (or any) IP address if needed
+    when connecting to the server;
+
+  - Provide a reliable return IP address to the servers in multi-site LBs;
+
+  - Offload the server thanks to buffers and possibly short-lived connections
+    to reduce their concurrent connection count and their memory footprint;
+
+  - Optimize TCP stacks (e.g. SACK), congestion control, and reduce RTT impacts;
+
+  - Support different protocol families on both sides (e.g. IPv4/IPv6/Unix);
+
+  - Timeout enforcement : HAProxy supports multiple levels of timeouts depending
+    on the stage the connection is, so that a dead client or server, or an
+    attacker cannot be granted resources for too long;
+
+  - Protocol validation: HTTP, SSL, or payload are inspected and invalid
+    protocol elements are rejected, unless instructed to accept them anyway;
+
+  - Policy enforcement : ensure that only what is allowed may be forwarded;
+
+  - Both incoming and outgoing connections may be limited to certain network
+    namespaces (Linux only), making it easy to build a cross-container,
+    multi-tenant load balancer;
+
+  - PROXY protocol presents the client's IP address to the server even for
+    non-HTTP traffic. This is an HAProxy extension that was adopted by a number
+    of third-party products by now, at least these ones at the time of writing :
+      - client : haproxy, stud, stunnel, exaproxy, ELB, squid
+      - server : haproxy, stud, postfix, exim, nginx, squid, node.js, varnish
+
+
+3.3.2. Basic features : SSL
+---------------------------
+
+HAProxy's SSL stack is recognized as one of the most featureful according to
+Google's engineers (http://istlsfastyet.com/). The most commonly used features
+making it quite complete are :
+
+  - SNI-based multi-hosting with no limit on sites count and focus on
+    performance. At least one deployment is known for running 50000 domains
+    with their respective certificates;
+
+  - support for wildcard certificates reduces the need for many certificates ;
+
+  - certificate-based client authentication with configurable policies on
+    failure to present a valid certificate. This allows to present a different
+    server farm to regenerate the client certificate for example;
+
+  - authentication of the backend server ensures the backend server is the real
+    one and not a man in the middle;
+
+  - authentication with the backend server lets the backend server know it's
+    really the expected haproxy node that is connecting to it;
+
+  - TLS NPN and ALPN extensions make it possible to reliably offload SPDY/HTTP2
+    connections and pass them in clear text to backend servers;
+
+  - OCSP stapling further reduces first page load time by delivering inline an
+    OCSP response when the client requests a Certificate Status Request;
+
+  - Dynamic record sizing provides both high performance and low latency, and
+    significantly reduces page load time by letting the browser start to fetch
+    new objects while packets are still in flight;
+
+  - permanent access to all relevant SSL/TLS layer information for logging,
+    access control, reporting etc. These elements can be embedded into HTTP
+    header or even as a PROXY protocol extension so that the offloaded server
+    gets all the information it would have had if it performed the SSL
+    termination itself.
+
+  - Detect, log and block certain known attacks even on vulnerable SSL libs,
+    such as the Heartbleed attack affecting certain versions of OpenSSL.
+
+  - support for stateless session resumption (RFC 5077 TLS Ticket extension).
+    TLS tickets can be updated from CLI which provides them means to implement
+    Perfect Forward Secrecy by frequently rotating the tickets.
+
+
+3.3.3. Basic features : Monitoring
+----------------------------------
+
+HAProxy focuses a lot on availability. As such it cares about servers state,
+and about reporting its own state to other network components :
+
+  - Servers' state is continuously monitored using per-server parameters. This
+    ensures the path to the server is operational for regular traffic;
+
+  - Health checks support two hysteresis for up and down transitions in order
+    to protect against state flapping;
+
+  - Checks can be sent to a different address/port/protocol : this makes it
+    easy to check a single service that is considered representative of multiple
+    ones, for example the HTTPS port for an HTTP+HTTPS server.
+
+  - Servers can track other servers and go down simultaneously : this ensures
+    that servers hosting multiple services can fail atomically and that no one
+    will be sent to a partially failed server;
+
+  - Agents may be deployed on the server to monitor load and health : a server
+    may be interested in reporting its load, operational status, administrative
+    status independently from what health checks can see. By running a simple
+    agent on the server, it's possible to consider the server's view of its own
+    health in addition to the health checks validating the whole path;
+
+  - Various check methods are available : TCP connect, HTTP request, SMTP hello,
+    SSL hello, LDAP, SQL, Redis, send/expect scripts, all with/without SSL;
+
+  - State change is notified in the logs and stats page with the failure reason
+    (e.g. the HTTP response received at the moment the failure was detected). An
+    e-mail can also be sent to a configurable address upon such a change ;
+
+  - Server state is also reported on the stats interface and can be used to take
+    routing decisions so that traffic may be sent to different farms depending
+    on their sizes and/or health (e.g. loss of an inter-DC link);
+
+  - HAProxy can use health check requests to pass information to the servers,
+    such as their names, weight, the number of other servers in the farm etc.
+    so that servers can adjust their response and decisions based on this
+    knowledge (e.g. postpone backups to keep more CPU available);
+
+  - Servers can use health checks to report more detailed state than just on/off
+    (e.g. I would like to stop, please stop sending new visitors);
+
+  - HAProxy itself can report its state to external components such as routers
+    or other load balancers, allowing to build very complete multi-path and
+    multi-layer infrastructures.
+
+
+3.3.4. Basic features : High availability
+-----------------------------------------
+
+Just like any serious load balancer, HAProxy cares a lot about availability to
+ensure the best global service continuity :
+
+  - Only valid servers are used ; the other ones are automatically evicted from
+    load balancing farms ; under certain conditions it is still possible to
+    force to use them though;
+
+  - Support for a graceful shutdown so that it is possible to take servers out
+    of a farm without affecting any connection;
+
+  - Backup servers are automatically used when active servers are down and
+    replace them so that sessions are not lost when possible. This also allows
+    to build multiple paths to reach the same server (e.g. multiple interfaces);
+
+  - Ability to return a global failed status for a farm when too many servers
+    are down. This, combined with the monitoring capabilities makes it possible
+    for an upstream component to choose a different LB node for a given service;
+
+  - Stateless design makes it easy to build clusters : by design, HAProxy does
+    its best to ensure the highest service continuity without having to store
+    information that could be lost in the event of a failure. This ensures that
+    a takeover is the most seamless possible;
+
+  - Integrates well with standard VRRP daemon keepalived : HAProxy easily tells
+    keepalived about its state and copes very well with floating virtual IP
+    addresses. Note: only use IP redundancy protocols (VRRP/CARP) over cluster-
+    based solutions (Heartbeat, ...) as they're the ones offering the fastest,
+    most seamless, and most reliable switchover.
+
+
+3.3.5. Basic features : Load balancing
+--------------------------------------
+
+HAProxy offers a fairly complete set of load balancing features, most of which
+are unfortunately not available in a number of other load balancing products :
+
+  - no less than 10 load balancing algorithms are supported, some of which apply
+    to input data to offer an infinite list of possibilities. The most common
+    ones are round-robin (for short connections, pick each server in turn),
+    leastconn (for long connections, pick the least recently used of the servers
+    with the lowest connection count), source (for SSL farms or terminal server
+    farms, the server directly depends on the client's source address), URI (for
+    HTTP caches, the server directly depends on the HTTP URI), hdr (the server
+    directly depends on the contents of a specific HTTP header field), first
+    (for short-lived virtual machines, all connections are packed on the
+    smallest possible subset of servers so that unused ones can be powered
+    down);
+
+  - all algorithms above support per-server weights so that it is possible to
+    accommodate from different server generations in a farm, or direct a small
+    fraction of the traffic to specific servers (debug mode, running the next
+    version of the software, etc);
+
+  - dynamic weights are supported for round-robin, leastconn and consistent
+    hashing ; this allows server weights to be modified on the fly from the CLI
+    or even by an agent running on the server;
+
+  - slow-start is supported whenever a dynamic weight is supported; this allows
+    a server to progressively take the traffic. This is an important feature
+    for fragile application servers which require to compile classes at runtime
+    as well as cold caches which need to fill up before being run at full
+    throttle;
+
+  - hashing can apply to various elements such as client's source address, URL
+    components, query string element, header field values, POST parameter, RDP
+    cookie;
+
+  - consistent hashing protects server farms against massive redistribution when
+    adding or removing servers in a farm. That's very important in large cache
+    farms and it allows slow-start to be used to refill cold caches;
+
+  - a number of internal metrics such as the number of connections per server,
+    per backend, the amount of available connection slots in a backend etc makes
+    it possible to build very advanced load balancing strategies.
+
+
+3.3.6. Basic features : Stickiness
+----------------------------------
+
+Application load balancing would be useless without stickiness. HAProxy provides
+a fairly comprehensive set of possibilities to maintain a visitor on the same
+server even across various events such as server addition/removal, down/up
+cycles, and some methods are designed to be resistant to the distance between
+multiple load balancing nodes in that they don't require any replication :
+
+  - stickiness information can be individually matched and learned from
+    different places if desired. For example a JSESSIONID cookie may be matched
+    both in a cookie and in the URL. Up to 8 parallel sources can be learned at
+    the same time and each of them may point to a different stick-table;
+
+  - stickiness information can come from anything that can be seen within a
+    request or response, including source address, TCP payload offset and
+    length, HTTP query string elements, header field values, cookies, and so
+    on.
+
+  - stick-tables are replicated between all nodes in a multi-master fashion;
+
+  - commonly used elements such as SSL-ID or RDP cookies (for TSE farms) are
+    directly accessible to ease manipulation;
+
+  - all sticking rules may be dynamically conditioned by ACLs;
+
+  - it is possible to decide not to stick to certain servers, such as backup
+    servers, so that when the nominal server comes back, it automatically takes
+    the load back. This is often used in multi-path environments;
+
+  - in HTTP it is often preferred not to learn anything and instead manipulate
+    a cookie dedicated to stickiness. For this, it's possible to detect,
+    rewrite, insert or prefix such a cookie to let the client remember what
+    server was assigned;
+
+  - the server may decide to change or clean the stickiness cookie on logout,
+    so that leaving visitors are automatically unbound from the server;
+
+  - using ACL-based rules it is also possible to selectively ignore or enforce
+    stickiness regardless of the server's state; combined with advanced health
+    checks, that helps admins verify that the server they're installing is up
+    and running before presenting it to the whole world;
+
+  - an innovative mechanism to set a maximum idle time and duration on cookies
+    ensures that stickiness can be smoothly stopped on devices which are never
+    closed (smartphones, TVs, home appliances) without having to store them on
+    persistent storage;
+
+  - multiple server entries may share the same stickiness keys so that
+    stickiness is not lost in multi-path environments when one path goes down;
+
+  - soft-stop ensures that only users with stickiness information will continue
+    to reach the server they've been assigned to but no new users will go there.
+
+
+3.3.7. Basic features : Logging
+-------------------------------
+
+Logging is an extremely important feature for a load balancer, first because a
+load balancer is often wrongly accused of causing the problems it reveals, and
+second because it is placed at a critical point in an infrastructure where all
+normal and abnormal activity needs to be analyzed and correlated with other
+components.
+
+HAProxy provides very detailed logs, with millisecond accuracy and the exact
+connection accept time that can be searched in firewalls logs (e.g. for NAT
+correlation). By default, TCP and HTTP logs are quite detailed and contain
+everything needed for troubleshooting, such as source IP address and port,
+frontend, backend, server, timers (request receipt duration, queue duration,
+connection setup time, response headers time, data transfer time), global
+process state, connection counts, queue status, retries count, detailed
+stickiness actions and disconnect reasons, header captures with a safe output
+encoding. It is then possible to extend or replace this format to include any
+sampled data, variables, captures, resulting in very detailed information. For
+example it is possible to log the number of cumulative requests or number of
+different URLs visited by a client.
+
+The log level may be adjusted per request using standard ACLs, so it is possible
+to automatically silent some logs considered as pollution and instead raise
+warnings when some abnormal behavior happen for a small part of the traffic
+(e.g. too many URLs or HTTP errors for a source address). Administrative logs
+are also emitted with their own levels to inform about the loss or recovery of a
+server for example.
+
+Each frontend and backend may use multiple independent log outputs, which eases
+multi-tenancy. Logs are preferably sent over UDP, maybe JSON-encoded, and are
+truncated after a configurable line length in order to guarantee delivery. But
+it is also possible to send them to stdout/stderr or any file descriptor, as
+well as to a ring buffer that a client can subscribe to in order to retrieve
+them.
+
+
+3.3.8. Basic features : Statistics
+----------------------------------
+
+HAProxy provides a web-based statistics reporting interface with authentication,
+security levels and scopes. It is thus possible to provide each hosted customer
+with his own page showing only his own instances. This page can be located in a
+hidden URL part of the regular web site so that no new port needs to be opened.
+This page may also report the availability of other HAProxy nodes so that it is
+easy to spot if everything works as expected at a glance. The view is synthetic
+with a lot of details accessible (such as error causes, last access and last
+change duration, etc), which are also accessible as a CSV table that other tools
+may import to draw graphs. The page may self-refresh to be used as a monitoring
+page on a large display. In administration mode, the page also allows to change
+server state to ease maintenance operations.
+
+A Prometheus exporter is also provided so that the statistics can be consumed
+in a different format depending on the deployment.
+
+
+3.4. Standard features
+----------------------
+
+In this section, some features that are very commonly used in HAProxy but are
+not necessarily present on other load balancers are enumerated.
+
+
+3.4.1. Standard features : Sampling and converting information
+--------------------------------------------------------------
+
+HAProxy supports information sampling using a wide set of "sample fetch
+functions". The principle is to extract pieces of information known as samples,
+for immediate use. This is used for stickiness, to build conditions, to produce
+information in logs or to enrich HTTP headers.
+
+Samples can be fetched from various sources :
+
+  - constants : integers, strings, IP addresses, binary blocks;
+
+  - the process : date, environment variables, server/frontend/backend/process
+    state, byte/connection counts/rates, queue length, random generator, ...
+
+  - variables : per-session, per-request, per-response variables;
+
+  - the client connection : source and destination addresses and ports, and all
+    related statistics counters;
+
+  - the SSL client session : protocol, version, algorithm, cipher, key size,
+    session ID, all client and server certificate fields, certificate serial,
+    SNI, ALPN, NPN, client support for certain extensions;
+
+  - request and response buffers contents : arbitrary payload at offset/length,
+    data length, RDP cookie, decoding of SSL hello type, decoding of TLS SNI;
+
+  - HTTP (request and response) : method, URI, path, query string arguments,
+    status code, headers values, positional header value, cookies, captures,
+    authentication, body elements;
+
+A sample may then pass through a number of operators known as "converters" to
+experience some transformation. A converter consumes a sample and produces a
+new one, possibly of a completely different type. For example, a converter may
+be used to return only the integer length of the input string, or could turn a
+string to upper case. Any arbitrary number of converters may be applied in
+series to a sample before final use. Among all available sample converters, the
+following ones are the most commonly used :
+
+  - arithmetic and logic operators : they make it possible to perform advanced
+    computation on input data, such as computing ratios, percentages or simply
+    converting from one unit to another one;
+
+  - IP address masks are useful when some addresses need to be grouped by larger
+    networks;
+
+  - data representation : URL-decode, base64, hex, JSON strings, hashing;
+
+  - string conversion : extract substrings at fixed positions, fixed length,
+    extract specific fields around certain delimiters, extract certain words,
+    change case, apply regex-based substitution;
+
+  - date conversion : convert to HTTP date format, convert local to UTC and
+    conversely, add or remove offset;
+
+  - lookup an entry in a stick table to find statistics or assigned server;
+
+  - map-based key-to-value conversion from a file (mostly used for geolocation).
+
+
+3.4.2. Standard features : Maps
+-------------------------------
+
+Maps are a powerful type of converter consisting in loading a two-columns file
+into memory at boot time, then looking up each input sample from the first
+column and either returning the corresponding pattern on the second column if
+the entry was found, or returning a default value. The output information also
+being a sample, it can in turn experience other transformations including other
+map lookups. Maps are most commonly used to translate the client's IP address
+to an AS number or country code since they support a longest match for network
+addresses but they can be used for various other purposes.
+
+Part of their strength comes from being updatable on the fly either from the CLI
+or from certain actions using other samples, making them capable of storing and
+retrieving information between subsequent accesses. Another strength comes from
+the binary tree based indexation which makes them extremely fast even when they
+contain hundreds of thousands of entries, making geolocation very cheap and easy
+to set up.
+
+
+3.4.3. Standard features : ACLs and conditions
+----------------------------------------------
+
+Most operations in HAProxy can be made conditional. Conditions are built by
+combining multiple ACLs using logic operators (AND, OR, NOT). Each ACL is a
+series of tests based on the following elements :
+
+  - a sample fetch method to retrieve the element to test ;
+
+  - an optional series of converters to transform the element ;
+
+  - a list of patterns to match against ;
+
+  - a matching method to indicate how to compare the patterns with the sample
+
+For example, the sample may be taken from the HTTP "Host" header, it could then
+be converted to lower case, then matched against a number of regex patterns
+using the regex matching method.
+
+Technically, ACLs are built on the same core as the maps, they share the exact
+same internal structure, pattern matching methods and performance. The only real
+difference is that instead of returning a sample, they only return "found" or
+or "not found". In terms of usage, ACL patterns may be declared inline in the
+configuration file and do not require their own file. ACLs may be named for ease
+of use or to make configurations understandable. A named ACL may be declared
+multiple times and it will evaluate all definitions in turn until one matches.
+
+About 13 different pattern matching methods are provided, among which IP address
+mask, integer ranges, substrings, regex. They work like functions, and just like
+with any programming language, only what is needed is evaluated, so when a
+condition involving an OR is already true, next ones are not evaluated, and
+similarly when a condition involving an AND is already false, the rest of the
+condition is not evaluated.
+
+There is no practical limit to the number of declared ACLs, and a handful of
+commonly used ones are provided. However experience has shown that setups using
+a lot of named ACLs are quite hard to troubleshoot and that sometimes using
+anonymous ACLs inline is easier as it requires less references out of the scope
+being analyzed.
+
+
+3.4.4. Standard features : Content switching
+--------------------------------------------
+
+HAProxy implements a mechanism known as content-based switching. The principle
+is that a connection or request arrives on a frontend, then the information
+carried with this request or connection are processed, and at this point it is
+possible to write ACLs-based conditions making use of these information to
+decide what backend will process the request. Thus the traffic is directed to
+one backend or another based on the request's contents. The most common example
+consists in using the Host header and/or elements from the path (sub-directories
+or file-name extensions) to decide whether an HTTP request targets a static
+object or the application, and to route static objects traffic to a backend made
+of fast and light servers, and all the remaining traffic to a more complex
+application server, thus constituting a fine-grained virtual hosting solution.
+This is quite convenient to make multiple technologies coexist as a more global
+solution.
+
+Another use case of content-switching consists in using different load balancing
+algorithms depending on various criteria. A cache may use a URI hash while an
+application would use round-robin.
+
+Last but not least, it allows multiple customers to use a small share of a
+common resource by enforcing per-backend (thus per-customer connection limits).
+
+Content switching rules scale very well, though their performance may depend on
+the number and complexity of the ACLs in use. But it is also possible to write
+dynamic content switching rules where a sample value directly turns into a
+backend name and without making use of ACLs at all. Such configurations have
+been reported to work fine at least with 300000 backends in production.
+
+
+3.4.5. Standard features : Stick-tables
+---------------------------------------
+
+Stick-tables are commonly used to store stickiness information, that is, to keep
+a reference to the server a certain visitor was directed to. The key is then the
+identifier associated with the visitor (its source address, the SSL ID of the
+connection, an HTTP or RDP cookie, the customer number extracted from the URL or
+from the payload, ...) and the stored value is then the server's identifier.
+
+Stick tables may use 3 different types of samples for their keys : integers,
+strings and addresses. Only one stick-table may be referenced in a proxy, and it
+is designated everywhere with the proxy name. Up to 8 keys may be tracked in
+parallel. The server identifier is committed during request or response
+processing once both the key and the server are known.
+
+Stick-table contents may be replicated in active-active mode with other HAProxy
+nodes known as "peers" as well as with the new process during a reload operation
+so that all load balancing nodes share the same information and take the same
+routing decision if client's requests are spread over multiple nodes.
+
+Since stick-tables are indexed on what allows to recognize a client, they are
+often also used to store extra information such as per-client statistics. The
+extra statistics take some extra space and need to be explicitly declared. The
+type of statistics that may be stored includes the input and output bandwidth,
+the number of concurrent connections, the connection rate and count over a
+period, the amount and frequency of errors, some specific tags and counters,
+etc. In order to support keeping such information without being forced to
+stick to a given server, a special "tracking" feature is implemented and allows
+to track up to 3 simultaneous keys from different tables at the same time
+regardless of stickiness rules. Each stored statistics may be searched, dumped
+and cleared from the CLI and adds to the live troubleshooting capabilities.
+
+While this mechanism can be used to surclass a returning visitor or to adjust
+the delivered quality of service depending on good or bad behavior, it is
+mostly used to fight against service abuse and more generally DDoS as it allows
+to build complex models to detect certain bad behaviors at a high processing
+speed.
+
+
+3.4.6. Standard features : Formatted strings
+--------------------------------------------
+
+There are many places where HAProxy needs to manipulate character strings, such
+as logs, redirects, header additions, and so on. In order to provide the
+greatest flexibility, the notion of Formatted strings was introduced, initially
+for logging purposes, which explains why it's still called "log-format". These
+strings contain escape characters allowing to introduce various dynamic data
+including variables and sample fetch expressions into strings, and even to
+adjust the encoding while the result is being turned into a string (for example,
+adding quotes). This provides a powerful way to build header contents, to build
+response data or even response templates, or to customize log lines.
+Additionally, in order to remain simple to build most common strings, about 50
+special tags are provided as shortcuts for information commonly used in logs.
+
+
+3.4.7. Standard features : HTTP rewriting and redirection
+---------------------------------------------------------
+
+Installing a load balancer in front of an application that was never designed
+for this can be a challenging task without the proper tools. One of the most
+commonly requested operation in this case is to adjust requests and response
+headers to make the load balancer appear as the origin server and to fix hard
+coded information. This comes with changing the path in requests (which is
+strongly advised against), modifying Host header field, modifying the Location
+response header field for redirects, modifying the path and domain attribute
+for cookies, and so on. It also happens that a number of servers are somewhat
+verbose and tend to leak too much information in the response, making them more
+vulnerable to targeted attacks. While it's theoretically not the role of a load
+balancer to clean this up, in practice it's located at the best place in the
+infrastructure to guarantee that everything is cleaned up.
+
+Similarly, sometimes the load balancer will have to intercept some requests and
+respond with a redirect to a new target URL. While some people tend to confuse
+redirects and rewriting, these are two completely different concepts, since the
+rewriting makes the client and the server see different things (and disagree on
+the location of the page being visited) while redirects ask the client to visit
+the new URL so that it sees the same location as the server.
+
+In order to do this, HAProxy supports various possibilities for rewriting and
+redirects, among which :
+
+  - regex-based URL and header rewriting in requests and responses. Regex are
+    the most commonly used tool to modify header values since they're easy to
+    manipulate and well understood;
+
+  - headers may also be appended, deleted or replaced based on formatted strings
+    so that it is possible to pass information there (e.g. client side TLS
+    algorithm and cipher);
+
+  - HTTP redirects can use any 3xx code to a relative, absolute, or completely
+    dynamic (formatted string) URI;
+
+  - HTTP redirects also support some extra options such as setting or clearing
+    a specific cookie, dropping the query string, appending a slash if missing,
+    and so on;
+
+  - a powerful "return" directive allows to customize every part of a response
+    like status, headers, body using dynamic contents or even template files.
+
+  - all operations support ACL-based conditions;
+
+
+3.4.8. Standard features : Server protection
+--------------------------------------------
+
+HAProxy does a lot to maximize service availability, and for this it takes
+large efforts to protect servers against overloading and attacks. The first
+and most important point is that only complete and valid requests are forwarded
+to the servers. The initial reason is that HAProxy needs to find the protocol
+elements it needs to stay synchronized with the byte stream, and the second
+reason is that until the request is complete, there is no way to know if some
+elements will change its semantics. The direct benefit from this is that servers
+are not exposed to invalid or incomplete requests. This is a very effective
+protection against slowloris attacks, which have almost no impact on HAProxy.
+
+Another important point is that HAProxy contains buffers to store requests and
+responses, and that by only sending a request to a server when it's complete and
+by reading the whole response very quickly from the local network, the server
+side connection is used for a very short time and this preserves server
+resources as much as possible.
+
+A direct extension to this is that HAProxy can artificially limit the number of
+concurrent connections or outstanding requests to a server, which guarantees
+that the server will never be overloaded even if it continuously runs at 100% of
+its capacity during traffic spikes. All excess requests will simply be queued to
+be processed when one slot is released. In the end, this huge resource savings
+most often ensures so much better server response times that it ends up actually
+being faster than by overloading the server. Queued requests may be redispatched
+to other servers, or even aborted in queue when the client aborts, which also
+protects the servers against the "reload effect", where each click on "reload"
+by a visitor on a slow-loading page usually induces a new request and maintains
+the server in an overloaded state.
+
+The slow-start mechanism also protects restarting servers against high traffic
+levels while they're still finalizing their startup or compiling some classes.
+
+Regarding the protocol-level protection, it is possible to relax the HTTP parser
+to accept non standard-compliant but harmless requests or responses and even to
+fix them. This allows bogus applications to be accessible while a fix is being
+developed. In parallel, offending messages are completely captured with a
+detailed report that help developers spot the issue in the application. The most
+dangerous protocol violations are properly detected and dealt with and fixed.
+For example malformed requests or responses with two Content-length headers are
+either fixed if the values are exactly the same, or rejected if they differ,
+since it becomes a security problem. Protocol inspection is not limited to HTTP,
+it is also available for other protocols like TLS or RDP.
+
+When a protocol violation or attack is detected, there are various options to
+respond to the user, such as returning the common "HTTP 400 bad request",
+closing the connection with a TCP reset, or faking an error after a long delay
+("tarpit") to confuse the attacker. All of these contribute to protecting the
+servers by discouraging the offending client from pursuing an attack that
+becomes very expensive to maintain.
+
+HAProxy also proposes some more advanced options to protect against accidental
+data leaks and session crossing. Not only it can log suspicious server responses
+but it will also log and optionally block a response which might affect a given
+visitors' confidentiality. One such example is a cacheable cookie appearing in a
+cacheable response and which may result in an intermediary cache to deliver it
+to another visitor, causing an accidental session sharing.
+
+
+3.5. Advanced features
+----------------------
+
+3.5.1. Advanced features : Management
+-------------------------------------
+
+HAProxy is designed to remain extremely stable and safe to manage in a regular
+production environment. It is provided as a single executable file which doesn't
+require any installation process. Multiple versions can easily coexist, meaning
+that it's possible (and recommended) to upgrade instances progressively by
+order of importance instead of migrating all of them at once. Configuration
+files are easily versioned. Configuration checking is done off-line so it
+doesn't require to restart a service that will possibly fail. During
+configuration checks, a number of advanced mistakes may be detected (e.g. a rule
+hiding another one, or stickiness that will not work) and detailed warnings and
+configuration hints are proposed to fix them. Backwards configuration file
+compatibility goes very far away in time, with version 1.5 still fully
+supporting configurations for versions 1.1 written 13 years before, and 1.6
+only dropping support for almost unused, obsolete keywords that can be done
+differently. The configuration and software upgrade mechanism is smooth and non
+disruptive in that it allows old and new processes to coexist on the system,
+each handling its own connections. System status, build options, and library
+compatibility are reported on startup.
+
+Some advanced features allow an application administrator to smoothly stop a
+server, detect when there's no activity on it anymore, then take it off-line,
+stop it, upgrade it and ensure it doesn't take any traffic while being upgraded,
+then test it again through the normal path without opening it to the public, and
+all of this without touching HAProxy at all. This ensures that even complicated
+production operations may be done during opening hours with all technical
+resources available.
+
+The process tries to save resources as much as possible, uses memory pools to
+save on allocation time and limit memory fragmentation, releases payload buffers
+as soon as their contents are sent, and supports enforcing strong memory limits
+above which connections have to wait for a buffer to become available instead of
+allocating more memory. This system helps guarantee memory usage in certain
+strict environments.
+
+A command line interface (CLI) is available as a UNIX or TCP socket, to perform
+a number of operations and to retrieve troubleshooting information. Everything
+done on this socket doesn't require a configuration change, so it is mostly used
+for temporary changes. Using this interface it is possible to change a server's
+address, weight and status, to consult statistics and clear counters, dump and
+clear stickiness tables, possibly selectively by key criteria, dump and kill
+client-side and server-side connections, dump captured errors with a detailed
+analysis of the exact cause and location of the error, dump, add and remove
+entries from ACLs and maps, update TLS shared secrets, apply connection limits
+and rate limits on the fly to arbitrary frontends (useful in shared hosting
+environments), and disable a specific frontend to release a listening port
+(useful when daytime operations are forbidden and a fix is needed nonetheless).
+Updating certificates and their configuration on the fly is permitted, as well
+as enabling and consulting traces of every processing step of the traffic.
+
+For environments where SNMP is mandatory, at least two agents exist, one is
+provided with the HAProxy sources and relies on the Net-SNMP Perl module.
+Another one is provided with the commercial packages and doesn't require Perl.
+Both are roughly equivalent in terms of coverage.
+
+It is often recommended to install 4 utilities on the machine where HAProxy is
+deployed :
+
+  - socat (in order to connect to the CLI, though certain forks of netcat can
+    also do it to some extents);
+
+  - halog from the latest HAProxy version : this is the log analysis tool, it
+    parses native TCP and HTTP logs extremely fast (1 to 2 GB per second) and
+    extracts useful information and statistics such as requests per URL, per
+    source address, URLs sorted by response time or error rate, termination
+    codes etc. It was designed to be deployed on the production servers to
+    help troubleshoot live issues so it has to be there ready to be used;
+
+  - tcpdump : this is highly recommended to take the network traces needed to
+    troubleshoot an issue that was made visible in the logs. There is a moment
+    where application and haproxy's analysis will diverge and the network traces
+    are the only way to say who's right and who's wrong. It's also fairly common
+    to detect bugs in network stacks and hypervisors thanks to tcpdump;
+
+  - strace : it is tcpdump's companion. It will report what HAProxy really sees
+    and will help sort out the issues the operating system is responsible for
+    from the ones HAProxy is responsible for. Strace is often requested when a
+    bug in HAProxy is suspected;
+
+
+3.5.2. Advanced features : System-specific capabilities
+-------------------------------------------------------
+
+Depending on the operating system HAProxy is deployed on, certain extra features
+may be available or needed. While it is supported on a number of platforms,
+HAProxy is primarily developed on Linux, which explains why some features are
+only available on this platform.
+
+The transparent bind and connect features, the support for binding connections
+to a specific network interface, as well as the ability to bind multiple
+processes to the same IP address and ports are only available on Linux and BSD
+systems, though only Linux performs a kernel-side load balancing of the incoming
+requests between the available processes.
+
+On Linux, there are also a number of extra features and optimizations including
+support for network namespaces (also known as "containers") allowing HAProxy to
+be a gateway between all containers, the ability to set the MSS, Netfilter marks
+and IP TOS field on the client side connection, support for TCP FastOpen on the
+listening side, TCP user timeouts to let the kernel quickly kill connections
+when it detects the client has disappeared before the configured timeouts, TCP
+splicing to let the kernel forward data between the two sides of a connections
+thus avoiding multiple memory copies, the ability to enable the "defer-accept"
+bind option to only get notified of an incoming connection once data become
+available in the kernel buffers, and the ability to send the request with the
+ACK confirming a connect (sometimes called "piggy-back") which is enabled with
+the "tcp-smart-connect" option. On Linux, HAProxy also takes great care of
+manipulating the TCP delayed ACKs to save as many packets as possible on the
+network.
+
+Some systems have an unreliable clock which jumps back and forth in the past
+and in the future. This used to happen with some NUMA systems where multiple
+processors didn't see the exact same time of day, and recently it became more
+common in virtualized environments where the virtual clock has no relation with
+the real clock, resulting in huge time jumps (sometimes up to 30 seconds have
+been observed). This causes a lot of trouble with respect to timeout enforcement
+in general. Due to this flaw of these systems, HAProxy maintains its own
+monotonic clock which is based on the system's clock but where drift is measured
+and compensated for. This ensures that even with a very bad system clock, timers
+remain reasonably accurate and timeouts continue to work. Note that this problem
+affects all the software running on such systems and is not specific to HAProxy.
+The common effects are spurious timeouts or application freezes. Thus if this
+behavior is detected on a system, it must be fixed, regardless of the fact that
+HAProxy protects itself against it.
+
+On Linux, a new starting process may communicate with the previous one to reuse
+its listening file descriptors so that the listening sockets are never
+interrupted during the process's replacement.
+
+
+3.5.3. Advanced features : Scripting
+------------------------------------
+
+HAProxy can be built with support for the Lua embedded language, which opens a
+wide area of new possibilities related to complex manipulation of requests or
+responses, routing decisions, statistics processing and so on. Using Lua it is
+even possible to establish parallel connections to other servers to exchange
+information. This way it becomes possible (though complex) to develop an
+authentication system for example. Please refer to the documentation in the file
+"doc/lua-api/index.rst" for more information on how to use Lua.
+
+
+3.5.4. Advanced features: Tracing
+---------------------------------
+
+At any moment an administrator may connect over the CLI and enable tracing in
+various internal subsystems. Various levels of details are provided by default
+so that in practice anything between one line per request to 500 lines per
+request can be retrieved. Filters as well as an automatic capture on/off/pause
+mechanism are available so that it really is possible to wait for a certain
+event and watch it in detail. This is extremely convenient to diagnose protocol
+violations from faulty servers and clients, or denial of service attacks.
+
+
+3.6. Sizing
+-----------
+
+Typical CPU usage figures show 15% of the processing time spent in HAProxy
+versus 85% in the kernel in TCP or HTTP close mode, and about 30% for HAProxy
+versus 70% for the kernel in HTTP keep-alive mode. This means that the operating
+system and its tuning have a strong impact on the global performance.
+
+Usages vary a lot between users, some focus on bandwidth, other ones on request
+rate, others on connection concurrency, others on SSL performance. This section
+aims at providing a few elements to help with this task.
+
+It is important to keep in mind that every operation comes with a cost, so each
+individual operation adds its overhead on top of the other ones, which may be
+negligible in certain circumstances, and which may dominate in other cases.
+
+When processing the requests from a connection, we can say that :
+
+  - forwarding data costs less than parsing request or response headers;
+
+  - parsing request or response headers cost less than establishing then closing
+    a connection to a server;
+
+  - establishing an closing a connection costs less than a TLS resume operation;
+
+  - a TLS resume operation costs less than a full TLS handshake with a key
+    computation;
+
+  - an idle connection costs less CPU than a connection whose buffers hold data;
+
+  - a TLS context costs even more memory than a connection with data;
+
+So in practice, it is cheaper to process payload bytes than header bytes, thus
+it is easier to achieve high network bandwidth with large objects (few requests
+per volume unit) than with small objects (many requests per volume unit). This
+explains why maximum bandwidth is always measured with large objects, while
+request rate or connection rates are measured with small objects.
+
+Some operations scale well on multiple processes spread over multiple CPUs,
+and others don't scale as well. Network bandwidth doesn't scale very far because
+the CPU is rarely the bottleneck for large objects, it's mostly the network
+bandwidth and data buses to reach the network interfaces. The connection rate
+doesn't scale well over multiple processors due to a few locks in the system
+when dealing with the local ports table. The request rate over persistent
+connections scales very well as it doesn't involve much memory nor network
+bandwidth and doesn't require to access locked structures. TLS key computation
+scales very well as it's totally CPU-bound. TLS resume scales moderately well,
+but reaches its limits around 4 processes where the overhead of accessing the
+shared table offsets the small gains expected from more power.
+
+The performance numbers one can expect from a very well tuned system are in the
+following range. It is important to take them as orders of magnitude and to
+expect significant variations in any direction based on the processor, IRQ
+setting, memory type, network interface type, operating system tuning and so on.
+
+The following numbers were found on a Core i7 running at 3.7 GHz equipped with
+a dual-port 10 Gbps NICs running Linux kernel 3.10, HAProxy 1.6 and OpenSSL
+1.0.2. HAProxy was running as a single process on a single dedicated CPU core,
+and two extra cores were dedicated to network interrupts :
+
+  - 20 Gbps of maximum network bandwidth in clear text for objects 256 kB or
+    higher, 10 Gbps for 41kB or higher;
+
+  - 4.6 Gbps of TLS traffic using AES256-GCM cipher with large objects;
+
+  - 83000 TCP connections per second from client to server;
+
+  - 82000 HTTP connections per second from client to server;
+
+  - 97000 HTTP requests per second in server-close mode (keep-alive with the
+    client, close with the server);
+
+  - 243000 HTTP requests per second in end-to-end keep-alive mode;
+
+  - 300000 filtered TCP connections per second (anti-DDoS)
+
+  - 160000 HTTPS requests per second in keep-alive mode over persistent TLS
+    connections;
+
+  - 13100 HTTPS requests per second using TLS resumed connections;
+
+  - 1300 HTTPS connections per second using TLS connections renegotiated with
+    RSA2048;
+
+  - 20000 concurrent saturated connections per GB of RAM, including the memory
+    required for system buffers; it is possible to do better with careful tuning
+    but this result it easy to achieve.
+
+  - about 8000 concurrent TLS connections (client-side only) per GB of RAM,
+    including the memory required for system buffers;
+
+  - about 5000 concurrent end-to-end TLS connections (both sides) per GB of
+    RAM including the memory required for system buffers;
+
+A more recent benchmark featuring the multi-thread enabled HAProxy 2.4 on a
+64-core ARM Graviton2 processor in AWS reached 2 million HTTPS requests per
+second at sub-millisecond response time, and 100 Gbps of traffic:
+
+  https://www.haproxy.com/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/
+
+Thus a good rule of thumb to keep in mind is that the request rate is divided
+by 10 between TLS keep-alive and TLS resume, and between TLS resume and TLS
+renegotiation, while it's only divided by 3 between HTTP keep-alive and HTTP
+close. Another good rule of thumb is to remember that a high frequency core
+with AES instructions can do around 20 Gbps of AES-GCM per core.
+
+Another good rule of thumb is to consider that on the same server, HAProxy will
+be able to saturate :
+
+  - about 5-10 static file servers or caching proxies;
+
+  - about 100 anti-virus proxies;
+
+  - and about 100-1000 application servers depending on the technology in use.
+
+
+3.7. How to get HAProxy
+-----------------------
+
+HAProxy is an open source project covered by the GPLv2 license, meaning that
+everyone is allowed to redistribute it provided that access to the sources is
+also provided upon request, especially if any modifications were made.
+
+HAProxy evolves as a main development branch called "master" or "mainline", from
+which new branches are derived once the code is considered stable. A lot of web
+sites run some development branches in production on a voluntarily basis, either
+to participate to the project or because they need a bleeding edge feature, and
+their feedback is highly valuable to fix bugs and judge the overall quality and
+stability of the version being developed.
+
+The new branches that are created when the code is stable enough constitute a
+stable version and are generally maintained for several years, so that there is
+no emergency to migrate to a newer branch even when you're not on the latest.
+Once a stable branch is issued, it may only receive bug fixes, and very rarely
+minor feature updates when that makes users' life easier. All fixes that go into
+a stable branch necessarily come from the master branch. This guarantees that no
+fix will be lost after an upgrade. For this reason, if you fix a bug, please
+make the patch against the master branch, not the stable branch. You may even
+discover it was already fixed. This process also ensures that regressions in a
+stable branch are extremely rare, so there is never any excuse for not upgrading
+to the latest version in your current branch.
+
+Branches are numbered with two digits delimited with a dot, such as "1.6".
+Since 1.9, branches with an odd second digit are mostly focused on sensitive
+technical updates and more aimed at advanced users because they are likely to
+trigger more bugs than the other ones. They are maintained for about a year
+only and must not be deployed where they cannot be rolled back in emergency. A
+complete version includes one or two sub-version numbers indicating the level of
+fix. For example, version 1.5.14 is the 14th fix release in branch 1.5 after
+version 1.5.0 was issued. It contains 126 fixes for individual bugs, 24 updates
+on the documentation, and 75 other backported patches, most of which were needed
+to fix the aforementioned 126 bugs. An existing feature may never be modified
+nor removed in a stable branch, in order to guarantee that upgrades within the
+same branch will always be harmless.
+
+HAProxy is available from multiple sources, at different release rhythms :
+
+  - The official community web site : http://www.haproxy.org/ : this site
+    provides the sources of the latest development release, all stable releases,
+    as well as nightly snapshots for each branch. The release cycle is not fast,
+    several months between stable releases, or between development snapshots.
+    Very old versions are still supported there. Everything is provided as
+    sources only, so whatever comes from there needs to be rebuilt and/or
+    repackaged;
+
+  - GitHub : https://github.com/haproxy/haproxy/ : this is the mirror for the
+    development branch only, which provides integration with the issue tracker,
+    continuous integration and code coverage tools. This is exclusively for
+    contributors;
+
+  - A number of operating systems such as Linux distributions and BSD ports.
+    These systems generally provide long-term maintained versions which do not
+    always contain all the fixes from the official ones, but which at least
+    contain the critical fixes. It often is a good option for most users who do
+    not seek advanced configurations and just want to keep updates easy;
+
+  - Commercial versions from http://www.haproxy.com/ : these are supported
+    professional packages built for various operating systems or provided as
+    appliances, based on the latest stable versions and including a number of
+    features backported from the next release for which there is a strong
+    demand. It is the best option for users seeking the latest features with
+    the reliability of a stable branch, the fastest response time to fix bugs,
+    or simply support contracts on top of an open source product;
+
+
+In order to ensure that the version you're using is the latest one in your
+branch, you need to proceed this way :
+
+  - verify which HAProxy executable you're running : some systems ship it by
+    default and administrators install their versions somewhere else on the
+    system, so it is important to verify in the startup scripts which one is
+    used;
+
+  - determine which source your HAProxy version comes from. For this, it's
+    generally sufficient to type "haproxy -v". A development version will
+    appear like this, with the "dev" word after the branch number :
+
+      HAProxy version 2.4-dev18-a5357c-137 2021/05/09 - https://haproxy.org/
+
+    A stable version will appear like this, as well as unmodified stable
+    versions provided by operating system vendors :
+
+      HAProxy version 1.5.14 2015/07/02
+
+    And a nightly snapshot of a stable version will appear like this with an
+    hexadecimal sequence after the version, and with the date of the snapshot
+    instead of the date of the release :
+
+      HAProxy version 1.5.14-e4766ba 2015/07/29
+
+    Any other format may indicate a system-specific package with its own
+    patch set. For example HAProxy Enterprise versions will appear with the
+    following format (<branch>-<latest commit>-<revision>) :
+
+      HAProxy version 1.5.0-994126-357 2015/07/02
+
+    Please note that historically versions prior to 2.4 used to report the
+    process name with a hyphen between "HA" and "Proxy", including those above
+    which were adjusted to show the correct format only, so better ignore this
+    word or use a relaxed match in scripts. Additionally, modern versions add
+    a URL linking to the project's home.
+
+    Finally, versions 2.1 and above will include a "Status" line indicating
+    whether the version is safe for production or not, and if so, till when, as
+    well as a link to the list of known bugs affecting this version.
+
+  - for system-specific packages, you have to check with your vendor's package
+    repository or update system to ensure that your system is still supported,
+    and that fixes are still provided for your branch. For community versions
+    coming from haproxy.org, just visit the site, verify the status of your
+    branch and compare the latest version with yours to see if you're on the
+    latest one. If not you can upgrade. If your branch is not maintained
+    anymore, you're definitely very late and will have to consider an upgrade
+    to a more recent branch (carefully read the README when doing so).
+
+HAProxy will have to be updated according to the source it came from. Usually it
+follows the system vendor's way of upgrading a package. If it was taken from
+sources, please read the README file in the sources directory after extracting
+the sources and follow the instructions for your operating system.
+
+
+4. Companion products and alternatives
+--------------------------------------
+
+HAProxy integrates fairly well with certain products listed below, which is why
+they are mentioned here even if not directly related to HAProxy.
+
+
+4.1. Apache HTTP server
+-----------------------
+
+Apache is the de-facto standard HTTP server. It's a very complete and modular
+project supporting both file serving and dynamic contents. It can serve as a
+frontend for some application servers. It can even proxy requests and cache
+responses. In all of these use cases, a front load balancer is commonly needed.
+Apache can work in various modes, some being heavier than others. Certain
+modules still require the heavier pre-forked model and will prevent Apache from
+scaling well with a high number of connections. In this case HAProxy can provide
+a tremendous help by enforcing the per-server connection limits to a safe value
+and will significantly speed up the server and preserve its resources that will
+be better used by the application.
+
+Apache can extract the client's address from the X-Forwarded-For header by using
+the "mod_rpaf" extension. HAProxy will automatically feed this header when
+"option forwardfor" is specified in its configuration. HAProxy may also offer a
+nice protection to Apache when exposed to the internet, where it will better
+resist a wide number of types of DoS attacks.
+
+
+4.2. NGINX
+----------
+
+NGINX is the second de-facto standard HTTP server. Just like Apache, it covers a
+wide range of features. NGINX is built on a similar model as HAProxy so it has
+no problem dealing with tens of thousands of concurrent connections. When used
+as a gateway to some applications (e.g. using the included PHP FPM) it can often
+be beneficial to set up some frontend connection limiting to reduce the load
+on the PHP application. HAProxy will clearly be useful there both as a regular
+load balancer and as the traffic regulator to speed up PHP by decongesting
+it. Also since both products use very little CPU thanks to their event-driven
+architecture, it's often easy to install both of them on the same system. NGINX
+implements HAProxy's PROXY protocol, thus it is easy for HAProxy to pass the
+client's connection information to NGINX so that the application gets all the
+relevant information. Some benchmarks have also shown that for large static
+file serving, implementing consistent hash on HAProxy in front of NGINX can be
+beneficial by optimizing the OS' cache hit ratio, which is basically multiplied
+by the number of server nodes.
+
+
+4.3. Varnish
+------------
+
+Varnish is a smart caching reverse-proxy, probably best described as a web
+application accelerator. Varnish doesn't implement SSL/TLS and wants to dedicate
+all of its CPU cycles to what it does best. Varnish also implements HAProxy's
+PROXY protocol so that HAProxy can very easily be deployed in front of Varnish
+as an SSL offloader as well as a load balancer and pass it all relevant client
+information. Also, Varnish naturally supports decompression from the cache when
+a server has provided a compressed object, but doesn't compress however. HAProxy
+can then be used to compress outgoing data when backend servers do not implement
+compression, though it's rarely a good idea to compress on the load balancer
+unless the traffic is low.
+
+When building large caching farms across multiple nodes, HAProxy can make use of
+consistent URL hashing to intelligently distribute the load to the caching nodes
+and avoid cache duplication, resulting in a total cache size which is the sum of
+all caching nodes. In addition, caching of very small dumb objects for a short
+duration on HAProxy can sometimes save network round trips and reduce the CPU
+load on both the HAProxy and the Varnish nodes. This is only possible is no
+processing is done on these objects on Varnish (this is often referred to as
+the notion of "favicon cache", by which a sizeable percentage of useless
+downstream requests can sometimes be avoided). However do not enable HAProxy
+caching for a long time (more than a few seconds) in front of any other cache,
+that would significantly complicate troubleshooting without providing really
+significant savings.
+
+
+4.4. Alternatives
+-----------------
+
+Linux Virtual Server (LVS or IPVS) is the layer 4 load balancer included within
+the Linux kernel. It works at the packet level and handles TCP and UDP. In most
+cases it's more a complement than an alternative since it doesn't have layer 7
+knowledge at all.
+
+Pound is another well-known load balancer. It's much simpler and has much less
+features than HAProxy but for many very basic setups both can be used. Its
+author has always focused on code auditability first and wants to maintain the
+set of features low. Its thread-based architecture scales less well with high
+connection counts, but it's a good product.
+
+Pen is a quite light load balancer. It supports SSL, maintains persistence using
+a fixed-size table of its clients' IP addresses. It supports a packet-oriented
+mode allowing it to support direct server return and UDP to some extents. It is
+meant for small loads (the persistence table only has 2048 entries).
+
+NGINX can do some load balancing to some extents, though it's clearly not its
+primary function. Production traffic is used to detect server failures, the
+load balancing algorithms are more limited, and the stickiness is very limited.
+But it can make sense in some simple deployment scenarios where it is already
+present. The good thing is that since it integrates very well with HAProxy,
+there's nothing wrong with adding HAProxy later when its limits have been
+reached.
+
+Varnish also does some load balancing of its backend servers and does support
+real health checks. It doesn't implement stickiness however, so just like with
+NGINX, as long as stickiness is not needed that can be enough to start with.
+And similarly, since HAProxy and Varnish integrate so well together, it's easy
+to add it later into the mix to complement the feature set.
+
+
+5. Contacts
+-----------
+
+If you want to contact the developers or any community member about anything,
+the best way to do it usually is via the mailing list by sending your message
+to haproxy@formilux.org. Please note that this list is public and its archives
+are public as well so you should avoid disclosing sensitive information. A
+thousand of users of various experience levels are present there and even the
+most complex questions usually find an optimal response relatively quickly.
+Suggestions are welcome too. For users having difficulties with e-mail, a
+Discourse platform is available at http://discourse.haproxy.org/ . However
+please keep in mind that there are less people reading questions there and that
+most are handled by a really tiny team. In any case, please be patient and
+respectful with those who devote their spare time helping others.
+
+I you believe you've found a bug but are not sure, it's best reported on the
+mailing list. If you're quite convinced you've found a bug, that your version
+is up-to-date in its branch, and you already have a GitHub account, feel free
+to go directly to https://github.com/haproxy/haproxy/ and file an issue with
+all possibly available details. Again, this is public so be careful not to post
+information you might later regret. Since the issue tracker presents itself as
+a very long thread, please avoid pasting very long dumps (a few hundreds lines
+or more) and attach them instead.
+
+If you've found what you're absolutely certain can be considered a critical
+security issue that would put many users in serious trouble if discussed in a
+public place, then you can send it with the reproducer to security@haproxy.org.
+A small team of trusted developers will receive it and will be able to propose
+a fix. We usually don't use embargoes and once a fix is available it gets
+merged. In some rare circumstances it can happen that a release is coordinated
+with software vendors. Please note that this process usually messes up with
+eveyone's work, and that rushed up releases can sometimes introduce new bugs,
+so it's best avoided unless strictly necessary; as such, there is often little
+consideration for reports that needlessly cause such extra burden, and the best
+way to see your work credited usually is to provide a working fix, which will
+appear in changelogs.