From b46aad6df449445a9fc4aa7b32bd40005438e3f7 Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sat, 13 Apr 2024 14:18:05 +0200 Subject: Adding upstream version 2.9.5. Signed-off-by: Daniel Baumann --- doc/51Degrees-device-detection.txt | 174 + doc/DeviceAtlas-device-detection.txt | 82 + doc/SOCKS4.protocol.txt | 1 + doc/SPOE.txt | 1255 ++ doc/WURFL-device-detection.txt | 71 + doc/acl.fig | 229 + doc/architecture.txt | 1448 ++ doc/coding-style.txt | 1566 ++ doc/configuration.txt | 26732 +++++++++++++++++++++++ doc/cookie-options.txt | 25 + doc/design-thoughts/binding-possibilities.txt | 167 + doc/design-thoughts/connection-reuse.txt | 224 + doc/design-thoughts/http_load_time.url | 5 + doc/design-thoughts/pool-debugging.txt | 243 + doc/design-thoughts/thread-group.txt | 655 + doc/gpl.txt | 340 + doc/haproxy.1 | 227 + doc/internals/acl.txt | 82 + doc/internals/api/appctx.txt | 142 + doc/internals/api/buffer-api.txt | 653 + doc/internals/api/event_hdl.txt | 1015 + doc/internals/api/filters.txt | 1188 + doc/internals/api/htx-api.txt | 570 + doc/internals/api/initcalls.txt | 366 + doc/internals/api/ist.txt | 167 + doc/internals/api/layers.txt | 190 + doc/internals/api/list.txt | 195 + doc/internals/api/pools.txt | 585 + doc/internals/api/scheduler.txt | 228 + doc/internals/body-parsing.txt | 165 + doc/internals/connect-status.txt | 28 + doc/internals/connection-header.txt | 196 + doc/internals/connection-scale.txt | 44 + doc/internals/fd-migration.txt | 138 + doc/internals/hashing.txt | 83 + doc/internals/list.fig | 599 + doc/internals/list.png | Bin 0 -> 33618 bytes doc/internals/listener-states.fig | 150 + doc/internals/listener-states.png | Bin 0 -> 36142 bytes doc/internals/lua_socket.fig | 113 + doc/internals/lua_socket.pdf | Bin 0 -> 14969 bytes doc/internals/muxes.fig | 401 + doc/internals/muxes.pdf | Bin 0 -> 12984 bytes doc/internals/muxes.png | Bin 0 -> 32209 bytes doc/internals/muxes.svg | 911 + doc/internals/notes-layers.txt | 330 + doc/internals/notes-poll-connect.txt | 93 + doc/internals/notes-pollhup.txt | 281 + doc/internals/notes-polling.txt | 192 + doc/internals/pattern.dia | Bin 0 -> 5631 bytes doc/internals/pattern.pdf | Bin 0 -> 37269 bytes doc/internals/polling-states.fig | 59 + doc/internals/sched.fig | 748 + doc/internals/sched.pdf | Bin 0 -> 25334 bytes doc/internals/sched.png | Bin 0 -> 150457 bytes doc/internals/sched.svg | 1204 + doc/internals/ssl_cert.dia | Bin 0 -> 6700 bytes doc/internals/stats-v2.txt | 8 + doc/internals/stconn-close.txt | 74 + doc/internals/stream-sock-states.fig | 535 + doc/intro.txt | 1700 ++ doc/lgpl.txt | 504 + doc/linux-syn-cookies.txt | 106 + doc/lua-api/Makefile | 153 + doc/lua-api/_static/channel.fig | 55 + doc/lua-api/_static/channel.png | Bin 0 -> 18457 bytes doc/lua-api/conf.py | 242 + doc/lua-api/index.rst | 4491 ++++ doc/lua.txt | 972 + doc/management.txt | 4521 ++++ doc/netscaler-client-ip-insertion-protocol.txt | 55 + doc/network-namespaces.txt | 106 + doc/peers-v2.0.txt | 294 + doc/peers.txt | 491 + doc/proxy-protocol.txt | 1051 + doc/queuing.fig | 192 + doc/regression-testing.txt | 706 + doc/seamless_reload.txt | 62 + 78 files changed, 60878 insertions(+) create mode 100644 doc/51Degrees-device-detection.txt create mode 100644 doc/DeviceAtlas-device-detection.txt create mode 100644 doc/SOCKS4.protocol.txt create mode 100644 doc/SPOE.txt create mode 100644 doc/WURFL-device-detection.txt create mode 100644 doc/acl.fig create mode 100644 doc/architecture.txt create mode 100644 doc/coding-style.txt create mode 100644 doc/configuration.txt create mode 100644 doc/cookie-options.txt create mode 100644 doc/design-thoughts/binding-possibilities.txt create mode 100644 doc/design-thoughts/connection-reuse.txt create mode 100644 doc/design-thoughts/http_load_time.url create mode 100644 doc/design-thoughts/pool-debugging.txt create mode 100644 doc/design-thoughts/thread-group.txt create mode 100644 doc/gpl.txt create mode 100644 doc/haproxy.1 create mode 100644 doc/internals/acl.txt create mode 100644 doc/internals/api/appctx.txt create mode 100644 doc/internals/api/buffer-api.txt create mode 100644 doc/internals/api/event_hdl.txt create mode 100644 doc/internals/api/filters.txt create mode 100644 doc/internals/api/htx-api.txt create mode 100644 doc/internals/api/initcalls.txt create mode 100644 doc/internals/api/ist.txt create mode 100644 doc/internals/api/layers.txt create mode 100644 doc/internals/api/list.txt create mode 100644 doc/internals/api/pools.txt create mode 100644 doc/internals/api/scheduler.txt create mode 100644 doc/internals/body-parsing.txt create mode 100644 doc/internals/connect-status.txt create mode 100644 doc/internals/connection-header.txt create mode 100644 doc/internals/connection-scale.txt create mode 100644 doc/internals/fd-migration.txt create mode 100644 doc/internals/hashing.txt create mode 100644 doc/internals/list.fig create mode 100644 doc/internals/list.png create mode 100644 doc/internals/listener-states.fig create mode 100644 doc/internals/listener-states.png create mode 100644 doc/internals/lua_socket.fig create mode 100644 doc/internals/lua_socket.pdf create mode 100644 doc/internals/muxes.fig create mode 100644 doc/internals/muxes.pdf create mode 100644 doc/internals/muxes.png create mode 100644 doc/internals/muxes.svg create mode 100644 doc/internals/notes-layers.txt create mode 100644 doc/internals/notes-poll-connect.txt create mode 100644 doc/internals/notes-pollhup.txt create mode 100644 doc/internals/notes-polling.txt create mode 100644 doc/internals/pattern.dia create mode 100644 doc/internals/pattern.pdf create mode 100644 doc/internals/polling-states.fig create mode 100644 doc/internals/sched.fig create mode 100644 doc/internals/sched.pdf create mode 100644 doc/internals/sched.png create mode 100644 doc/internals/sched.svg create mode 100644 doc/internals/ssl_cert.dia create mode 100644 doc/internals/stats-v2.txt create mode 100644 doc/internals/stconn-close.txt create mode 100644 doc/internals/stream-sock-states.fig create mode 100644 doc/intro.txt create mode 100644 doc/lgpl.txt create mode 100644 doc/linux-syn-cookies.txt create mode 100644 doc/lua-api/Makefile create mode 100644 doc/lua-api/_static/channel.fig create mode 100644 doc/lua-api/_static/channel.png create mode 100644 doc/lua-api/conf.py create mode 100644 doc/lua-api/index.rst create mode 100644 doc/lua.txt create mode 100644 doc/management.txt create mode 100644 doc/netscaler-client-ip-insertion-protocol.txt create mode 100644 doc/network-namespaces.txt create mode 100644 doc/peers-v2.0.txt create mode 100644 doc/peers.txt create mode 100644 doc/proxy-protocol.txt create mode 100644 doc/queuing.fig create mode 100644 doc/regression-testing.txt create mode 100644 doc/seamless_reload.txt (limited to 'doc') diff --git a/doc/51Degrees-device-detection.txt b/doc/51Degrees-device-detection.txt new file mode 100644 index 0000000..2e31274 --- /dev/null +++ b/doc/51Degrees-device-detection.txt @@ -0,0 +1,174 @@ +51Degrees Device Detection +-------------------------- + +You can also include 51Degrees for inbuilt device detection enabling attributes +such as screen size (physical & pixels), supported input methods, release date, +hardware vendor and model, browser information, and device price among many +others. Such information can be used to improve the user experience of a web +site by tailoring the page content, layout and business processes to the +precise characteristics of the device. Such customisations improve profit by +making it easier for customers to get to the information or services they +need. Attributes of the device making a web request can be added to HTTP +headers as configurable parameters. + +In order to enable 51Degrees download the 51Degrees source code from the +official git repository : + + - either use the proven stable but frozen 3.2.10 version which + supports the Trie algorithm : + + git clone https://github.com/51Degrees/Device-Detection.git -b v3.2.10 + + - use newer 3.2.12.12 version which continues to receive database + updates and supports a new Hash Trie algorithm, but which is not + compatible with older Trie databases : + + git clone https://github.com/51Degrees/Device-Detection.git -b v3.2.12 + + - or use the latest 51Degrees version 4 with 51Degrees Hash algorithm, + not compatible with older databases : + + git clone --recurse-submodules https://github.com/51Degrees/device-detection-cxx.git + +then run 'make' with USE_51DEGREES, optionally 51DEGREES_VER=4 (if using +51Degrees version 4), and 51DEGREES_SRC set. Both 51DEGREES_INC and +51DEGREES_LIB may additionally be used to force specific different paths for +.o and .h, but will default to 51DEGREES_SRC. Make sure to replace +'51D_REPO_PATH' with the path to the 51Degrees repository. + +51Degrees provide 4 different detection algorithms: + + 1. Pattern - balances main memory usage and CPU. + 2. Trie - a very high performance detection solution which uses more main + memory than Pattern. + 3. Hash Trie - replaces Trie, 3x faster, 80% lower memory consumption and + tuning options. + 4. 51Degrees V4 Hash - only with 51Degrees Device Detection V4. + +To make with 51Degrees Pattern algorithm use the following command line. + + $ make TARGET= USE_51DEGREES=1 51DEGREES_SRC='51D_REPO_PATH'/src/pattern + +To use the 51Degrees Trie algorithm use the following command line. + + $ make TARGET= USE_51DEGREES=1 51DEGREES_SRC='51D_REPO_PATH'/src/trie + +To build with the 51Degrees Device Detection V4 use the following command line. + + $ make TARGET= USE_51DEGREES=1 51DEGREES_VER=4 51DEGREES_SRC='51D_REPO_PATH'/src + +A data file containing information about devices, browsers, operating systems +and their associated signatures is then needed. 51Degrees provide a free +database with Github repo for this purpose. These free data files are located +in '51D_REPO_PATH'/data with the extensions .dat for Pattern data and .trie for +Trie data. Free Hash Trie data file can be obtained by signing up for a licence +key at https://51degrees.com/products/store/on-premise-device-detection. +If using the 51degrees version 4, the free hash data file is located in +'51D_REPO_PATH'/device-detection-data with the .hash extension. + +For HAProxy developers who need to verify that their changes didn't affect the +51Degrees implementation, a dummy library is provided in the +"addons/51degrees/dummy" directory. This does not function, but implements the +API such that the 51Degrees module can be used (but not return any meaningful +information). To test either Pattern or Hash Trie, or the 51Degrees version 4 +Hash algorithm, build with: + + $ make TARGET= USE_51DEGREES=1 51DEGREES_SRC=addons/51degrees/dummy/pattern +or + $ make TARGET= USE_51DEGREES=1 51DEGREES_SRC=addons/51degrees/dummy/trie +or + $ make TARGET= USE_51DEGREES=1 51DEGREES_VER=4 51DEGREES_SRC=addons/51degrees/dummy/v4hash + +respectively. + +The configuration file needs to set the following parameters: + + global + 51degrees-data-file path to the Pattern, Trie or V4 Hash data file + 51degrees-property-name-list list of 51Degrees properties to detect + 51degrees-property-separator separator to use between values + 51degrees-cache-size LRU-based cache size (disabled by default) + +The following is an example of the settings for Pattern. + + global + 51degrees-data-file '51D_REPO_PATH'/data/51Degrees-LiteV3.2.dat + 51degrees-property-name-list IsTablet DeviceType IsMobile + 51degrees-property-separator , + 51degrees-cache-size 10000 + +HAProxy needs a way to pass device information to the backend servers. This is +done by using the 51d converter or fetch method, which intercepts the HTTP +headers and creates some new headers. This is controlled in the frontend +http-in section. + +The following is an example which adds two new HTTP headers prefixed X-51D- + + frontend http-in + bind *:8081 + default_backend servers + http-request set-header X-51D-DeviceTypeMobileTablet %[51d.all(DeviceType,IsMobile,IsTablet)] + http-request set-header X-51D-Tablet %[51d.all(IsTablet)] + +Here, two headers are created with 51Degrees data, X-51D-DeviceTypeMobileTablet +and X-51D-Tablet. Any number of headers can be created this way and can be +named anything. 51d.all( ) invokes the 51degrees fetch. It can be passed up to +five property names of values to return. Values will be returned in the same +order, separated by the 51-degrees-property-separator configured earlier. If a +property name can't be found the value 'NoData' is returned instead. + +In addition to the device properties three additional properties related to the +validity of the result can be returned when used with the Pattern method. The +following example shows how Method, Difference and Rank could be included as one +new HTTP header X-51D-Stats. + + frontend http-in + ... + http-request set-header X-51D-Stats %[51d.all(Method,Difference,Rank)] + +These values indicate how confident 51Degrees is in the result that that was +returned. More information is available on the 51Degrees web site at: + + https://51degrees.com/support/documentation/pattern + +The above 51d.all fetch method uses all available HTTP headers for detection. A +modest performance improvement can be obtained by only passing one HTTP header +to the detection method with the 51d.single converter. The following example +uses the User-Agent HTTP header only for detection. + + frontend http-in + ... + http-request set-header X-51D-DeviceTypeMobileTablet %[req.fhdr(User-Agent),51d.single(DeviceType,IsMobile,IsTablet)] + +Any HTTP header could be used inplace of User-Agent by changing the parameter +provided to req.fhdr. + +When compiled to use the Trie detection method the trie format data file needs +to be provided. Changing the extension of the data file from dat to trie will +use the correct data. + + global + 51degrees-data-file '51D_REPO_PATH'/data/51Degrees-LiteV3.2.trie + +When used with Trie the Method, Difference and Rank properties are not +available. + +When using the 51Degrees V4 Hash algorithm, the hash format data file needs +to be provided as in the following example. + + global + 51degrees-data-file '51D_REPO_PATH'/device-detection-data/51Degrees-LiteV4.1.hash + +The free Lite data file contains information about screen size in pixels and +whether the device is a mobile. A full list of available properties is located +on the 51Degrees web site at: + + https://51degrees.com/resources/property-dictionary + +Some properties are only available in the paid for Premium and Enterprise +versions of 51Degrees. These data sets not only contain more properties but +are updated weekly and daily and contain signatures for 100,000s of different +device combinations. For more information see the data options comparison web +page: + + https://51degrees.com/compare-data-options diff --git a/doc/DeviceAtlas-device-detection.txt b/doc/DeviceAtlas-device-detection.txt new file mode 100644 index 0000000..b600918 --- /dev/null +++ b/doc/DeviceAtlas-device-detection.txt @@ -0,0 +1,82 @@ +DeviceAtlas Device Detection +---------------------------- + +In order to add DeviceAtlas Device Detection support, you would need to download +the API source code from https://deviceatlas.com/deviceatlas-haproxy-module. +The build supports the USE_PCRE and USE_PCRE2 options. Once extracted : + + $ make TARGET= USE_PCRE=1 (or USE_PCRE2=1) USE_DEVICEATLAS=1 DEVICEATLAS_SRC= + +Optionally DEVICEATLAS_INC and DEVICEATLAS_LIB may be set to override the path +to the include files and libraries respectively if they're not in the source +directory. However, if the API had been installed beforehand, DEVICEATLAS_SRC +can be omitted. Note that the DeviceAtlas C API version supported is the 2.4.0 +at minimum. + +For HAProxy developers who need to verify that their changes didn't accidentally +break the DeviceAtlas code, it is possible to build a dummy library provided in +the addons/deviceatlas/dummy directory and to use it as an alternative for the +full library. This will not provide the full functionalities, it will just allow +haproxy to start with a deviceatlas configuration, which generally is enough to +validate API changes : + + $ make TARGET= USE_PCRE=1 USE_DEVICEATLAS=1 DEVICEATLAS_SRC=$PWD/addons/deviceatlas/dummy + +These are supported DeviceAtlas directives (see doc/configuration.txt) : + - deviceatlas-json-file . + - deviceatlas-log-level (0 to 3, level of information returned by + the API, 0 by default). + - deviceatlas-property-separator (character used to separate the + properties produced by the API, | by default). + +Sample configuration : + + global + deviceatlas-json-file + + ... + frontend + bind *:8881 + default_backend servers + +There are two distinct methods available, one which leverages all HTTP headers +and one which uses only a single HTTP header for the detection. The former +method is highly recommended and more accurate. There are several possible use +cases. + +# To transmit the DeviceAtlas data downstream to the target application + +All HTTP headers via the sample / fetch + + http-request set-header X-DeviceAtlas-Data %[da-csv-fetch(primaryHardwareType,osName,osVersion,browserName,browserVersion,browserRenderingEngine)] + +Single HTTP header (e.g. User-Agent) via the converter + + http-request set-header X-DeviceAtlas-Data %[req.fhdr(User-Agent),da-csv-conv(primaryHardwareType,osName,osVersion,browserName,browserVersion,browserRenderingEngine)] + +# Mobile content switching with ACL + +All HTTP headers + + acl is_mobile da-csv-fetch(mobileDevice) 1 + +Single HTTP header + + acl device_type_tablet req.fhdr(User-Agent),da-csv-conv(primaryHardwareType) "Tablet" + +Optionally a JSON download scheduler is provided to allow a data file being +fetched automatically in a daily basis without restarting HAProxy : + + $ cd addons/deviceatlas && make [DEVICEATLAS_SRC=] + +Similarly, if the DeviceAtlas API is installed, DEVICEATLAS_SRC can be omitted. + + $ ./dadwsch -u JSON data file URL e.g. "https://deviceatlas.com/getJSON?licencekey=&format=zip&data=my&index=web" \ + [-p download directory path /tmp by default] \ + [-d scheduled hour of download, hour when the service is launched by default] + +Noted it needs to be started before HAProxy. + + +Please find more information about DeviceAtlas and the detection methods at +https://deviceatlas.com/resources . diff --git a/doc/SOCKS4.protocol.txt b/doc/SOCKS4.protocol.txt new file mode 100644 index 0000000..06aee8a --- /dev/null +++ b/doc/SOCKS4.protocol.txt @@ -0,0 +1 @@ +Please reference to "https://www.openssh.com/txt/socks4.protocol". \ No newline at end of file diff --git a/doc/SPOE.txt b/doc/SPOE.txt new file mode 100644 index 0000000..cc6d8dd --- /dev/null +++ b/doc/SPOE.txt @@ -0,0 +1,1255 @@ + ----------------------------------------------- + Stream Processing Offload Engine (SPOE) + Version 1.2 + ( Last update: 2020-06-13 ) + ----------------------------------------------- + Author : Christopher Faulet + Contact : cfaulet at haproxy dot com + + +SUMMARY +-------- + + 0. Terms + 1. Introduction + 2. SPOE configuration + 2.1. SPOE scope + 2.2. "spoe-agent" section + 2.3. "spoe-message" section + 2.4. "spoe-group" section + 2.5. Example + 3. SPOP specification + 3.1. Data types + 3.2. Frames + 3.2.1. Frame capabilities + 3.2.2. Frame types overview + 3.2.3. Workflow + 3.2.4. Frame: HAPROXY-HELLO + 3.2.5. Frame: AGENT-HELLO + 3.2.6. Frame: NOTIFY + 3.2.7. Frame: ACK + 3.2.8. Frame: HAPROXY-DISCONNECT + 3.2.9. Frame: AGENT-DISCONNECT + 3.3. Events & messages + 3.4. Actions + 3.5. Errors & timeouts + 4. Logging + + +0. Terms +--------- + +* SPOE : Stream Processing Offload Engine. + + A SPOE is a filter talking to servers managed by a SPOA to offload the + stream processing. An engine is attached to a proxy. A proxy can have + several engines. Each engine is linked to an agent and only one. + +* SPOA : Stream Processing Offload Agent. + + A SPOA is a service that will receive info from a SPOE to offload the + stream processing. An agent manages several servers. It uses a backend to + reference all of them. By extension, these servers can also be called + agents. + +* SPOP : Stream Processing Offload Protocol, used by SPOEs to talk to SPOA + servers. + + This protocol is used by engines to talk to agents. It is an in-house + binary protocol described in this documentation. + + +1. Introduction +---------------- + +SPOE is a feature introduced in HAProxy 1.7. It makes possible the +communication with external components to retrieve some info. The idea started +with the problems caused by most ldap libs not working fine in event-driven +systems (often at least the connect() is blocking). So, it is hard to properly +implement Single Sign On solution (SSO) in HAProxy. The SPOE will ease this +kind of processing, or we hope so. + +Now, the aim of SPOE is to allow any kind of offloading on the streams. First +releases won't do lot of things. As we will see, there are few handled events +and even less actions supported. Actually, for now, the SPOE can offload the +processing before "tcp-request content", "tcp-response content", "http-request" +and "http-response" rules. And it only supports variables definition. But, in +spite of these limited features, we can easily imagine to implement SSO +solution, ip reputation or ip geolocation services. + +Some example implementations in various languages are linked to from the +HAProxy Wiki page dedicated to this mechanism: + + https://github.com/haproxy/wiki/wiki/SPOE:-Stream-Processing-Offloading-Engine + +2. SPOE configuration +---------------------- + +Because SPOE is implemented as a filter, To use it, you must declare a "filter +spoe" line in a proxy section (frontend/backend/listen) : + + frontend my-front + ... + filter spoe [engine ] config + ... + +The "config" parameter is mandatory. It specifies the SPOE configuration +file. The engine name is optional. It can be set to declare the scope to use in +the SPOE configuration. So it is possible to use the same SPOE configuration +for several engines. If no name is provided, the SPOE configuration must not +contain any scope directive. + +We use a separate configuration file on purpose. By commenting SPOE filter +line, you completely disable the feature, including the parsing of sections +reserved to SPOE. This is also a way to keep the HAProxy configuration clean. + +A SPOE configuration file must contains, at least, the SPOA configuration +("spoe-agent" section) and SPOE messages/groups ("spoe-message" or "spoe-group" +sections) attached to this agent. + +IMPORTANT : The configuration of a SPOE filter must be located in a dedicated +file. But the backend used by a SPOA must be declared in HAProxy configuration +file. + +2.1. SPOE scope +------------------------- + +If you specify an engine name on the SPOE filter line, then you need to define +scope in the SPOE configuration with the same name. You can have several SPOE +scope in the same file. In each scope, you must define one and only one +"spoe-agent" section to configure the SPOA linked to your SPOE and several +"spoe-message" and "spoe-group" sections to describe, respectively, messages and +group of messages sent to servers managed by your SPOA. + +A SPOE scope starts with this kind of line : + + [] + +where is the same engine name specified on the SPOE filter line. The +scope ends when the file ends or when another scope is found. + + Example : + [my-first-engine] + spoe-agent my-agent + ... + spoe-message msg1 + ... + spoe-message msg2 + ... + spoe-group grp1 + ... + spoe-group grp2 + ... + + [my-second-engine] + ... + +If no engine name is provided on the SPOE filter line, no SPOE scope must be +found in the SPOE configuration file. All the file is considered to be in the +same anonymous and implicit scope. + +The engine name must be uniq for a proxy. If no engine name is provided on the +SPOE filter line, the SPOE agent name is used by default. + +2.2. "spoe-agent" section +-------------------------- + +For each engine, you must define one and only one "spoe-agent" section. In this +section, you will declare SPOE messages and the backend you will use. You will +also set timeouts and options to customize your agent's behaviour. + + +spoe-agent + Create a new SPOA with the name . It must have one and only one + "spoe-agent" definition by SPOE scope. + + Arguments : + is the name of the agent section. + + following keywords are supported : + - groups + - log + - maxconnrate + - maxerrrate + - max-frame-size + - max-waiting-frames + - messages + - [no] option async + - [no] option dontlog-normal + - [no] option pipelining + - [no] option send-frag-payload + - option continue-on-error + - option force-set-var + - option set-on-error + - option set-process-time + - option set-total-time + - option var-prefix + - register-var-names + - timeout hello|idle|processing + - use-backend + + +groups ... + Declare the list of SPOE groups that an agent will handle. + + Arguments : + is the name of a SPOE group. + + Groups declared here must be found in the same engine scope, else an error is + triggered during the configuration parsing. You can have many "groups" lines. + + See also: "spoe-group" section. + + +log global +log
[len ] [format ] [ []] +no log + Enable per-instance logging of events and traffic. + + Prefix : + no should be used when the logger list must be flushed. + + See the HAProxy Configuration Manual for details about this option. + +maxconnrate + Set the maximum number of connections per second to . The SPOE will + stop to open new connections if the maximum is reached and will wait to + acquire an existing one. So it is important to set "timeout hello" to a + relatively small value. + + +maxerrrate + Set the maximum number of errors per second to . The SPOE will stop + its processing if the maximum is reached. + + +max-frame-size + Set the maximum allowed size for frames exchanged between HAProxy and SPOA. + It must be in the range [256, tune.bufsize-4] (4 bytes are reserved for the + frame length). By default, it is set to (tune.bufsize-4). + +max-waiting-frames + Set the maximum number of frames waiting for an acknowledgement on the same + connection. This value is only used when the pipelinied or asynchronous + exchanges between HAProxy and SPOA are enabled. By default, it is set to 20. + +messages ... + Declare the list of SPOE messages that an agent will handle. + + Arguments : + is the name of a SPOE message. + + Messages declared here must be found in the same engine scope, else an error + is triggered during the configuration parsing. You can have many "messages" + lines. + + See also: "spoe-message" section. + + +option async +no option async + Enable or disable the support of asynchronous exchanges between HAProxy and + SPOA. By default, this option is enabled. + + +option continue-on-error + Do not stop the events processing when an error occurred on a stream. + + By default, for a specific stream, when an abnormal/unexpected error occurs, + the SPOE is disabled for all the transaction. So if you have several events + configured, such error on an event will disabled all following. For TCP + streams, this will disable the SPOE for the whole session. For HTTP streams, + this will disable it for the transaction (request and response). + + When set, this option bypass this behaviour and only the current event will + be ignored. + + +option dontlog-normal +no option dontlog-normal + Enable or disable logging of normal, successful processing. + + Arguments : none + + See also: "log" and section 4 about logging. + + +option force-set-var + By default, SPOE filter only register already known variables (mainly from + parsing of the configuration), and process-wide variables (those of scope + "proc") cannot be created. If you want that haproxy trusts the agent and + registers all variables (ex: can be useful for LUA workload), activate this + option. + + Caution : this option opens to a variety of attacks such as a rogue SPOA that + asks to register too many variables. + + +option pipelining +no option pipelining + Enable or disable the support of pipelined exchanges between HAProxy and + SPOA. By default, this option is enabled. + + +option send-frag-payload +no option send-frag-payload + Enable or disable the sending of fragmented payload to SPOA. By default, this + option is enabled. + + +option set-on-error + Define the variable to set when an error occurred during an event processing. + + Arguments : + + is the variable name, without the scope. The name may only + contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. + + This variable will only be set when an error occurred in the scope of the + transaction. As for all other variables define by the SPOE, it will be + prefixed. So, if your variable name is "error" and your prefix is + "my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error". + + When set, the variable is an integer representing the error reason. For values + under 256, it represents an error coming from the engine. Below 256, it + reports a SPOP error. In this case, to retrieve the right SPOP status code, + you must remove 256 to this value. Here are possible values: + + * 1 a timeout occurred during the event processing. + + * 2 an error was triggered during the resources allocation. + + * 3 the frame payload exceeds the frame size and it cannot be + fragmented. + + * 4 the fragmentation of a payload is aborted. + + * 5 The frame processing has been interrupted by HAProxy. + + * 255 an unknown error occurred during the event processing. + + * 256+N a SPOP error occurred during the event processing (see section + "Errors & timeouts"). + + Note that if "option continue-on-error" is set, the variable is not + automatically removed between events processing. + + See also: "option continue-on-error", "option var-prefix". + + +option set-process-time + Define the variable to set to report the processing time of the last event or + group. + + Arguments : + + is the variable name, without the scope. The name may only + contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. + + This variable will be set in the scope of the transaction. As for all other + variables define by the SPOE, it will be prefixed. So, if your variable name + is "process_time" and your prefix is "my_spoe_pfx", the variable will be + "txn.my_spoe_pfx.process_time". + + When set, the variable is an integer representing the delay to process the + event or the group, in milliseconds. From the stream point of view, it is the + latency added by the SPOE processing for the last handled event or group. + + If several events or groups are processed for the same stream, this value + will be overrideen. + + See also: "option set-total-time". + + +option set-total-time + Define the variable to set to report the total processing time SPOE for a + stream. + + Arguments : + + is the variable name, without the scope. The name may only + contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. + + This variable will be set in the scope of the transaction. As for all other + variables define by the SPOE, it will be prefixed. So, if your variable name + is "total_time" and your prefix is "my_spoe_pfx", the variable will be + "txn.my_spoe_pfx.total_time". + + When set, the variable is an integer representing the sum of processing times + for a stream, in milliseconds. From the stream point of view, it is the + latency added by the SPOE processing. + + If several events or groups are processed for the same stream, this value + will be updated. + + See also: "option set-process-time". + + +option var-prefix + Define the prefix used when variables are set by an agent. + + Arguments : + + is the prefix used to limit the scope of variables set by an + agent. + + To avoid conflict with other variables defined by HAProxy, all variables + names will be prefixed. By default, the "spoe-agent" name is used. This + option can be used to customize it. + + The prefix will be added between the variable scope and its name, separated + by a '.'. It may only contain characters 'a-z', 'A-Z', '0-9', '.' and '_', as + for variables name. In HAProxy configuration, you need to use this prefix as + a part of the variables name. For example, if an agent define the variable + "myvar" in the "txn" scope, with the prefix "my_spoe_pfx", then you should + use "txn.my_spoe_pfx.myvar" name in your HAProxy configuration. + + By default, an agent will never set new variables at runtime: It can only set + new value for existing ones. If you want a different behaviour, see + force-set-var option and register-var-names directive. + +register-var-names ... + Register some variable names. By default, an agent will not be allowed to set + new variables at runtime. This rule can be totally relaxed by setting the + option "force-set-var". If you know all the variables you will need, this + directive is a good way to register them without letting an agent doing what + it want. This is only required if these variables are not referenced anywhere + in the HAProxy configuration or the SPOE one. + + Arguments: + is a variable name without the scope. The name may only + contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. + + The prefix will be automatically added during the registration. You can have + many "register-var-names" lines. + + See also: "option force-set-var", "option var-prefix". + +timeout hello + Set the maximum time to wait for an agent to receive the AGENT-HELLO frame. + It is applied on the stream that handle the connection with the agent. + + Arguments : + is the timeout value specified in milliseconds by default, but + can be in any other unit if the number is suffixed by the unit, + as explained at the top of this document. + + This timeout is an applicative timeout. It differ from "timeout connect" + defined on backends. + + +timeout idle + Set the maximum time to wait for an agent to close an idle connection. It is + applied on the stream that handle the connection with the agent. + + Arguments : + is the timeout value specified in milliseconds by default, but + can be in any other unit if the number is suffixed by the unit, + as explained at the top of this document. + + +timeout processing + Set the maximum time to wait for a stream to process an event, i.e to acquire + a stream to talk with an agent, to encode all messages, to send the NOTIFY + frame, to receive the corresponding acknowledgement and to process all + actions. It is applied on the stream that handle the client and the server + sessions. + + Arguments : + is the timeout value specified in milliseconds by default, but + can be in any other unit if the number is suffixed by the unit, + as explained at the top of this document. + + +use-backend + Specify the backend to use. It must be defined. + + Arguments : + is the name of a valid "backend" section. + + +2.3. "spoe-message" section +---------------------------- + +To offload the stream processing, SPOE will send messages with specific +information at a specific moment in the stream life and will wait for +corresponding replies to know what to do. + + +spoe-message + Create a new SPOE message with the name . + + Arguments : + is the name of the SPOE message. + + Here you define a message that can be referenced in a "spoe-agent" + section. Following keywords are supported : + - acl + - args + - event + + See also: "spoe-agent" section. + + +acl [flags] [operator] ... + Declare or complete an access list. + + See section 7 about ACL usage in the HAProxy Configuration Manual. + + +args [name=] ... + Define arguments passed into the SPOE message. + + Arguments : + is a sample expression. + + When the message is processed, if a sample expression is not available, it is + set to NULL. Arguments are processed in their declaration order and added in + the message in that order. It is possible to declare named arguments. + + For example: + args frontend=fe_id src dst + + +event [ { if | unless } ] + Set the event that triggers sending of the message. It may optionally be + followed by an ACL-based condition, in which case it will only be evaluated + if the condition is true. A SPOE message can only be sent on one event. If + several events are defined, only the last one is considered. + + ACL-based conditions are executed in the context of the stream that handle + the client and the server connections. + + Arguments : + is the event name. + is a standard ACL-based condition. + + Supported events are: + - on-client-session + - on-server-session + - on-frontend-tcp-request + - on-backend-tcp-request + - on-tcp-response + - on-frontend-http-request + - on-backend-http-request + - on-http-response + + See section "Events & Messages" for more details about supported events. + See section 7 about ACL usage in the HAProxy Configuration Manual. + +2.4. "spoe-group" section +-------------------------- + +This section can be used to declare a group of SPOE messages. Unlike messages +referenced in a "spoe-agent" section, messages inside a group are not sent on a +specific event. The sending must be triggered by TCP or HTTP rules, from the +HAProxy configuration. + + +spoe-group + Create a new SPOE group with the name . + + Arguments : + is the name of the SPOE group. + + Here you define a group of SPOE messages that can be referenced in a + "spoe-agent" section. Following keywords are supported : + - messages + + See also: "spoe-agent" and "spoe-message" sections. + + +messages ... + Declare the list of SPOE messages belonging to the group. + + Arguments : + is the name of a SPOE message. + + Messages declared here must be found in the same engine scope, else an error + is triggered during the configuration parsing. Furthermore, a message belongs + at most to a group. You can have many "messages" lines. + + See also: "spoe-message" section. + + +2.5. Example +------------- + +Here is a simple but complete example that sends client-ip address to a ip +reputation service. This service can set the variable "ip_score" which is an +integer between 0 and 100, indicating its reputation (100 means totally safe +and 0 a blacklisted IP with no doubt). + + ### + ### HAProxy configuration + frontend www + mode http + bind *:80 + + filter spoe engine ip-reputation config spoe-ip-reputation.conf + + # Reject connection if the IP reputation is under 20 + tcp-request content reject if { var(sess.iprep.ip_score) -m int lt 20 } + + default_backend http-servers + + backend http-servers + mode http + server http A.B.C.D:80 + + backend iprep-servers + mode tcp + balance roundrobin + + timeout connect 5s # greater than hello timeout + timeout server 3m # greater than idle timeout + + server iprep1 A1.B1.C1.D1:12345 + server iprep2 A2.B2.C2.D2:12345 + + #### + ### spoe-ip-reputation.conf + [ip-reputation] + + spoe-agent iprep-agent + messages get-ip-reputation + + option var-prefix iprep + + timeout hello 2s + timeout idle 2m + timeout processing 10ms + + use-backend iprep-servers + + spoe-message get-ip-reputation + args ip=src + event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } + + +3. SPOP specification +---------------------- + +3.1. Data types +---------------- + +Here is the bytewise representation of typed data: + + TYPED-DATA : + +Supported types and their representation are: + + TYPE | ID | DESCRIPTION + -----------------------------+-----+---------------------------------- + NULL | 0 | NULL : <0> + Boolean | 1 | BOOL : <1+FLAG> + 32bits signed integer | 2 | INT32 : <2> + 32bits unsigned integer | 3 | UINT32 : <3> + 64bits signed integer | 4 | INT64 : <4> + 32bits unsigned integer | 5 | UNIT64 : <5> + IPV4 | 6 | IPV4 : <6> + IPV6 | 7 | IPV6 : <7> + String | 8 | STRING : <8> + Binary | 9 | BINARY : <9> + 10 -> 15 unused/reserved | - | - + -----------------------------+-----+---------------------------------- + +Variable-length integer (varint) are encoded using Peers encoding: + + + 0 <= X < 240 : 1 byte (7.875 bits) [ XXXX XXXX ] + 240 <= X < 2288 : 2 bytes (11 bits) [ 1111 XXXX ] [ 0XXX XXXX ] + 2288 <= X < 264432 : 3 bytes (18 bits) [ 1111 XXXX ] [ 1XXX XXXX ] [ 0XXX XXXX ] + 264432 <= X < 33818864 : 4 bytes (25 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*2 [ 0XXX XXXX ] + 33818864 <= X < 4328786160 : 5 bytes (32 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*3 [ 0XXX XXXX ] + ... + +For booleans, the value (true or false) is the first bit in the FLAGS +bitfield. if this bit is set to 0, then the boolean is evaluated as false, +otherwise, the boolean is evaluated as true. + +3.2. Frames +------------ + +Exchange between HAProxy and agents are made using FRAME packets. All frames +must be prefixed with their size encoded on 4 bytes in network byte order: + + + +A frame always starts with its type, on one byte, followed by metadata +containing flags, on 4 bytes and a two variable-length integer representing the +stream identifier and the frame identifier inside the stream: + + FRAME : + METADATA : + +Then comes the frame payload. Depending on the frame type, the payload can be +of three types: a simple key/value list, a list of messages or a list of +actions. + + FRAME-PAYLOAD : | | + + LIST-OF-MESSAGES : [ ... ] + MESSAGE-NAME : + + LIST-OF-ACTIONS : [ ... ] + ACTION-ARGS : [ ... ] + + KV-LIST : [ ... ] + KV-NAME : + KV-VALUE : + + FLAGS : + + Flags are a 32 bits field. They are encoded on 4 bytes in network byte + order, where the bit 0 is the LSB. + + 0 1 2-31 + +---+---+----------+ + | | A | | + | F | B | | + | I | O | RESERVED | + | N | R | | + | | T | | + +---+---+----------+ + + FIN: Indicates that this is the final payload fragment. The first fragment + may also be the final fragment. + + ABORT: Indicates that the processing of the current frame must be + cancelled. This bit should be set on frames with a fragmented + payload. It can be ignore for frames with an unfragemnted + payload. When it is set, the FIN bit must also be set. + + +Frames cannot exceed a maximum size negotiated between HAProxy and agents +during the HELLO handshake. Most of time, payload will be small enough to send +it in one frame. But when supported by the peer, it will be possible to +fragment huge payload on many frames. This ability is announced during the +HELLO handshake and it can be asynmetric (supported by agents but not by +HAProxy or the opposite). The following rules apply to fragmentation: + + * An unfragemnted payload consists of a single frame with the FIN bit set. + + * A fragemented payload consists of several frames with the FIN bit clear and + terminated by a single frame with the FIN bit set. All these frames must + share the same STREAM-ID and FRAME-ID. The first frame must set the right + FRAME-TYPE (e.g, NOTIFY). The following frames must have an unset type (0). + +Beside the support of fragmented payload by a peer, some payload must not be +fragmented. See below for details. + +IMPORTANT : The maximum size supported by peers for a frame must be greater +than or equal to 256 bytes. + +3.2.1. Frame capabilities +-------------------------- + +Here are the list of official capabilities that HAProxy and agents can support: + + * fragmentation: This is the ability for a peer to support fragmented + payload in received frames. This is an asymmectical + capability, it only concerns the peer that announces + it. This is the responsibility to the other peer to use it + or not. + + * pipelining: This is the ability for a peer to decouple NOTIFY and ACK + frames. This is a symmectical capability. To be used, it must + be supported by HAProxy and agents. Unlike HTTP pipelining, the + ACK frames can be send in any order, but always on the same TCP + connection used for the corresponding NOTIFY frame. + + * async: This ability is similar to the pipelining, but here any TCP + connection established between HAProxy and the agent can be used to + send ACK frames. if an agent accepts connections from multiple + HAProxy, it can use the "engine-id" value to group TCP + connections. See details about HAPROXY-HELLO frame. + +Unsupported or unknown capabilities are silently ignored, when possible. + +NOTE: HAProxy does not support the fragmentation for now. This means it is not + able to handle fragmented frames. However, if an agent announces the + fragmentation support, HAProxy may choose to send fragemented frames. + +3.2.2. Frame types overview +---------------------------- + +Here are types of frame supported by SPOE. Frames sent by HAProxy come first, +then frames sent by agents : + + TYPE | ID | DESCRIPTION + -----------------------------+-----+------------------------------------- + UNSET | 0 | Used for all frames but the first when a + | | payload is fragmented. + -----------------------------+-----+------------------------------------- + HAPROXY-HELLO | 1 | Sent by HAProxy when it opens a + | | connection on an agent. + | | + HAPROXY-DISCONNECT | 2 | Sent by HAProxy when it want to close + | | the connection or in reply to an + | | AGENT-DISCONNECT frame + | | + NOTIFY | 3 | Sent by HAProxy to pass information + | | to an agent + -----------------------------+-----+------------------------------------- + AGENT-HELLO | 101 | Reply to a HAPROXY-HELLO frame, when + | | the connection is established + | | + AGENT-DISCONNECT | 102 | Sent by an agent just before closing + | | the connection + | | + ACK | 103 | Sent to acknowledge a NOTIFY frame + -----------------------------+-----+------------------------------------- + +Unknown frames may be silently skipped. + +3.2.3. Workflow +---------------- + + * Successful HELLO handshake: + + HAPROXY AGENT SRV + | HAPROXY-HELLO | + | (healthcheck: false) | + | --------------------------> | + | | + | AGENT-HELLO | + | <-------------------------- | + | | + + * Successful HELLO healthcheck: + + HAPROXY AGENT SRV + | HAPROXY-HELLO | + | (healthcheck: true) | + | --------------------------> | + | | + | AGENT-HELLO + close() | + | <-------------------------- | + | | + + + * Error encountered by agent during the HELLO handshake: + + HAPROXY AGENT SRV + | HAPROXY-HELLO | + | --------------------------> | + | | + | DISCONNECT + close() | + | <-------------------------- | + | | + + * Error encountered by HAProxy during the HELLO handshake: + + HAPROXY AGENT SRV + | HAPROXY-HELLO | + | --------------------------> | + | | + | AGENT-HELLO | + | <-------------------------- | + | | + | DISCONNECT | + | --------------------------> | + | | + | DISCONNECT + close() | + | <-------------------------- | + | | + + * Notify / Ack exchange (unfragmented payload): + + HAPROXY AGENT SRV + | NOTIFY | + | --------------------------> | + | | + | ACK | + | <-------------------------- | + | | + + * Notify / Ack exchange (fragmented payload): + + HAPROXY AGENT SRV + | NOTIFY (frag 1) | + | --------------------------> | + | | + | UNSET (frag 2) | + | --------------------------> | + | ... | + | UNSET (frag N) | + | --------------------------> | + | | + | ACK | + | <-------------------------- | + | | + + * Aborted fragmentation of a NOTIFY frame: + + HAPROXY AGENT SRV + | ... | + | UNSET (frag X) | + | --------------------------> | + | | + | ACK/ABORT | + | <-------------------------- | + | | + | UNSET (frag X+1) | + | -----------X | + | | + | | + + * Connection closed by haproxy: + + HAPROXY AGENT SRV + | DISCONNECT | + | --------------------------> | + | | + | DISCONNECT + close() | + | <-------------------------- | + | | + + * Connection closed by agent: + + HAPROXY AGENT SRV + | DISCONNECT + close() | + | <-------------------------- | + | | + +3.2.4. Frame: HAPROXY-HELLO +---------------------------- + +This frame is the first one exchanged between HAProxy and an agent, when the +connection is established. The payload of this frame is a KV-LIST. It cannot be +fragmented. STREAM-ID and FRAME-ID are must be set 0. + +Following items are mandatory in the KV-LIST: + + * "supported-versions" + + Last SPOP major versions supported by HAProxy. It is a comma-separated list + of versions, following the format "Major.Minor". Spaces must be ignored, if + any. When a major version is announced by HAProxy, it means it also support + all previous minor versions. + + Example: "2.0, 1.5" means HAProxy supports SPOP 2.0 and 1.0 to 1.5 + + * "max-frame-size" + + This is the maximum size allowed for a frame. The HAPROXY-HELLO frame must + be lower or equal to this value. + + * "capabilities" + + This a comma-separated list of capabilities supported by HAProxy. Spaces + must be ignored, if any. + +Following optional items can be added in the KV-LIST: + + * "healthcheck" + + If this item is set to TRUE, then the HAPROXY-HELLO frame is sent during a + SPOE health check. When set to FALSE, this item can be ignored. + + * "engine-id" + + This is a uniq string that identify a SPOE engine. + +To finish the HELLO handshake, the agent must return an AGENT-HELLO frame with +its supported SPOP version, the lower value between its maximum size allowed +for a frame and the HAProxy one and capabilities it supports. If an error +occurs or if an incompatibility is detected with the agent configuration, an +AGENT-DISCONNECT frame must be returned. + +3.2.5. Frame: AGENT-HELLO +-------------------------- + +This frame is sent in reply to a HAPROXY-HELLO frame to finish a HELLO +handshake. As for HAPROXY-HELLO frame, STREAM-ID and FRAME-ID are also set +0. The payload of this frame is a KV-LIST and it cannot be fragmented. + +Following items are mandatory in the KV-LIST: + + * "version" + + This is the SPOP version the agent supports. It must follow the format + "Major.Minor" and it must be lower or equal than one of major versions + announced by HAProxy. + + * "max-frame-size" + + This is the maximum size allowed for a frame. It must be lower or equal to + the value in the HAPROXY-HELLO frame. This value will be used for all + subsequent frames. + + * "capabilities" + + This a comma-separated list of capabilities supported by agent. Spaces must + be ignored, if any. + +At this time, if everything is ok for HAProxy (supported version and valid +max-frame-size value), the HELLO handshake is successfully completed. Else, +HAProxy sends a HAPROXY-DISCONNECT frame with the corresponding error. + +If "healthcheck" item was set to TRUE in the HAPROXY-HELLO frame, the agent can +safely close the connection without DISCONNECT frame. In all cases, HAProxy +will close the connection at the end of the health check. + +3.2.6. Frame: NOTIFY +--------------------- + +Information are sent to the agents inside NOTIFY frames. These frames are +attached to a stream, so STREAM-ID and FRAME-ID must be set. The payload of +NOTIFY frames is a LIST-OF-MESSAGES and, if supported by agents, it can be +fragmented. + +NOTIFY frames must be acknowledge by agents sending an ACK frame, repeating +right STREAM-ID and FRAME-ID. + +3.2.7. Frame: ACK +------------------ + +ACK frames must be sent by agents to reply to NOTIFY frames. STREAM-ID and +FRAME-ID found in a NOTIFY frame must be reuse in the corresponding ACK +frame. The payload of ACK frames is a LIST-OF-ACTIONS and, if supported by +HAProxy, it can be fragmented. + +3.2.8. Frame: HAPROXY-DISCONNECT +--------------------------------- + +If an error occurs, at anytime, from the HAProxy side, a HAPROXY-DISCONNECT +frame is sent with information describing the error. HAProxy will wait an +AGENT-DISCONNECT frame in reply. All other frames will be ignored. The agent +must then close the socket. + +The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and +FRAME-ID are must be set 0. + +Following items are mandatory in the KV-LIST: + + * "status-code" + + This is the code corresponding to the error. + + * "message" + + This is a textual message describing the error. + +For more information about known errors, see section "Errors & timeouts" + +3.2.9. Frame: AGENT-DISCONNECT +------------------------------- + +If an error occurs, at anytime, from the agent size, a AGENT-DISCONNECT frame +is sent, with information describing the error. such frame is also sent in reply +to a HAPROXY-DISCONNECT. The agent must close the socket just after sending +this frame. + +The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and +FRAME-ID are must be set 0. + +Following items are mandatory in the KV-LIST: + + * "status-code" + + This is the code corresponding to the error. + + * "message" + + This is a textual message describing the error. + +For more information about known errors, see section "Errors & timeouts" + +3.3. Events & Messages +----------------------- + +Information about streams are sent in NOTIFY frames. You can specify which kind +of information to send by defining "spoe-message" sections in your SPOE +configuration file. for each "spoe-message" there will be a message in a NOTIFY +frame when the right event is triggered. + +A NOTIFY frame is sent for an specific event when there is at least one +"spoe-message" attached to this event. All messages for an event will be added +in the same NOTIFY frame. + +Here is the list of supported events: + + * on-client-session is triggered when a new client session is created. + This event is only available for SPOE filters + declared in a frontend or a listen section. + + * on-frontend-tcp-request is triggered just before the evaluation of + "tcp-request content" rules on the frontend side. + This event is only available for SPOE filters + declared in a frontend or a listen section. + + * on-backend-tcp-request is triggered just before the evaluation of + "tcp-request content" rules on the backend side. + This event is skipped for SPOE filters declared + in a listen section. + + * on-frontend-http-request is triggered just before the evaluation of + "http-request" rules on the frontend side. This + event is only available for SPOE filters declared + in a frontend or a listen section. + + * on-backend-http-request is triggered just before the evaluation of + "http-request" rules on the backend side. This + event is skipped for SPOE filters declared in a + listen section. + + * on-server-session is triggered when the session with the server is + established. + + * on-tcp-response is triggered just before the evaluation of + "tcp-response content" rules. + + * on-http-response is triggered just before the evaluation of + "http-response" rules. + + +The stream processing will loop on these events, when triggered, waiting the +agent reply. + +3.4. Actions +------------- + +An agent must acknowledge each NOTIFY frame by sending the corresponding ACK +frame. Actions can be added in these frames to dynamically take action on the +processing of a stream. + +Here is the list of supported actions: + + * set-var set the value for an existing variable. 3 arguments must be + attached to this action: the variable scope (proc, sess, txn, + req or res), the variable name (a string) and its value. + + ACTION-SET-VAR : + + SET-VAR : <1> + NB-ARGS : <3> + VAR-SCOPE : | | | | + VAR-NAME : + VAR-VALUE : + + PROCESS : <0> + SESSION : <1> + TRANSACTION : <2> + REQUEST : <3> + RESPONSE : <4> + + * unset-var unset the value for an existing variable. 2 arguments must be + attached to this action: the variable scope (proc, sess, txn, + req or res) and the variable name (a string). + + ACTION-UNSET-VAR : + + UNSET-VAR : <2> + NB-ARGS : <2> + VAR-SCOPE : | | | | + VAR-NAME : + + PROCESS : <0> + SESSION : <1> + TRANSACTION : <2> + REQUEST : <3> + RESPONSE : <4> + + +NOTE: Name of the variables will be automatically prefixed by HAProxy to avoid + name clashes with other variables used in HAProxy. Moreover, unknown + variable will be silently ignored. + +3.5. Errors & timeouts +---------------------- + +Here is the list of all known errors: + + STATUS CODE | DESCRIPTION + ----------------+-------------------------------------------------------- + 0 | normal (no error occurred) + 1 | I/O error + 2 | A timeout occurred + 3 | frame is too big + 4 | invalid frame received + 5 | version value not found + 6 | max-frame-size value not found + 7 | capabilities value not found + 8 | unsupported version + 9 | max-frame-size too big or too small + 10 | payload fragmentation is not supported + 11 | invalid interlaced frames + 12 | frame-id not found (it does not match any referenced frame) + 13 | resource allocation error + 99 | an unknown error occurrde + ----------------+-------------------------------------------------------- + +An agent can define its own errors using a not yet assigned status code. + +IMPORTANT NOTE: By default, for a specific stream, when an abnormal/unexpected + error occurs, the SPOE is disabled for all the transaction. So + if you have several events configured, such error on an event + will disabled all following. For TCP streams, this will + disable the SPOE for the whole session. For HTTP streams, this + will disable it for the transaction (request and response). + See 'option continue-on-error' to bypass this limitation. + +To avoid a stream to wait undefinetly, you must carefully choose the +acknowledgement timeout. In most of cases, it will be quiet low. But it depends +on the responsivness of your service. + +You must also choose idle timeout carefully. Because connection with your +service depends on the backend configuration used by the SPOA, it is important +to use a lower value for idle timeout than the server timeout. Else the +connection will be closed by HAProxy. The same is true for hello timeout. You +should choose a lower value than the connect timeout. + +4. Logging +----------- + +Activity of an SPOE is logged using HAProxy's logger. The messages are logged +in the context of the streams that handle the client and the server +connections. A message is emitted for each event or group handled by an +SPOE. Depending on the status code, the log level will be different. In the +normal case, when no error occurred, the message is logged with the level +LOG_NOTICE. Otherwise, the message is logged with the level LOG_WARNING. + +The messages are logged using the agent's logger, if defined, and use the +following format: + + SPOE: [AGENT] sid=STREAM-ID st=STATUS-CODE reqT/qT/wT/resT/pT \ + / / / + + AGENT is the agent name + TYPE is EVENT of GROUP + NAME is the event or the group name + STREAM-ID is an integer, the unique id of the stream + STATUS_CODE is the processing's status code + reqT/qT/wT/resT/pT are the following time events: + + * reqT : the encoding time. It includes ACLs processing, if any. For + fragmented frames, it is the sum of all fragments. + * qT : the delay before the request gets out the sending queue. For + fragmented frames, it is the sum of all fragments. + * wT : the delay before the response is received. No fragmentation + supported here. + * resT : the delay to process the response. No fragmentation supported + here. + * pT : the delay to process the event or the group. From the stream + point of view, it is the latency added by the SPOE processing. + It is more or less the sum of values above. + + is the numbers of idle SPOE applets + is the numbers of SPOE applets + is the numbers of streams waiting to send data + is the numbers of streams waiting for a ack + is the numbers of processing errors + is the numbers of events/groups processed + + +For all these time events, -1 means the processing was interrupted before the +end. So -1 for the queue time means the request was never dequeued. For +fragmented frames it is harder to know when the interruption happened. + +/* + * Local variables: + * fill-column: 79 + * End: + */ diff --git a/doc/WURFL-device-detection.txt b/doc/WURFL-device-detection.txt new file mode 100644 index 0000000..4786e22 --- /dev/null +++ b/doc/WURFL-device-detection.txt @@ -0,0 +1,71 @@ +Scientiamobile WURFL Device Detection +------------------------------------- + +You can also include WURFL for inbuilt device detection enabling attributes. + +WURFL is a high-performance and low-memory footprint mobile device detection +software component that can quickly and accurately detect over 500 capabilities +of visiting devices. It can differentiate between portable mobile devices, desktop devices, +SmartTVs and any other types of devices on which a web browser can be installed. + +In order to add WURFL device detection support, you would need to download Scientiamobile +InFuze C API and install it on your system. Refer to www.scientiamobile.com to obtain a valid +InFuze license. +Compile haproxy as shown : + + $ make TARGET= USE_WURFL=1 + +Optionally WURFL_DEBUG=1 may be set to increase logs verbosity + +For HAProxy developers who need to verify that their changes didn't accidentally +break the WURFL code, it is possible to build a dummy library provided in the +addons/wurfl/dummy directory and to use it as an alternative for the full library. +This will not provide the full functionalities, it will just allow haproxy to +start with a wurfl configuration, which generally is enough to validate API +changes : + + $ make -C addons/wurfl/dummy + $ make TARGET= USE_WURFL=1 WURFL_INC=$PWD/addons/wurfl/dummy WURFL_LIB=$PWD/addons/wurfl/dummy + +These are the supported WURFL directives (see doc/configuration.txt) : +- wurfl-data-file +- wurfl-information-list [] (list of WURFL capabilities, + virtual capabilities, property names we plan to use in injected headers) +- wurfl-information-list-separator (character that will be + used to separate values in a response header, ',' by default). +- wurfl-cache-size (Sets the WURFL caching strategy) +- wurfl-patch-file [] (Sets the paths to custom WURFL patch files) + +Sample configuration : + + global + wurfl-data-file /usr/share/wurfl/wurfl.zip + + wurfl-information-list wurfl_id model_name + + #wurfl-information-list-separator | + + ## single LRU cache + #wurfl-cache-size 100000 + ## no cache + #wurfl-cache-size 0 + + #wurfl-patch-file + + ... + frontend + bind *:8888 + default_backend servers + +There are two distinct methods available to transmit the WURFL data downstream +to the target application: + +All data listed in wurfl-information-list + + http-request set-header X-WURFL-All %[wurfl-get-all()] + +A subset of data listed in wurfl-information-list + + http-request set-header X-WURFL-Properties %[wurfl-get(wurfl_id,is_tablet)] + +Please find more information about WURFL and the detection methods at https://www.scientiamobile.com diff --git a/doc/acl.fig b/doc/acl.fig new file mode 100644 index 0000000..253a053 --- /dev/null +++ b/doc/acl.fig @@ -0,0 +1,229 @@ +#FIG 3.2 Produced by xfig version 3.2.5-alpha5 +Portrait +Center +Metric +A4 +100.00 +Single +-2 +1200 2 +6 2430 1080 2700 2250 +1 2 0 1 0 11 52 -1 20 0.000 1 0.0000 2587 1687 113 563 2474 1687 2700 1687 +4 1 0 50 -1 16 8 1.5708 4 120 840 2610 1710 tcp-req inspect\001 +-6 +6 5805 1080 6255 2250 +1 2 0 1 0 29 52 -1 20 0.000 1 0.0000 6052 1687 203 563 5849 1687 6255 1687 +4 1 0 50 -1 16 8 1.5708 4 90 300 6030 1710 HTTP\001 +4 1 0 50 -1 16 8 1.5708 4 120 615 6165 1710 processing\001 +-6 +6 1575 3375 1800 4500 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 1575 3375 1800 3375 1800 4500 1575 4500 1575 3375 +4 1 0 50 -1 16 8 1.5708 4 120 735 1710 3960 http-resp out\001 +-6 +6 2025 3375 2250 4500 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 2025 3375 2250 3375 2250 4500 2025 4500 2025 3375 +4 1 0 50 -1 16 8 1.5708 4 120 735 2160 3960 http-resp out\001 +-6 +6 810 3600 1080 4230 +4 1 0 50 -1 16 8 1.5708 4 105 555 900 3915 Response\001 +4 1 0 50 -1 16 8 1.5708 4 105 450 1065 3915 to client\001 +-6 +6 720 1350 1035 2070 +4 1 0 50 -1 16 8 1.5708 4 120 540 855 1710 Requests \001 +4 1 0 50 -1 16 8 1.5708 4 105 645 1020 1710 from clients\001 +-6 +6 7695 1350 8010 1980 +4 1 0 50 -1 16 8 1.5708 4 120 510 7830 1665 Requests\001 +4 1 0 50 -1 16 8 1.5708 4 105 555 7995 1665 to servers\001 +-6 +6 7785 3600 8055 4230 +4 1 0 50 -1 16 8 1.5708 4 105 555 7875 3915 Response\001 +4 1 0 50 -1 16 8 1.5708 4 105 630 8055 3915 from server\001 +-6 +1 2 0 1 0 11 52 -1 20 0.000 1 0.0000 1687 1687 113 563 1574 1687 1800 1687 +1 2 0 1 0 11 52 -1 20 0.000 1 0.0000 7087 3937 113 563 6974 3937 7200 3937 +1 2 0 1 0 29 52 -1 20 0.000 1 0.0000 4072 3937 203 563 3869 3937 4275 3937 +1 2 0 1 0 29 52 -1 20 0.000 1 0.0000 2903 3937 203 563 2700 3937 3106 3937 +2 3 0 1 0 6 54 -1 20 0.000 0 0 -1 0 0 9 + 1485 900 1485 2475 4140 2475 4140 1035 6390 1035 6390 2340 + 6840 2340 6840 900 1485 900 +2 3 0 1 0 2 54 -1 20 0.000 0 0 -1 0 0 9 + 4365 1035 4365 2475 7290 2475 7290 900 6840 900 6840 2340 + 5715 2340 5715 1035 4365 1035 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 4950 1125 5175 1125 5175 2250 4950 2250 4950 1125 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 5400 1125 5625 1125 5625 2250 5400 2250 5400 1125 +2 2 0 1 0 11 52 -1 20 0.000 0 0 -1 0 0 5 + 2025 1125 2250 1125 2250 2250 2025 2250 2025 1125 +2 2 0 1 0 11 52 -1 20 0.000 0 0 -1 0 0 5 + 2925 1125 3150 1125 3150 2250 2925 2250 2925 1125 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 1125 1710 1575 1710 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 1125 1935 1575 1755 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 1125 1485 1575 1665 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 3825 1125 4050 1125 4050 2250 3825 2250 3825 1125 +2 2 0 1 0 6 50 -1 20 0.000 0 0 -1 0 0 5 + 1575 450 2025 450 2025 540 1575 540 1575 450 +2 2 0 1 0 2 50 -1 20 0.000 0 0 -1 0 0 5 + 1575 675 2025 675 2025 765 1575 765 1575 675 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 3150 450 3600 450 3600 540 3150 540 3150 450 +2 2 0 1 0 29 50 -1 20 0.000 0 0 -1 0 0 5 + 3150 675 3600 675 3600 765 3150 765 3150 675 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 6525 1125 6750 1125 6750 2250 6525 2250 6525 1125 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 7200 1665 7650 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 7200 1620 7650 1530 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 7200 1710 7650 1800 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 6975 1125 7200 1125 7200 2250 6975 2250 6975 1125 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 3375 1125 3600 1125 3600 2250 3375 2250 3375 1125 +2 2 0 1 0 11 52 -1 20 0.000 0 0 -1 0 0 5 + 4500 1125 4725 1125 4725 2250 4500 2250 4500 1125 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 1800 1665 2025 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 2250 1665 2475 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 2700 1665 2925 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 3150 1665 3375 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 3600 1665 3825 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 4725 1665 4950 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 5175 1665 5400 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 5625 1665 5850 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 6750 1665 6975 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 6255 1665 6525 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 4050 1665 4500 1665 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 4050 1620 4500 1530 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 4050 1710 4500 1800 +2 2 0 1 0 11 52 -1 20 0.000 0 0 -1 0 0 5 + 6525 3375 6750 3375 6750 4500 6525 4500 6525 3375 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 6075 3375 6300 3375 6300 4500 6075 4500 6075 3375 +2 3 0 1 0 2 54 -1 20 0.000 0 0 -1 0 0 9 + 7290 3150 7290 4725 5985 4725 5985 3285 2385 3285 2385 4590 + 1935 4590 1935 3150 7290 3150 +2 3 0 1 0 6 54 -1 20 0.000 0 0 -1 0 0 9 + 1935 3150 1485 3150 1485 4725 5985 4725 5985 3285 5085 3285 + 5085 4590 1935 4590 1935 3150 +2 2 0 1 0 11 52 -1 20 0.000 0 0 -1 0 0 5 + 5625 3375 5850 3375 5850 4500 5625 4500 5625 3375 +2 2 0 1 0 29 52 -1 20 0.000 0 0 -1 0 0 5 + 5175 3375 5400 3375 5400 4500 5175 4500 5175 3375 +2 1 0 1 0 0 54 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 7650 3915 7200 3915 +2 1 0 1 0 0 54 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 1575 3915 1125 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 6975 3915 6750 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 6525 3915 6300 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 6075 3915 5850 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 5625 3915 5400 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 2025 3915 1800 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 5175 3915 4275 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 3870 3915 3105 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 30.00 60.00 + 2700 3915 2250 3915 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 3 + 1 1 1.00 30.00 60.00 + 3465 2250 3465 2880 2970 3465 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 4 + 1 1 1.00 30.00 60.00 + 5040 2250 5040 2655 3600 2880 3015 3510 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 4 + 1 1 1.00 30.00 60.00 + 6075 2250 6075 2565 3645 2925 3060 3555 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 4 + 1 1 1.00 30.00 60.00 + 6615 2250 6615 2610 3690 2970 3060 3645 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 4 + 1 1 1.00 30.00 60.00 + 7065 2250 7065 2655 3735 3015 3060 3690 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 4 + 1 1 1.00 30.00 60.00 + 5265 3375 5265 2970 3825 3105 3105 3780 +2 1 0 1 0 0 50 -1 -1 0.000 0 0 -1 1 0 4 + 1 1 1.00 30.00 60.00 + 6165 3375 6165 2835 3780 3060 3105 3735 +4 1 0 50 -1 16 8 1.5708 4 120 630 2160 1710 tcp-request\001 +4 1 0 50 -1 16 8 1.5708 4 120 870 3060 1710 tcp-req content\001 +4 1 0 50 -1 16 8 1.5708 4 120 600 5085 1710 http-req in\001 +4 1 0 50 -1 16 8 1.5708 4 105 690 3960 1710 use-backend\001 +4 1 0 50 -1 16 8 1.5708 4 75 570 5535 1710 use-server\001 +4 1 0 50 -1 16 8 1.5708 4 120 360 1710 1710 accept\001 +4 0 0 50 -1 18 6 0.0000 4 90 435 2115 540 frontend\001 +4 0 0 50 -1 18 6 0.0000 4 90 405 2115 765 backend\001 +4 0 0 50 -1 18 6 0.0000 4 105 150 3735 540 tcp\001 +4 0 0 50 -1 18 6 0.0000 4 105 450 3735 765 http only\001 +4 2 0 50 -1 18 6 0.0000 4 90 435 4050 2430 frontend\001 +4 0 0 50 -1 18 6 0.0000 4 90 405 4455 2430 backend\001 +4 1 0 50 -1 16 8 1.5708 4 120 675 6660 1710 http-req out\001 +4 1 0 50 -1 16 8 1.5708 4 120 675 7110 1710 http-req out\001 +4 1 0 50 -1 16 8 1.5708 4 120 600 3510 1710 http-req in\001 +4 1 0 50 -1 16 8 1.5708 4 120 870 4635 1710 tcp-req content\001 +4 1 0 50 -1 16 8 1.5708 4 120 660 6210 3960 http-resp in\001 +4 1 0 50 -1 16 8 1.5708 4 120 930 6660 3960 tcp-resp content\001 +4 1 0 50 -1 16 8 1.5708 4 120 900 7110 3960 tcp-resp inspect\001 +4 1 0 50 -1 16 8 1.5708 4 120 930 5760 3960 tcp-resp content\001 +4 1 0 50 -1 16 8 1.5708 4 120 660 5310 3960 http-resp in\001 +4 0 0 50 -1 18 6 0.0000 4 90 405 6075 4680 backend\001 +4 1 0 50 -1 16 8 1.5708 4 90 300 4050 3960 HTTP\001 +4 1 0 50 -1 16 8 1.5708 4 120 615 4185 3960 processing\001 +4 1 0 50 -1 16 8 1.5708 4 90 300 2835 3915 Error\001 +4 1 0 50 -1 16 8 1.5708 4 120 615 2970 3915 processing\001 +4 2 0 50 -1 18 6 0.0000 4 90 435 5895 4680 frontend\001 diff --git a/doc/architecture.txt b/doc/architecture.txt new file mode 100644 index 0000000..c37632f --- /dev/null +++ b/doc/architecture.txt @@ -0,0 +1,1448 @@ + ------------------- + HAProxy + Architecture Guide + ------------------- + version 1.1.34 + willy tarreau + 2006/01/29 + + +This document provides real world examples with working configurations. +Please note that except stated otherwise, global configuration parameters +such as logging, chrooting, limits and time-outs are not described here. + +=================================================== +1. Simple HTTP load-balancing with cookie insertion +=================================================== + +A web application often saturates the front-end server with high CPU loads, +due to the scripting language involved. It also relies on a back-end database +which is not much loaded. User contexts are stored on the server itself, and +not in the database, so that simply adding another server with simple IP/TCP +load-balancing would not work. + + +-------+ + |clients| clients and/or reverse-proxy + +---+---+ + | + -+-----+--------+---- + | _|_db + +--+--+ (___) + | web | (___) + +-----+ (___) + 192.168.1.1 192.168.1.2 + + +Replacing the web server with a bigger SMP system would cost much more than +adding low-cost pizza boxes. The solution is to buy N cheap boxes and install +the application on them. Install haproxy on the old one which will spread the +load across the new boxes. + + 192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2 + -------+-----------+-----+-----+-----+--------+---- + | | | | | _|_db + +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___) + | LB1 | | A | | B | | C | | D | (___) + +-----+ +---+ +---+ +---+ +---+ (___) + haproxy 4 cheap web servers + + +Config on haproxy (LB1) : +------------------------- + + listen webfarm 192.168.1.1:80 + mode http + balance roundrobin + cookie SERVERID insert indirect + option httpchk HEAD /index.html HTTP/1.0 + server webA 192.168.1.11:80 cookie A check + server webB 192.168.1.12:80 cookie B check + server webC 192.168.1.13:80 cookie C check + server webD 192.168.1.14:80 cookie D check + + +Description : +------------- + - LB1 will receive clients requests. + - if a request does not contain a cookie, it will be forwarded to a valid + server + - in return, a cookie "SERVERID" will be inserted in the response holding the + server name (eg: "A"). + - when the client comes again with the cookie "SERVERID=A", LB1 will know that + it must be forwarded to server A. The cookie will be removed so that the + server does not see it. + - if server "webA" dies, the requests will be sent to another valid server + and a cookie will be reassigned. + + +Flows : +------- + +(client) (haproxy) (server A) + >-- GET /URI1 HTTP/1.0 ------------> | + ( no cookie, haproxy forwards in load-balancing mode. ) + | >-- GET /URI1 HTTP/1.0 ----------> + | <-- HTTP/1.0 200 OK -------------< + ( the proxy now adds the server cookie in return ) + <-- HTTP/1.0 200 OK ---------------< | + Set-Cookie: SERVERID=A | + >-- GET /URI2 HTTP/1.0 ------------> | + Cookie: SERVERID=A | + ( the proxy sees the cookie. it forwards to server A and deletes it ) + | >-- GET /URI2 HTTP/1.0 ----------> + | <-- HTTP/1.0 200 OK -------------< + ( the proxy does not add the cookie in return because the client knows it ) + <-- HTTP/1.0 200 OK ---------------< | + >-- GET /URI3 HTTP/1.0 ------------> | + Cookie: SERVERID=A | + ( ... ) + + +Limits : +-------- + - if clients use keep-alive (HTTP/1.1), only the first response will have + a cookie inserted, and only the first request of each session will be + analyzed. This does not cause trouble in insertion mode because the cookie + is put immediately in the first response, and the session is maintained to + the same server for all subsequent requests in the same session. However, + the cookie will not be removed from the requests forwarded to the servers, + so the server must not be sensitive to unknown cookies. If this causes + trouble, you can disable keep-alive by adding the following option : + + option httpclose + + - if for some reason the clients cannot learn more than one cookie (eg: the + clients are indeed some home-made applications or gateways), and the + application already produces a cookie, you can use the "prefix" mode (see + below). + + - LB1 becomes a very sensible server. If LB1 dies, nothing works anymore. + => you can back it up using keepalived (see below) + + - if the application needs to log the original client's IP, use the + "forwardfor" option which will add an "X-Forwarded-For" header with the + original client's IP address. You must also use "httpclose" to ensure + that you will rewrite every requests and not only the first one of each + session : + + option httpclose + option forwardfor + + - if the application needs to log the original destination IP, use the + "originalto" option which will add an "X-Original-To" header with the + original destination IP address. You must also use "httpclose" to ensure + that you will rewrite every requests and not only the first one of each + session : + + option httpclose + option originalto + + The web server will have to be configured to use this header instead. + For example, on apache, you can use LogFormat for this : + + LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b " combined + CustomLog /var/log/httpd/access_log combined + +Hints : +------- +Sometimes on the internet, you will find a few percent of the clients which +disable cookies on their browser. Obviously they have troubles everywhere on +the web, but you can still help them access your site by using the "source" +balancing algorithm instead of the "roundrobin". It ensures that a given IP +address always reaches the same server as long as the number of servers remains +unchanged. Never use this behind a proxy or in a small network, because the +distribution will be unfair. However, in large internal networks, and on the +internet, it works quite well. Clients which have a dynamic address will not +be affected as long as they accept the cookie, because the cookie always has +precedence over load balancing : + + listen webfarm 192.168.1.1:80 + mode http + balance source + cookie SERVERID insert indirect + option httpchk HEAD /index.html HTTP/1.0 + server webA 192.168.1.11:80 cookie A check + server webB 192.168.1.12:80 cookie B check + server webC 192.168.1.13:80 cookie C check + server webD 192.168.1.14:80 cookie D check + + +================================================================== +2. HTTP load-balancing with cookie prefixing and high availability +================================================================== + +Now you don't want to add more cookies, but rather use existing ones. The +application already generates a "JSESSIONID" cookie which is enough to track +sessions, so we'll prefix this cookie with the server name when we see it. +Since the load-balancer becomes critical, it will be backed up with a second +one in VRRP mode using keepalived under Linux. + +Download the latest version of keepalived from this site and install it +on each load-balancer LB1 and LB2 : + + http://www.keepalived.org/ + +You then have a shared IP between the two load-balancers (we will still use the +original IP). It is active only on one of them at any moment. To allow the +proxy to bind to the shared IP on Linux 2.4, you must enable it in /proc : + +# echo 1 >/proc/sys/net/ipv4/ip_nonlocal_bind + + + shared IP=192.168.1.1 + 192.168.1.3 192.168.1.4 192.168.1.11-192.168.1.14 192.168.1.2 + -------+------------+-----------+-----+-----+-----+--------+---- + | | | | | | _|_db + +--+--+ +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___) + | LB1 | | LB2 | | A | | B | | C | | D | (___) + +-----+ +-----+ +---+ +---+ +---+ +---+ (___) + haproxy haproxy 4 cheap web servers + keepalived keepalived + + +Config on both proxies (LB1 and LB2) : +-------------------------------------- + + listen webfarm 192.168.1.1:80 + mode http + balance roundrobin + cookie JSESSIONID prefix + option httpclose + option forwardfor + option httpchk HEAD /index.html HTTP/1.0 + server webA 192.168.1.11:80 cookie A check + server webB 192.168.1.12:80 cookie B check + server webC 192.168.1.13:80 cookie C check + server webD 192.168.1.14:80 cookie D check + + +Notes: the proxy will modify EVERY cookie sent by the client and the server, +so it is important that it can access to ALL cookies in ALL requests for +each session. This implies that there is no keep-alive (HTTP/1.1), thus the +"httpclose" option. Only if you know for sure that the client(s) will never +use keep-alive (eg: Apache 1.3 in reverse-proxy mode), you can remove this +option. + + +Configuration for keepalived on LB1/LB2 : +----------------------------------------- + + vrrp_script chk_haproxy { # Requires keepalived-1.1.13 + script "killall -0 haproxy" # cheaper than pidof + interval 2 # check every 2 seconds + weight 2 # add 2 points of prio if OK + } + + vrrp_instance VI_1 { + interface eth0 + state MASTER + virtual_router_id 51 + priority 101 # 101 on master, 100 on backup + virtual_ipaddress { + 192.168.1.1 + } + track_script { + chk_haproxy + } + } + + +Description : +------------- + - LB1 is VRRP master (keepalived), LB2 is backup. Both monitor the haproxy + process, and lower their prio if it fails, leading to a failover to the + other node. + - LB1 will receive clients requests on IP 192.168.1.1. + - both load-balancers send their checks from their native IP. + - if a request does not contain a cookie, it will be forwarded to a valid + server + - in return, if a JESSIONID cookie is seen, the server name will be prefixed + into it, followed by a delimiter ('~') + - when the client comes again with the cookie "JSESSIONID=A~xxx", LB1 will + know that it must be forwarded to server A. The server name will then be + extracted from cookie before it is sent to the server. + - if server "webA" dies, the requests will be sent to another valid server + and a cookie will be reassigned. + + +Flows : +------- + +(client) (haproxy) (server A) + >-- GET /URI1 HTTP/1.0 ------------> | + ( no cookie, haproxy forwards in load-balancing mode. ) + | >-- GET /URI1 HTTP/1.0 ----------> + | X-Forwarded-For: 10.1.2.3 + | <-- HTTP/1.0 200 OK -------------< + ( no cookie, nothing changed ) + <-- HTTP/1.0 200 OK ---------------< | + >-- GET /URI2 HTTP/1.0 ------------> | + ( no cookie, haproxy forwards in lb mode, possibly to another server. ) + | >-- GET /URI2 HTTP/1.0 ----------> + | X-Forwarded-For: 10.1.2.3 + | <-- HTTP/1.0 200 OK -------------< + | Set-Cookie: JSESSIONID=123 + ( the cookie is identified, it will be prefixed with the server name ) + <-- HTTP/1.0 200 OK ---------------< | + Set-Cookie: JSESSIONID=A~123 | + >-- GET /URI3 HTTP/1.0 ------------> | + Cookie: JSESSIONID=A~123 | + ( the proxy sees the cookie, removes the server name and forwards + to server A which sees the same cookie as it previously sent ) + | >-- GET /URI3 HTTP/1.0 ----------> + | Cookie: JSESSIONID=123 + | X-Forwarded-For: 10.1.2.3 + | <-- HTTP/1.0 200 OK -------------< + ( no cookie, nothing changed ) + <-- HTTP/1.0 200 OK ---------------< | + ( ... ) + +Hints : +------- +Sometimes, there will be some powerful servers in the farm, and some smaller +ones. In this situation, it may be desirable to tell haproxy to respect the +difference in performance. Let's consider that WebA and WebB are two old +P3-1.2 GHz while WebC and WebD are shiny new Opteron-2.6 GHz. If your +application scales with CPU, you may assume a very rough 2.6/1.2 performance +ratio between the servers. You can inform haproxy about this using the "weight" +keyword, with values between 1 and 256. It will then spread the load the most +smoothly possible respecting those ratios : + + server webA 192.168.1.11:80 cookie A weight 12 check + server webB 192.168.1.12:80 cookie B weight 12 check + server webC 192.168.1.13:80 cookie C weight 26 check + server webD 192.168.1.14:80 cookie D weight 26 check + + +======================================================== +2.1 Variations involving external layer 4 load-balancers +======================================================== + +Instead of using a VRRP-based active/backup solution for the proxies, +they can also be load-balanced by a layer4 load-balancer (eg: Alteon) +which will also check that the services run fine on both proxies : + + | VIP=192.168.1.1 + +----+----+ + | Alteon | + +----+----+ + | + 192.168.1.3 | 192.168.1.4 192.168.1.11-192.168.1.14 192.168.1.2 + -------+-----+------+-----------+-----+-----+-----+--------+---- + | | | | | | _|_db + +--+--+ +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___) + | LB1 | | LB2 | | A | | B | | C | | D | (___) + +-----+ +-----+ +---+ +---+ +---+ +---+ (___) + haproxy haproxy 4 cheap web servers + + +Config on both proxies (LB1 and LB2) : +-------------------------------------- + + listen webfarm 0.0.0.0:80 + mode http + balance roundrobin + cookie JSESSIONID prefix + option httpclose + option forwardfor + option httplog + option dontlognull + option httpchk HEAD /index.html HTTP/1.0 + server webA 192.168.1.11:80 cookie A check + server webB 192.168.1.12:80 cookie B check + server webC 192.168.1.13:80 cookie C check + server webD 192.168.1.14:80 cookie D check + +The "dontlognull" option is used to prevent the proxy from logging the health +checks from the Alteon. If a session exchanges no data, then it will not be +logged. + +Config on the Alteon : +---------------------- + + /c/slb/real 11 + ena + name "LB1" + rip 192.168.1.3 + /c/slb/real 12 + ena + name "LB2" + rip 192.168.1.4 + /c/slb/group 10 + name "LB1-2" + metric roundrobin + health tcp + add 11 + add 12 + /c/slb/virt 10 + ena + vip 192.168.1.1 + /c/slb/virt 10/service http + group 10 + + +Note: the health-check on the Alteon is set to "tcp" to prevent the proxy from +forwarding the connections. It can also be set to "http", but for this the +proxy must specify a "monitor-net" with the Alteons' addresses, so that the +Alteon can really check that the proxies can talk HTTP but without forwarding +the connections to the end servers. Check next section for an example on how to +use monitor-net. + + +============================================================ +2.2 Generic TCP relaying and external layer 4 load-balancers +============================================================ + +Sometimes it's useful to be able to relay generic TCP protocols (SMTP, TSE, +VNC, etc...), for example to interconnect private networks. The problem comes +when you use external load-balancers which need to send periodic health-checks +to the proxies, because these health-checks get forwarded to the end servers. +The solution is to specify a network which will be dedicated to monitoring +systems and must not lead to a forwarding connection nor to any log, using the +"monitor-net" keyword. Note: this feature expects a version of haproxy greater +than or equal to 1.1.32 or 1.2.6. + + + | VIP=172.16.1.1 | + +----+----+ +----+----+ + | Alteon1 | | Alteon2 | + +----+----+ +----+----+ + 192.168.1.252 | GW=192.168.1.254 | 192.168.1.253 + | | + ------+---+------------+--+-----------------> TSE farm : 192.168.1.10 + 192.168.1.1 | | 192.168.1.2 + +--+--+ +--+--+ + | LB1 | | LB2 | + +-----+ +-----+ + haproxy haproxy + + +Config on both proxies (LB1 and LB2) : +-------------------------------------- + + listen tse-proxy + bind :3389,:1494,:5900 # TSE, ICA and VNC at once. + mode tcp + balance roundrobin + server tse-farm 192.168.1.10 + monitor-net 192.168.1.252/31 + +The "monitor-net" option instructs the proxies that any connection coming from +192.168.1.252 or 192.168.1.253 will not be logged nor forwarded and will be +closed immediately. The Alteon load-balancers will then see the proxies alive +without perturbating the service. + +Config on the Alteon : +---------------------- + + /c/l3/if 1 + ena + addr 192.168.1.252 + mask 255.255.255.0 + /c/slb/real 11 + ena + name "LB1" + rip 192.168.1.1 + /c/slb/real 12 + ena + name "LB2" + rip 192.168.1.2 + /c/slb/group 10 + name "LB1-2" + metric roundrobin + health tcp + add 11 + add 12 + /c/slb/virt 10 + ena + vip 172.16.1.1 + /c/slb/virt 10/service 1494 + group 10 + /c/slb/virt 10/service 3389 + group 10 + /c/slb/virt 10/service 5900 + group 10 + + +Special handling of SSL : +------------------------- +Sometimes, you want to send health-checks to remote systems, even in TCP mode, +in order to be able to failover to a backup server in case the first one is +dead. Of course, you can simply enable TCP health-checks, but it sometimes +happens that intermediate firewalls between the proxies and the remote servers +acknowledge the TCP connection themselves, showing an always-up server. Since +this is generally encountered on long-distance communications, which often +involve SSL, an SSL health-check has been implemented to work around this issue. +It sends SSL Hello messages to the remote server, which in turns replies with +SSL Hello messages. Setting it up is very easy : + + listen tcp-syslog-proxy + bind :1514 # listen to TCP syslog traffic on this port (SSL) + mode tcp + balance roundrobin + option ssl-hello-chk + server syslog-prod-site 192.168.1.10 check + server syslog-back-site 192.168.2.10 check backup + + +========================================================= +3. Simple HTTP/HTTPS load-balancing with cookie insertion +========================================================= + +This is the same context as in example 1 above, but the web +server uses HTTPS. + + +-------+ + |clients| clients + +---+---+ + | + -+-----+--------+---- + | _|_db + +--+--+ (___) + | SSL | (___) + | web | (___) + +-----+ + 192.168.1.1 192.168.1.2 + + +Since haproxy does not handle SSL, this part will have to be extracted from the +servers (freeing even more resources) and installed on the load-balancer +itself. Install haproxy and apache+mod_ssl on the old box which will spread the +load between the new boxes. Apache will work in SSL reverse-proxy-cache. If the +application is correctly developed, it might even lower its load. However, +since there now is a cache between the clients and haproxy, some security +measures must be taken to ensure that inserted cookies will not be cached. + + + 192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2 + -------+-----------+-----+-----+-----+--------+---- + | | | | | _|_db + +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___) + | LB1 | | A | | B | | C | | D | (___) + +-----+ +---+ +---+ +---+ +---+ (___) + apache 4 cheap web servers + mod_ssl + haproxy + + +Config on haproxy (LB1) : +------------------------- + + listen 127.0.0.1:8000 + mode http + balance roundrobin + cookie SERVERID insert indirect nocache + option httpchk HEAD /index.html HTTP/1.0 + server webA 192.168.1.11:80 cookie A check + server webB 192.168.1.12:80 cookie B check + server webC 192.168.1.13:80 cookie C check + server webD 192.168.1.14:80 cookie D check + + +Description : +------------- + - apache on LB1 will receive clients requests on port 443 + - it forwards it to haproxy bound to 127.0.0.1:8000 + - if a request does not contain a cookie, it will be forwarded to a valid + server + - in return, a cookie "SERVERID" will be inserted in the response holding the + server name (eg: "A"), and a "Cache-control: private" header will be added + so that the apache does not cache any page containing such cookie. + - when the client comes again with the cookie "SERVERID=A", LB1 will know that + it must be forwarded to server A. The cookie will be removed so that the + server does not see it. + - if server "webA" dies, the requests will be sent to another valid server + and a cookie will be reassigned. + +Notes : +------- + - if the cookie works in "prefix" mode, there is no need to add the "nocache" + option because it is an application cookie which will be modified, and the + application flags will be preserved. + - if apache 1.3 is used as a front-end before haproxy, it always disables + HTTP keep-alive on the back-end, so there is no need for the "httpclose" + option on haproxy. + - configure apache to set the X-Forwarded-For header itself, and do not do + it on haproxy if you need the application to know about the client's IP. + + +Flows : +------- + +(apache) (haproxy) (server A) + >-- GET /URI1 HTTP/1.0 ------------> | + ( no cookie, haproxy forwards in load-balancing mode. ) + | >-- GET /URI1 HTTP/1.0 ----------> + | <-- HTTP/1.0 200 OK -------------< + ( the proxy now adds the server cookie in return ) + <-- HTTP/1.0 200 OK ---------------< | + Set-Cookie: SERVERID=A | + Cache-Control: private | + >-- GET /URI2 HTTP/1.0 ------------> | + Cookie: SERVERID=A | + ( the proxy sees the cookie. it forwards to server A and deletes it ) + | >-- GET /URI2 HTTP/1.0 ----------> + | <-- HTTP/1.0 200 OK -------------< + ( the proxy does not add the cookie in return because the client knows it ) + <-- HTTP/1.0 200 OK ---------------< | + >-- GET /URI3 HTTP/1.0 ------------> | + Cookie: SERVERID=A | + ( ... ) + + + +======================================== +3.1. Alternate solution using Stunnel +======================================== + +When only SSL is required and cache is not needed, stunnel is a cheaper +solution than Apache+mod_ssl. By default, stunnel does not process HTTP and +does not add any X-Forwarded-For header, but there is a patch on the official +haproxy site to provide this feature to recent stunnel versions. + +This time, stunnel will only process HTTPS and not HTTP. This means that +haproxy will get all HTTP traffic, so haproxy will have to add the +X-Forwarded-For header for HTTP traffic, but not for HTTPS traffic since +stunnel will already have done it. We will use the "except" keyword to tell +haproxy that connections from local host already have a valid header. + + + 192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2 + -------+-----------+-----+-----+-----+--------+---- + | | | | | _|_db + +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___) + | LB1 | | A | | B | | C | | D | (___) + +-----+ +---+ +---+ +---+ +---+ (___) + stunnel 4 cheap web servers + haproxy + + +Config on stunnel (LB1) : +------------------------- + + cert=/etc/stunnel/stunnel.pem + setuid=stunnel + setgid=proxy + + socket=l:TCP_NODELAY=1 + socket=r:TCP_NODELAY=1 + + [https] + accept=192.168.1.1:443 + connect=192.168.1.1:80 + xforwardedfor=yes + + +Config on haproxy (LB1) : +------------------------- + + listen 192.168.1.1:80 + mode http + balance roundrobin + option forwardfor except 192.168.1.1 + cookie SERVERID insert indirect nocache + option httpchk HEAD /index.html HTTP/1.0 + server webA 192.168.1.11:80 cookie A check + server webB 192.168.1.12:80 cookie B check + server webC 192.168.1.13:80 cookie C check + server webD 192.168.1.14:80 cookie D check + +Description : +------------- + - stunnel on LB1 will receive clients requests on port 443 + - it forwards them to haproxy bound to port 80 + - haproxy will receive HTTP client requests on port 80 and decrypted SSL + requests from Stunnel on the same port. + - stunnel will add the X-Forwarded-For header + - haproxy will add the X-Forwarded-For header for everyone except the local + address (stunnel). + + +======================================== +4. Soft-stop for application maintenance +======================================== + +When an application is spread across several servers, the time to update all +instances increases, so the application seems jerky for a longer period. + +HAProxy offers several solutions for this. Although it cannot be reconfigured +without being stopped, nor does it offer any external command, there are other +working solutions. + + +========================================= +4.1 Soft-stop using a file on the servers +========================================= + +This trick is quite common and very simple: put a file on the server which will +be checked by the proxy. When you want to stop the server, first remove this +file. The proxy will see the server as failed, and will not send it any new +session, only the old ones if the "persist" option is used. Wait a bit then +stop the server when it does not receive anymore connections. + + + listen 192.168.1.1:80 + mode http + balance roundrobin + cookie SERVERID insert indirect + option httpchk HEAD /running HTTP/1.0 + server webA 192.168.1.11:80 cookie A check inter 2000 rise 2 fall 2 + server webB 192.168.1.12:80 cookie B check inter 2000 rise 2 fall 2 + server webC 192.168.1.13:80 cookie C check inter 2000 rise 2 fall 2 + server webD 192.168.1.14:80 cookie D check inter 2000 rise 2 fall 2 + option persist + redispatch + contimeout 5000 + + +Description : +------------- + - every 2 seconds, haproxy will try to access the file "/running" on the + servers, and declare the server as down after 2 attempts (4 seconds). + - only the servers which respond with a 200 or 3XX response will be used. + - if a request does not contain a cookie, it will be forwarded to a valid + server + - if a request contains a cookie for a failed server, haproxy will insist + on trying to reach the server anyway, to let the user finish what they were + doing. ("persist" option) + - if the server is totally stopped, the connection will fail and the proxy + will rebalance the client to another server ("redispatch") + +Usage on the web servers : +-------------------------- +- to start the server : + # /etc/init.d/httpd start + # touch /home/httpd/www/running + +- to soft-stop the server + # rm -f /home/httpd/www/running + +- to completely stop the server : + # /etc/init.d/httpd stop + +Limits +------ +If the server is totally powered down, the proxy will still try to reach it +for those clients who still have a cookie referencing it, and the connection +attempt will expire after 5 seconds ("contimeout"), and only after that, the +client will be redispatched to another server. So this mode is only useful +for software updates where the server will suddenly refuse the connection +because the process is stopped. The problem is the same if the server suddenly +crashes. All of its users will be fairly perturbated. + + +================================== +4.2 Soft-stop using backup servers +================================== + +A better solution which covers every situation is to use backup servers. +Version 1.1.30 fixed a bug which prevented a backup server from sharing +the same cookie as a standard server. + + + listen 192.168.1.1:80 + mode http + balance roundrobin + redispatch + cookie SERVERID insert indirect + option httpchk HEAD / HTTP/1.0 + server webA 192.168.1.11:80 cookie A check port 81 inter 2000 + server webB 192.168.1.12:80 cookie B check port 81 inter 2000 + server webC 192.168.1.13:80 cookie C check port 81 inter 2000 + server webD 192.168.1.14:80 cookie D check port 81 inter 2000 + + server bkpA 192.168.1.11:80 cookie A check port 80 inter 2000 backup + server bkpB 192.168.1.12:80 cookie B check port 80 inter 2000 backup + server bkpC 192.168.1.13:80 cookie C check port 80 inter 2000 backup + server bkpD 192.168.1.14:80 cookie D check port 80 inter 2000 backup + +Description +----------- +Four servers webA..D are checked on their port 81 every 2 seconds. The same +servers named bkpA..D are checked on the port 80, and share the exact same +cookies. Those servers will only be used when no other server is available +for the same cookie. + +When the web servers are started, only the backup servers are seen as +available. On the web servers, you need to redirect port 81 to local +port 80, either with a local proxy (eg: a simple haproxy tcp instance), +or with iptables (linux) or pf (openbsd). This is because we want the +real web server to reply on this port, and not a fake one. Eg, with +iptables : + + # /etc/init.d/httpd start + # iptables -t nat -A PREROUTING -p tcp --dport 81 -j REDIRECT --to-port 80 + +A few seconds later, the standard server is seen up and haproxy starts to send +it new requests on its real port 80 (only new users with no cookie, of course). + +If a server completely crashes (even if it does not respond at the IP level), +both the standard and backup servers will fail, so clients associated to this +server will be redispatched to other live servers and will lose their sessions. + +Now if you want to enter a server into maintenance, simply stop it from +responding on port 81 so that its standard instance will be seen as failed, +but the backup will still work. Users will not notice anything since the +service is still operational : + + # iptables -t nat -D PREROUTING -p tcp --dport 81 -j REDIRECT --to-port 80 + +The health checks on port 81 for this server will quickly fail, and the +standard server will be seen as failed. No new session will be sent to this +server, and existing clients with a valid cookie will still reach it because +the backup server will still be up. + +Now wait as long as you want for the old users to stop using the service, and +once you see that the server does not receive any traffic, simply stop it : + + # /etc/init.d/httpd stop + +The associated backup server will in turn fail, and if any client still tries +to access this particular server, they will be redispatched to any other valid +server because of the "redispatch" option. + +This method has an advantage : you never touch the proxy when doing server +maintenance. The people managing the servers can make them disappear smoothly. + + +4.2.1 Variations for operating systems without any firewall software +-------------------------------------------------------------------- + +The downside is that you need a redirection solution on the server just for +the health-checks. If the server OS does not support any firewall software, +this redirection can also be handled by a simple haproxy in tcp mode : + + global + daemon + quiet + pidfile /var/run/haproxy-checks.pid + listen 0.0.0.0:81 + mode tcp + dispatch 127.0.0.1:80 + contimeout 1000 + clitimeout 10000 + srvtimeout 10000 + +To start the web service : + + # /etc/init.d/httpd start + # haproxy -f /etc/haproxy/haproxy-checks.cfg + +To soft-stop the service : + + # kill $( +------------------------------------------------------------ + +A number of contributors are often embarrassed with coding style issues, they +don't always know if they're doing it right, especially since the coding style +has elvoved along the years. What is explained here is not necessarily what is +applied in the code, but new code should as much as possible conform to this +style. Coding style fixes happen when code is replaced. It is useless to send +patches to fix coding style only, they will be rejected, unless they belong to +a patch series which needs these fixes prior to get code changes. Also, please +avoid fixing coding style in the same patches as functional changes, they make +code review harder. + +A good way to quickly validate your patch before submitting it is to pass it +through the Linux kernel's checkpatch.pl utility which can be downloaded here : + + http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl + +Running it with the following options relaxes its checks to accommodate to the +extra degree of freedom that is tolerated in HAProxy's coding style compared to +the stricter style used in the kernel : + + checkpatch.pl -q --max-line-length=160 --no-tree --no-signoff \ + --ignore=LEADING_SPACE,CODE_INDENT,DEEP_INDENTATION \ + --ignore=ELSE_AFTER_BRACE < patch + +You can take its output as hints instead of strict rules, but in general its +output will be accurate and it may even spot some real bugs. + +When modifying a file, you must accept the terms of the license of this file +which is recalled at the top of the file, or is explained in the LICENSE file, +or if not stated, defaults to LGPL version 2.1 or later for files in the +'include' directory, and GPL version 2 or later for all other files. + +When adding a new file, you must add a copyright banner at the top of the +file with your real name, e-mail address and a reminder of the license. +Contributions under incompatible licenses or too restrictive licenses might +get rejected. If in doubt, please apply the principle above for existing files. + +All code examples below will intentionally be prefixed with " | " to mark +where the code aligns with the first column, and tabs in this document will be +represented as a series of 8 spaces so that it displays the same everywhere. + + +1) Indentation and alignment +---------------------------- + +1.1) Indentation +---------------- + +Indentation and alignment are two completely different things that people often +get wrong. Indentation is used to mark a sub-level in the code. A sub-level +means that a block is executed in the context of another block (eg: a function +or a condition) : + + | main(int argc, char **argv) + | { + | int i; + | + | if (argc < 2) + | exit(1); + | } + +In the example above, the code belongs to the main() function and the exit() +call belongs to the if statement. Indentation is made with tabs (\t, ASCII 9), +which allows any developer to configure their preferred editor to use their +own tab size and to still get the text properly indented. Exactly one tab is +used per sub-level. Tabs may only appear at the beginning of a line or after +another tab. It is illegal to put a tab after some text, as it mangles displays +in a different manner for different users (particularly when used to align +comments or values after a #define). If you're tempted to put a tab after some +text, then you're doing it wrong and you need alignment instead (see below). + +Note that there are places where the code was not properly indented in the +past. In order to view it correctly, you may have to set your tab size to 8 +characters. + + +1.2) Alignment +-------------- + +Alignment is used to continue a line in a way to makes things easier to group +together. By definition, alignment is character-based, so it uses spaces. Tabs +would not work because for one tab there would not be as many characters on all +displays. For instance, the arguments in a function declaration may be broken +into multiple lines using alignment spaces : + + | int http_header_match2(const char *hdr, const char *end, + | const char *name, int len) + | { + | ... + | } + +In this example, the "const char *name" part is aligned with the first +character of the group it belongs to (list of function arguments). Placing it +here makes it obvious that it's one of the function's arguments. Multiple lines +are easy to handle this way. This is very common with long conditions too : + + | if ((len < eol - sol) && + | (sol[len] == ':') && + | (strncasecmp(sol, name, len) == 0)) { + | ctx->del = len; + | } + +If we take again the example above marking tabs with "[-Tabs-]" and spaces +with "#", we get this : + + | [-Tabs-]if ((len < eol - sol) && + | [-Tabs-]####(sol[len] == ':') && + | [-Tabs-]####(strncasecmp(sol, name, len) == 0)) { + | [-Tabs-][-Tabs-]ctx->del = len; + | [-Tabs-]} + +It is worth noting that some editors tend to confuse indentations and alignment. +Emacs is notoriously known for this brokenness, and is responsible for almost +all of the alignment mess. The reason is that Emacs only counts spaces, tries +to fill as many as possible with tabs and completes with spaces. Once you know +it, you just have to be careful, as alignment is not used much, so generally it +is just a matter of replacing the last tab with 8 spaces when this happens. + +Indentation should be used everywhere there is a block or an opening brace. It +is not possible to have two consecutive closing braces on the same column, it +means that the innermost was not indented. + +Right : + + | main(int argc, char **argv) + | { + | if (argc > 1) { + | printf("Hello\n"); + | } + | exit(0); + | } + +Wrong : + + | main(int argc, char **argv) + | { + | if (argc > 1) { + | printf("Hello\n"); + | } + | exit(0); + | } + +A special case applies to switch/case statements. Due to my editor's settings, +I've been used to align "case" with "switch" and to find it somewhat logical +since each of the "case" statements opens a sublevel belonging to the "switch" +statement. But indenting "case" after "switch" is accepted too. However in any +case, whatever follows the "case" statement must be indented, whether or not it +contains braces : + + | switch (*arg) { + | case 'A': { + | int i; + | for (i = 0; i < 10; i++) + | printf("Please stop pressing 'A'!\n"); + | break; + | } + | case 'B': + | printf("You pressed 'B'\n"); + | break; + | case 'C': + | case 'D': + | printf("You pressed 'C' or 'D'\n"); + | break; + | default: + | printf("I don't know what you pressed\n"); + | } + + +2) Braces +--------- + +Braces are used to delimit multiple-instruction blocks. In general it is +preferred to avoid braces around single-instruction blocks as it reduces the +number of lines : + +Right : + + | if (argc >= 2) + | exit(0); + +Wrong : + + | if (argc >= 2) { + | exit(0); + | } + +But it is not that strict, it really depends on the context. It happens from +time to time that single-instruction blocks are enclosed within braces because +it makes the code more symmetrical, or more readable. Example : + + | if (argc < 2) { + | printf("Missing argument\n"); + | exit(1); + | } else { + | exit(0); + | } + +Braces are always needed to declare a function. A function's opening brace must +be placed at the beginning of the next line : + +Right : + + | int main(int argc, char **argv) + | { + | exit(0); + | } + +Wrong : + + | int main(int argc, char **argv) { + | exit(0); + | } + +Note that a large portion of the code still does not conforms to this rule, as +it took years to get all authors to adapt to this more common standard which +is now preferred, as it avoids visual confusion when function declarations are +broken on multiple lines : + +Right : + + | int foo(const char *hdr, const char *end, + | const char *name, const char *err, + | int len) + | { + | int i; + +Wrong : + + | int foo(const char *hdr, const char *end, + | const char *name, const char *err, + | int len) { + | int i; + +Braces should always be used where there might be an ambiguity with the code +later. The most common example is the stacked "if" statement where an "else" +may be added later at the wrong place breaking the code, but it also happens +with comments or long arguments in function calls. In general, if a block is +more than one line long, it should use braces. + +Dangerous code waiting of a victim : + + | if (argc < 2) + | /* ret must not be negative here */ + | if (ret < 0) + | return -1; + +Wrong change : + + | if (argc < 2) + | /* ret must not be negative here */ + | if (ret < 0) + | return -1; + | else + | return 0; + +It will do this instead of what your eye seems to tell you : + + | if (argc < 2) + | /* ret must not be negative here */ + | if (ret < 0) + | return -1; + | else + | return 0; + +Right : + + | if (argc < 2) { + | /* ret must not be negative here */ + | if (ret < 0) + | return -1; + | } + | else + | return 0; + +Similarly dangerous example : + + | if (ret < 0) + | /* ret must not be negative here */ + | complain(); + | init(); + +Wrong change to silent the annoying message : + + | if (ret < 0) + | /* ret must not be negative here */ + | //complain(); + | init(); + +... which in fact means : + + | if (ret < 0) + | init(); + + +3) Breaking lines +----------------- + +There is no strict rule for line breaking. Some files try to stick to the 80 +column limit, but given that various people use various tab sizes, it does not +make much sense. Also, code is sometimes easier to read with less lines, as it +represents less surface on the screen (since each new line adds its tabs and +spaces). The rule is to stick to the average line length of other lines. If you +are working in a file which fits in 80 columns, try to keep this goal in mind. +If you're in a function with 120-chars lines, there is no reason to add many +short lines, so you can make longer lines. + +In general, opening a new block should lead to a new line. Similarly, multiple +instructions should be avoided on the same line. But some constructs make it +more readable when those are perfectly aligned : + +A copy-paste bug in the following construct will be easier to spot : + + | if (omult % idiv == 0) { omult /= idiv; idiv = 1; } + | if (idiv % omult == 0) { idiv /= omult; omult = 1; } + | if (imult % odiv == 0) { imult /= odiv; odiv = 1; } + | if (odiv % imult == 0) { odiv /= imult; imult = 1; } + +than in this one : + + | if (omult % idiv == 0) { + | omult /= idiv; + | idiv = 1; + | } + | if (idiv % omult == 0) { + | idiv /= omult; + | omult = 1; + | } + | if (imult % odiv == 0) { + | imult /= odiv; + | odiv = 1; + | } + | if (odiv % imult == 0) { + | odiv /= imult; + | imult = 1; + | } + +What is important is not to mix styles. For instance there is nothing wrong +with having many one-line "case" statements as long as most of them are this +short like below : + + | switch (*arg) { + | case 'A': ret = 1; break; + | case 'B': ret = 2; break; + | case 'C': ret = 4; break; + | case 'D': ret = 8; break; + | default : ret = 0; break; + | } + +Otherwise, prefer to have the "case" statement on its own line as in the +example in section 1.2 about alignment. In any case, avoid to stack multiple +control statements on the same line, so that it will never be the needed to +add two tab levels at once : + +Right : + + | switch (*arg) { + | case 'A': + | if (ret < 0) + | ret = 1; + | break; + | default : ret = 0; break; + | } + +Wrong : + + | switch (*arg) { + | case 'A': if (ret < 0) + | ret = 1; + | break; + | default : ret = 0; break; + | } + +Right : + + | if (argc < 2) + | if (ret < 0) + | return -1; + +or Right : + + | if (argc < 2) + | if (ret < 0) return -1; + +but Wrong : + + | if (argc < 2) if (ret < 0) return -1; + + +When complex conditions or expressions are broken into multiple lines, please +do ensure that alignment is perfectly appropriate, and group all main operators +on the same side (which you're free to choose as long as it does not change for +every block. Putting binary operators on the right side is preferred as it does +not mangle with alignment but various people have their preferences. + +Right : + + | if ((txn->flags & TX_NOT_FIRST) && + | ((req->flags & BF_FULL) || + | req->r < req->lr || + | req->r > req->data + req->size - global.tune.maxrewrite)) { + | return 0; + | } + +Right : + + | if ((txn->flags & TX_NOT_FIRST) + | && ((req->flags & BF_FULL) + | || req->r < req->lr + | || req->r > req->data + req->size - global.tune.maxrewrite)) { + | return 0; + | } + +Wrong : + + | if ((txn->flags & TX_NOT_FIRST) && + | ((req->flags & BF_FULL) || + | req->r < req->lr + | || req->r > req->data + req->size - global.tune.maxrewrite)) { + | return 0; + | } + +If it makes the result more readable, parenthesis may even be closed on their +own line in order to align with the opening one. Note that should normally not +be needed because such code would be too complex to be digged into. + +The "else" statement may either be merged with the closing "if" brace or lie on +its own line. The later is preferred but it adds one extra line to each control +block which is annoying in short ones. However, if the "else" is followed by an +"if", then it should really be on its own line and the rest of the if/else +blocks must follow the same style. + +Right : + + | if (a < b) { + | return a; + | } + | else { + | return b; + | } + +Right : + + | if (a < b) { + | return a; + | } else { + | return b; + | } + +Right : + + | if (a < b) { + | return a; + | } + | else if (a != b) { + | return b; + | } + | else { + | return 0; + | } + +Wrong : + + | if (a < b) { + | return a; + | } else if (a != b) { + | return b; + | } else { + | return 0; + | } + +Wrong : + + | if (a < b) { + | return a; + | } + | else if (a != b) { + | return b; + | } else { + | return 0; + | } + + +4) Spacing +---------- + +Correctly spacing code is very important. When you have to spot a bug at 3am, +you need it to be clear. When you expect other people to review your code, you +want it to be clear and don't want them to get nervous when trying to find what +you did. + +Always place spaces around all binary or ternary operators, commas, as well as +after semi-colons and opening braces if the line continues : + +Right : + + | int ret = 0; + | /* if (x >> 4) { x >>= 4; ret += 4; } */ + | ret += (x >> 4) ? (x >>= 4, 4) : 0; + | val = ret + ((0xFFFFAA50U >> (x << 1)) & 3) + 1; + +Wrong : + + | int ret=0; + | /* if (x>>4) {x>>=4;ret+=4;} */ + | ret+=(x>>4)?(x>>=4,4):0; + | val=ret+((0xFFFFAA50U>>(x<<1))&3)+1; + +Never place spaces after unary operators (&, *, -, !, ~, ++, --) nor cast, as +they might be confused with they binary counterpart, nor before commas or +semicolons : + +Right : + + | bit = !!(~len++ ^ -(unsigned char)*x); + +Wrong : + + | bit = ! ! (~len++ ^ - (unsigned char) * x) ; + +Note that "sizeof" is a unary operator which is sometimes considered as a +language keyword, but in no case it is a function. It does not require +parenthesis so it is sometimes followed by spaces and sometimes not when +there are no parenthesis. Most people do not really care as long as what +is written is unambiguous. + +Braces opening a block must be preceded by one space unless the brace is +placed on the first column : + +Right : + + | if (argc < 2) { + | } + +Wrong : + + | if (argc < 2){ + | } + +Do not add unneeded spaces inside parenthesis, they just make the code less +readable. + +Right : + + | if (x < 4 && (!y || !z)) + | break; + +Wrong : + + | if ( x < 4 && ( !y || !z ) ) + | break; + +Language keywords must all be followed by a space. This is true for control +statements (do, for, while, if, else, return, switch, case), and for types +(int, char, unsigned). As an exception, the last type in a cast does not take +a space before the closing parenthesis). The "default" statement in a "switch" +construct is generally just followed by the colon. However the colon after a +"case" or "default" statement must be followed by a space. + +Right : + + | if (nbargs < 2) { + | printf("Missing arg at %c\n", *(char *)ptr); + | for (i = 0; i < 10; i++) beep(); + | return 0; + | } + | switch (*arg) { + +Wrong : + + | if(nbargs < 2){ + | printf("Missing arg at %c\n", *(char*)ptr); + | for(i = 0; i < 10; i++)beep(); + | return 0; + | } + | switch(*arg) { + +Function calls are different, the opening parenthesis is always coupled to the +function name without any space. But spaces are still needed after commas : + +Right : + + | if (!init(argc, argv)) + | exit(1); + +Wrong : + + | if (!init (argc,argv)) + | exit(1); + + +5) Excess or lack of parenthesis +-------------------------------- + +Sometimes there are too many parenthesis in some formulas, sometimes there are +too few. There are a few rules of thumb for this. The first one is to respect +the compiler's advice. If it emits a warning and asks for more parenthesis to +avoid confusion, follow the advice at least to shut the warning. For instance, +the code below is quite ambiguous due to its alignment : + + | if (var1 < 2 || var2 < 2 && + | var3 != var4) { + | /* fail */ + | return -3; + | } + +Note that this code does : + + | if (var1 < 2 || (var2 < 2 && var3 != var4)) { + | /* fail */ + | return -3; + | } + +But maybe the author meant : + + | if ((var1 < 2 || var2 < 2) && var3 != var4) { + | /* fail */ + | return -3; + | } + +A second rule to put parenthesis is that people don't always know operators +precedence too well. Most often they have no issue with operators of the same +category (eg: booleans, integers, bit manipulation, assignment) but once these +operators are mixed, it causes them all sort of issues. In this case, it is +wise to use parenthesis to avoid errors. One common error concerns the bit +shift operators because they're used to replace multiplies and divides but +don't have the same precedence : + +The expression : + + | x = y * 16 + 5; + +becomes : + + | x = y << 4 + 5; + +which is wrong because it is equivalent to : + + | x = y << (4 + 5); + +while the following was desired instead : + + | x = (y << 4) + 5; + +It is generally fine to write boolean expressions based on comparisons without +any parenthesis. But on top of that, integer expressions and assignments should +then be protected. For instance, there is an error in the expression below +which should be safely rewritten : + +Wrong : + + | if (var1 > 2 && var1 < 10 || + | var1 > 2 + 256 && var2 < 10 + 256 || + | var1 > 2 + 1 << 16 && var2 < 10 + 2 << 16) + | return 1; + +Right (may remove a few parenthesis depending on taste) : + + | if ((var1 > 2 && var1 < 10) || + | (var1 > (2 + 256) && var2 < (10 + 256)) || + | (var1 > (2 + (1 << 16)) && var2 < (10 + (1 << 16)))) + | return 1; + +The "return" statement is not a function, so it takes no argument. It is a +control statement which is followed by the expression to be returned. It does +not need to be followed by parenthesis : + +Wrong : + + | int ret0() + | { + | return(0); + | } + +Right : + + | int ret0() + | { + | return 0; + | } + +Parenthesisis are also found in type casts. Type casting should be avoided as +much as possible, especially when it concerns pointer types. Casting a pointer +disables the compiler's type checking and is the best way to get caught doing +wrong things with data not the size you expect. If you need to manipulate +multiple data types, you can use a union instead. If the union is really not +convenient and casts are easier, then try to isolate them as much as possible, +for instance when initializing function arguments or in another function. Not +proceeding this way causes huge risks of not using the proper pointer without +any notification, which is especially true during copy-pastes. + +Wrong : + + | void *check_private_data(void *arg1, void *arg2) + | { + | char *area; + | + | if (*(int *)arg1 > 1000) + | return NULL; + | if (memcmp(*(const char *)arg2, "send(", 5) != 0)) + | return NULL; + | area = malloc(*(int *)arg1); + | if (!area) + | return NULL; + | memcpy(area, *(const char *)arg2 + 5, *(int *)arg1); + | return area; + | } + +Right : + + | void *check_private_data(void *arg1, void *arg2) + | { + | char *area; + | int len = *(int *)arg1; + | const char *msg = arg2; + | + | if (len > 1000) + | return NULL; + | if (memcmp(msg, "send(", 5) != 0) + | return NULL; + | area = malloc(len); + | if (!area) + | return NULL; + | memcpy(area, msg + 5, len); + | return area; + | } + + +6) Ambiguous comparisons with zero or NULL +------------------------------------------ + +In C, '0' has no type, or it has the type of the variable it is assigned to. +Comparing a variable or a return value with zero means comparing with the +representation of zero for this variable's type. For a boolean, zero is false. +For a pointer, zero is NULL. Very often, to make things shorter, it is fine to +use the '!' unary operator to compare with zero, as it is shorter and easier to +remind or understand than a plain '0'. Since the '!' operator is read "not", it +helps read code faster when what follows it makes sense as a boolean, and it is +often much more appropriate than a comparison with zero which makes an equal +sign appear at an undesirable place. For instance : + + | if (!isdigit(*c) && !isspace(*c)) + | break; + +is easier to understand than : + + | if (isdigit(*c) == 0 && isspace(*c) == 0) + | break; + +For a char this "not" operator can be reminded as "no remaining char", and the +absence of comparison to zero implies existence of the tested entity, hence the +simple strcpy() implementation below which automatically stops once the last +zero is copied : + + | void my_strcpy(char *d, const char *s) + | { + | while ((*d++ = *s++)); + | } + +Note the double parenthesis in order to avoid the compiler telling us it looks +like an equality test. + +For a string or more generally any pointer, this test may be understood as an +existence test or a validity test, as the only pointer which will fail to +validate equality is the NULL pointer : + + | area = malloc(1000); + | if (!area) + | return -1; + +However sometimes it can fool the reader. For instance, strcmp() precisely is +one of such functions whose return value can make one think the opposite due to +its name which may be understood as "if strings compare...". Thus it is strongly +recommended to perform an explicit comparison with zero in such a case, and it +makes sense considering that the comparison's operator is the same that is +wanted to compare the strings (note that current config parser lacks a lot in +this regards) : + + strcmp(a, b) == 0 <=> a == b + strcmp(a, b) != 0 <=> a != b + strcmp(a, b) < 0 <=> a < b + strcmp(a, b) > 0 <=> a > b + +Avoid this : + + | if (strcmp(arg, "test")) + | printf("this is not a test\n"); + | + | if (!strcmp(arg, "test")) + | printf("this is a test\n"); + +Prefer this : + + | if (strcmp(arg, "test") != 0) + | printf("this is not a test\n"); + | + | if (strcmp(arg, "test") == 0) + | printf("this is a test\n"); + + +7) System call returns +---------------------- + +This is not directly a matter of coding style but more of bad habits. It is +important to check for the correct value upon return of syscalls. The proper +return code indicating an error is described in its man page. There is no +reason to consider wider ranges than what is indicated. For instance, it is +common to see such a thing : + + | if ((fd = open(file, O_RDONLY)) < 0) + | return -1; + +This is wrong. The man page says that -1 is returned if an error occurred. It +does not suggest that any other negative value will be an error. It is possible +that a few such issues have been left in existing code. They are bugs for which +fixes are accepted, even though they're currently harmless since open() is not +known for returning negative values at the moment. + + +8) Declaring new types, names and values +---------------------------------------- + +Please refrain from using "typedef" to declare new types, they only obfuscate +the code. The reader never knows whether he's manipulating a scalar type or a +struct. For instance it is not obvious why the following code fails to build : + + | int delay_expired(timer_t exp, timer_us_t now) + | { + | return now >= exp; + | } + +With the types declared in another file this way : + + | typedef unsigned int timer_t; + | typedef struct timeval timer_us_t; + +This cannot work because we're comparing a scalar with a struct, which does +not make sense. Without a typedef, the function would have been written this +way without any ambiguity and would not have failed : + + | int delay_expired(unsigned int exp, struct timeval *now) + | { + | return now >= exp->tv_sec; + | } + +Declaring special values may be done using enums. Enums are a way to define +structured integer values which are related to each other. They are perfectly +suited for state machines. While the first element is always assigned the zero +value, not everybody knows that, especially people working with multiple +languages all the day. For this reason it is recommended to explicitly force +the first value even if it's zero. The last element should be followed by a +comma if it is planned that new elements might later be added, this will make +later patches shorter. Conversely, if the last element is placed in order to +get the number of possible values, it must not be followed by a comma and must +be preceded by a comment : + + | enum { + | first = 0, + | second, + | third, + | fourth, + | }; + + + | enum { + | first = 0, + | second, + | third, + | fourth, + | /* nbvalues must always be placed last */ + | nbvalues + | }; + +Structure names should be short enough not to mangle function declarations, +and explicit enough to avoid confusion (which is the most important thing). + +Wrong : + + | struct request_args { /* arguments on the query string */ + | char *name; + | char *value; + | struct misc_args *next; + | }; + +Right : + + | struct qs_args { /* arguments on the query string */ + | char *name; + | char *value; + | struct qs_args *next; + | } + + +When declaring new functions or structures, please do not use CamelCase, which +is a style where upper and lower case are mixed in a single word. It causes a +lot of confusion when words are composed from acronyms, because it's hard to +stick to a rule. For instance, a function designed to generate an ISN (initial +sequence number) for a TCP/IP connection could be called : + + - generateTcpipIsn() + - generateTcpIpIsn() + - generateTcpIpISN() + - generateTCPIPISN() + etc... + +None is right, none is wrong, these are just preferences which might change +along the code. Instead, please use an underscore to separate words. Lowercase +is preferred for the words, but if acronyms are upcased it's not dramatic. The +real advantage of this method is that it creates unambiguous levels even for +short names. + +Valid examples : + + - generate_tcpip_isn() + - generate_tcp_ip_isn() + - generate_TCPIP_ISN() + - generate_TCP_IP_ISN() + +Another example is easy to understand when 3 arguments are involved in naming +the function : + +Wrong (naming conflict) : + + | /* returns A + B * C */ + | int mulABC(int a, int b, int c) + | { + | return a + b * c; + | } + | + | /* returns (A + B) * C */ + | int mulABC(int a, int b, int c) + | { + | return (a + b) * c; + | } + +Right (unambiguous naming) : + + | /* returns A + B * C */ + | int mul_a_bc(int a, int b, int c) + | { + | return a + b * c; + | } + | + | /* returns (A + B) * C */ + | int mul_ab_c(int a, int b, int c) + | { + | return (a + b) * c; + | } + +Whenever you manipulate pointers, try to declare them as "const", as it will +save you from many accidental misuses and will only cause warnings to be +emitted when there is a real risk. In the examples below, it is possible to +call my_strcpy() with a const string only in the first declaration. Note that +people who ignore "const" are often the ones who cast a lot and who complain +from segfaults when using strtok() ! + +Right : + + | void my_strcpy(char *d, const char *s) + | { + | while ((*d++ = *s++)); + | } + | + | void say_hello(char *dest) + | { + | my_strcpy(dest, "hello\n"); + | } + +Wrong : + + | void my_strcpy(char *d, char *s) + | { + | while ((*d++ = *s++)); + | } + | + | void say_hello(char *dest) + | { + | my_strcpy(dest, "hello\n"); + | } + + +9) Getting macros right +----------------------- + +It is very common for macros to do the wrong thing when used in a way their +author did not have in mind. For this reason, macros must always be named with +uppercase letters only. This is the only way to catch the developer's eye when +using them, so that they double-check whether they are taking a risk or not. First, +macros must never ever be terminated by a semi-colon, or they will close the +wrong block once in a while. For instance, the following will cause a build +error before the "else" due to the double semi-colon : + +Wrong : + + | #define WARN printf("warning\n"); + | ... + | if (a < 0) + | WARN; + | else + | a--; + +Right : + + | #define WARN printf("warning\n") + +If multiple instructions are needed, then use a do { } while (0) block, which +is the only construct which respects *exactly* the semantics of a single +instruction : + + | #define WARN do { printf("warning\n"); log("warning\n"); } while (0) + | ... + | + | if (a < 0) + | WARN; + | else + | a--; + +Second, do not put unprotected control statements in macros, they will +definitely cause bugs : + +Wrong : + + | #define WARN if (verbose) printf("warning\n") + | ... + | if (a < 0) + | WARN; + | else + | a--; + +Which is equivalent to the undesired form below : + + | if (a < 0) + | if (verbose) + | printf("warning\n"); + | else + | a--; + +Right way to do it : + + | #define WARN do { if (verbose) printf("warning\n"); } while (0) + | ... + | if (a < 0) + | WARN; + | else + | a--; + +Which is equivalent to : + + | if (a < 0) + | do { if (verbose) printf("warning\n"); } while (0); + | else + | a--; + +Macro parameters must always be surrounded by parenthesis, and must never be +duplicated in the same macro unless explicitly stated. Also, macros must not be +defined with operators without surrounding parenthesis. The MIN/MAX macros are +a pretty common example of multiple misuses, but this happens as early as when +using bit masks. Most often, in case of any doubt, try to use inline functions +instead. + +Wrong : + + | #define MIN(a, b) a < b ? a : b + | + | /* returns 2 * min(a,b) + 1 */ + | int double_min_p1(int a, int b) + | { + | return 2 * MIN(a, b) + 1; + | } + +What this will do : + + | int double_min_p1(int a, int b) + | { + | return 2 * a < b ? a : b + 1; + | } + +Which is equivalent to : + + | int double_min_p1(int a, int b) + | { + | return (2 * a) < b ? a : (b + 1); + | } + +The first thing to fix is to surround the macro definition with parenthesis to +avoid this mistake : + + | #define MIN(a, b) (a < b ? a : b) + +But this is still not enough, as can be seen in this example : + + | /* compares either a or b with c */ + | int min_ab_c(int a, int b, int c) + | { + | return MIN(a ? a : b, c); + | } + +Which is equivalent to : + + | int min_ab_c(int a, int b, int c) + | { + | return (a ? a : b < c ? a ? a : b : c); + | } + +Which in turn means a totally different thing due to precedence : + + | int min_ab_c(int a, int b, int c) + | { + | return (a ? a : ((b < c) ? (a ? a : b) : c)); + | } + +This can be fixed by surrounding *each* argument in the macro with parenthesis: + + | #define MIN(a, b) ((a) < (b) ? (a) : (b)) + +But this is still not enough, as can be seen in this example : + + | int min_ap1_b(int a, int b) + | { + | return MIN(++a, b); + | } + +Which is equivalent to : + + | int min_ap1_b(int a, int b) + | { + | return ((++a) < (b) ? (++a) : (b)); + | } + +Again, this is wrong because "a" is incremented twice if below b. The only way +to fix this is to use a compound statement and to assign each argument exactly +once to a local variable of the same type : + + | #define MIN(a, b) ({ typeof(a) __a = (a); typeof(b) __b = (b); \ + | ((__a) < (__b) ? (__a) : (__b)); \ + | }) + +At this point, using static inline functions is much cleaner if a single type +is to be used : + + | static inline int min(int a, int b) + | { + | return a < b ? a : b; + | } + + +10) Includes +------------ + +Includes are as much as possible listed in alphabetically ordered groups : + - the includes more or less system-specific (sys/*, netinet/*, ...) + - the libc-standard includes (those without any path component) + - includes from the local "import" subdirectory + - includes from the local "haproxy" subdirectory + +Each section is just visually delimited from the other ones using an empty +line. The two first ones above may be merged into a single section depending on +developer's preference. Please do not copy-paste include statements from other +files. Having too many includes significantly increases build time and makes it +hard to find which ones are needed later. Just include what you need and if +possible in alphabetical order so that when something is missing, it becomes +obvious where to look for it and where to add it. + +All files should include because this is where build options +are prepared. + +HAProxy header files are split in two, those exporting the types only (named +with a trailing "-t") and those exporting variables, functions and inline +functions. Types, structures, enums and #defines must go into the types files +which are the only ones that may be included by othertype files. Function +prototypes and inlined functions must go into the main files. This split is +because of inlined functions which cross-reference types from other files, +which cause a chicken-and-egg problem if the functions and types are declared +at the same place. + +Include files must be protected against multiple inclusion using the common +#ifndef/#define/#endif trick with a tag derived from the include file and its +location. + + +11) Comments +------------ + +Comments are preferably of the standard 'C' form using /* */. The C++ form "//" +are tolerated for very short comments (eg: a word or two) but should be avoided +as much as possible. Multi-line comments are made with each intermediate line +starting with a star aligned with the first one, as in this example : + + | /* + | * This is a multi-line + | * comment. + | */ + +If multiple code lines need a short comment, try to align them so that you can +have multi-line sentences. This is rarely needed, only for really complex +constructs. + +Do not tell what you're doing in comments, but explain why you're doing it if +it seems not to be obvious. Also *do* indicate at the top of function what they +accept and what they don't accept. For instance, strcpy() only accepts output +buffers at least as large as the input buffer, and does not support any NULL +pointer. There is nothing wrong with that if the caller knows it. + +Wrong use of comments : + + | int flsnz8(unsigned int x) + | { + | int ret = 0; /* initialize ret */ + | if (x >> 4) { x >>= 4; ret += 4; } /* add 4 to ret if needed */ + | return ret + ((0xFFFFAA50U >> (x << 1)) & 3) + 1; /* add ??? */ + | } + | ... + | bit = ~len + (skip << 3) + 9; /* update bit */ + +Right use of comments : + + | /* This function returns the position of the highest bit set in the lowest + | * byte of , between 0 and 7. It only works if is non-null. It uses + | * a 32-bit value as a lookup table to return one of 4 values for the + | * highest 16 possible 4-bit values. + | */ + | int flsnz8(unsigned int x) + | { + | int ret = 0; + | if (x >> 4) { x >>= 4; ret += 4; } + | return ret + ((0xFFFFAA50U >> (x << 1)) & 3) + 1; + | } + | ... + | bit = ~len + (skip << 3) + 9; /* (skip << 3) + (8 - len), saves 1 cycle */ + + +12) Use of assembly +------------------- + +There are many projects where use of assembly code is not welcome. There is no +problem with use of assembly in haproxy, provided that : + + a) an alternate C-form is provided for architectures not covered + b) the code is small enough and well commented enough to be maintained + +It is important to take care of various incompatibilities between compiler +versions, for instance regarding output and cloberred registers. There are +a number of documentations on the subject on the net. Anyway if you are +fiddling with assembly, you probably know that already. + +Example : + | /* gcc does not know when it can safely divide 64 bits by 32 bits. Use this + | * function when you know for sure that the result fits in 32 bits, because + | * it is optimal on x86 and on 64bit processors. + | */ + | static inline unsigned int div64_32(unsigned long long o1, unsigned int o2) + | { + | unsigned int result; + | #ifdef __i386__ + | asm("divl %2" + | : "=a" (result) + | : "A"(o1), "rm"(o2)); + | #else + | result = o1 / o2; + | #endif + | return result; + | } + + +13) Pointers +------------ + +A lot could be said about pointers, there's enough to fill entire books. Misuse +of pointers is one of the primary reasons for bugs in haproxy, and this rate +has significantly increased with the use of threads. Moreover, bogus pointers +cause the hardest to analyse bugs, because usually they result in modifications +to reassigned areas or accesses to unmapped areas, and in each case, bugs that +strike very far away from where they were located. Some bugs have already taken +up to 3 weeks of full time analysis, which has a severe impact on the project's +ability to make forward progress on important features. For this reason, code +that doesn't look robust enough or that doesn't follow some of the rules below +will be rejected, and may even be reverted after being merged if the trouble is +detected late! + + +13.1) No test before freeing +---------------------------- + +All platforms where haproxy is supported have a well-defined and documented +behavior for free(NULL), which is to do nothing at all. In other words, free() +does test for the pointer's nullity. As such, there is no point in testing +if a pointer is NULL or not before calling free(). And further, you must not +do it, because it adds some confusion to the reader during debugging sessions, +making one think that the code's authors weren't very sure about what they +were doing. This will not cause a bug but will result in your code to get +rejected. + +Wrong call to free : + + | static inline int blah_free(struct blah *blah) + | { + | if (blah->str1) + | free(blah->str1); + | if (blah->str2) + | free(blah->str2); + | free(blah); + | } + +Correct call to free : + + | static inline int blah_free(struct blah *blah) + | { + | free(blah->str1); + | free(blah->str2); + | free(blah); + | } + + +13.2) No dangling pointers +-------------------------- + +Pointers are very commonly used as booleans: if they're not NULL, then the +area they point to is valid and may be used. This is convenient for many things +and is even emphasized with threads where they can atomically be swapped with +another value (even NULL), and as such provide guaranteed atomic resource +allocation and sharing. + +The problem with this is when someone forgets to delete a pointer when an area +is no longer valid, because this may result in the pointer being accessed later +and pointing to a wrong location, one that was reallocated for something else +and causing all sort of nastiness like crashes or memory corruption. Moreover, +thanks to the memory pools, it is extremely likely that a just released pointer +will be reassigned to a similar object with comparable values (flags etc) at +the same positions, making tests apparently succeed for a while. Some such bugs +have gone undetected for several years. + +The rule is pretty simple: + + +-----------------------------------------------------------------+ + | NO REACHABLE POINTER MAY EVER POINT TO AN UNREACHABLE LOCATION. | + +-----------------------------------------------------------------+ + +By "reachable pointer", here we mean a pointer that is accessible from a +reachable structure or a global variable. This means that any pointer found +anywhere in any structure in the code may always be dereferenced. This can +seem obvious but this is not always enforced. + +This means that when freeing an area, the pointer that was used to find that +area must be overwritten with NULL, and all other such pointers must as well +if any. It is one case where one can find more convenient to write the NULL +on the same line as the call to free() to make things easier to check. Be +careful about any potential "if" when doing this. + +Wrong use of free : + + | static inline int blah_recycle(struct blah *blah) + | { + | free(blah->str1); + | free(blah->str2); + | } + +Correct use of free : + + | static inline int blah_recycle(struct blah *blah) + | { + | free(blah->str1); blah->str1 = NULL; + | free(blah->str2); blah->str2 = NULL; + | } + +Sometimes the code doesn't permit this to be done. It is not a matter of code +but a matter of architecture. Example: + +Initialization: + + | static struct foo *foo_init() + | { + | struct foo *foo; + | struct bar *bar; + | + | foo = pool_alloc(foo_head); + | bar = pool_alloc(bar_head); + | if (!foo || !bar) + | goto fail; + | foo->bar = bar; + | ... + | } + +Scheduled task 1: + + | static inline int foo_timeout(struct foo *foo) + | { + | free(foo->bar); + | free(foo); + | } + +Scheduled task 2: + + | static inline int bar_timeout(struct bar *bar) + | { + | free(bar); + | } + +Here it's obvious that if "bar" times out, it will be freed but its pointer in +"foo" will remain here, and if foo times out just after, it will lead to a +double free. Or worse, if another instance allocates a pointer and receives bar +again, when foo times out, it will release the old bar pointer which now points +to a new object, and the code using that new object will crash much later, or +even worse, will share the same area as yet another instance having inherited +that pointer again. + +Here this simply means that the data model is wrong. If bar may be freed alone, +it MUST have a pointer to foo so that bar->foo->bar is set to NULL to let foo +finish its life peacefully. This also means that the code dealing with foo must +be written in a way to support bar's leaving. + + +13.3) Don't abuse pointers as booleans +-------------------------------------- + +Given the common use of a pointer to know if the area it points to is valid, +there is a big incentive in using such pointers as booleans to describe +something a bit higher level, like "is the user authenticated". This must not +be done. The reason stems from the points above. Initially this perfectly +matches and the code is simple. Then later some extra options need to be added, +and more pointers are needed, all allocated together. At this point they all +start to become their own booleans, supposedly always equivalent, but if that +were true, they would be a single area with a single pointer. And things start +to fall apart with some code areas relying on one pointer for the condition and +other ones relying on other pointers. Pointers may be substituted with "flags" +or "present in list" etc here. And from this point, things quickly degrade with +pointers needing to remain set even if pointing to wrong areas, just for the +sake of not being NULL and not breaking some assumptions. At this point the +bugs are already there and the code is not trustable anymore. + +The only way to avoid this is to strictly respect this rule: pointers do not +represent a functionality but a storage area. Of course it is very frequent to +consider that if an optional string is not set, a feature is not enabled. This +can be fine to some extents. But as soon as any slightest condition is added +anywhere into the mux, the code relying on the pointer must be replaced with +something else so that the pointer may live its own life and be released (and +reset) earlier if needed. + + +13.4) Mixing const and non-const +-------------------------------- + +Something often encountered, especially when assembling error messages, is +functions that collect strings, assemble them into larger messages and free +everything. The problem here is that if strings are defined as variables, there +will rightfully be build warnings when reporting string constants such as bare +keywords or messages, and if strings are defined as constants, it is not +possible to free them. The temptation is sometimes huge to force some free() +calls on casted strings. Do not do that! It will inevitably lead to someone +getting caught passing a constant string that will make the process crash (if +lucky). Document the expectations, indicate that all arguments must be freeable +and that the caller must be capable of strdup(), and make your function support +NULLs and document it (so that callers can deal with a failing strdup() on +allocation error). + +One valid alternative is to use a secondary channel to indicate whether the +message may be freed or not. A flag in a complex structure can be used for this +purpose, for example. If you are certain that your strings are aligned to a +certain number of bytes, it can be possible to instrument the code to use the +lowest bit to indicate the need to free (e.g. by always adding one to every +const string). But such a solution will require good enough instrumentation so +that it doesn't constitute a new set of traps. + + +13.5) No pointer casts +---------------------- + +Except in rare occasions caused by legacy APIs (e.g. sockaddr) or special cases +which explicitly require a form of aliasing, there is no valid reason for +casting pointers, and usually this is used to hide other problems that will +strike later. The only suitable type of cast is the cast from the generic void* +used to store a context for example. But in C, there is no need to cast to nor +from void*, so this is not required. However those coming from C++ tend to be +used to this practice, and others argue that it makes the intent more visible. + +As a corollary, do not abuse void*. Placing void* everywhere to avoid casting +is a bad practice as well. The use of void* is only for generic functions or +structures which do not have a limited set of types supported. When only a few +types are supported, generally their type can be passed using a side channel, +and the void* can be turned into a union that makes the code more readable and +more verifiable. + +An alternative in haproxy is to use a pointer to an obj_type enum. Usually it +is placed at the beginning of a structure. It works like a void* except that +the type is read directly from the object. This is convenient when a small set +of remote objects may be attached to another one because a single of them will +match a non-null pointer (e.g. a connection or an applet). + +Example: + + | static inline int blah_free(struct blah *blah) + | { + | /* only one of them (at most) will not be null */ + | pool_free(pool_head_connection, objt_conn(blah->target)); + | pool_free(pool_head_appctx, objt_appctx(blah->target)); + | pool_free(pool_head_stream, objt_stream(blah->target)); + | blah->target = NULL; + | } + + +13.6) Extreme caution when using non-canonical pointers +------------------------------------------------------- + +It can be particularly convenient to embed some logic in the unused bits or +code points of a pointer. Indeed, when it is known by design that a given +pointer will always follow a certain alignment, a few lower bits will always +remain zero, and as such may be used as optional flags. For example, the ebtree +code uses the lowest bit to differentiate left/right attachments to the parent +and node/leaf in branches. It is also known that values very close to NULL will +never represent a valid pointer, and the thread-safe MT_LIST code uses this to +lock visited pointers. + +There are a few rules to respect in order to do this: + - the deviations from the canonical pointers must be exhaustively documented + where the pointer type is defined, and the whole control logic with its + implications and possible and impossible cases must be enumerated as well ; + + - make sure that the operations will work on every supported platform, which + includes 32-bit platforms where structures may be aligned on as little as + 32-bit. 32-bit alignment leaves only two LSB available. When doing so, make + sure the target structures are not labelled with the "packed" attribute, or + that they're always perfectly aligned. All platforms where haproxy runs + have their NULL pointer mapped at address zero, and use page sizes at least + 4096 bytes large, leaving all values form 1 to 4095 unused. Anything + outside of this is unsafe. In particular, never use negative numbers to + represent a supposedly invalid address. On 32-bits platforms it will often + correspond to a system address or a special page. Always try a variety of + platforms when doing such a thing. + + - the code must not use such pointers as booleans anymore even if it is known + that "it works" because that keeps a doubt open for the reviewer. Only the + canonical pointer may be tested. There can be a rare exception which is if + this is on a critical path where severe performance degradation may result + from this. In this case, *each* of the checks must be duly documented and + the equivalent BUG_ON() instances must be placed to prove the claim. + + - some inline functions (or macros) must be used to turn the pointers to/from + their canonical form so that the regular code doesn't have to see the + operations, and so that the representation may be easily adjusted in the + future. A few comments indicating to a human how to turn a pointer back and + forth from inside a debugger will be appreciated, as macros often end up + not being trivially readable nor directly usable. + + - do not use int types to cast the pointers, this will only work on 32-bit + platforms. While "long" is usually fine, it is not recommended anymore due + to the Windows platform being LLP64 and having it set to 32 bits. And + "long long" isn't good either for always being 64 bits. More suitable types + are ptrdiff_t or size_t. Note that while those were not available everywhere + in the early days of hparoxy, size_t is now heavily used and known to work + everywhere. And do not perform the operations on the pointers, only on the + integer types (and cast back again). Some compilers such as gcc are + extremely picky about this and will often emit wrong code when they see + equality conditions they believe is impossible and decide to optimize them + away. + + +13.7) Pointers in unions +------------------------ + +Before placing multiple aliasing pointers inside a same union, there MUST be a +SINGLE well-defined way to figure them out from each other. It may be thanks to +a side-channel information (as done in the samples with a defined type), it may +be based on in-area information (as done using obj_types), or any other trusted +solution. In any case, if pointers are mixed with any other type (integer or +float) in a union, there must be a very simple way to distinguish them, and not +a platform-dependent nor compiler-dependent one. diff --git a/doc/configuration.txt b/doc/configuration.txt new file mode 100644 index 0000000..a1f15fc --- /dev/null +++ b/doc/configuration.txt @@ -0,0 +1,26732 @@ + ---------------------- + HAProxy + Configuration Manual + ---------------------- + version 2.9 + 2024/02/15 + + +This document covers the configuration language as implemented in the version +specified above. It does not provide any hints, examples, or advice. For such +documentation, please refer to the Reference Manual or the Architecture Manual. +The summary below is meant to help you find sections by name and navigate +through the document. + +Note to documentation contributors : + This document is formatted with 80 columns per line, with even number of + spaces for indentation and without tabs. Please follow these rules strictly + so that it remains easily printable everywhere. If a line needs to be + printed verbatim and does not fit, please end each line with a backslash + ('\') and continue on next line, indented by two characters. It is also + sometimes useful to prefix all output lines (logs, console outputs) with 3 + closing angle brackets ('>>>') in order to emphasize the difference between + inputs and outputs when they may be ambiguous. If you add sections, + please update the summary below for easier searching. + + +Summary +------- + +1. Quick reminder about HTTP +1.1. The HTTP transaction model +1.2. Terminology +1.3. HTTP request +1.3.1. The request line +1.3.2. The request headers +1.4. HTTP response +1.4.1. The response line +1.4.2. The response headers + +2. Configuring HAProxy +2.1. Configuration file format +2.2. Quoting and escaping +2.3. Environment variables +2.4. Conditional blocks +2.5. Time format +2.6. Size format +2.7. Examples + +3. Global parameters +3.1. Process management and security +3.2. Performance tuning +3.3. Debugging +3.4. Userlists +3.5. Peers +3.6. Mailers +3.7. Programs +3.8. HTTP-errors +3.9. Rings +3.10. Log forwarding +3.11. HTTPClient tuning + +4. Proxies +4.1. Proxy keywords matrix +4.2. Alphabetically sorted keywords reference +4.3. Actions keywords matrix +4.4. Alphabetically sorted actions reference + +5. Bind and server options +5.1. Bind options +5.2. Server and default-server options +5.3. Server DNS resolution +5.3.1. Global overview +5.3.2. The resolvers section + +6. Cache +6.1. Limitation +6.2. Setup +6.2.1. Cache section +6.2.2. Proxy section + +7. Using ACLs and fetching samples +7.1. ACL basics +7.1.1. Matching booleans +7.1.2. Matching integers +7.1.3. Matching strings +7.1.4. Matching regular expressions (regexes) +7.1.5. Matching arbitrary data blocks +7.1.6. Matching IPv4 and IPv6 addresses +7.2. Using ACLs to form conditions +7.3. Fetching samples +7.3.1. Converters +7.3.2. Fetching samples from internal states +7.3.3. Fetching samples at Layer 4 +7.3.4. Fetching samples at Layer 5 +7.3.5. Fetching samples from buffer contents (Layer 6) +7.3.6. Fetching HTTP samples (Layer 7) +7.3.7. Fetching samples for developers +7.4. Pre-defined ACLs + +8. Logging +8.1. Log levels +8.2. Log formats +8.2.1. Default log format +8.2.2. TCP log format +8.2.3. HTTP log format +8.2.4. HTTPS log format +8.2.5. Error log format +8.2.6. Custom log format +8.3. Advanced logging options +8.3.1. Disabling logging of external tests +8.3.2. Logging before waiting for the stream to terminate +8.3.3. Raising log level upon errors +8.3.4. Disabling logging of successful connections +8.4. Timing events +8.5. Stream state at disconnection +8.6. Non-printable characters +8.7. Capturing HTTP cookies +8.8. Capturing HTTP headers +8.9. Examples of logs + +9. Supported filters +9.1. Trace +9.2. HTTP compression +9.3. Stream Processing Offload Engine (SPOE) +9.4. Cache +9.5. fcgi-app +9.6. OpenTracing +9.7. Bandwidth limitation + +10. FastCGI applications +10.1. Setup +10.1.1. Fcgi-app section +10.1.2. Proxy section +10.1.3. Example +10.2. Default parameters +10.3. Limitations + +11. Address formats +11.1. Address family prefixes +11.2. Socket type prefixes +11.3. Protocol prefixes + + +1. Quick reminder about HTTP +---------------------------- + +When HAProxy is running in HTTP mode, both the request and the response are +fully analyzed and indexed, thus it becomes possible to build matching criteria +on almost anything found in the contents. + +However, it is important to understand how HTTP requests and responses are +formed, and how HAProxy decomposes them. It will then become easier to write +correct rules and to debug existing configurations. + +First, HTTP is standardized by a series of RFC that HAProxy follows as closely +as possible: + - RFC 9110: HTTP Semantics (explains the meaning of protocol elements) + - RFC 9111: HTTP Caching (explains the rules to follow for an HTTP cache) + - RFC 9112: HTTP/1.1 (representation, interoperability rules, security) + - RFC 9113: HTTP/2 (representation, interoperability rules, security) + - RFC 9114: HTTP/3 (representation, interoperability rules, security) + +In addition to these, RFC 8999 to 9002 specify the QUIC transport layer used by +the HTTP/3 protocol. + + +1.1. The HTTP transaction model +------------------------------- + +The HTTP protocol is transaction-driven. This means that each request will lead +to one and only one response. Originally, with version 1.0 of the protocol, +there was a single request per connection: a TCP connection is established from +the client to the server, a request is sent by the client over the connection, +the server responds, and the connection is closed. A new request then involves +a new connection : + + [CON1] [REQ1] ... [RESP1] [CLO1] [CON2] [REQ2] ... [RESP2] [CLO2] ... + +In this mode, often called the "HTTP close" mode, there are as many connection +establishments as there are HTTP transactions. Since the connection is closed +by the server after the response, the client does not need to know the content +length, it considers that the response is complete when the connection closes. +This also means that if some responses are truncated due to network errors, the +client could mistakenly think a response was complete, and this used to cause +truncated images to be rendered on screen sometimes. + +Due to the transactional nature of the protocol, it was possible to improve it +to avoid closing a connection between two subsequent transactions. In this mode +however, it is mandatory that the server indicates the content length for each +response so that the client does not wait indefinitely. For this, a special +header is used: "Content-length". This mode is called the "keep-alive" mode, +and arrived with HTTP/1.1 (some HTTP/1.0 agents support it), and connections +that are reused between requests are called "persistent connections": + + [CON] [REQ1] ... [RESP1] [REQ2] ... [RESP2] [CLO] ... + +Its advantages are a reduced latency between transactions, less processing +power required on the server side, and the ability to detect a truncated +response. It is generally faster than the close mode, but not always because +some clients often limit their concurrent connections to a smaller value, and +this compensates less for poor network connectivity. Also, some servers have to +keep the connection alive for a long time waiting for a possible new request +and may experience a high memory usage due to the high number of connections, +and closing too fast may break some requests that arrived at the moment the +connection was closed. + +In this mode, the response size needs to be known upfront so that's not always +possible with dynamically generated or compressed contents. For this reason +another mode was implemented, the "chunked mode", where instead of announcing +the size of the whole size at once, the sender only advertises the size of the +next "chunk" of response it already has in a buffer, and can terminate at any +moment with a zero-sized chunk. In this mode, the Content-Length header is not +used. + +Another improvement in the communications is the pipelining mode. It still uses +keep-alive, but the client does not wait for the first response to send the +second request. This is useful for fetching large number of images composing a +page : + + [CON] [REQ1] [REQ2] ... [RESP1] [RESP2] [CLO] ... + +This can obviously have a tremendous benefit on performance because the network +latency is eliminated between subsequent requests. Many HTTP agents do not +correctly support pipelining since there is no way to associate a response with +the corresponding request in HTTP. For this reason, it is mandatory for the +server to reply in the exact same order as the requests were received. In +practice, after several attempts by various clients to deploy it, it has been +totally abandoned for its lack of reliability on certain servers. But it is +mandatory for servers to support it. + +The next improvement is the multiplexed mode, as implemented in HTTP/2 and +HTTP/3. In this mode, multiple transactions (i.e. request-response pairs) are +transmitted in parallel over a single connection, and they all progress at +their own speed, independent from each other. With multiplexed protocols, a new +notion of "stream" was introduced, to represent these parallel communications +happening over the same connection. Each stream is generally assigned a unique +identifier for a given connection, that is used by both endpoints to know where +to deliver the data. It is fairly common for clients to start many (up to 100, +sometimes more) streams in parallel over a same connection, and let the server +sort them out and respond in any order depending on what response is available. +The main benefit of the multiplexed mode is that it significantly reduces the +number of round trips, and speeds up page loading time over high latency +networks. It is sometimes visibles on sites using many images, where all images +appear to load in parallel. + +These protocols have also improved their efficiency by adopting some mechanisms +to compress header fields in order to reduce the number of bytes on the wire, +so that without the appropriate tools, they are not realistically manipulable +by hand nor readable to the naked eye like HTTP/1 was. For this reason, various +examples of HTTP messages continue to be represented in literature (including +this document) using the HTTP/1 syntax even for newer versions of the protocol. + +HTTP/2 suffers from some design limitations, such as packet losses affecting +all streams at once, and if a client takes too much time to retrieve an object +(e.g. needs to store it on disk), it may slow down its retrieval and make it +impossible during this time to access the data that is pending behind it. This +is called "head of line blocking" or "HoL blocking" or sometimes just "HoL". + +HTTP/3 is implemented over QUIC, itself implemented over UDP. QUIC solves the +head of line blocking at the transport level by means of independently handled +streams. Indeed, when experiencing loss, an impacted stream does not affect the +other streams, and all of them can be accessed in parallel. + +By default HAProxy operates in keep-alive mode with regards to persistent +connections: for each connection it processes each request and response, and +leaves the connection idle on both sides between the end of a response and the +start of a new request. When it receives HTTP/2 connections from a client, it +processes all the requests in parallel and leaves the connection idling, +waiting for new requests, just as if it was a keep-alive HTTP connection. + +HAProxy essentially supports 3 connection modes : + - keep alive : all requests and responses are processed, and the client + facing and server facing connections are kept alive for new + requests. This is the default and suits the modern web and + modern protocols (HTTP/2 and HTTP/3). + + - server close : the server-facing connection is closed after the response. + + - close : the connection is actively closed after end of response on + both sides. + +In addition to this, by default, the server-facing connection is reusable by +any request from any client, as mandated by the HTTP protocol specification, so +any information pertaining to a specific client has to be passed along with +each request if needed (e.g. client's source adress etc). When HTTP/2 is used +with a server, by default HAProxy will dedicate this connection to the same +client to avoid the risk of head of line blocking between clients. + + +1.2. Terminology +---------------- + +Inside HAProxy, the terminology has evolved a bit over the ages to follow the +evolutions of the HTTP protocol and its usages. While originally there was no +significant difference between a connection, a session, a stream or a +transaction, these ones clarified over time to match closely what exists in the +modern versions of the HTTP protocol, though some terms remain visible in the +configuration or the command line interface for the purpose of historical +compatibility. + +Here are some definitions that apply to the current version of HAProxy: + + - connection: a connection is a single, bidiractional communication channel + between a remote agent (client or server) and haproxy, at the lowest level + possible. Usually it corresponds to a TCP socket established between a pair + of IP and ports. On the client-facing side, connections are the very first + entities that are instantiated when a client connects to haproxy, and rules + applying at the connection level are the earliest ones that apply. + + - session: a session adds some context information associated with a + connection. This includes and information specific to the transport layer + (e.g. TLS keys etc), or variables. This term has long been used inside + HAProxy to denote end-to-end HTTP/1.0 communications between two ends, and + as such it remains visible in the name of certain CLI commands or + statistics, despite representing streams nowadays, but the help messages + and descriptions try to make this unambiguous. It is still valid when it + comes to network-level terminology (e.g. TCP sessions inside the operating + systems, or TCP sessions across a firewall), or for non-HTTP user-level + applications (e.g. a telnet session or an SSH session). It must not be + confused with "application sessions" that are used to store a full user + context in a cookie and require to be sent to the same server. + + - stream: a stream exactly corresponds to an end-to-end bidirectional + communication at the application level, where analysis and transformations + may be applied. In HTTP, it contains a single request and its associated + response, and is instantiated by the arrival of the request and is finished + with the end of delivery of the response. In this context there is a 1:1 + relation between such a stream and the stream of a multiplexed protocol. In + TCP communications there is a single stream per connection. + + - transaction: a transaction is only a pair of a request and the associated + response. The term was used in conjunction with sessions before the streams + but nowadays there is a 1:1 relation between a transaction and a stream. It + is essentially visible in the variables' scope "txn" which is valid during + the whole transaction, hence the stream. + + - request: it designates the traffic flowing from the client to the server. + It is mainly used for HTTP to indicate where operations are performed. This + term also exists for TCP operations to indicate where data are processed. + Requests often appear in counters as a unit of traffic or activity. They do + not always imply a response (e.g. due to errors), but since there is no + spontaneous responses without requests, requests remain a relevant metric + of the overall activity. In TCP there are as many requests as connections. + + - response: this designates the traffic flowing from the server to the + client, or sometimes from HAProxy to the client, when HAProxy produces the + response itself (e.g. an HTTP redirect). + + - service: this generally indicates some internal processing in HAProxy that + does not require a server, such as the stats page, the cache, or some Lua + code to implement a small application. A service usually reads a request, + performs some operations and produces a response. + + +1.3. HTTP request +----------------- + +First, let's consider this HTTP request : + + Line Contents + number + 1 GET /serv/login.php?lang=en&profile=2 HTTP/1.1 + 2 Host: www.mydomain.com + 3 User-agent: my small browser + 4 Accept: image/jpeg, image/gif + 5 Accept: image/png + + +1.3.1. The Request line +----------------------- + +Line 1 is the "request line". It is always composed of 3 fields : + + - a METHOD : GET + - a URI : /serv/login.php?lang=en&profile=2 + - a version tag : HTTP/1.1 + +All of them are delimited by what the standard calls LWS (linear white spaces), +which are commonly spaces, but can also be tabs or line feeds/carriage returns +followed by spaces/tabs. The method itself cannot contain any colon (':') and +is limited to alphabetic letters. All those various combinations make it +desirable that HAProxy performs the splitting itself rather than leaving it to +the user to write a complex or inaccurate regular expression. + +The URI itself can have several forms : + + - A "relative URI" : + + /serv/login.php?lang=en&profile=2 + + It is a complete URL without the host part. This is generally what is + received by servers, reverse proxies and transparent proxies. + + - An "absolute URI", also called a "URL" : + + http://192.168.0.12:8080/serv/login.php?lang=en&profile=2 + + It is composed of a "scheme" (the protocol name followed by '://'), a host + name or address, optionally a colon (':') followed by a port number, then + a relative URI beginning at the first slash ('/') after the address part. + This is generally what proxies receive, but a server supporting HTTP/1.1 + must accept this form too. + + - a star ('*') : this form is only accepted in association with the OPTIONS + method and is not relayable. It is used to inquiry a next hop's + capabilities. + + - an address:port combination : 192.168.0.12:80 + This is used with the CONNECT method, which is used to establish TCP + tunnels through HTTP proxies, generally for HTTPS, but sometimes for + other protocols too. + +In a relative URI, two sub-parts are identified. The part before the question +mark is called the "path". It is typically the relative path to static objects +on the server. The part after the question mark is called the "query string". +It is mostly used with GET requests sent to dynamic scripts and is very +specific to the language, framework or application in use. + +HTTP/3 and HTTP/3 do not convey a version information with the request, so the +version is assumed to be the same as the one of the underlying protocol (i.e. +"HTTP/2"). In addition, these protocols do not send a request line as one part, +but split it into individual fields called "pseudo-headers", whose name start +with a colon, and which are conveniently reassembled by HAProxy into an +equivalent request line. For this reason, request lines found in logs may +slightly differ between HTTP/1.x and HTTP/2 or HTTP/3. + + +1.3.2. The request headers +-------------------------- + +The headers start at the second line. They are composed of a name at the +beginning of the line, immediately followed by a colon (':'). Traditionally, +an LWS is added after the colon but that's not required. Then come the values. +Multiple identical headers may be folded into one single line, delimiting the +values with commas, provided that their order is respected. This is commonly +encountered in the "Cookie:" field. A header may span over multiple lines if +the subsequent lines begin with an LWS. In the example in 1.3, lines 4 and 5 +define a total of 3 values for the "Accept:" header. Finally, all LWS at the +beginning or at the end of a header are ignored and are not part of the value, +as per the specification. + +Contrary to a common misconception, header names are not case-sensitive, and +their values are not either if they refer to other header names (such as the +"Connection:" header). In HTTP/2 and HTTP/3, header names are always sent in +lower case, as can be seen when running in debug mode. Internally, all header +names are normalized to lower case so that HTTP/1.x and HTTP/2 or HTTP/3 use +the exact same representation, and they are sent as-is on the other side. This +explains why an HTTP/1.x request typed with camel case is delivered in lower +case. + +The end of the headers is indicated by the first empty line. People often say +that it's a double line feed, which is not exact, even if a double line feed +is one valid form of empty line. + +Fortunately, HAProxy takes care of all these complex combinations when indexing +headers, checking values and counting them, so there is no reason to worry +about the way they could be written, but it is important not to accuse an +application of being buggy if it does unusual, valid things. + +Important note: + As suggested by RFC7231, HAProxy normalizes headers by replacing line breaks + in the middle of headers by LWS in order to join multi-line headers. This + is necessary for proper analysis and helps less capable HTTP parsers to work + correctly and not to be fooled by such complex constructs. + + +1.4. HTTP response +------------------ + +An HTTP response looks very much like an HTTP request. Both are called HTTP +messages. Let's consider this HTTP response : + + Line Contents + number + 1 HTTP/1.1 200 OK + 2 Content-length: 350 + 3 Content-Type: text/html + +As a special case, HTTP supports so called "Informational responses" as status +codes 1xx. These messages are special in that they don't convey any part of the +response, they're just used as sort of a signaling message to ask a client to +continue to post its request for instance. In the case of a status 100 response +the requested information will be carried by the next non-100 response message +following the informational one. This implies that multiple responses may be +sent to a single request, and that this only works when keep-alive is enabled +(1xx messages appeared in HTTP/1.1). HAProxy handles these messages and is able +to correctly forward and skip them, and only process the next non-100 response. +As such, these messages are neither logged nor transformed, unless explicitly +state otherwise. Status 101 messages indicate that the protocol is changing +over the same connection and that HAProxy must switch to tunnel mode, just as +if a CONNECT had occurred. Then the Upgrade header would contain additional +information about the type of protocol the connection is switching to. + + +1.4.1. The response line +------------------------ + +Line 1 is the "response line". It is always composed of 3 fields : + + - a version tag : HTTP/1.1 + - a status code : 200 + - a reason : OK + +The status code is always 3-digit. The first digit indicates a general status : + - 1xx = informational message to be skipped (e.g. 100, 101) + - 2xx = OK, content is following (e.g. 200, 206) + - 3xx = OK, no content following (e.g. 302, 304) + - 4xx = error caused by the client (e.g. 401, 403, 404) + - 5xx = error caused by the server (e.g. 500, 502, 503) + +Status codes greater than 599 must not be emitted in communications, though +certain agents may produce them in logs to report their internal statuses. +Please refer to RFC9110 for the detailed meaning of all such codes. HTTP/2 and +above do not have a version tag and use the ":status" pseudo-header to report +the status code. + +The "reason" field is just a hint, but is not parsed by clients. Anything can +be found there, but it's a common practice to respect the well-established +messages. It can be composed of one or multiple words, such as "OK", "Found", +or "Authentication Required". It does not exist in HTTP/2 and above and is +not emitted there. When a response from HTTP/2 or above is transmitted to an +HTTP/1 client, HAProxy will produce such a common reason field that matches +the status code. + +HAProxy may emit the following status codes by itself : + + Code When / reason + 200 access to stats page, and when replying to monitoring requests + 301 when performing a redirection, depending on the configured code + 302 when performing a redirection, depending on the configured code + 303 when performing a redirection, depending on the configured code + 307 when performing a redirection, depending on the configured code + 308 when performing a redirection, depending on the configured code + 400 for an invalid or too large request + 401 when an authentication is required to perform the action (when + accessing the stats page) + 403 when a request is forbidden by a "http-request deny" rule + 404 when the requested resource could not be found + 408 when the request timeout strikes before the request is complete + 410 when the requested resource is no longer available and will not + be available again + 500 when HAProxy encounters an unrecoverable internal error, such as a + memory allocation failure, which should never happen + 501 when HAProxy is unable to satisfy a client request because of an + unsupported feature + 502 when the server returns an empty, invalid or incomplete response, or + when an "http-response deny" rule blocks the response. + 503 when no server was available to handle the request, or in response to + monitoring requests which match the "monitor fail" condition + 504 when the response timeout strikes before the server responds + +The error 4xx and 5xx codes above may be customized (see "errorloc" in section +4.2). Other status codes can be emitted on purpose by specific actions (see the +"deny", "return" and "redirect" actions in section 4.3 for example). + + +1.4.2. The response headers +--------------------------- + +Response headers work exactly like request headers, and as such, HAProxy uses +the same parsing function for both. Please refer to paragraph 1.3.2 for more +details. + + +2. Configuring HAProxy +---------------------- + +2.1. Configuration file format +------------------------------ + +HAProxy's configuration process involves 3 major sources of parameters : + + - the arguments from the command-line, which always take precedence + - the configuration file(s), whose format is described here + - the running process's environment, in case some environment variables are + explicitly referenced + +The configuration file follows a fairly simple hierarchical format which obey +a few basic rules: + + 1. a configuration file is an ordered sequence of statements + + 2. a statement is a single non-empty line before any unprotected "#" (hash) + + 3. a line is a series of tokens or "words" delimited by unprotected spaces or + tab characters + + 4. the first word or sequence of words of a line is one of the keywords or + keyword sequences listed in this document + + 5. all other words are all arguments of the first one, some being well-known + keywords listed in this document, others being values, references to other + parts of the configuration, or expressions + + 6. certain keywords delimit a section inside which only a subset of keywords + are supported + + 7. a section ends at the end of a file or on a special keyword starting a new + section + +This is all that is needed to know to write a simple but reliable configuration +generator, but this is not enough to reliably parse any configuration nor to +figure how to deal with certain corner cases. + +First, there are a few consequences of the rules above. Rule 6 and 7 imply that +the keywords used to define a new section are valid everywhere and cannot have +a different meaning in a specific section. These keywords are always a single +word (as opposed to a sequence of words), and traditionally the section that +follows them is designated using the same name. For example when speaking about +the "global section", it designates the section of configuration that follows +the "global" keyword. This usage is used a lot in error messages to help locate +the parts that need to be addressed. + +A number of sections create an internal object or configuration space, which +requires to be distinguished from other ones. In this case they will take an +extra word which will set the name of this particular section. For some of them +the section name is mandatory. For example "frontend foo" will create a new +section of type "frontend" named "foo". Usually a name is specific to its +section and two sections of different types may use the same name, but this is +not recommended as it tends to complexify configuration management. + +A direct consequence of rule 7 is that when multiple files are read at once, +each of them must start with a new section, and the end of each file will end +a section. A file cannot contain sub-sections nor end an existing section and +start a new one. + +Rule 1 mentioned that ordering matters. Indeed, some keywords create directives +that can be repeated multiple times to create ordered sequences of rules to be +applied in a certain order. For example "tcp-request" can be used to alternate +"accept" and "reject" rules on varying criteria. As such, a configuration file +processor must always preserve a section's ordering when editing a file. The +ordering of sections usually does not matter except for the global section +which must be placed before other sections, but it may be repeated if needed. +In addition, some automatic identifiers may automatically be assigned to some +of the created objects (e.g. proxies), and by reordering sections, their +identifiers will change. These ones appear in the statistics for example. As +such, the configuration below will assign "foo" ID number 1 and "bar" ID number +2, which will be swapped if the two sections are reversed: + + listen foo + bind :80 + + listen bar + bind :81 + +Another important point is that according to rules 2 and 3 above, empty lines, +spaces, tabs, and comments following and unprotected "#" character are not part +of the configuration as they are just used as delimiters. This implies that the +following configurations are strictly equivalent: + + global#this is the global section + daemon#daemonize + frontend foo + mode http # or tcp + +and: + + global + daemon + + # this is the public web frontend + frontend foo + mode http + +The common practice is to align to the left only the keyword that initiates a +new section, and indent (i.e. prepend a tab character or a few spaces) all +other keywords so that it's instantly visible that they belong to the same +section (as done in the second example above). Placing comments before a new +section helps the reader decide if it's the desired one. Leaving a blank line +at the end of a section also visually helps spotting the end when editing it. + +Tabs are very convenient for indent but they do not copy-paste well. If spaces +are used instead, it is recommended to avoid placing too many (2 to 4) so that +editing in field doesn't become a burden with limited editors that do not +support automatic indent. + +In the early days it used to be common to see arguments split at fixed tab +positions because most keywords would not take more than two arguments. With +modern versions featuring complex expressions this practice does not stand +anymore, and is not recommended. + + +2.2. Quoting and escaping +------------------------- + +In modern configurations, some arguments require the use of some characters +that were previously considered as pure delimiters. In order to make this +possible, HAProxy supports character escaping by prepending a backslash ('\') +in front of the character to be escaped, weak quoting within double quotes +('"') and strong quoting within single quotes ("'"). + +This is pretty similar to what is done in a number of programming languages and +very close to what is commonly encountered in Bourne shell. The principle is +the following: while the configuration parser cuts the lines into words, it +also takes care of quotes and backslashes to decide whether a character is a +delimiter or is the raw representation of this character within the current +word. The escape character is then removed, the quotes are removed, and the +remaining word is used as-is as a keyword or argument for example. + +If a backslash is needed in a word, it must either be escaped using itself +(i.e. double backslash) or be strongly quoted. + +Escaping outside quotes is achieved by preceding a special character by a +backslash ('\'): + + \ to mark a space and differentiate it from a delimiter + \# to mark a hash and differentiate it from a comment + \\ to use a backslash + \' to use a single quote and differentiate it from strong quoting + \" to use a double quote and differentiate it from weak quoting + +In addition, a few non-printable characters may be emitted using their usual +C-language representation: + + \n to insert a line feed (LF, character \x0a or ASCII 10 decimal) + \r to insert a carriage return (CR, character \x0d or ASCII 13 decimal) + \t to insert a tab (character \x09 or ASCII 9 decimal) + \xNN to insert character having ASCII code hex NN (e.g \x0a for LF). + +Weak quoting is achieved by surrounding double quotes ("") around the character +or sequence of characters to protect. Weak quoting prevents the interpretation +of: + + space or tab as a word separator + ' single quote as a strong quoting delimiter + # hash as a comment start + +Weak quoting permits the interpretation of environment variables (which are not +evaluated outside of quotes) by preceding them with a dollar sign ('$'). If a +dollar character is needed inside double quotes, it must be escaped using a +backslash. + +Strong quoting is achieved by surrounding single quotes ('') around the +character or sequence of characters to protect. Inside single quotes, nothing +is interpreted, it's the efficient way to quote regular expressions. + +As a result, here is the matrix indicating how special characters can be +entered in different contexts (unprintable characters are replaced with their +name within angle brackets). Note that some characters that may only be +represented escaped have no possible representation inside single quotes, +hence its absence there: + + Character | Unquoted | Weakly quoted | Strongly quoted + -----------+---------------+-----------------------------+----------------- + | \, \x09 | "", "\", "\x09" | '' + -----------+---------------+-----------------------------+----------------- + | \n, \x0a | "\n", "\x0a" | + -----------+---------------+-----------------------------+----------------- + | \r, \x0d | "\r", "\x0d" | + -----------+---------------+-----------------------------+----------------- + | \, \x20 | "", "\", "\x20" | '' + -----------+---------------+-----------------------------+----------------- + " | \", \x22 | "\"", "\x22" | '"' + -----------+---------------+-----------------------------+----------------- + # | \#, \x23 | "#", "\#", "\x23" | '#' + -----------+---------------+-----------------------------+----------------- + $ | $, \$, \x24 | "\$", "\x24" | '$' + -----------+---------------+-----------------------------+----------------- + ' | \', \x27 | "'", "\'", "\x27" | + -----------+---------------+-----------------------------+----------------- + \ | \\, \x5c | "\\", "\x5c" | '\' + -----------+---------------+-----------------------------+----------------- + + Example: + # those are all strictly equivalent: + log-format %{+Q}o\ %t\ %s\ %{-Q}r + log-format "%{+Q}o %t %s %{-Q}r" + log-format '%{+Q}o %t %s %{-Q}r' + log-format "%{+Q}o %t"' %s %{-Q}r' + log-format "%{+Q}o %t"' %s'\ %{-Q}r + +There is one particular case where a second level of quoting or escaping may be +necessary. Some keywords take arguments within parenthesis, sometimes delimited +by commas. These arguments are commonly integers or predefined words, but when +they are arbitrary strings, it may be required to perform a separate level of +escaping to disambiguate the characters that belong to the argument from the +characters that are used to delimit the arguments themselves. A pretty common +case is the "regsub" converter. It takes a regular expression in argument, and +if a closing parenthesis is needed inside, this one will require to have its +own quotes. + +The keyword argument parser is exactly the same as the top-level one regarding +quotes, except that the \#, \$, and \xNN escapes are not processed. But what is +not always obvious is that the delimiters used inside must first be escaped or +quoted so that they are not resolved at the top level. + +Let's take this example making use of the "regsub" converter which takes 3 +arguments, one regular expression, one replacement string and one set of flags: + + # replace all occurrences of "foo" with "blah" in the path: + http-request set-path %[path,regsub(foo,blah,g)] + +Here no special quoting was necessary. But if now we want to replace either +"foo" or "bar" with "blah", we'll need the regular expression "(foo|bar)". We +cannot write: + + http-request set-path %[path,regsub((foo|bar),blah,g)] + +because we would like the string to cut like this: + + http-request set-path %[path,regsub((foo|bar),blah,g)] + |---------|----|-| + arg1 _/ / / + arg2 __________/ / + arg3 ______________/ + +but actually what is passed is a string between the opening and closing +parenthesis then garbage: + + http-request set-path %[path,regsub((foo|bar),blah,g)] + |--------|--------| + arg1=(foo|bar _/ / + trailing garbage _________/ + +The obvious solution here seems to be that the closing parenthesis needs to be +quoted, but alone this will not work, because as mentioned above, quotes are +processed by the top-level parser which will resolve them before processing +this word: + + http-request set-path %[path,regsub("(foo|bar)",blah,g)] + ------------ -------- ---------------------------------- + word1 word2 word3=%[path,regsub((foo|bar),blah,g)] + +So we didn't change anything for the argument parser at the second level which +still sees a truncated regular expression as the only argument, and garbage at +the end of the string. By escaping the quotes they will be passed unmodified to +the second level: + + http-request set-path %[path,regsub(\"(foo|bar)\",blah,g)] + ------------ -------- ------------------------------------ + word1 word2 word3=%[path,regsub("(foo|bar)",blah,g)] + |---------||----|-| + arg1=(foo|bar) _/ / / + arg2=blah ___________/ / + arg3=g _______________/ + +Another approach consists in using single quotes outside the whole string and +double quotes inside (so that the double quotes are not stripped again): + + http-request set-path '%[path,regsub("(foo|bar)",blah,g)]' + ------------ -------- ---------------------------------- + word1 word2 word3=%[path,regsub("(foo|bar)",blah,g)] + |---------||----|-| + arg1=(foo|bar) _/ / / + arg2 ___________/ / + arg3 _______________/ + +When using regular expressions, it can happen that the dollar ('$') character +appears in the expression or that a backslash ('\') is used in the replacement +string. In this case these ones will also be processed inside the double quotes +thus single quotes are preferred (or double escaping). Example: + + http-request set-path '%[path,regsub("^/(here)(/|$)","my/\1",g)]' + ------------ -------- ----------------------------------------- + word1 word2 word3=%[path,regsub("^/(here)(/|$)","my/\1",g)] + |-------------| |-----||-| + arg1=(here)(/|$) _/ / / + arg2=my/\1 ________________/ / + arg3 ______________________/ + +Remember that backslashes are not escape characters within single quotes and +that the whole word above is already protected against them using the single +quotes. Conversely, if double quotes had been used around the whole expression, +single the dollar character and the backslashes would have been resolved at top +level, breaking the argument contents at the second level. + +Unfortunately, since single quotes can't be escaped inside of strong quoting, +if you need to include single quotes in your argument, you will need to escape +or quote them twice. There are a few ways to do this: + + http-request set-var(txn.foo) str("\\'foo\\'") + http-request set-var(txn.foo) str(\"\'foo\'\") + http-request set-var(txn.foo) str(\\\'foo\\\') + +When in doubt, simply do not use quotes anywhere, and start to place single or +double quotes around arguments that require a comma or a closing parenthesis, +and think about escaping these quotes using a backslash if the string contains +a dollar or a backslash. Again, this is pretty similar to what is used under +a Bourne shell when double-escaping a command passed to "eval". For API writers +the best is probably to place escaped quotes around each and every argument, +regardless of their contents. Users will probably find that using single quotes +around the whole expression and double quotes around each argument provides +more readable configurations. + + +2.3. Environment variables +-------------------------- + +HAProxy's configuration supports environment variables. Those variables are +interpreted only within double quotes. Variables are expanded during the +configuration parsing. Variable names must be preceded by a dollar ("$") and +optionally enclosed with braces ("{}") similarly to what is done in Bourne +shell. Variable names can contain alphanumerical characters or the character +underscore ("_") but should not start with a digit. If the variable contains a +list of several values separated by spaces, it can be expanded as individual +arguments by enclosing the variable with braces and appending the suffix '[*]' +before the closing brace. It is also possible to specify a default value to +use when the variable is not set, by appending that value after a dash '-' +next to the variable name. Note that the default value only replaces non +existing variables, not empty ones. + + Example: + + bind "fd@${FD_APP1}" + + log "${LOCAL_SYSLOG-127.0.0.1}:514" local0 notice # send to local server + + user "$HAPROXY_USER" + +Some variables are defined by HAProxy, they can be used in the configuration +file, or could be inherited by a program (See 3.7. Programs): + +* HAPROXY_LOCALPEER: defined at the startup of the process which contains the + name of the local peer. (See "-L" in the management guide.) + +* HAPROXY_CFGFILES: list of the configuration files loaded by HAProxy, + separated by semicolons. Can be useful in the case you specified a + directory. + +* HAPROXY_HTTP_LOG_FMT: contains the value of the default HTTP log format as + defined in section 8.2.3 "HTTP log format". It can be used to override the + default log format without having to copy the whole original definition. + + Example: + # Add the rule that gave the final verdict to the log + log-format "${HAPROXY_TCP_LOG_FMT} lr=last_rule_file:last_rule_line" + +* HAPROXY_HTTPS_LOG_FMT: similar to HAPROXY_HTTP_LOG_FMT but for HTTPS log + format as defined in section 8.2.4 "HTTPS log format". + +* HAPROXY_TCP_LOG_FMT: similar to HAPROXY_HTTP_LOG_FMT but for TCP log format + as defined in section 8.2.2 "TCP log format". + +* HAPROXY_MWORKER: In master-worker mode, this variable is set to 1. + +* HAPROXY_CLI: configured listeners addresses of the stats socket for every + processes, separated by semicolons. + +* HAPROXY_MASTER_CLI: In master-worker mode, listeners addresses of the master + CLI, separated by semicolons. + +* HAPROXY_STARTUP_VERSION: contains the version used to start, in master-worker + mode this is the version which was used to start the master, even after + updating the binary and reloading. + +* HAPROXY_BRANCH: contains the HAProxy branch version (such as "2.8"). It does + not contain the full version number. It can be useful in case of migration + if resources (such as maps or certificates) are in a path containing the + branch number. + +In addition, some pseudo-variables are internally resolved and may be used as +regular variables. Pseudo-variables always start with a dot ('.'), and are the +only ones where the dot is permitted. The current list of pseudo-variables is: + +* .FILE: the name of the configuration file currently being parsed. + +* .LINE: the line number of the configuration file currently being parsed, + starting at one. + +* .SECTION: the name of the section currently being parsed, or its type if the + section doesn't have a name (e.g. "global"), or an empty string before the + first section. + +These variables are resolved at the location where they are parsed. For example +if a ".LINE" variable is used in a "log-format" directive located in a defaults +section, its line number will be resolved before parsing and compiling the +"log-format" directive, so this same line number will be reused by subsequent +proxies. + +This way it is possible to emit information to help locate a rule in variables, +logs, error statuses, health checks, header values, or even to use line numbers +to name some config objects like servers for example. + +See also "external-check command" for other variables. + + +2.4. Conditional blocks +----------------------- + +It may sometimes be convenient to be able to conditionally enable or disable +some arbitrary parts of the configuration, for example to enable/disable SSL or +ciphers, enable or disable some pre-production listeners without modifying the +configuration, or adjust the configuration's syntax to support two distinct +versions of HAProxy during a migration.. HAProxy brings a set of nestable +preprocessor-like directives which allow to integrate or ignore some blocks of +text. These directives must be placed on their own line and they act on the +lines that follow them. Two of them support an expression, the other ones only +switch to an alternate block or end a current level. The 4 following directives +are defined to form conditional blocks: + + - .if + - .elif + - .else + - .endif + +The ".if" directive nests a new level, ".elif" stays at the same level, ".else" +as well, and ".endif" closes a level. Each ".if" must be terminated by a +matching ".endif". The ".elif" may only be placed after ".if" or ".elif", and +there is no limit to the number of ".elif" that may be chained. There may be +only one ".else" per ".if" and it must always be after the ".if" or the last +".elif" of a block. + +Comments may be placed on the same line if needed after a '#', they will be +ignored. The directives are tokenized like other configuration directives, and +as such it is possible to use environment variables in conditions. + +Conditions can also be evaluated on startup with the -cc parameter. +See "3. Starting HAProxy" in the management doc. + +The conditions are either an empty string (which then returns false), or an +expression made of any combination of: + + - the integer zero ('0'), always returns "false" + - a non-nul integer (e.g. '1'), always returns "true". + - a predicate optionally followed by argument(s) in parenthesis. + - a condition placed between a pair of parenthesis '(' and ')' + - an exclamation mark ('!') preceding any of the non-empty elements above, + and which will negate its status. + - expressions combined with a logical AND ('&&'), which will be evaluated + from left to right until one returns false + - expressions combined with a logical OR ('||'), which will be evaluated + from right to left until one returns true + +Note that like in other languages, the AND operator has precedence over the OR +operator, so that "A && B || C && D" evalues as "(A && B) || (C && D)". + +The list of currently supported predicates is the following: + + - defined() : returns true if an environment variable + exists, regardless of its contents + + - feature() : returns true if feature is listed as present + in the features list reported by "haproxy -vv" + (which means a appears after a '+') + + - streq(,) : returns true only if the two strings are equal + - strneq(,) : returns true only if the two strings differ + - strstr(,) : returns true only if the second string is found in the first one + + - version_atleast(): returns true if the current haproxy version is + at least as recent as otherwise false. The + version syntax is the same as shown by "haproxy -v" + and missing components are assumed as being zero. + + - version_before() : returns true if the current haproxy version is + strictly older than otherwise false. The + version syntax is the same as shown by "haproxy -v" + and missing components are assumed as being zero. + + - enabled() : returns true if the option is enabled at + run-time. Only a subset of options are supported: + POLL, EPOLL, KQUEUE, EVPORTS, SPLICE, + GETADDRINFO, REUSEPORT, FAST-FORWARD, + SERVER-SSL-VERIFY-NONE + +Example: + + .if defined(HAPROXY_MWORKER) + listen mwcli_px + bind :1111 + ... + .endif + + .if strneq("$SSL_ONLY",yes) + bind :80 + .endif + + .if streq("$WITH_SSL",yes) + .if feature(OPENSSL) + bind :443 ssl crt ... + .endif + .endif + + .if feature(OPENSSL) && (streq("$WITH_SSL",yes) || streq("$SSL_ONLY",yes)) + bind :443 ssl crt ... + .endif + + .if version_atleast(2.4-dev19) + profiling.memory on + .endif + + .if !feature(OPENSSL) + .alert "SSL support is mandatory" + .endif + +Four other directives are provided to report some status: + + - .diag "message" : emit this message only when in diagnostic mode (-dD) + - .notice "message" : emit this message at level NOTICE + - .warning "message" : emit this message at level WARNING + - .alert "message" : emit this message at level ALERT + +Messages emitted at level WARNING may cause the process to fail to start if the +"strict-mode" is enabled. Messages emitted at level ALERT will always cause a +fatal error. These can be used to detect some inappropriate conditions and +provide advice to the user. + +Example: + + .if "${A}" + .if "${B}" + .notice "A=1, B=1" + .elif "${C}" + .notice "A=1, B=0, C=1" + .elif "${D}" + .warning "A=1, B=0, C=0, D=1" + .else + .alert "A=1, B=0, C=0, D=0" + .endif + .else + .notice "A=0" + .endif + + .diag "WTA/2021-05-07: replace 'redirect' with 'return' after switch to 2.4" + http-request redirect location /goaway if ABUSE + + +2.5. Time format +---------------- + +Some parameters involve values representing time, such as timeouts. These +values are generally expressed in milliseconds (unless explicitly stated +otherwise) but may be expressed in any other unit by suffixing the unit to the +numeric value. It is important to consider this because it will not be repeated +for every keyword. Supported units are : + + - us : microseconds. 1 microsecond = 1/1000000 second + - ms : milliseconds. 1 millisecond = 1/1000 second. This is the default. + - s : seconds. 1s = 1000ms + - m : minutes. 1m = 60s = 60000ms + - h : hours. 1h = 60m = 3600s = 3600000ms + - d : days. 1d = 24h = 1440m = 86400s = 86400000ms + + +2.6. Size format +---------------- + +Some parameters involve values representing size, such as bandwidth limits. +These values are generally expressed in bytes (unless explicitly stated +otherwise) but may be expressed in any other unit by suffixing the unit to the +numeric value. It is important to consider this because it will not be repeated +for every keyword. Supported units are case insensitive : + + - k : kilobytes. 1 kilobyte = 1024 bytes + - m : megabytes. 1 megabyte = 1048576 bytes + - g : gigabytes. 1 gigabyte = 1073741824 bytes + +Both time and size formats require integers, decimal notation is not allowed. + + +2.7. Examples +------------- + + # Simple configuration for an HTTP proxy listening on port 80 on all + # interfaces and forwarding requests to a single backend "servers" with a + # single server "server1" listening on 127.0.0.1:8000 + global + daemon + maxconn 256 + + defaults + mode http + timeout connect 5000ms + timeout client 50000ms + timeout server 50000ms + + frontend http-in + bind *:80 + default_backend servers + + backend servers + server server1 127.0.0.1:8000 maxconn 32 + + + # The same configuration defined with a single listen block. Shorter but + # less expressive, especially in HTTP mode. + global + daemon + maxconn 256 + + defaults + mode http + timeout connect 5000ms + timeout client 50000ms + timeout server 50000ms + + listen http-in + bind *:80 + server server1 127.0.0.1:8000 maxconn 32 + + +Assuming haproxy is in $PATH, test these configurations in a shell with: + + $ sudo haproxy -f configuration.conf -c + + +3. Global parameters +-------------------- + +Parameters in the "global" section are process-wide and often OS-specific. They +are generally set once for all and do not need being changed once correct. Some +of them have command-line equivalents. + +The following keywords are supported in the "global" section : + + * Process management and security + - 51degrees-allow-unmatched + - 51degrees-cache-size + - 51degrees-data-file + - 51degrees-difference + - 51degrees-drift + - 51degrees-property-name-list + - 51degrees-property-separator + - 51degrees-use-performance-graph + - 51degrees-use-predictive-graph + - ca-base + - chroot + - cluster-secret + - cpu-map + - crt-base + - daemon + - default-path + - description + - deviceatlas-json-file + - deviceatlas-log-level + - deviceatlas-properties-cookie + - deviceatlas-separator + - expose-experimental-directives + - external-check + - fd-hard-limit + - gid + - grace + - group + - h1-accept-payload-with-any-method + - h1-case-adjust + - h1-case-adjust-file + - h2-workaround-bogus-websocket-clients + - hard-stop-after + - insecure-fork-wanted + - insecure-setuid-wanted + - issuers-chain-path + - localpeer + - log + - log-send-hostname + - log-tag + - lua-load + - lua-load-per-thread + - lua-prepend-path + - mworker-max-reloads + - nbthread + - node + - numa-cpu-mapping + - pidfile + - pp2-never-send-local + - presetenv + - prealloc-fd + - resetenv + - set-dumpable + - set-var + - setenv + - ssl-default-bind-ciphers + - ssl-default-bind-ciphersuites + - ssl-default-bind-client-sigalgs + - ssl-default-bind-curves + - ssl-default-bind-options + - ssl-default-bind-sigalgs + - ssl-default-server-ciphers + - ssl-default-server-ciphersuites + - ssl-default-server-client-sigalgs + - ssl-default-server-curves + - ssl-default-server-options + - ssl-default-server-sigalgs + - ssl-dh-param-file + - ssl-propquery + - ssl-provider + - ssl-provider-path + - ssl-server-verify + - ssl-skip-self-issued-ca + - stats + - strict-limits + - uid + - ulimit-n + - unix-bind + - unsetenv + - user + - wurfl-cache-size + - wurfl-data-file + - wurfl-information-list + - wurfl-information-list-separator + + * Performance tuning + - busy-polling + - max-spread-checks + - maxcompcpuusage + - maxcomprate + - maxconn + - maxconnrate + - maxpipes + - maxsessrate + - maxsslconn + - maxsslrate + - maxzlibmem + - no-memory-trimming + - noepoll + - noevports + - nogetaddrinfo + - nokqueue + - nopoll + - noreuseport + - nosplice + - profiling.tasks + - server-state-base + - server-state-file + - spread-checks + - ssl-engine + - ssl-mode-async + - tune.buffers.limit + - tune.buffers.reserve + - tune.bufsize + - tune.comp.maxlevel + - tune.disable-fast-forward + - tune.disable-zero-copy-forwarding + - tune.events.max-events-at-once + - tune.fail-alloc + - tune.fd.edge-triggered + - tune.h1.zero-copy-fwd-recv + - tune.h1.zero-copy-fwd-send + - tune.h2.be.initial-window-size + - tune.h2.be.max-concurrent-streams + - tune.h2.fe.initial-window-size + - tune.h2.fe.max-concurrent-streams + - tune.h2.fe.max-total-streams + - tune.h2.header-table-size + - tune.h2.initial-window-size + - tune.h2.max-concurrent-streams + - tune.h2.max-frame-size + - tune.h2.zero-copy-fwd-send + - tune.http.cookielen + - tune.http.logurilen + - tune.http.maxhdr + - tune.idle-pool.shared + - tune.idletimer + - tune.lua.forced-yield + - tune.lua.maxmem + - tune.lua.service-timeout + - tune.lua.session-timeout + - tune.lua.task-timeout + - tune.lua.log.loggers + - tune.lua.log.stderr + - tune.max-checks-per-thread + - tune.maxaccept + - tune.maxpollevents + - tune.maxrewrite + - tune.memory.hot-size + - tune.pattern.cache-size + - tune.peers.max-updates-at-once + - tune.pipesize + - tune.pool-high-fd-ratio + - tune.pool-low-fd-ratio + - tune.pt.zero-copy-forwarding + - tune.quic.frontend.conn-tx-buffers.limit + - tune.quic.frontend.max-idle-timeout + - tune.quic.frontend.max-streams-bidi + - tune.quic.max-frame-loss + - tune.quic.retry-threshold + - tune.quic.socket-owner + - tune.quic.zero-copy-fwd-send + - tune.rcvbuf.backend + - tune.rcvbuf.client + - tune.rcvbuf.frontend + - tune.rcvbuf.server + - tune.recv_enough + - tune.runqueue-depth + - tune.sched.low-latency + - tune.sndbuf.backend + - tune.sndbuf.client + - tune.sndbuf.frontend + - tune.sndbuf.server + - tune.stick-counters + - tune.ssl.cachesize + - tune.ssl.capture-buffer-size + - tune.ssl.capture-cipherlist-size (deprecated) + - tune.ssl.default-dh-param + - tune.ssl.force-private-cache + - tune.ssl.hard-maxrecord + - tune.ssl.keylog + - tune.ssl.lifetime + - tune.ssl.maxrecord + - tune.ssl.ssl-ctx-cache-size + - tune.ssl.ocsp-update.maxdelay + - tune.ssl.ocsp-update.mindelay + - tune.vars.global-max-size + - tune.vars.proc-max-size + - tune.vars.reqres-max-size + - tune.vars.sess-max-size + - tune.vars.txn-max-size + - tune.zlib.memlevel + - tune.zlib.windowsize + + * Debugging + - anonkey + - quiet + - zero-warning + + * HTTPClient + - httpclient.resolvers.disabled + - httpclient.resolvers.id + - httpclient.resolvers.prefer + - httpclient.retries + - httpclient.ssl.ca-file + - httpclient.ssl.verify + - httpclient.timeout.connect + +3.1. Process management and security +------------------------------------ + +51degrees-data-file + The path of the 51Degrees data file to provide device detection services. The + file should be unzipped and accessible by HAProxy with relevant permissions. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES. + +51degrees-property-name-list [ ...] + A list of 51Degrees property names to be load from the dataset. A full list + of names is available on the 51Degrees website: + https://51degrees.com/resources/property-dictionary + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES. + +51degrees-property-separator + A char that will be appended to every property value in a response header + containing 51Degrees results. If not set that will be set as ','. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES. + +51degrees-cache-size + Sets the size of the 51Degrees converter cache to entries. This + is an LRU cache which reminds previous device detections and their results. + By default, this cache is disabled. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES. + +51degrees-use-performance-graph { on | off } + Enables ('on') or disables ('off') the use of the performance graph in + the detection process. The default value depends on 51Degrees library. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES and 51DEGREES_VER=4. + +51degrees-use-predictive-graph { on | off } + Enables ('on') or disables ('off') the use of the predictive graph in + the detection process. The default value depends on 51Degrees library. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES and 51DEGREES_VER=4. + +51degrees-drift + Sets the drift value that a detection can allow. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES and 51DEGREES_VER=4. + +51degrees-difference + Sets the difference value that a detection can allow. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES and 51DEGREES_VER=4. + +51degrees-allow-unmatched { on | off } + Enables ('on') or disables ('off') the use of unmatched nodes in the + detection process. The default value depends on 51Degrees library. + + Please note that this option is only available when HAProxy has been + compiled with USE_51DEGREES and 51DEGREES_VER=4. + +ca-base + Assigns a default directory to fetch SSL CA certificates and CRLs from when a + relative path is used with "ca-file", "ca-verify-file" or "crl-file" + directives. Absolute locations specified in "ca-file", "ca-verify-file" and + "crl-file" prevail and ignore "ca-base". + +chroot + Changes current directory to and performs a chroot() there before + dropping privileges. This increases the security level in case an unknown + vulnerability would be exploited, since it would make it very hard for the + attacker to exploit the system. This only works when the process is started + with superuser privileges. It is important to ensure that is both + empty and non-writable to anyone. + +close-spread-time