diff options
Diffstat (limited to 'doc/management.txt')
| -rw-r--r-- | doc/management.txt | 253 |
1 files changed, 202 insertions, 51 deletions
diff --git a/doc/management.txt b/doc/management.txt index 9cbc772..d036018 100644 --- a/doc/management.txt +++ b/doc/management.txt @@ -1,7 +1,7 @@ ------------------------ HAProxy Management Guide ------------------------ - version 2.9 + version 3.0 This document describes how to start, stop, manage, and troubleshoot HAProxy, @@ -32,10 +32,12 @@ Summary 9.3. Unix Socket commands 9.4. Master CLI 9.4.1. Master CLI commands +9.5. Stats-file 10. Tricks for easier configuration management 11. Well-known traps to avoid 12. Debugging and performance issues 13. Security considerations +13.1. Linux capabilities support 1. Prerequisites @@ -49,7 +51,7 @@ familiar with troubleshooting utilities such as strace and tcpdump. 2. Quick reminder about HAProxy's architecture ---------------------------------------------- -HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means is +HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means it uses event multiplexing to schedule all of its activities instead of relying on the system to schedule between multiple activities. Most of the time it runs as a single process, so the output of "ps aux" on a system will report only one @@ -128,7 +130,7 @@ followed by one of more letters, and possibly followed by one or multiple extra arguments. Without any option, HAProxy displays the help page with a reminder about supported options. Available options may vary slightly based on the operating system. A fair number of these options overlap with an equivalent one -if the "global" section. In this case, the command line always has precedence +in the "global" section. In this case, the command line always has precedence over the configuration file, so that the command line can be used to quickly enforce some settings without touching the configuration files. The current list of options is : @@ -230,6 +232,11 @@ list of options is : getaddrinfo() exist on various systems and cause anomalies that are difficult to troubleshoot. + -dI : enable the insecure fork. This is the equivalent of the + "insecure-fork-wanted" in the global section. It can be useful when running + all the reg-tests with ASAN which need to fork addr2line to resolve the + addresses. + -dK<class[,class]*> : dumps the list of registered keywords in each class. The list of classes is available with "-dKhelp". All classes may be dumped using "-dKall", otherwise a selection of those shown in the help can be @@ -407,16 +414,20 @@ list of options is : detect protocol violations from clients or servers. An optional argument can be used to specify a list of various trace configurations using ',' as separator. Each element activates one or all trace sources. Additionally, - level and verbosity can be optionally specified on each element using ':' as - inner separator with trace name. - - -m <limit> : limit the total allocatable memory to <limit> megabytes across - all processes. This may cause some connection refusals or some slowdowns - depending on the amount of memory needed for normal operations. This is - mostly used to force the processes to work in a constrained resource usage - scenario. It is important to note that the memory is not shared between - processes, so in a multi-process scenario, this value is first divided by - global.nbproc before forking. + level and verbosity can be optionally specified on each element using ':' + as inner separator with trace name. When entering an invalid verbosity or + level name, the list of available keywords is presented. For example it can + be convenient to pass 'help' for each field to consult the list first. + + -m <limit> : limit allocatable memory, which is used to keep process's data, + to <limit> megabytes. This may cause some connection refusals or some + slowdowns depending on the amount of memory needed for normal operations. + This is mostly used to force haproxy process to work in a constrained + resource consumption scenario. It is important to note that the memory is + not shared between haproxy processes and a child process created via fork() + system call inherits its parent's resource limits. So, in a master-worker + mode this memory limit is separately applied to the master and its forked + worker process. -n <limit> : limits the per-process connection limit to <limit>. This is equivalent to the global section's keyword "maxconn". It has precedence @@ -450,7 +461,7 @@ list of options is : -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after boot completion to terminate them immediately without finishing what they were doing. <pid> is a list of pids to signal (one per argument). The list - is ends on any option starting with a "-". It is not a problem if the list + ends on any option starting with a "-". It is not a problem if the list of pids is empty, so that it can be built on the fly based on the result of a command like "pidof" or "pgrep". @@ -462,11 +473,16 @@ list of options is : -x <unix_socket> : connect to the specified socket and try to retrieve any listening sockets from the old process, and use them instead of trying to bind new ones. This is useful to avoid missing any new connection when - reloading the configuration on Linux. The capability must be enable on the - stats socket using "expose-fd listeners" in your configuration. - In master-worker mode, the master will use this option upon a reload with - the "sockpair@" syntax, which allows the master to connect directly to a - worker without using stats socket declared in the configuration. + reloading the configuration on Linux. + + Without master-worker mode, the capability must be enable on the stats + socket using "expose-fd listeners" in your configuration. + + In master-worker mode, it does not need "expose-fd listeners", the master + will use automatically this option upon a reload with the "sockpair@" + syntax, which allows the master to connect directly to a worker without using + any stats socket declared in the configuration. If you want to disable this, + you can pass -x /dev/null. A safe way to start HAProxy from an init file consists in forcing the daemon mode, storing existing pids to a pid file and using this pid file to notify @@ -1553,7 +1569,7 @@ Limitations do exist: the length of the whole buffer passed to the CLI must not be greater than tune.bfsize and the pattern "<<" must not be glued to the last word of the line. -When entering a paylod while in interactive mode, the prompt will change from +When entering a payload while in interactive mode, the prompt will change from "> " to "+ ". It is important to understand that when multiple haproxy processes are started @@ -1586,7 +1602,7 @@ abort ssl crl-file <crlfile> See also "set ssl crl-file" and "commit ssl crl-file". add acl [@<ver>] <acl> <pattern> - Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by + Add an entry into the acl <acl>. <acl> is the #<id> or the <name> returned by "show acl". This command does not verify if the entry already exists. Entries are added to the current version of the ACL, unless a specific version is specified with "@<ver>". This version number must have preliminary been @@ -1595,7 +1611,7 @@ add acl [@<ver>] <acl> <pattern> added with a specific version number will not match until a "commit acl" operation is performed on them. They may however be consulted using the "show acl @<ver>" command, and cleared using a "clear acl @<ver>" command. - This command cannot be used if the reference <acl> is a file also used with + This command cannot be used if the reference <acl> is a name also used with a map. In this case, the "add map" command must be used instead. add map [@<ver>] <map> <key> <value> @@ -1692,7 +1708,6 @@ add server <backend>/<server> [args]* - crt - disabled - downinter - - enabled - error-limit - fall - fastinter @@ -1788,15 +1803,15 @@ clear counters all and can only be issued on sockets configured for level "admin". clear acl [@<ver>] <acl> - Remove all entries from the acl <acl>. <acl> is the #<id> or the <file> - returned by "show acl". Note that if the reference <acl> is a file and is + Remove all entries from the acl <acl>. <acl> is the #<id> or the <name> + returned by "show acl". Note that if the reference <acl> is a name and is shared with a map, this map will be also cleared. By default only the current version of the ACL is cleared (the one being matched against). However it is possible to specify another version using '@' followed by this version. clear map [@<ver>] <map> - Remove all entries from the map <map>. <map> is the #<id> or the <file> - returned by "show map". Note that if the reference <map> is a file and is + Remove all entries from the map <map>. <map> is the #<id> or the <name> + returned by "show map". Note that if the reference <map> is a name and is shared with a acl, this acl will be also cleared. By default only the current version of the map is cleared (the one being matched against). However it is possible to specify another version using '@' followed by this version. @@ -1851,7 +1866,7 @@ clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ] commit acl @<ver> <acl> Commit all changes made to version <ver> of ACL <acl>, and deletes all past - versions. <acl> is the #<id> or the <file> returned by "show acl". The + versions. <acl> is the #<id> or the <name> returned by "show acl". The version number must be between "curr_ver"+1 and "next_ver" as reported in "show acl". The contents to be committed to the ACL can be consulted with "show acl @<ver> <acl>" if desired. The specified version number has normally @@ -1861,12 +1876,12 @@ commit acl @<ver> <acl> and all entries in the new version to become visible. It is also possible to use this command to perform an atomic removal of all visible entries of an ACL by calling "prepare acl" first then committing without adding any - entries. This command cannot be used if the reference <acl> is a file also + entries. This command cannot be used if the reference <acl> is a name also used as a map. In this case, the "commit map" command must be used instead. commit map @<ver> <map> Commit all changes made to version <ver> of map <map>, and deletes all past - versions. <map> is the #<id> or the <file> returned by "show map". The + versions. <map> is the #<id> or the <name> returned by "show map". The version number must be between "curr_ver"+1 and "next_ver" as reported in "show map". The contents to be committed to the map can be consulted with "show map @<ver> <map>" if desired. The specified version number has normally @@ -1903,7 +1918,7 @@ commit ssl cert <filename> Commit a temporary SSL certificate update transaction. In the case of an existing certificate (in a "Used" state in "show ssl - cert"), generate every SSL contextes and SNIs it need, insert them, and + cert"), generate every SSL contexts and SNIs it needs, insert them, and remove the previous ones. Replace in memory the previous SSL certificates everywhere the <filename> was used in the configuration. Upon failure it doesn't remove or insert anything. Once the temporary transaction is @@ -1952,16 +1967,16 @@ debug dev <command> [args]* del acl <acl> [<key>|#<ref>] Delete all the acl entries from the acl <acl> corresponding to the key <key>. - <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used, + <acl> is the #<id> or the <name> returned by "show acl". If the <ref> is used, this command delete only the listed reference. The reference can be found with - listing the content of the acl. Note that if the reference <acl> is a file and + listing the content of the acl. Note that if the reference <acl> is a name and is shared with a map, the entry will be also deleted in the map. del map <map> [<key>|#<ref>] Delete all the map entries from the map <map> corresponding to the key <key>. - <map> is the #<id> or the <file> returned by "show map". If the <ref> is used, + <map> is the #<id> or the <name> returned by "show map". If the <ref> is used, this command delete only the listed reference. The reference can be found with - listing the content of the map. Note that if the reference <map> is a file and + listing the content of the map. Note that if the reference <map> is a name and is shared with a acl, the entry will be also deleted in the map. del ssl ca-file <cafile> @@ -1992,7 +2007,7 @@ del server <backend>/<server> Remove a server attached to the backend <backend>. All servers are eligible, except servers which are referenced by other configuration elements. The server must be put in maintenance mode prior to its deletion. The operation - is cancelled if the serveur still has active or idle connection or its + is cancelled if the server still has active or idle connection or its connection queue is not empty. disable agent <backend>/<server> @@ -2060,6 +2075,10 @@ disable server <backend>/<server> This command is restricted and can only be issued on sockets configured for level "admin". +dump stats-file + Generate a stats-file which can be used to preload haproxy counters values on + startup. See "Stats-file" section for more detail. + enable agent <backend>/<server> Resume auxiliary agent check that was temporarily stopped. @@ -2142,7 +2161,7 @@ expert-mode [on|off] get map <map> <value> get acl <acl> <value> Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl> - are the #<id> or the <file> returned by "show map" or "show acl". This command + are the #<id> or the <name> returned by "show map" or "show acl". This command returns all the matching patterns associated with this map. This is useful for debugging maps and ACLs. The output format is composed by one line par matching type. Each line is composed by space-delimited series of words. @@ -2219,7 +2238,7 @@ new ssl crl-file <crlfile> prepare acl <acl> Allocate a new version number in ACL <acl> for atomic replacement. <acl> is - the #<id> or the <file> returned by "show acl". The new version number is + the #<id> or the <name> returned by "show acl". The new version number is shown in response after "New version created:". This number will then be usable to prepare additions of new entries into the ACL which will then atomically replace the current ones once committed. It is reported as @@ -2227,12 +2246,12 @@ prepare acl <acl> unused versions will automatically be removed once a more recent version is committed. Version numbers are unsigned 32-bit values which wrap at the end, so care must be taken when comparing them in an external program. This - command cannot be used if the reference <acl> is a file also used as a map. + command cannot be used if the reference <acl> is a name also used as a map. In this case, the "prepare map" command must be used instead. prepare map <map> Allocate a new version number in map <map> for atomic replacement. <map> is - the #<id> or the <file> returned by "show map". The new version number is + the #<id> or the <name> returned by "show map". The new version number is shown in response after "New version created:". This number will then be usable to prepare additions of new entries into the map which will then atomically replace the current ones once committed. It is reported as @@ -2281,7 +2300,7 @@ set anon global-key <key> set map <map> [<key>|#<ref>] <value> Modify the value corresponding to each key <key> in a map <map>. <map> is the - #<id> or <file> returned by "show map". If the <ref> is used in place of + #<id> or <name> returned by "show map". If the <ref> is used in place of <key>, only the entry pointed by <ref> is changed. The new value is <value>. set maxconn frontend <frontend> <value> @@ -2547,7 +2566,7 @@ set weight <backend>/<server> <weight>[%] show acl [[@<ver>] <acl>] Dump info about acl converters. Without argument, the list of all available acls is returned. If a <acl> is specified, its contents are dumped. <acl> is - the #<id> or <file>. By default the current version of the ACL is shown (the + the #<id> or <name>. By default the current version of the ACL is shown (the version currently being matched against and reported as 'curr_ver' in the ACL list). It is possible to instead dump other versions by prepending '@<ver>' before the ACL's identifier. The version works as a filter and non-existing @@ -2930,7 +2949,7 @@ show libs show map [[@<ver>] <map>] Dump info about map converters. Without argument, the list of all available maps is returned. If a <map> is specified, its contents are dumped. <map> is - the #<id> or <file>. By default the current version of the map is shown (the + the #<id> or <name>. By default the current version of the map is shown (the version currently being matched against and reported as 'curr_ver' in the map list). It is possible to instead dump other versions by prepending '@<ver>' before the map's identifier. The version works as a filter and non-existing @@ -3068,14 +3087,22 @@ show resolvers [<resolvers section id>] too_big: too big response outdated: number of response arrived too late (after another name server) -show quic [oneline|full] [all] +show quic [<format>] [<filter>] Dump information on all active QUIC frontend connections. This command is restricted and can only be issued on sockets configured for levels "operator" - or "admin". An optional format can be specified as first argument to control - the verbosity. Currently supported values are "oneline" which is the default - if format is unspecified or "full". By default, connections on closing or - draining state are not displayed. Use the extra argument "all" to include - them in the output. + or "admin". + + An optional argument can be specified to control the verbosity. Its value can + be interpreted in different way. The first possibility is to used predefined + values, "oneline" for the default format and "full" to display all + information. Alternatively, a list of comma-delimited fields can be specified + to restrict output. Currently supported values are "tp", "sock", "pktns", + "cc" and "mux". + + The final argument is used to restrict or extend the connection list. By + default, connections on closing or draining state are not displayed. Use the + extra argument "all" to include them in the output. It's also possible to + restrict to a single connection by specifying its hexadecimal address. show servers conn [<backend>] Dump the current and idle connections state of the servers belonging to the @@ -3998,6 +4025,37 @@ update ssl ocsp-response <certfile> local tree, its contents will be displayed on the standard output. The format is the same as the one described in "show ssl ocsp-response". +wait { -h | <delay> } [<condition> [<args>...]] + In its simplest form without any condition, this simply waits for the + requested delay before continuing. This can be used to collect metrics around + a specific interval. + + With a condition and optional arguments, the command will wait for the + specified condition to be satisfied, to unrecoverably fail, or to remain + unsatisfied for the whole <delay> duration. The supported conditions are: + + - srv-removable <proxy>/<server> : this will wait for the specified server to + be removable, i.e. be in maintenance and no longer have any connection on + it. Some conditions will never be accepted (e.g. not in maintenance) and + will cause the report of a specific error message indicating what condition + is not met. The server might even have been removed in parallel and no + longer exit. If everything is OK before the delay, a success is returned + and the operation is terminated. + + The default unit for the delay is milliseconds, though other units are + accepted if suffixed with the usual timer units (us, ms, s, m, h, d). When + used with the 'socat' utility, do not forget to extend socat's close timeout + to cover the wait time. Passing "-h" as the first or second argument provides + the command's usage. + Example: + $ socat -t20 /path/to/socket - <<< "show activity; wait 10s; show activity" + + $ socat -t5 /path/to/socket - <<< " + disable server px/srv1 + shutdown sessions server px/srv1 + wait 2s srv-removable px/srv1 + del server px/srv1" + 9.4. Master CLI --------------- @@ -4122,7 +4180,7 @@ reload return a reload status, once the reload was performed. Be careful with the timeout if a tool is used to parse it, it is only returned once the configuration is parsed and the new worker is forked. The "socat" command uses - a timeout of 0.5s by default so it will quits before showing the message if + a timeout of 0.5s by default so it will quit before showing the message if the reload is too long. "ncat" does not have a timeout by default. When compiled with USE_SHM_OPEN=1, the reload command is also able to dump the startup-logs of the master. @@ -4189,6 +4247,29 @@ show startup-logs Those messages are also dumped with the "reload" command. + +9.5. Stats-file +-------------- + +A so-called stats-file can be used to preload internal haproxy counters on +process startup with non-null values. Its main purpose is to preserve +statistics for worker processes across reloads. Only an excerpt of all the +exposed haproxy statistics is present in a stats-file as it only makes sense to +preload metric-type values. + +For the moment, only proxy counters are supported in stats-file. This allows to +preload values for frontends, backends, servers and listeners. However only +objects instances with a non-empty GUID are stored in a stats-file. This +guarantees that value will be preloaded for object with matching type and GUID, +even if other parameters differ. + +The CLI command "dump stats-file" purpose is to generate a stats-file. Format +of the stats-file is internally defined and freely subject to future changes +and extension. It is designed to be compatible at least across adjacent +haproxy stable branch releases, but may require optional extra configuration +when loading a stats-file to a process running on an older version. + + 10. Tricks for easier configuration management ---------------------------------------------- @@ -4208,7 +4289,7 @@ using regular expressions involving the dollar symbol). Environment variables also make it convenient to write configurations which are expected to work on various sites where only the address changes. It can also -permit to remove passwords from some configs. Example below where the the file +permit to remove passwords from some configs. Example below where the file "site1.env" file is sourced by the init script upon startup : $ cat site1.env @@ -4520,3 +4601,73 @@ A safe configuration will have : stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600 +13.1. Linux capabilities support +------------------------------ + +Since version v2.9 haproxy supports Linux capabilities. If the binary is +compiled with USE_LINUX_CAP=1, it is able to preserve capabilities given in +'setcap' keyword during switching from root user to a non-root. + +Since version v3.1 haproxy also checks if capabilities given in 'setcap' +keyword were set in its binary file Permitted set by administrator +(capget syscall). If this a case it performs transition of these capabilities +in its process Effective set (capset syscall), while running as a non-root +user. + +This was done to avoid all potential use cases when haproxy starts and runs as +root: transparent proxy mode, binding to privileged ports. + +'setcap' keyword supports following network capabilities: +- cap_net_admin: transparent proxying, binding socket to a specific network + interface, using set-mark action; +- cap_net_raw (subset of cap_net_admin): transparent proxying; +- cap_net_bind_service: binding socket to a specific network interface; +- cap_sys_admin: creating socket in a specific network namespace. + +Haproxy never does the transition of these capabilities from its Permitted set +to the Effective, if they are not listed as 'setcap' argument. See more +information about 'setcap' keyword and supported capabilities in the chapter +3.1 Process management and security in the Configuration guide. + +Administrator may add needed capabilities in the haproxy binary file Permitted +set with the following command: + +Example: + # setcap cap_net_admin,cap_net_bind_service=p /usr/local/sbin/haproxy + +Added capabilities will be seen in process Permitted set after its start. +If the same capabilities are the arguments of 'setcap' keyword, they could be +also seen in the process Effective set. This could be check with the following +command: + +Example: + # grep Cap /proc/<haproxy PID>/status + CapInh: 0000000000000000 + CapPrm: 0000000000001400 + CapEff: 0000000000001400 + CapBnd: 000001ffffffffff + CapAmb: 0000000000000000 + +See more details about setcap and capabilities sets in Linux man pages +(capabilities(7)). + +In some use cases like transparent proxying or creating socket in a specific +network namespace, configuration file parser detects that cap_net_raw or +cap_sys_admin or some other supported capabilities are needed. Then, during +the initialization stage, haproxy process checks, if these capabilities could +be put in its Effective set. If it's not possible due to capget or capset +syscall failure (restrictions set on syscalls by some security modules like +SELinux, Seccomp, etc), process emits diagnostic warnings (start with -dD). + +Due to support of many different platforms with different system settings, +it's impossible for the parser to deduce from the configuration file, if +binding to privileged ports will be done. So, in the case of insufficient +privileges (run as non-root) process will terminate only with an alert +message like below. It's up to a user to recheck its configuration and haproxy +binary capabilities set. + +Example: + $ haproxy -dD -f haproxy.cfg + ... + [ALERT] (96797) : Binding [haproxy.cfg:36] for frontend fe: cannot bind socket (Permission denied) for [0.0.0.0:80] + [ALERT] (96797) : [haproxy.main()] Some protocols failed to start their listeners! Exiting. |