summaryrefslogtreecommitdiffstats
path: root/doc/design-thoughts
diff options
context:
space:
mode:
Diffstat (limited to 'doc/design-thoughts')
-rw-r--r--doc/design-thoughts/backends-v0.txt27
-rw-r--r--doc/design-thoughts/backends.txt125
-rw-r--r--doc/design-thoughts/be-fe-changes.txt74
-rw-r--r--doc/design-thoughts/binding-possibilities.txt167
-rw-r--r--doc/design-thoughts/config-language.txt262
-rw-r--r--doc/design-thoughts/connection-reuse.txt224
-rw-r--r--doc/design-thoughts/connection-sharing.txt31
-rw-r--r--doc/design-thoughts/dynamic-buffers.txt41
-rw-r--r--doc/design-thoughts/entities-v2.txt276
-rw-r--r--doc/design-thoughts/how-it-works.txt60
-rw-r--r--doc/design-thoughts/http2.txt277
-rw-r--r--doc/design-thoughts/http_load_time.url5
-rw-r--r--doc/design-thoughts/pool-debugging.txt243
-rw-r--r--doc/design-thoughts/rate-shaping.txt90
-rw-r--r--doc/design-thoughts/sess_par_sec.txt13
-rw-r--r--doc/design-thoughts/thread-group.txt528
16 files changed, 2443 insertions, 0 deletions
diff --git a/doc/design-thoughts/backends-v0.txt b/doc/design-thoughts/backends-v0.txt
new file mode 100644
index 0000000..d350e22
--- /dev/null
+++ b/doc/design-thoughts/backends-v0.txt
@@ -0,0 +1,27 @@
+1 type générique "entité", avec les attributs suivants :
+
+ - frontend *f
+ - l7switch *s
+ - backend *b
+
+des types spécifiques sont simplement des entités avec certains
+de ces champs remplis et pas forcément tous :
+
+ listen = f [s] b
+ frontend = f [s]
+ l7switch = s
+ backend = [s] b
+
+Ensuite, les traitements sont évalués dans l'ordre :
+ - listen -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back, ou on travaille avec le back local.
+ - frontend -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back
+ - l7switch -> on évalue ses règles, potentiellement on branche vers d'autres listen, l7 ou backends
+ - backend -> s'il a des règles l7, on les évalue (quitte à changer encore de backend) puis on traite.
+
+Les requêtes sont traitées dans l'ordre des chaînages f->s*->b, et les réponses doivent être
+traitées dans l'ordre inverse b->s*->f. Penser aux réécritures de champs Host à l'aller et
+Location en retour.
+
+D'autre part, prévoir des "profils" plutôt que des blocs de nouveaux paramètres par défaut.
+Ca permettra d'avoir plein de jeux de paramètres par défaut à utiliser dans chacun de ces
+types.
diff --git a/doc/design-thoughts/backends.txt b/doc/design-thoughts/backends.txt
new file mode 100644
index 0000000..d2474ce
--- /dev/null
+++ b/doc/design-thoughts/backends.txt
@@ -0,0 +1,125 @@
+There has been a lot of confusion during the development because of the
+backends and frontends.
+
+What we want :
+
+- being able to still use a listener as it has always been working
+
+- being able to write a rule stating that we will *change* the backend when we
+ match some pattern. Only one jump is allowed.
+
+- being able to write a "use_filters_from XXX" line stating that we will ignore
+ any filter in the current listener, and that those from XXX will be borrowed
+ instead. A warning would be welcome for options which will silently get
+ masked. This is used to factor configuration.
+
+- being able to write a "use_backend_from XXX" line stating that we will ignore
+ any server and timeout config in the current listener, and that those from
+ XXX will be borrowed instead. A warning would be welcome for options which
+ will silently get masked. This is used to factor configuration.
+
+
+
+Example :
+---------
+
+ | # frontend HTTP/80
+ | listen fe_http 1.1.1.1:80
+ | use_filters_from default_http
+ | use_backend_from appli1
+ |
+ | # frontend HTTPS/443
+ | listen fe_https 1.1.1.1:443
+ | use_filters_from default_https
+ | use_backend_from appli1
+ |
+ | # frontend HTTP/8080
+ | listen fe_http-dev 1.1.1.1:8080
+ | reqadd "X-proto: http"
+ | reqisetbe "^Host: www1" appli1
+ | reqisetbe "^Host: www2" appli2
+ | reqisetbe "^Host: www3" appli-dev
+ | use_backend_from appli1
+ |
+ |
+ | # filters default_http
+ | listen default_http
+ | reqadd "X-proto: http"
+ | reqisetbe "^Host: www1" appli1
+ | reqisetbe "^Host: www2" appli2
+ |
+ | # filters default_https
+ | listen default_https
+ | reqadd "X-proto: https"
+ | reqisetbe "^Host: www1" appli1
+ | reqisetbe "^Host: www2" appli2
+ |
+ |
+ | # backend appli1
+ | listen appli1
+ | reqidel "^X-appli1:.*"
+ | reqadd "Via: appli1"
+ | balance roundrobin
+ | cookie app1
+ | server srv1
+ | server srv2
+ |
+ | # backend appli2
+ | listen appli2
+ | reqidel "^X-appli2:.*"
+ | reqadd "Via: appli2"
+ | balance roundrobin
+ | cookie app2
+ | server srv1
+ | server srv2
+ |
+ | # backend appli-dev
+ | listen appli-dev
+ | reqadd "Via: appli-dev"
+ | use_backend_from appli2
+ |
+ |
+
+
+Now we clearly see multiple things :
+------------------------------------
+
+ - a frontend can EITHER have filters OR reference a use_filter
+
+ - a backend can EITHER have servers OR reference a use_backend
+
+ - we want the evaluation to cross TWO levels per request. When a request is
+ being processed, it keeps track of its "frontend" side (where it came
+ from), and of its "backend" side (where the server-side parameters have
+ been found).
+
+ - the use_{filters|backend} have nothing to do with how the request is
+ decomposed.
+
+
+Conclusion :
+------------
+
+ - a proxy is always its own frontend. It also has 2 parameters :
+ - "fi_prm" : pointer to the proxy holding the filters (itself by default)
+ - "be_prm" : pointer to the proxy holding the servers (itself by default)
+
+ - a request has a frontend (fe) and a backend (be). By default, the backend
+ is initialized to the frontend. Everything related to the client side is
+ accessed through ->fe. Everything related to the server side is accessed
+ through ->be.
+
+ - request filters are first called from ->fe then ->be. Since only the
+ filters can change ->be, it is possible to iterate the filters on ->be
+ only and stop when ->be does not change anymore.
+
+ - response filters are first called from ->be then ->fe IF (fe != be).
+
+
+When we parse the configuration, we immediately configure ->fi and ->be for
+all proxies.
+
+Upon session creation, s->fe and s->be are initialized to the proxy. Filters
+are executed via s->fe->fi_prm and s->be->fi_prm. Servers are found in
+s->be->be_prm.
+
diff --git a/doc/design-thoughts/be-fe-changes.txt b/doc/design-thoughts/be-fe-changes.txt
new file mode 100644
index 0000000..f242f8a
--- /dev/null
+++ b/doc/design-thoughts/be-fe-changes.txt
@@ -0,0 +1,74 @@
+- PR_O_TRANSP => FE !!! devra peut-être changer vu que c'est un complément du mode dispatch.
+- PR_O_NULLNOLOG => FE
+- PR_O_HTTP_CLOSE => FE. !!! mettre BE aussi !!!
+- PR_O_TCP_CLI_KA => FE
+
+- PR_O_FWDFOR => BE. FE aussi ?
+- PR_O_FORCE_CLO => BE
+- PR_O_PERSIST => BE
+- PR_O_COOK_RW, PR_O_COOK_INS, PR_O_COOK_PFX, PR_O_COOK_POST => BE
+- PR_O_COOK_NOC, PR_O_COOK_IND => BE
+- PR_O_ABRT_CLOSE => BE
+- PR_O_REDISP => BE
+- PR_O_BALANCE, PR_O_BALANCE_RR, PR_O_BALANCE_SH => BE
+- PR_O_CHK_CACHE => BE
+- PR_O_TCP_SRV_KA => BE
+- PR_O_BIND_SRC => BE
+- PR_O_TPXY_MASK => BE
+
+
+- PR_MODE_TCP : BE côté serveur, FE côté client
+
+- nbconn -> fe->nbconn, be->nbconn.
+ Pb: rendre impossible le fait que (fe == be) avant de faire ça,
+ sinon on va compter les connexions en double. Ce ne sera possible
+ que lorsque les FE et BE seront des entités distinctes. On va donc
+ commencer par laisser uniquement fe->nbconn (vu que le fe ne change
+ pas), et modifier ceci plus tard, ne serait-ce que pour prendre en
+ compte correctement les minconn/maxconn.
+ => solution : avoir beconn et feconn dans chaque proxy.
+
+- failed_conns, failed_secu (réponses bloquées), failed_resp... : be
+ Attention: voir les cas de ERR_SRVCL, il semble que parfois on
+ indique ça alors qu'il y a un write error côté client (ex: ligne
+ 2044 dans proto_http).
+
+ => be et pas be->beprm
+
+- logs du backup : ->be (idem)
+
+- queue : be
+
+- logs/debug : srv toujours associé à be (ex: proxy->id:srv->id). Rien
+ pour le client pour le moment. D'une manière générale, les erreurs
+ provoquées côté serveur vont sur BE et celles côté client vont sur
+ FE.
+- logswait & LW_BYTES : FE (puisqu'on veut savoir si on logue tout de suite)
+
+- messages d'erreurs personnalisés (errmsg, ...) -> fe
+
+- monitor_uri -> fe
+- uri_auth -> (fe->firpm puis be->fiprm). Utilisation de ->be
+
+- req_add, req_exp => fe->fiprm, puis be->fiprm
+- req_cap, rsp_cap -> fe->fiprm
+- rsp_add, rsp_exp => be->fiprm, devrait être fait ensuite aussi sur fe->fiprm
+- capture_name, capture_namelen : fe->fiprm
+
+ Ce n'est pas la solution idéale, mais au moins la capture et configurable
+ par les filtres du FE et ne bouge pas lorsque le BE est réassigné. Cela
+ résoud aussi un pb d'allocation mémoire.
+
+
+- persistance (appsessions, cookiename, ...) -> be
+- stats:scope "." = fe (celui par lequel on arrive)
+ !!!ERREUR!!! => utiliser be pour avoir celui qui a été validé par
+ l'uri_auth.
+
+
+--------- corrections à effectuer ---------
+
+- remplacement de headers : parser le header et éventuellement le supprimer puis le(les) rajouter.
+- session->proto.{l4state,l7state,l7substate} pour CLI et SRV
+- errorloc : si définie dans backend, la prendre, sinon dans front.
+- logs : faire be sinon fe.
diff --git a/doc/design-thoughts/binding-possibilities.txt b/doc/design-thoughts/binding-possibilities.txt
new file mode 100644
index 0000000..3f5e432
--- /dev/null
+++ b/doc/design-thoughts/binding-possibilities.txt
@@ -0,0 +1,167 @@
+2013/10/10 - possibilities for setting source and destination addresses
+
+
+When establishing a connection to a remote device, this device is designated
+as a target, which designates an entity defined in the configuration. A same
+target appears only once in a configuration, and multiple targets may share
+the same settings if needed.
+
+The following types of targets are currently supported :
+
+ - listener : all connections with this type of target come from clients ;
+ - server : connections to such targets are for "server" lines ;
+ - peer : connections to such target address "peer" lines in "peers"
+ sections ;
+ - proxy : these targets are used by "dispatch", "option transparent"
+ or "option http_proxy" statements.
+
+A connection might not be reused between two different targets, even if all
+parameters seem similar. One of the reason is that some parameters are specific
+to the target and are not easy or not cheap to compare (eg: bind to interface,
+mss, ...).
+
+A number of source and destination addresses may be set for a given target.
+
+ - listener :
+ - the "from" address:port is set by accept()
+
+ - the "to" address:port is set if conn_get_to_addr() is called
+
+ - peer :
+ - the "from" address:port is not set
+
+ - the "to" address:port is static and dependent only on the peer
+
+ - server :
+ - the "from" address may be set alone when "source" is used with
+ a forced IP address, or when "usesrc clientip" is used.
+
+ - the "from" port may be set only combined with the address when
+ "source" is used with IP:port, IP:port-range or "usesrc client" is
+ used. Note that in this case, both the address and the port may be
+ 0, meaning that the kernel will pick the address or port and that
+ the final value might not match the one explicitly set (eg:
+ important for logging).
+
+ - the "from" address may be forced from a header which implies it
+ may change between two consecutive requests on the same connection.
+
+ - the "to" address and port are set together when connecting to a
+ regular server, or by copying the client's IP address when
+ "server 0.0.0.0" is used. Note that the destination port may be
+ an offset applied to the original destination port.
+
+ - proxy :
+ - the "from" address may be set alone when "source" is used with a
+ forced IP address or when "usesrc clientip" is used.
+
+ - the "from" port may be set only combined with the address when
+ "source" is used with IP:port or with "usesrc client". There is
+ no ip:port range for a proxy as of now. Same comment applies as
+ above when port and/or address are 0.
+
+ - the "from" address may be forced from a header which implies it
+ may change between two consecutive requests on the same connection.
+
+ - the "to" address and port are set together, either by configuration
+ when "dispatch" is used, or dynamically when "transparent" is used
+ (1:1 with client connection) or "option http_proxy" is used, where
+ each client request may lead to a different destination address.
+
+
+At the moment, there are some limits in what might happen between multiple
+concurrent requests to a same target.
+
+ - peers parameter do not change, so no problem.
+
+ - server parameters may change in this way :
+ - a connection may require a source bound to an IP address found in a
+ header, which will fall back to the "source" settings if the address
+ is not found in this header. This means that the source address may
+ switch between a dynamically forced IP address and another forced
+ IP and/or port range.
+
+ - if the element is not found (eg: header), the remaining "forced"
+ source address might very well be empty (unset), so the connection
+ reuse is acceptable when switching in that direction.
+
+ - it is not possible to switch between client and clientip or any of
+ these and hdr_ip() because they're exclusive.
+
+ - using a source address/port belonging to a port range is compatible
+ with connection reuse because there is a single range per target, so
+ switching from a range to another range means we remain in the same
+ range.
+
+ - destination address may currently not change since the only possible
+ case for dynamic destination address setting is the transparent mode,
+ reproducing the client's destination address.
+
+ - proxy parameters may change in this way :
+ - a connection may require a source bound to an IP address found in a
+ header, which will fall back to the "source" settings if the address
+ is not found in this header. This means that the source address may
+ switch between a dynamically forced IP address and another forced
+ IP and/or port range.
+
+ - if the element is not found (eg: header), the remaining "forced"
+ source address might very well be empty (unset), so the connection
+ reuse is acceptable when switching in that direction.
+
+ - it is not possible to switch between client and clientip or any of
+ these and hdr_ip() because they're exclusive.
+
+ - proxies do not support port ranges at the moment.
+
+ - destination address might change in the case where "option http_proxy"
+ is used.
+
+So, for each source element (IP, port), we want to know :
+ - if the element was assigned by static configuration (eg: ":80")
+ - if the element was assigned from a connection-specific value (eg: usesrc clientip)
+ - if the element was assigned from a configuration-specific range (eg: 1024-65535)
+ - if the element was assigned from a request-specific value (eg: hdr_ip(xff))
+ - if the element was not assigned at all
+
+For the destination, we want to know :
+ - if the element was assigned by static configuration (eg: ":80")
+ - if the element was assigned from a connection-specific value (eg: transparent)
+ - if the element was assigned from a request-specific value (eg: http_proxy)
+
+We don't need to store the information about the origin of the dynamic value
+since we have the value itself. So in practice we have :
+ - default value, unknown (not yet checked with getsockname/getpeername)
+ - default value, known (check done)
+ - forced value (known)
+ - forced range (known)
+
+We can't do that on an ip:port basis because the port may be fixed regardless
+of the address and conversely.
+
+So that means :
+
+ enum {
+ CO_ADDR_NONE = 0, /* not set, unknown value */
+ CO_ADDR_KNOWN = 1, /* not set, known value */
+ CO_ADDR_FIXED = 2, /* fixed value, known */
+ CO_ADDR_RANGE = 3, /* from assigned range, known */
+ } conn_addr_values;
+
+ unsigned int new_l3_src_status:2;
+ unsigned int new_l4_src_status:2;
+ unsigned int new_l3_dst_status:2;
+ unsigned int new_l4_dst_status:2;
+
+ unsigned int cur_l3_src_status:2;
+ unsigned int cur_l4_src_status:2;
+ unsigned int cur_l3_dsp_status:2;
+ unsigned int cur_l4_dst_status:2;
+
+ unsigned int new_family:2;
+ unsigned int cur_family:2;
+
+Note: this obsoletes CO_FL_ADDR_FROM_SET and CO_FL_ADDR_TO_SET. These flags
+must be changed to individual l3+l4 checks ORed between old and new values,
+or better, set to cur only which will inherit new.
+
+In the connection, these values may be merged in the same word as err_code.
diff --git a/doc/design-thoughts/config-language.txt b/doc/design-thoughts/config-language.txt
new file mode 100644
index 0000000..20c4fbd
--- /dev/null
+++ b/doc/design-thoughts/config-language.txt
@@ -0,0 +1,262 @@
+Prévoir des commandes en plusieurs mots clés.
+Par exemple :
+
+ timeout connection XXX
+ connection scale XXX
+
+On doit aussi accepter les préfixes :
+
+ tim co XXX
+ co sca XXX
+
+Prévoir de ranger les combinaisons dans un tableau. On doit même
+pouvoir effectuer un mapping simplifiant le parseur.
+
+
+Pour les filtres :
+
+
+ <direction> <where> <what> <operator> <pattern> <action> [ <args>* ]
+
+ <direction> = [ req | rsp ]
+ <where> = [ in | out ]
+ <what> = [ line | LINE | METH | URI | h(hdr) | H(hdr) | c(cookie) | C(cookie) ]
+ <operator> = [ == | =~ | =* | =^ | =/ | != | !~ | !* | !^ | !/ ]
+ <pattern> = "<string>"
+ <action> = [ allow | permit | deny | delete | replace | switch | add | set | redir ]
+ <args> = optional action args
+
+ examples:
+
+ req in URI =^ "/images" switch images
+ req in h(host) =* ".mydomain.com" switch mydomain
+ req in h(host) =~ "localhost(.*)" replace "www\1"
+
+ alternative :
+
+ <direction> <where> <action> [not] <what> [<operator> <pattern> [ <args>* ]]
+
+ req in switch URI =^ "/images" images
+ req in switch h(host) =* ".mydomain.com" mydomain
+ req in replace h(host) =~ "localhost(.*)" "www\1"
+ req in delete h(Connection)
+ req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
+ req out set h(Connection) "close"
+ req out add line "Server: truc"
+
+
+ <direction> <action> <where> [not] <what> [<operator> <pattern> [ <args>* ]] ';' <action2> <what2>
+
+ req in switch URI =^ "/images/" images ; replace "/"
+ req in switch h(host) =* ".mydomain.com" mydomain
+ req in replace h(host) =~ "localhost(.*)" "www\1"
+ req in delete h(Connection)
+ req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
+ req out set h(Connection) "close"
+ req out add line == "Server: truc"
+
+
+Extension avec des ACL :
+
+ req in acl(meth_valid) METH =~ "(GET|POST|HEAD|OPTIONS)"
+ req in acl(meth_options) METH == "OPTIONS"
+ req in acl(uri_slash) URI =^ "/"
+ req in acl(uri_star) URI == "*"
+
+ req in deny acl !(meth_options && uri_star || meth_valid && uri_slash)
+
+Peut-être plus simplement :
+
+ acl meth_valid METH =~ "(GET|POST|HEAD|OPTIONS)"
+ acl meth_options METH == "OPTIONS"
+ acl uri_slash URI =^ "/"
+ acl uri_star URI == "*"
+
+ req in deny not acl(meth_options uri_star, meth_valid uri_slash)
+
+ req in switch URI =^ "/images/" images ; replace "/"
+ req in switch h(host) =* ".mydomain.com" mydomain
+ req in replace h(host) =~ "localhost(.*)" "www\1"
+ req in delete h(Connection)
+ req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
+ req out set h(Connection) "close"
+ req out add line == "Server: truc"
+
+Prévoir le cas du "if" pour exécuter plusieurs actions :
+
+ req in if URI =^ "/images/" then replace "/" ; switch images
+
+Utiliser les noms en majuscules/minuscules pour indiquer si on veut prendre
+en compte la casse ou non :
+
+ if uri =^ "/watch/" setbe watch rebase "/watch/" "/"
+ if uri =* ".jpg" setbe images
+ if uri =~ ".*dll.*" deny
+ if HOST =* ".mydomain.com" setbe mydomain
+ etc...
+
+Another solution would be to have a dedicated keyword to URI remapping. It
+would both rewrite the URI and optionally switch to another backend.
+
+ uriremap "/watch/" "/" watch
+ uriremap "/chat/" "/" chat
+ uriremap "/event/" "/event/" event
+
+Or better :
+
+ uriremap "/watch/" watch "/"
+ uriremap "/chat/" chat "/"
+ uriremap "/event/" event
+
+For the URI, using a regex is sometimes useful (eg: providing a set of possible prefixes.
+
+
+Sinon, peut-être que le "switch" peut prendre un paramètre de mapping pour la partie matchée :
+
+ req in switch URI =^ "/images/" images:"/"
+
+
+2007/03/31 - Besoins plus précis.
+
+1) aucune extension de branchement ou autre dans les "listen", c'est trop complexe.
+
+Distinguer les données entrantes (in) et sortantes (out).
+
+Le frontend ne voit que les requetes entrantes et les réponses sortantes.
+Le backend voir les requêtes in/out et les réponses in/out.
+Le frontend permet les branchements d'ensembles de filtres de requêtes vers
+d'autres. Le frontend et les ensembles de filtres de requêtes peuvent brancher
+vers un backend.
+
+-----------+--------+----------+----------+---------+----------+
+ \ Where | | | | | |
+ \______ | Listen | Frontend | ReqRules | Backend | RspRules |
+ \| | | | | |
+Capability | | | | | |
+-----------+--------+----------+----------+---------+----------+
+Frontend | X | X | | | |
+-----------+--------+----------+----------+---------+----------+
+FiltReqIn | X | X | X | X | |
+-----------+--------+----------+----------+---------+----------+
+JumpFiltReq| X | X | X | | | \
+-----------+--------+----------+----------+---------+----------+ > = ReqJump
+SetBackend | X | X | X | | | /
+-----------+--------+----------+----------+---------+----------+
+FiltReqOut | | | | X | |
+-----------+--------+----------+----------+---------+----------+
+FiltRspIn | X | | | X | X |
+-----------+--------+----------+----------+---------+----------+
+JumpFiltRsp| | | | X | X |
+-----------+--------+----------+----------+---------+----------+
+FiltRspOut | | X | | X | X |
+-----------+--------+----------+----------+---------+----------+
+Backend | X | | | X | |
+-----------+--------+----------+----------+---------+----------+
+
+En conclusion
+-------------
+
+Il y a au moins besoin de distinguer 8 fonctionnalités de base :
+ - capacité à recevoir des connexions (frontend)
+ - capacité à filtrer les requêtes entrantes
+ - capacité à brancher vers un backend ou un ensemble de règles de requêtes
+ - capacité à filtrer les requêtes sortantes
+ - capacité à filtrer les réponses entrantes
+ - capacité à brancher vers un autre ensemble de règles de réponses
+ - capacité à filtrer la réponse sortante
+ - capacité à gérer des serveurs (backend)
+
+Remarque
+--------
+ - on a souvent besoin de pouvoir appliquer un petit traitement sur un ensemble
+ host/uri/autre. Le petit traitement peut consister en quelques filtres ainsi
+ qu'une réécriture du couple (host,uri).
+
+
+Proposition : ACL
+
+Syntaxe :
+---------
+
+ acl <name> <what> <operator> <value> ...
+
+Ceci créera une acl référencée sous le nom <name> qui sera validée si
+l'application d'au moins une des valeurs <value> avec l'opérateur <operator>
+sur le sujet <what> est validée.
+
+Opérateurs :
+------------
+
+Toujours 2 caractères :
+
+ [=!][~=*^%/.]
+
+Premier caractère :
+ '=' : OK si test valide
+ '!' : OK si test échoué.
+
+Second caractère :
+ '~' : compare avec une regex
+ '=' : compare chaîne à chaîne
+ '*' : compare la fin de la chaîne (ex: =* ".mydomain.com")
+ '^' : compare le début de la chaîne (ex: =^ "/images/")
+ '%' : recherche une sous-chaîne
+ '/' : compare avec un mot entier en acceptant le '/' comme délimiteur
+ '.' : compare avec un mot entier en acceptant el '.' comme délimiteur
+
+Ensuite on exécute une action de manière conditionnelle si l'ensemble des ACLs
+mentionnées sont validées (ou invalidées pour celles précédées d'un "!") :
+
+ <what> <where> <action> on [!]<aclname> ...
+
+
+Exemple :
+---------
+
+ acl www_pub host =. www www01 dev preprod
+ acl imghost host =. images
+ acl imgdir uri =/ img
+ acl imagedir uri =/ images
+ acl msie h(user-agent) =% "MSIE"
+
+ set_host "images" on www_pub imgdir
+ remap_uri "/img" "/" on www_pub imgdir
+ remap_uri "/images" "/" on www_pub imagedir
+ setbe images on imghost
+ reqdel "Cookie" on all
+
+
+
+Actions possibles :
+
+ req {in|out} {append|delete|rem|add|set|rep|mapuri|rewrite|reqline|deny|allow|setbe|tarpit}
+ resp {in|out} {append|delete|rem|add|set|rep|maploc|rewrite|stsline|deny|allow}
+
+ req in append <line>
+ req in delete <line_regex>
+ req in rem <header>
+ req in add <header> <new_value>
+ req in set <header> <new_value>
+ req in rep <header> <old_value> <new_value>
+ req in mapuri <old_uri_prefix> <new_uri_prefix>
+ req in rewrite <old_uri_regex> <new_uri>
+ req in reqline <old_req_regex> <new_req>
+ req in deny
+ req in allow
+ req in tarpit
+ req in setbe <backend>
+
+ resp out maploc <old_location_prefix> <new_loc_prefix>
+ resp out stsline <old_sts_regex> <new_sts_regex>
+
+Les chaînes doivent être délimitées par un même caractère au début et à la fin,
+qui doit être échappé s'il est présent dans la chaîne. Tout ce qui se trouve
+entre le caractère de fin et les premiers espace est considéré comme des
+options passées au traitement. Par exemple :
+
+ req in rep host /www/i /www/
+ req in rep connection /keep-alive/i "close"
+
+Il serait pratique de pouvoir effectuer un remap en même temps qu'un setbe.
+
+Captures: les séparer en in/out. Les rendre conditionnelles ?
diff --git a/doc/design-thoughts/connection-reuse.txt b/doc/design-thoughts/connection-reuse.txt
new file mode 100644
index 0000000..4eb22f6
--- /dev/null
+++ b/doc/design-thoughts/connection-reuse.txt
@@ -0,0 +1,224 @@
+2015/08/06 - server connection sharing
+
+Improvements on the connection sharing strategies
+-------------------------------------------------
+
+4 strategies are currently supported :
+ - never
+ - safe
+ - aggressive
+ - always
+
+The "aggressive" and "always" strategies take into account the fact that the
+connection has already been reused at least once or not. The principle is that
+second requests can be used to safely "validate" connection reuse on newly
+added connections, and that such validated connections may be used even by
+first requests from other sessions. A validated connection is a connection
+which has already been reused, hence proving that it definitely supports
+multiple requests. Such connections are easy to verify : after processing the
+response, if the txn already had the TX_NOT_FIRST flag, then it was not the
+first request over that connection, and it is validated as safe for reuse.
+Validated connections are put into a distinct list : server->safe_conns.
+
+Incoming requests with TX_NOT_FIRST first pick from the regular idle_conns
+list so that any new idle connection is validated as soon as possible.
+
+Incoming requests without TX_NOT_FIRST only pick from the safe_conns list for
+strategy "aggressive", guaranteeing that the server properly supports connection
+reuse, or first from the safe_conns list, then from the idle_conns list for
+strategy "always".
+
+Connections are always stacked into the list (LIFO) so that there are higher
+changes to convert recent connections and to use them. This will first optimize
+the likeliness that the connection works, and will avoid TCP metrics from being
+lost due to an idle state, and/or the congestion window to drop and the
+connection going to slow start mode.
+
+
+Handling connections in pools
+-----------------------------
+
+A per-server "pool-max" setting should be added to permit disposing unused idle
+connections not attached anymore to a session for use by future requests. The
+principle will be that attached connections are queued from the front of the
+list while the detached connections will be queued from the tail of the list.
+
+This way, most reused connections will be fairly recent and detached connections
+will most often be ignored. The number of detached idle connections in the lists
+should be accounted for (pool_used) and limited (pool_max).
+
+After some time, a part of these detached idle connections should be killed.
+For this, the list is walked from tail to head and connections without an owner
+may be evicted. It may be useful to have a per-server pool_min setting
+indicating how many idle connections should remain in the pool, ready for use
+by new requests. Conversely, a pool_low metric should be kept between eviction
+runs, to indicate the lowest amount of detached connections that were found in
+the pool.
+
+For eviction, the principle of a half-life is appealing. The principle is
+simple : over a period of time, half of the connections between pool_min and
+pool_low should be gone. Since pool_low indicates how many connections were
+remaining unused over a period, it makes sense to kill some of them.
+
+In order to avoid killing thousands of connections in one run, the purge
+interval should be split into smaller batches. Let's call N the ratio of the
+half-life interval and the effective interval.
+
+The algorithm consists in walking over them from the end every interval and
+killing ((pool_low - pool_min) + 2 * N - 1) / (2 * N). It ensures that half
+of the unused connections are killed over the half-life period, in N batches
+of population/2N entries at most.
+
+Unsafe connections should be evicted first. There should be quite few of them
+since most of them are probed and become safe. Since detached connections are
+quickly recycled and attached to a new session, there should not be too many
+detached connections in the pool, and those present there may be killed really
+quickly.
+
+Another interesting point of pools is that when a pool-max is not null, then it
+makes sense to automatically enable pretend-keep-alive on non-private connections
+going to the server in order to be able to feed them back into the pool. With
+the "aggressive" or "always" strategies, it can allow clients making a single
+request over their connection to share persistent connections to the servers.
+
+
+
+2013/10/17 - server connection management and reuse
+
+Current state
+-------------
+
+At the moment, a connection entity is needed to carry any address
+information. This means in the following situations, we need a server
+connection :
+
+- server is elected and the server's destination address is set
+
+- transparent mode is elected and the destination address is set from
+ the incoming connection
+
+- proxy mode is enabled, and the destination's address is set during
+ the parsing of the HTTP request
+
+- connection to the server fails and must be retried on the same
+ server using the same parameters, especially the destination
+ address (SN_ADDR_SET not removed)
+
+
+On the accepting side, we have further requirements :
+
+- allocate a clean connection without a stream interface
+
+- incrementally set the accepted connection's parameters without
+ clearing it, and keep track of what is set (eg: getsockname).
+
+- initialize a stream interface in established mode
+
+- attach the accepted connection to a stream interface
+
+
+This means several things :
+
+- the connection has to be allocated on the fly the first time it is
+ needed to store the source or destination address ;
+
+- the connection has to be attached to the stream interface at this
+ moment ;
+
+- it must be possible to incrementally set some settings on the
+ connection's addresses regardless of the connection's current state
+
+- the connection must not be released across connection retries ;
+
+- it must be possible to clear a connection's parameters for a
+ redispatch without having to detach/attach the connection ;
+
+- we need to allocate a connection without an existing stream interface
+
+So on the accept() side, it looks like this :
+
+ fd = accept();
+ conn = new_conn();
+ get_some_addr_info(&conn->addr);
+ ...
+ si = new_si();
+ si_attach_conn(si, conn);
+ si_set_state(si, SI_ST_EST);
+ ...
+ get_more_addr_info(&conn->addr);
+
+On the connect() side, it looks like this :
+
+ si = new_si();
+ while (!properly_connected) {
+ if (!(conn = si->end)) {
+ conn = new_conn();
+ conn_clear(conn);
+ si_attach_conn(si, conn);
+ }
+ else {
+ if (connected) {
+ f = conn->flags & CO_FL_XPRT_TRACKED;
+ conn->flags &= ~CO_FL_XPRT_TRACKED;
+ conn_close(conn);
+ conn->flags |= f;
+ }
+ if (!correct_dest)
+ conn_clear(conn);
+ }
+ set_some_addr_info(&conn->addr);
+ si_set_state(si, SI_ST_CON);
+ ...
+ set_more_addr_info(&conn->addr);
+ conn->connect();
+ if (must_retry) {
+ close_conn(conn);
+ }
+ }
+
+Note: we need to be able to set the control and transport protocols.
+On outgoing connections, this is set once we know the destination address.
+On incoming connections, this is set the earliest possible (once we know
+the source address).
+
+The problem analysed below was solved on 2013/10/22
+
+| ==> the real requirement is to know whether a connection is still valid or not
+| before deciding to close it. CO_FL_CONNECTED could be enough, though it
+| will not indicate connections that are still waiting for a connect to occur.
+| This combined with CO_FL_WAIT_L4_CONN and CO_FL_WAIT_L6_CONN should be OK.
+|
+| Alternatively, conn->xprt could be used for this, but needs some careful checks
+| (it's used by conn_full_close at least).
+|
+| Right now, conn_xprt_close() checks conn->xprt and sets it to NULL.
+| conn_full_close() also checks conn->xprt and sets it to NULL, except
+| that the check on ctrl is performed within xprt. So conn_xprt_close()
+| followed by conn_full_close() will not close the file descriptor.
+| Note that conn_xprt_close() is never called, maybe we should kill it ?
+|
+| Note: at the moment, it's problematic to leave conn->xprt to NULL before doing
+| xprt_init() because we might end up with a pending file descriptor. Or at
+| least with some transport not de-initialized. We might thus need
+| conn_xprt_close() when conn_xprt_init() fails.
+|
+| The fd should be conditioned by ->ctrl only, and the transport layer by ->xprt.
+|
+| - conn_prepare_ctrl(conn, ctrl)
+| - conn_prepare_xprt(conn, xprt)
+| - conn_prepare_data(conn, data)
+|
+| Note: conn_xprt_init() needs conn->xprt so it's not a problem to set it early.
+|
+| One problem might be with conn_xprt_close() not being able to know if xprt_init()
+| was called or not. That's where it might make sense to only set ->xprt during init.
+| Except that it does not fly with outgoing connections (xprt_init is called after
+| connect()).
+|
+| => currently conn_xprt_close() is only used by ssl_sock.c and decides whether
+| to do something based on ->xprt_ctx which is set by ->init() from xprt_init().
+| So there is nothing to worry about. We just need to restore conn_xprt_close()
+| and rely on ->ctrl to close the fd instead of ->xprt.
+|
+| => we have the same issue with conn_ctrl_close() : when is the fd supposed to be
+| valid ? On outgoing connections, the control is set much before the fd...
diff --git a/doc/design-thoughts/connection-sharing.txt b/doc/design-thoughts/connection-sharing.txt
new file mode 100644
index 0000000..99be1cd
--- /dev/null
+++ b/doc/design-thoughts/connection-sharing.txt
@@ -0,0 +1,31 @@
+2014/10/28 - Server connection sharing
+
+For HTTP/2 we'll have to use multiplexed connections to the servers and to
+share them between multiple streams. We'll also have to do this for H/1, but
+with some variations since H1 doesn't offer connection status verification.
+
+In order to validate that an idle connection is still usable, it is desirable
+to periodically send health checks over it. Normally, idle connections are
+meant to be heavily used, so there is no reason for having them idle for a long
+time. Thus we have two possibilities :
+
+ - either we time them out after some inactivity, this saves server resources ;
+ - or we check them after some inactivity. For this we can send the server-
+ side HTTP health check (only when the server uses HTTP checks), and avoid
+ using that to mark the server down, and instead consider the connection as
+ dead.
+
+For HTTP/2 we'll have to send pings periodically over these connections, so
+it's worth considering a per-connection task to validate that the channel still
+works.
+
+In the current model, a connection necessarily belongs to a session, so it's
+not really possible to share them, at best they can be exchanged, but that
+doesn't make much sense as it means that it could disturb parallel traffic.
+
+Thus we need to have a per-server list of idle connections and a max-idle-conn
+setting to kill them when there are too many. In the case of H/1 it is also
+advisable to consider that if a connection was created to pass a first non-
+idempotent request while other idle connections were still existing, then a
+connection will have to be killed in order not to exceed the limit.
+
diff --git a/doc/design-thoughts/dynamic-buffers.txt b/doc/design-thoughts/dynamic-buffers.txt
new file mode 100644
index 0000000..564d868
--- /dev/null
+++ b/doc/design-thoughts/dynamic-buffers.txt
@@ -0,0 +1,41 @@
+2014/10/30 - dynamic buffer management
+
+Since HTTP/2 processing will significantly increase the need for buffering, it
+becomes mandatory to be able to support dynamic buffer allocation. This also
+means that at any moment some buffer allocation will fail and that a task or an
+I/O operation will have to be paused for the time needed to allocate a buffer.
+
+There are 3 places where buffers are needed :
+
+ - receive side of a stream interface. A connection notifies about a pending
+ recv() and the SI calls the receive function to put the data into a buffer.
+ Here the buffer will have to be picked from a pool first, and if the
+ allocation fails, the I/O will have to temporarily be disabled, the
+ connection will have to subscribe for buffer release notification to be
+ woken up once a buffer is available again. It's important to keep in mind
+ that buffer availability doesn't necessarily mean a desire to enable recv
+ again, just that recv is not paused anymore for resource reasons.
+
+ - receive side of a stream interface when the other end point is an applet.
+ The applet wants to write into the buffer and for this the buffer needs to
+ be allocated as well. It is the same as above except that it is the applet
+ which is put to a pause. Since the applet might be at the core of the task
+ itself, it could become tricky to handle the situation correctly. Stats and
+ peers are in this situation.
+
+ - Tx of a task : some tasks perform spontaneous writes to a buffer. Checks
+ are an example of this. The checks will have to be able to sleep while a
+ buffer is being awaited.
+
+One important point is that such pauses must not prevent the task from timing
+out. There it becomes difficult because in the case of a time out, we could
+want to emit a timeout error message and for this, require a buffer. So it is
+important to keep the ability not to send messages upon error processing, and
+to be able to give up and stop waiting for buffers.
+
+The refill mechanism needs to be designed in a thread-safe way because this
+will become one of the rare cases of inter-task activity. Thus it is important
+to ensure that checking the state of the task and passing of the freshly
+released buffer are performed atomically, and that in case the task doesn't
+want it anymore, it is responsible for passing it to the next one.
+
diff --git a/doc/design-thoughts/entities-v2.txt b/doc/design-thoughts/entities-v2.txt
new file mode 100644
index 0000000..91c4fa9
--- /dev/null
+++ b/doc/design-thoughts/entities-v2.txt
@@ -0,0 +1,276 @@
+2012/07/05 - Connection layering and sequencing
+
+
+An FD has a state :
+ - CLOSED
+ - READY
+ - ERROR (?)
+ - LISTEN (?)
+
+A connection has a state :
+ - CLOSED
+ - ACCEPTED
+ - CONNECTING
+ - ESTABLISHED
+ - ERROR
+
+A stream interface has a state :
+ - INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO
+
+Note that CON and CER might be replaced by EST if the connection state is used
+instead. CON might even be more suited than EST to indicate that a connection
+is known.
+
+
+si_shutw() must do :
+
+ data_shutw()
+ if (shutr) {
+ data_close()
+ ctrl_shutw()
+ ctrl_close()
+ }
+
+si_shutr() must do :
+ data_shutr()
+ if (shutw) {
+ data_close()
+ ctrl_shutr()
+ ctrl_close()
+ }
+
+Each of these steps may fail, in which case the step must be retained and the
+operations postponed in an asynchronous task.
+
+The first asynchronous data_shut() might already fail so it is mandatory to
+save the other side's status with the connection in order to let the async task
+know whether the 3 next steps must be performed.
+
+The connection (or perhaps the FD) needs to know :
+ - the desired close operations : DSHR, DSHW, CSHR, CSHW
+ - the completed close operations : DSHR, DSHW, CSHR, CSHW
+
+
+On the accept() side, we probably need to know :
+ - if a header is expected (eg: accept-proxy)
+ - if this header is still being waited for
+ => maybe both info might be combined into one bit
+
+ - if a data-layer accept() is expected
+ - if a data-layer accept() has been started
+ - if a data-layer accept() has been performed
+ => possibly 2 bits, to indicate the need to free()
+
+On the connect() side, we need to know :
+ - the desire to send a header (eg: send-proxy)
+ - if this header has been sent
+ => maybe both info might be combined
+
+ - if a data-layer connect() is expected
+ - if a data-layer connect() has been started
+ - if a data-layer connect() has been completed
+ => possibly 2 bits, to indicate the need to free()
+
+On the response side, we also need to know :
+ - the desire to send a header (eg: health check response for monitor-net)
+ - if this header was sent
+ => might be the same as sending a header over a new connection
+
+Note: monitor-net has precedence over proxy proto and data layers. Same for
+ health mode.
+
+For multi-step operations, use 2 bits :
+ 00 = operation not desired, not performed
+ 10 = operation desired, not started
+ 11 = operation desired, started but not completed
+ 01 = operation desired, started and completed
+
+ => X != 00 ==> operation desired
+ X & 01 ==> operation at least started
+ X & 10 ==> operation not completed
+
+Note: no way to store status information for error reporting.
+
+Note2: it would be nice if "tcp-request connection" rules could work at the
+connection level, just after headers ! This means support for tracking stick
+tables, possibly not too much complicated.
+
+
+Proposal for incoming connection sequence :
+
+- accept()
+- if monitor-net matches or if mode health => try to send response
+- if accept-proxy, wait for proxy request
+- if tcp-request connection, process tcp rules and possibly keep the
+ pointer to stick-table
+- if SSL is enabled, switch to SSL handshake
+- then switch to DATA state and instantiate a session
+
+We just need a map of handshake handlers on the connection. They all manage the
+FD status themselves and set the callbacks themselves. If their work succeeds,
+they remove themselves from the list. If it fails, they remain subscribed and
+enable the required polling until they are woken up again or the timeout strikes.
+
+Identified handshake handlers for incoming connections :
+ - HH_HEALTH (tries to send OK and dies)
+ - HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK)
+ - HH_SEND_OK (tries to send "OK" and dies)
+ - HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies)
+ - HH_ACCEPT_PROXY (waits for PROXY line and parses it)
+ - HH_TCP_RULES (processes TCP rules)
+ - HH_SSL_HS (starts SSL handshake)
+ - HH_ACCEPT_SESSION (instantiates a session)
+
+Identified handshake handlers for outgoing connections :
+ - HH_SEND_PROXY (tries to build and send the PROXY line)
+ - HH_SSL_HS (starts SSL handshake)
+
+For the pollers, we could check that handshake handlers are not 0 and decide to
+call a generic connection handshake handler instead of usual callbacks. Problem
+is that pollers don't know connections, they know fds. So entities which manage
+handlers should update change the FD callbacks accordingly.
+
+With a bit of care, we could have :
+ - HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies)
+ => merges HEALTH, SEND_OK and SEND_HTTP_OK
+
+It sounds like the ctrl vs data state for the connection are per-direction
+(eg: support an async ctrl shutw while still reading data).
+
+Also support shutr/shutw status at L4/L7.
+
+In practice, what we really need is :
+
+shutdown(conn) =
+ conn.data.shut()
+ conn.ctrl.shut()
+ conn.fd.shut()
+
+close(conn) =
+ conn.data.close()
+ conn.ctrl.close()
+ conn.fd.close()
+
+With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have :
+
+ HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection
+
+The connection has to be closed at 3 places after a successful response :
+ - DATA (RSSL over RTCP)
+ - CTRL (RTCP to close connection to server)
+ - SOCK (FD to close connection to second process)
+
+Externally, the connection is seen with very few flags :
+ - SHR
+ - SHW
+ - ERR
+
+We don't need a CLOSED flag as a connection must always be detached when it's closed.
+
+The internal status doesn't need to be exposed :
+ - FD allocated (Y/N)
+ - CTRL initialized (Y/N)
+ - CTRL connected (Y/N)
+ - CTRL handlers done (Y/N)
+ - CTRL failed (Y/N)
+ - CTRL shutr (Y/N)
+ - CTRL shutw (Y/N)
+ - DATA initialized (Y/N)
+ - DATA connected (Y/N)
+ - DATA handlers done (Y/N)
+ - DATA failed (Y/N)
+ - DATA shutr (Y/N)
+ - DATA shutw (Y/N)
+
+(note that having flags for operations needing to be completed might be easier)
+--------------
+
+Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds
+very unlikely since the only functions manipulating this are in the code of
+the data/ctrl handlers.
+
+FDSET/FDCLR cannot be directly controlled by the stream interface since it also
+depends on the DATA layer (WANT_READ/WANT_WRITE).
+
+But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA).
+
+Example: an SSL conn relies on an FD. The buffer is full, and wants the conn to
+stop reading. It must not stop the FD itself. It is the read function which
+should notice that it has nothing to do with a read wake-up, which needs to
+disable reading.
+
+Conversely, when calling conn->chk_rcv(), the reader might get a WANT_READ or
+even WANT_WRITE and adjust the FDs accordingly.
+
+------------------------
+
+OK, the problem is simple : we don't manipulate the FD at the right level.
+We should have :
+ ->connect(), ->chk_snd(), ->chk_rcv(), ->shutw(), ->shutr() which are
+ called from the upper layer (buffer)
+ ->recv(), ->send(), called from the lower layer
+
+Note that the SHR is *reported* by lower layer but can be forced by upper
+layer. In this case it's like a delayed abort. The difficulty consists in
+knowing the output data were correctly read. Probably we'd need to drain
+incoming data past the active shutr().
+
+The only four purposes of the top-down shutr() call are :
+ - acknowledge a shut read report : could probably be done better
+ - read timeout => disable reading : it's a delayed abort. We want to
+ report that the buffer is SHR, maybe even the connection, but the
+ FD clearly isn't.
+ - read abort due to error on the other side or desire to close (eg:
+ http-server-close) : delayed abort
+ - complete abort
+
+The active shutr() is problematic as we can't disable reading if we expect some
+exchanges for data acknowledgement. We probably need to drain data only until
+the shutw() has been performed and ACKed.
+
+A connection shut down for read would behave like this :
+
+ 1) bidir exchanges
+
+ 2) shutr() => read_abort_pending=1
+
+ 3) drain input, still send output
+
+ 4) shutw()
+
+ 5) drain input, wait for read0 or ack(shutw)
+
+ 6) close()
+
+--------------------- 2012/07/05 -------------------
+
+Communications must be performed this way :
+
+ connection <-> channel <-> connection
+
+A channel is composed of flags and stats, and may store data in either a buffer
+or a pipe. We need low-layer operations between sockets and buffers or pipes.
+Right now we only support sockets, but later we might support remote sockets
+and maybe pipes or shared memory segments.
+
+So we need :
+
+ - raw_sock_to_buf() => receive raw data from socket into buffer
+ - raw_sock_to_pipe => receive raw data from socket into pipe (splice in)
+ - raw_sock_from_buf() => send raw data from buffer to socket
+ - raw_sock_from_pipe => send raw data from pipe to socket (splice out)
+
+ - ssl_sock_to_buf() => receive ssl data from socket into buffer
+ - ssl_sock_to_pipe => receive ssl data from socket into a pipe (NULL)
+ - ssl_sock_from_buf() => send ssl data from buffer to socket
+ - ssl_sock_from_pipe => send ssl data from pipe to socket (NULL)
+
+These functions should set such status flags :
+
+#define ERR_IN 0x01
+#define ERR_OUT 0x02
+#define SHUT_IN 0x04
+#define SHUT_OUT 0x08
+#define EMPTY_IN 0x10
+#define FULL_OUT 0x20
+
diff --git a/doc/design-thoughts/how-it-works.txt b/doc/design-thoughts/how-it-works.txt
new file mode 100644
index 0000000..2d1cb89
--- /dev/null
+++ b/doc/design-thoughts/how-it-works.txt
@@ -0,0 +1,60 @@
+How it works ? (unfinished and inexact)
+
+For TCP and HTTP :
+
+- listeners create listening sockets with a READ callback pointing to the
+ protocol-specific accept() function.
+
+- the protocol-specific accept() function then accept()'s the connection and
+ instantiates a "server TCP socket" (which is dedicated to the client side),
+ and configures it (non_block, get_original_dst, ...).
+
+For TCP :
+- in case of pure TCP, a request buffer is created, as well as a "client TCP
+ socket", which tries to connect to the server.
+
+- once the connection is established, the response buffer is allocated and
+ connected to both ends.
+
+- both sockets are set to "autonomous mode" so that they only wake up their
+ supervising session when they encounter a special condition (error or close).
+
+
+For HTTP :
+- in case of HTTP, a request buffer is created with the "HOLD" flag set and
+ a read limit to support header rewriting (may be this one will be removed
+ eventually because it's better to limit only to the buffer size and report
+ an error when rewritten data overflows)
+
+- a "flow analyzer" is attached to the buffer (or possibly multiple flow
+ analyzers). For the request, the flow analyzer is "http_lb_req". The flow
+ analyzer is a function which gets called when new data is present and
+ blocked. It has a timeout (request timeout). It can also be bypassed on
+ demand.
+
+- when the "http_lb_req" has received the whole request, it creates a client
+ socket with all the parameters needed to try to connect to the server. When
+ the connection establishes, the response buffer is allocated on the fly,
+ put to HOLD mode, and a an "http_lb_resp" flow analyzer is attached to the
+ buffer.
+
+
+For client-side HTTPS :
+
+- the accept() function must completely instantiate a TCP socket + an SSL
+ reader. It is when the SSL session is complete that we call the
+ protocol-specific accept(), and create its buffer.
+
+
+
+
+Conclusions
+-----------
+
+- we need a generic TCP accept() function with a lot of flags set by the
+ listener, to tell it what info we need to get at the accept() time, and
+ what flags will have to be set on the socket.
+
+- once the TCP accept() function ends, it wakes up the protocol supervisor
+ which is in charge of creating the buffers, etc, switch states, etc...
+
diff --git a/doc/design-thoughts/http2.txt b/doc/design-thoughts/http2.txt
new file mode 100644
index 0000000..c21ac10
--- /dev/null
+++ b/doc/design-thoughts/http2.txt
@@ -0,0 +1,277 @@
+2014/10/23 - design thoughts for HTTP/2
+
+- connections : HTTP/2 depends a lot more on a connection than HTTP/1 because a
+ connection holds a compression context (headers table, etc...). We probably
+ need to have an h2_conn struct.
+
+- multiple transactions will be handled in parallel for a given h2_conn. They
+ are called streams in HTTP/2 terminology.
+
+- multiplexing : for a given client-side h2 connection, we can have multiple
+ server-side h2 connections. And for a server-side h2 connection, we can have
+ multiple client-side h2 connections. Streams circulate in N-to-N fashion.
+
+- flow control : flow control will be applied between multiple streams. Special
+ care must be taken so that an H2 client cannot block some H2 servers by
+ sending requests spread over multiple servers to the point where one server
+ response is blocked and prevents other responses from the same server from
+ reaching their clients. H2 connection buffers must always be empty or nearly
+ empty. The per-stream flow control needs to be respected as well as the
+ connection's buffers. It is important to implement some fairness between all
+ the streams so that it's not always the same which gets the bandwidth when
+ the connection is congested.
+
+- some clients can be H1 with an H2 server (is this really needed ?). Most of
+ the initial use case will be H2 clients to H1 servers. It is important to keep
+ in mind that H1 servers do not do flow control and that we don't want them to
+ block transfers (eg: post upload).
+
+- internal tasks : some H2 clients will be internal tasks (eg: health checks).
+ Some H2 servers will be internal tasks (eg: stats, cache). The model must be
+ compatible with this use case.
+
+- header indexing : headers are transported compressed, with a reference to a
+ static or a dynamic header, or a literal, possibly huffman-encoded. Indexing
+ is specific to the H2 connection. This means there is no way any binary data
+ can flow between both sides, headers will have to be decoded according to the
+ incoming connection's context and re-encoded according to the outgoing
+ connection's context, which can significantly differ. In order to avoid the
+ parsing trouble we currently face, headers will have to be clearly split
+ between name and value. It is worth noting that neither the incoming nor the
+ outgoing connections' contexts will be of any use while processing the
+ headers. At best we can have some shortcuts for well-known names that map
+ well to the static ones (eg: use the first static entry with same name), and
+ maybe have a few special cases for static name+value as well. Probably we can
+ classify headers in such categories :
+
+ - static name + value
+ - static name + other value
+ - dynamic name + other value
+
+ This will allow for better processing in some specific cases. Headers
+ supporting a single value (:method, :status, :path, ...) should probably
+ be stored in a single location with a direct access. That would allow us
+ to retrieve a method using hdr[METHOD]. All such indexing must be performed
+ while parsing. That also means that HTTP/1 will have to be converted to this
+ representation very early in the parser and possibly converted back to H/1
+ after processing.
+
+ Header names/values will have to be placed in a small memory area that will
+ inevitably get fragmented as headers are rewritten. An automatic packing
+ mechanism must be implemented so that when there's no more room, headers are
+ simply defragmented/packet to a new table and the old one is released. Just
+ like for the static chunks, we need to have a few such tables pre-allocated
+ and ready to be swapped at any moment. Repacking must not change any index
+ nor affect the way headers are compressed so that it can happen late after a
+ retry (send-name-header for example).
+
+- header processing : can still happen on a (header, value) basis. Reqrep/
+ rsprep completely disappear and will have to be replaced with something else
+ to support renaming headers and rewriting url/path/...
+
+- push_promise : servers can push dummy requests+responses. They advertise
+ the stream ID in the push_promise frame indicating the associated stream ID.
+ This means that it is possible to initiate a client-server stream from the
+ information coming from the server and make the data flow as if the client
+ had made it. It's likely that we'll have to support two types of server
+ connections: those which support push and those which do not. That way client
+ streams will be distributed to existing server connections based on their
+ capabilities. It's important to keep in mind that PUSH will not be rewritten
+ in responses.
+
+- stream ID mapping : since the stream ID is per H2 connection, stream IDs will
+ have to be mapped. Thus a given stream is an entity with two IDs (one per
+ side). Or more precisely a stream has two end points, each one carrying an ID
+ when it ends on an HTTP2 connection. Also, for each stream ID we need to
+ quickly find the associated transaction in progress. Using a small quick
+ unique tree seems indicated considering the wide range of valid values.
+
+- frame sizes : frame have to be remapped between both sides as multiplexed
+ connections won't always have the same characteristics. Thus some frames
+ might be spliced and others will be sliced.
+
+- error processing : care must be taken to never break a connection unless it
+ is dead or corrupt at the protocol level. Stats counter must exist to observe
+ the causes. Timeouts are a great problem because silent connections might
+ die out of inactivity. Ping frames should probably be scheduled a few seconds
+ before the connection timeout so that an unused connection is verified before
+ being killed. Abnormal requests must be dealt with using RST_STREAM.
+
+- ALPN : ALPN must be observed on the client side, and transmitted to the server
+ side.
+
+- proxy protocol : proxy protocol makes little to no sense in a multiplexed
+ protocol. A per-stream equivalent will surely be needed if implementations
+ do not quickly generalize the use of Forward.
+
+- simplified protocol for local devices (eg: haproxy->varnish in clear and
+ without handshake, and possibly even with splicing if the connection's
+ settings are shared)
+
+- logging : logging must report a number of extra information such as the
+ stream ID, and whether the transaction was initiated by the client or by the
+ server (which can be deduced from the stream ID's parity). In case of push,
+ the number of the associated stream must also be reported.
+
+- memory usage : H2 increases memory usage by mandating use of 16384 bytes
+ frame size minimum. That means slightly more than 16kB of buffer in each
+ direction to process any frame. It will definitely have an impact on the
+ deployed maxconn setting in places using less than this (4..8kB are common).
+ Also, the header list is persistent per connection, so if we reach the same
+ size as the request, that's another 16kB in each direction, resulting in
+ about 48kB of memory where 8 were previously used. A more careful encoder
+ can work with a much smaller set even if that implies evicting entries
+ between multiple headers of the same message.
+
+- HTTP/1.0 should very carefully be transported over H2. Since there's no way
+ to pass version information in the protocol, the server could use some
+ features of HTTP/1.1 that are unsafe in HTTP/1.0 (compression, trailers,
+ ...).
+
+- host / :authority : ":authority" is the norm, and "host" will be absent when
+ H2 clients generate :authority. This probably means that a dummy Host header
+ will have to be produced internally from :authority and removed when passing
+ to H2 behind. This can cause some trouble when passing H2 requests to H1
+ proxies, because there's no way to know if the request should contain scheme
+ and authority in H1 or not based on the H2 request. Thus a "proxy" option
+ will have to be explicitly mentioned on HTTP/1 server lines. One of the
+ problem that it creates is that it's not longer possible to pass H/1 requests
+ to H/1 proxies without an explicit configuration. Maybe a table of the
+ various combinations is needed.
+
+ :scheme :authority host
+ HTTP/2 request present present absent
+ HTTP/1 server req absent absent present
+ HTTP/1 proxy req present present present
+
+ So in the end the issue is only with H/2 requests passed to H/1 proxies.
+
+- ping frames : they don't indicate any stream ID so by definition they cannot
+ be forwarded to any server. The H2 connection should deal with them only.
+
+There's a layering problem with H2. The framing layer has to be aware of the
+upper layer semantics. We can't simply re-encode HTTP/1 to HTTP/2 then pass
+it over a framing layer to mux the streams, the frame type must be passed below
+so that frames are properly arranged. Header encoding is connection-based and
+all streams using the same connection will interact in the way their headers
+are encoded. Thus the encoder *has* to be placed in the h2_conn entity, and
+this entity has to know for each stream what its headers are.
+
+Probably that we should remove *all* headers from transported data and move
+them on the fly to a parallel structure that can be shared between H1 and H2
+and consumed at the appropriate level. That means buffers only transport data.
+Trailers have to be dealt with differently.
+
+So if we consider an H1 request being forwarded between a client and a server,
+it would look approximately like this :
+
+ - request header + body land into a stream's receive buffer
+ - headers are indexed and stripped out so that only the body and whatever
+ follows remain in the buffer
+ - both the header index and the buffer with the body stay attached to the
+ stream
+ - the sender can rebuild the whole headers. Since they're found in a table
+ supposed to be stable, it can rebuild them as many times as desired and
+ will always get the same result, so it's safe to build them into the trash
+ buffer for immediate sending, just as we do for the PROXY protocol.
+ - the upper protocol should probably provide a build_hdr() callback which
+ when called by the socket layer, builds this header block based on the
+ current stream's header list, ready to be sent.
+ - the socket layer has to know how many bytes from the headers are left to be
+ forwarded prior to processing the body.
+ - the socket layer needs to consume only the acceptable part of the body and
+ must not release the buffer if any data remains in it (eg: pipelining over
+ H1). This is already handled by channel->o and channel->to_forward.
+ - we could possibly have another optional callback to send a preamble before
+ data, that could be used to send chunk sizes in H1. The danger is that it
+ absolutely needs to be stable if it has to be retried. But it could
+ considerably simplify de-chunking.
+
+When the request is sent to an H2 server, an H2 stream request must be made
+to the server, we find an existing connection whose settings are compatible
+with our needs (eg: tls/clear, push/no-push), and with a spare stream ID. If
+none is found, a new connection must be established, unless maxconn is reached.
+
+Servers must have a maxstream setting just like they have a maxconn. The same
+queue may be used for that.
+
+The "tcp-request content" ruleset must apply to the TCP layer. But with HTTP/2
+that becomes impossible (and useless). We still need something like the
+"tcp-request session" hook to apply just after the SSL handshake is done.
+
+It is impossible to defragment the body on the fly in HTTP/2. Since multiple
+messages are interleaved, we cannot wait for all of them and block the head of
+line. Thus if body analysis is required, it will have to use the stream's
+buffer, which necessarily implies a copy. That means that with each H2 end we
+necessarily have at least one copy. Sometimes we might be able to "splice" some
+bytes from one side to the other without copying into the stream buffer (same
+rules as for TCP splicing).
+
+In theory, only data should flow through the channel buffer, so each side's
+connector is responsible for encoding data (H1: linear/chunks, H2: frames).
+Maybe the same mechanism could be extrapolated to tunnels / TCP.
+
+Since we'd use buffers only for data (and for receipt of headers), we need to
+have dynamic buffer allocation.
+
+Thus :
+- Tx buffers do not exist. We allocate a buffer on the fly when we're ready to
+ send something that we need to build and that needs to be persistent in case
+ of partial send. H1 headers are built on the fly from the header table to a
+ temporary buffer that is immediately sent and whose amount of sent bytes is
+ the only information kept (like for PROXY protocol). H2 headers are more
+ complex since the encoding depends on what was successfully sent. Thus we
+ need to build them and put them into a temporary buffer that remains
+ persistent in case send() fails. It is possible to have a limited pool of
+ Tx buffers and refrain from sending if there is no more buffer available in
+ the pool. In that case we need a wake-up mechanism once a buffer is
+ available. Once the data are sent, the Tx buffer is then immediately recycled
+ in its pool. Note that no tx buffer being used (eg: for hdr or control) means
+ that we have to be able to serialize access to the connection and retry with
+ the same stream. It also means that a stream that times out while waiting for
+ the connector to read the second half of its request has to stay there, or at
+ least needs to be handled gracefully. However if the connector cannot read
+ the data to be sent, it means that the buffer is congested and the connection
+ is dead, so that probably means it can be killed.
+
+- Rx buffers have to be pre-allocated just before calling recv(). A connection
+ will first try to pick a buffer and disable reception if it fails, then
+ subscribe to the list of tasks waiting for an Rx buffer.
+
+- full Rx buffers might sometimes be moved around to the next buffer instead of
+ experiencing a copy. That means that channels and connectors must use the
+ same format of buffer, and that only the channel will have to see its
+ pointers adjusted.
+
+- Tx of data should be made as much as possible without copying. That possibly
+ means by directly looking into the connection buffer on the other side if
+ the local Tx buffer does not exist and the stream buffer is not allocated, or
+ even performing a splice() call between the two sides. One of the problem in
+ doing this is that it requires proper ordering of the operations (eg: when
+ multiple readers are attached to a same buffer). If the splitting occurs upon
+ receipt, there's no problem. If we expect to retrieve data directly from the
+ original buffer, it's harder since it contains various things in an order
+ which does not even indicate what belongs to whom. Thus possibly the only
+ mechanism to implement is the buffer permutation which guarantees zero-copy
+ and only in the 100% safe case. Also it's atomic and does not cause HOL
+ blocking.
+
+It makes sense to chose the frontend_accept() function right after the
+handshake ended. It is then possible to check the ALPN, the SNI, the ciphers
+and to accept to switch to the h2_conn_accept handler only if everything is OK.
+The h2_conn_accept handler will have to deal with the connection setup,
+initialization of the header table, exchange of the settings frames and
+preparing whatever is needed to fire new streams upon receipt of unknown
+stream IDs. Note: most of the time it will not be possible to splice() because
+we need to know in advance the amount of bytes to write the header, and here it
+will not be possible.
+
+H2 health checks must be seen as regular transactions/streams. The check runs a
+normal client which seeks an available stream from a server. The server then
+finds one on an existing connection or initiates a new H2 connection. The H2
+checks will have to be configurable for sharing streams or not. Another option
+could be to specify how many requests can be made over existing connections
+before insisting on getting a separate connection. Note that such separate
+connections might end up stacking up once released. So probably that they need
+to be recycled very quickly (eg: fix how many unused ones can exist max).
+
diff --git a/doc/design-thoughts/http_load_time.url b/doc/design-thoughts/http_load_time.url
new file mode 100644
index 0000000..f178e46
--- /dev/null
+++ b/doc/design-thoughts/http_load_time.url
@@ -0,0 +1,5 @@
+Excellent paper about page load time for keepalive on/off, pipelining,
+multiple host names, etc...
+
+http://www.die.net/musings/page_load_time/
+
diff --git a/doc/design-thoughts/pool-debugging.txt b/doc/design-thoughts/pool-debugging.txt
new file mode 100644
index 0000000..106e41c
--- /dev/null
+++ b/doc/design-thoughts/pool-debugging.txt
@@ -0,0 +1,243 @@
+2022-02-22 - debugging options with pools
+
+Two goals:
+ - help developers spot bugs as early as possible
+
+ - make the process more reliable in field, by killing sick ones as soon as
+ possible instead of letting them corrupt data, cause trouble, or even be
+ exploited.
+
+An allocated object may exist in 5 forms:
+ - in use: currently referenced and used by haproxy, 100% of its size are
+ dedicated to the application which can do absolutely anything with it,
+ but it may never touch anything before nor after that area.
+
+ - in cache: the object is neither referenced nor used anymore, but it sits
+ in a thread's cache. The application may not touch it at all anymore, and
+ some parts of it could even be unmapped. Only the current thread may safely
+ reach it, though others might find/release it when under thread isolation.
+ The thread cache needs some LRU linking that may be stored anywhere, either
+ inside the area, or outside. The parts surrounding the <size> parts remain
+ invisible to the application layer, and can serve as a protection.
+
+ - in shared cache: the object is neither referenced nor used anymore, but it
+ may be reached by any thread. Some parts of it could be unmapped. Any
+ thread may pick it but only one may find it, hence once grabbed, it is
+ guaranteed no other one will find it. The shared cache needs to set up a
+ linked list and a single pointer needs to be stored anywhere, either inside
+ or outside the area. The parts surrounding the <size> parts remain
+ invisible to the application layer, and can serve as a protection.
+
+ - in the system's memory allocator: the object is not known anymore from
+ haproxy. It may be reassigned in parts or totally to other pools or other
+ subsystems (e.g. crypto library). Some or all of it may be unmapped. The
+ areas surrounding the <size> parts are also part of the object from the
+ library's point of view and may be delivered to other areas. Tampering
+ with these may cause any other part to malfunction in dirty ways.
+
+ - in the OS only: the memory allocator gave it back to the OS.
+
+The following options need to be configurable:
+ - detect improper initialization: this is done by poisonning objects before
+ delivering them to the application.
+
+ - help figure where an object was allocated when in use: a pointer to the
+ call place will help. Pointing to the last pool_free() as well for the
+ same reasons when dealing with a UAF.
+
+ - detection of wrong pointer/pool when in use: a pointer to the pool before
+ or after the area will definitely help.
+
+ - detection of overflows when in use: a canary at the end of the area
+ (closest possible to <size>) will definitely help. The pool above can do
+ that job. Ideally, we should fill some data at the end so that even
+ unaligned sizes can be checked (e.g. a buffer that gets a zero appended).
+ If we just align on 2 pointers, writing the same pointer twice at the end
+ may do the job, but we won't necessarily have our bytes. Thus a particular
+ end-of-string pattern would be useful (e.g. ff55aa01) to fill it.
+
+ - detection of double free when in cache: similar to detection of wrong
+ pointer/pool when in use: the pointer at the end may simply be changed so
+ that it cannot match the pool anymore. By using a pointer to the caller of
+ the previous free() operation, we have the guarantee to see different
+ pointers, and this pointer can be inspected to figure where the object was
+ previously freed. An extra check may even distinguish a perfect double-free
+ (same caller) from just a wrong free (pointer differs from pool).
+
+ - detection of late corruption when in cache: keeping a copy of the
+ checksum of the whole area upon free() will do the job, but requires one
+ extra storage area for the checksum. Filling the area with a pattern also
+ does the job and doesn't require extra storage, but it loses the contents
+ and can be a bit slower. Sometimes losing the contents can be a feature,
+ especially when trying to detect late reads. Probably that both need to
+ be implemented. Note that if contents are not strictly needed, storing a
+ checksum inside the area does the job.
+
+ - preserve total contents in cache for debugging: losing some precious
+ information can be a problem.
+
+ - pattern filling of the area helps detect use-after-free in read-only mode.
+
+ - allocate cold first helps with both cases above.
+
+Uncovered:
+ - overflow/underflow when in cache/shared/libc: it belongs to use-after-free
+ pattern and such an error during regular use ought to be caught while the
+ object was still in use.
+
+ - integrity when in libc: not under our control anymore, this is a libc
+ problem.
+
+Arbitrable:
+ - integrity when in shared cache: unlikely to happen only then if it could
+ have happened in the local cache. Shared cache not often used anymore, thus
+ probably not worth the effort
+
+ - protection against double-free when in shared cache/libc: might be done for
+ a cheap price, probably worth being able to quickly tell that such an
+ object left the local cache (e.g. the mark points to the caller, but could
+ possibly just be incremented, hence still point to the same code location+1
+ byte when released. Calls are 4 bytes min on RISC, 5 on x86 so we do have
+ some margin by having a caller's location be +0,+1,+2 or +3.
+
+ - underflow when in use: hasn't been really needed over time but may change.
+
+ - detection of late corruption when in shared cache: checksum or area filling
+ are possible, but is this as relevant as it used to considering the less
+ common use of the shared cache ?
+
+Design considerations:
+ - object allocation when in use must remain minimal
+
+ - when in cache, there are 2 lists which the compiler expect to be at least
+ aligned each (e.g. if/when we start to use DWCAS).
+
+ - the original "pool debugging" feature covers both pool tracking, double-
+ free detection, overflow detection and caller info at the cost of a single
+ pointer placed immediately after the area.
+
+ - preserving the contents might be done by placing the cache links and the
+ shared cache's list outside of the area (either before or after). Placing
+ it before has the merit that the allocated object preserves the 4-ptr
+ alignment. But when a larger alignment is desired this often does not work
+ anymore. Placing it after requires some dynamic adjustment depending on the
+ object's size. If any protection is installed, this protection must be
+ placed before the links so that the list doesn't get randomly corrupted and
+ corrupts adjacent elements. Note that if protection is desired, the extra
+ waste is probably less critical.
+
+ - a link to the last caller might have to be stored somewhere. Without
+ preservation the free() caller may be placed anywhere while the alloc()
+ caller may only be placed outside. With preservation, again the free()
+ caller may be placed either before the object or after the mark at the end.
+ There is no particular need that both share the same location though it may
+ help. Note that when debugging is enabled, the free() caller doesn't need
+ to be duplicated and can continue to serve as the double-free detection.
+ Thus maybe in the end we only need to store the caller to the last alloc()
+ but not the free() since if we want it it's available via the pool debug.
+
+ - use-after-free detection: contents may be erased on free() and checked on
+ alloc(), but they can also be checksummed on free() and rechecked on
+ alloc(). In the latter case we need to store a checksum somewhere. Note
+ that with pure checksum we don't know what part was modified, but seeing
+ previous contents can be useful.
+
+Possibilities:
+
+1) Linked lists inside the area:
+
+ V size alloc
+ ---+------------------------------+-----------------+--
+ in use |##############################| (Pool) (Tracer) |
+ ---+------------------------------+-----------------+--
+
+ ---+--+--+------------------------+-----------------+--
+ in cache |L1|L2|########################| (Caller) (Sum) |
+ ---+--+--+------------------------+-----------------+--
+or:
+ ---+--+--+------------------------+-----------------+--
+ in cache |L1|L2|###################(sum)| (Caller) |
+ ---+--+--+------------------------+-----------------+--
+
+ ---+-+----------------------------+-----------------+--
+ in global |N|XXXX########################| (Caller) |
+ ---+-+----------------------------+-----------------+--
+
+
+2) Linked lists before the the area leave room for tracer and pool before
+ the area, but the canary must remain at the end, however the area will
+ be more difficult to keep aligned:
+
+ V head size alloc
+ ----+-+-+------------------------------+-----------------+--
+ in use |T|P|##############################| (canary) |
+ ----+-+-+------------------------------+-----------------+--
+
+ --+-----+------------------------------+-----------------+--
+ in cache |L1|L2|##############################| (Caller) (Sum) |
+ --+-----+------------------------------+-----------------+--
+
+ ------+-+------------------------------+-----------------+--
+ in global |N|##############################| (Caller) |
+ ------+-+------------------------------+-----------------+--
+
+
+3) Linked lists at the end of the area, might be shared with extra data
+ depending on the state:
+
+ V size alloc
+ ---+------------------------------+-----------------+--
+ in use |##############################| (Pool) (Tracer) |
+ ---+------------------------------+-----------------+--
+
+ ---+------------------------------+--+--+-----------+--
+ in cache |##############################|L1|L2| (Caller) (Sum)
+ ---+------------------------------+--+--+-----------+--
+
+ ---+------------------------------+-+---------------+--
+ in global |##############################|N| (Caller) |
+ ---+------------------------------+-+---------------+--
+
+This model requires a little bit of alignment at the end of the area, which is
+not incompatible with pattern filling and/or checksumming:
+ - preserving the area for post-mortem analysis means nothing may be placed
+ inside. In this case it could make sense to always store the last releaser.
+ - detecting late corruption may be done either with filling or checksumming,
+ but the simple fact of assuming a risk of corruption that needs to be
+ chased means we must not store the lists nor caller inside the area.
+
+Some models imply dedicating some place when in cache:
+ - preserving contents forces the lists to be prefixed or appended, which
+ leaves unused places when in use. Thus we could systematically place the
+ pool pointer and the caller in this case.
+
+ - if preserving contents is not desired, almost everything can be stored
+ inside when not in use. Then each situation's size should be calculated
+ so that the allocated size is known, and entries are filled from the
+ beginning while not in use, or after the size when in use.
+
+ - if poisonning is requested, late corruption might be detected but then we
+ don't want the list to be stored inside at the risk of being corrupted.
+
+Maybe just implement a few models:
+ - compact/optimal: put l1/l2 inside
+ - detect late corruption: fill/sum, put l1/l2 out
+ - preserve contents: put l1/l2 out
+ - corruption+preserve: do not fill, sum out
+ - poisonning: not needed on free if pattern filling is done.
+
+try2:
+ - poison on alloc to detect missing initialization: yes/no
+ (note: nothing to do if filling done)
+ - poison on free to detect use-after-free: yes/no
+ (note: nothing to do if filling done)
+ - check on alloc for corruption-after-free: yes/no
+ If content-preserving => sum, otherwise pattern filling; in
+ any case, move L1/L2 out.
+ - check for overflows: yes/no: use a canary after the area. The
+ canary can be the pointer to the pool.
+ - check for alloc caller: yes/no => always after the area
+ - content preservation: yes/no
+ (disables filling, moves lists out)
+ - improved caller tracking: used to detect double-free, may benefit
+ from content-preserving but not only.
diff --git a/doc/design-thoughts/rate-shaping.txt b/doc/design-thoughts/rate-shaping.txt
new file mode 100644
index 0000000..ca09408
--- /dev/null
+++ b/doc/design-thoughts/rate-shaping.txt
@@ -0,0 +1,90 @@
+2010/01/24 - Design of multi-criteria request rate shaping.
+
+We want to be able to rate-shape traffic on multiple cirteria. For instance, we
+may want to support shaping of per-host header requests, as well as per source.
+
+In order to achieve this, we will use checkpoints, one per criterion. Each of
+these checkpoints will consist in a test, a rate counter and a queue.
+
+A request reaches the checkpoint and checks the counter. If the counter is
+below the limit, it is updated and the request continues. If the limit is
+reached, the request attaches itself into the queue and sleeps. The sleep time
+is computed from the queue status, and updates the queue status.
+
+A task is dedicated to each queue. Its sole purpose is to be woken up when the
+next task may wake up, to check the frequency counter, wake as many requests as
+possible and update the counter. All the woken up requests are detached from
+the queue. Maybe the task dedicated to the queue can be avoided and replaced
+with all queued tasks's sleep counters, though this looks tricky. Or maybe it's
+just the first request in the queue that should be responsible for waking up
+other tasks, and not to forget to pass on this responsibility to next tasks if
+it leaves the queue.
+
+The woken up request then goes on evaluating other criteria and possibly sleeps
+again on another one. In the end, the task will have waited the amount of time
+required to pass all checkpoints, and all checkpoints will be able to maintain
+a permanent load of exactly their limit if enough streams flow through them.
+
+Since a request can only sleep in one queue at a time, it makes sense to use a
+linked list element in each session to attach it to any queue. It could very
+well be shared with the pendconn hooks which could then be part of the session.
+
+This mechanism could be used to rate-shape sessions and requests per backend
+and per server.
+
+When rate-shaping on dynamic criteria, such as the source IP address, we have
+to first extract the data pattern, then look it up in a table very similar to
+the stickiness tables, but with a frequency counter. At the checkpoint, the
+pattern is looked up, the entry created or refreshed, and the frequency counter
+updated and checked. Then the request either goes on or sleeps as described
+above, but if it sleeps, it's still in the checkpoint's queue, but with a date
+computed from the criterion's status.
+
+This means that we need 3 distinct features :
+
+ - optional pattern extraction
+ - per-pattern or per-queue frequency counter
+ - time-ordered queue with a task
+
+Based on past experiences with frequency counters, it does not appear very easy
+to exactly compute sleep delays in advance for multiple requests. So most
+likely we'll have to run per-criterion queues too, with only the head of the
+queue holding a wake-up timeout.
+
+This finally leads us to the following :
+
+ - optional pattern extraction
+ - per-pattern or per-queue frequency counter
+ - per-frequency counter queue
+ - head of the queue serves as a global queue timer.
+
+This brings us to a very flexible architecture :
+ - 1 list of rule-based checkpoints per frontend
+ - 1 list of rule-based checkpoints per backend
+ - 1 list of rule-based checkpoints per server
+
+Each of these lists have a lot of rules conditioned by ACLs, just like the
+use-backend rules, except that all rules are evaluated in turn.
+
+Since we might sometimes just want to enable that without setting any limit and
+just for enabling control in ACLs (or logging ?), we should probably try to
+find a flexible way of declaring just a counter without a queue.
+
+These checkpoints could be of two types :
+ - rate-limit (described here)
+ - concurrency-limit (very similar with the counter and no timer). This
+ feature would require to keep track of all accounted criteria in a
+ request so that they can be released upon request completion.
+
+It should be possible to define a max of requests in the queue, above which a
+503 is returned. The same applies for the max delay in the queue. We could have
+it per-task (currently it's the connection timeout) and abort tasks with a 503
+when the delay is exceeded.
+
+Per-server connection concurrency could be converted to use this mechanism
+which is very similar.
+
+The construct should be flexible enough so that the counters may be checked
+from ACLs. That would allow to reject connections or switch to an alternate
+backend when some limits are reached.
+
diff --git a/doc/design-thoughts/sess_par_sec.txt b/doc/design-thoughts/sess_par_sec.txt
new file mode 100644
index 0000000..e936374
--- /dev/null
+++ b/doc/design-thoughts/sess_par_sec.txt
@@ -0,0 +1,13 @@
+Graphe des nombres de traitements par seconde unité de temps avec
+ - un algo linéaire et très peu coûteux unitairement (0.01 ut)
+ - un algo en log(2) et 5 fois plus coûteux (0.05 ut)
+
+set yrange [0:1]
+plot [0:1000] 1/(1+0.01*x), 1/(1+0.05*log(x+1)/log(2))
+
+Graphe de la latence induite par ces traitements en unités de temps :
+
+set yrange [0:1000]
+plot [0:1000] x/(1+0.01*x), x/(1+0.05*log(x+1)/log(2))
+
+
diff --git a/doc/design-thoughts/thread-group.txt b/doc/design-thoughts/thread-group.txt
new file mode 100644
index 0000000..f1aad7b
--- /dev/null
+++ b/doc/design-thoughts/thread-group.txt
@@ -0,0 +1,528 @@
+Thread groups
+#############
+
+2021-07-13 - first draft
+==========
+
+Objective
+---------
+- support multi-socket systems with limited cache-line bouncing between
+ physical CPUs and/or L3 caches
+
+- overcome the 64-thread limitation
+
+- Support a reasonable number of groups. I.e. if modern CPUs arrive with
+ core complexes made of 8 cores, with 8 CC per chip and 2 chips in a
+ system, it makes sense to support 16 groups.
+
+
+Non-objective
+-------------
+- no need to optimize to the last possible cycle. I.e. some algos like
+ leastconn will remain shared across all threads, servers will keep a
+ single queue, etc. Global information remains global.
+
+- no stubborn enforcement of FD sharing. Per-server idle connection lists
+ can become per-group; listeners can (and should probably) be per-group.
+ Other mechanisms (like SO_REUSEADDR) can already overcome this.
+
+- no need to go beyond 64 threads per group.
+
+
+Identified tasks
+================
+
+General
+-------
+Everywhere tid_bit is used we absolutely need to find a complement using
+either the current group or a specific one. Thread debugging will need to
+be extended as masks are extensively used.
+
+
+Scheduler
+---------
+The global run queue and global wait queue must become per-group. This
+means that a task may only be queued into one of them at a time. It
+sounds like tasks may only belong to a given group, but doing so would
+bring back the original issue that it's impossible to perform remote wake
+ups.
+
+We could probably ignore the group if we don't need to set the thread mask
+in the task. the task's thread_mask is never manipulated using atomics so
+it's safe to complement it with a group.
+
+The sleeping_thread_mask should become per-group. Thus possibly that a
+wakeup may only be performed on the assigned group, meaning that either
+a task is not assigned, in which case it be self-assigned (like today),
+otherwise the tg to be woken up will be retrieved from the task itself.
+
+Task creation currently takes a thread mask of either tid_bit, a specific
+mask, or MAX_THREADS_MASK. How to create a task able to run anywhere
+(checks, Lua, ...) ?
+
+Profiling
+---------
+There should be one task_profiling_mask per thread group. Enabling or
+disabling profiling should be made per group (possibly by iterating).
+
+Thread isolation
+----------------
+Thread isolation is difficult as we solely rely on atomic ops to figure
+who can complete. Such operation is rare, maybe we could have a global
+read_mostly flag containing a mask of the groups that require isolation.
+Then the threads_want_rdv_mask etc can become per-group. However setting
+and clearing the bits will become problematic as this will happen in two
+steps hence will require careful ordering.
+
+FD
+--
+Tidbit is used in a number of atomic ops on the running_mask. If we have
+one fdtab[] per group, the mask implies that it's within the group.
+Theoretically we should never face a situation where an FD is reported nor
+manipulated for a remote group.
+
+There will still be one poller per thread, except that this time all
+operations will be related to the current thread_group. No fd may appear
+in two thread_groups at once, but we can probably not prevent that (e.g.
+delayed close and reopen). Should we instead have a single shared fdtab[]
+(less memory usage also) ? Maybe adding the group in the fdtab entry would
+work, but when does a thread know it can leave it ? Currently this is
+solved by running_mask and by update_mask. Having two tables could help
+with this (each table sees the FD in a different group with a different
+mask) but this looks overkill.
+
+There's polled_mask[] which needs to be decided upon. Probably that it
+should be doubled as well. Note, polled_mask left fdtab[] for cacheline
+alignment reasons in commit cb92f5cae4.
+
+If we have one fdtab[] per group, what *really* prevents from using the
+same FD in multiple groups ? _fd_delete_orphan() and fd_update_events()
+need to check for no-thread usage before closing the FD. This could be
+a limiting factor. Enabling could require to wake every poller.
+
+Shouldn't we remerge fdinfo[] with fdtab[] (one pointer + one int/short,
+used only during creation and close) ?
+
+Other problem, if we have one fdtab[] per TG, disabling/enabling an FD
+(e.g. pause/resume on listener) can become a problem if it's not necessarily
+on the current TG. We'll then need a way to figure that one. It sounds like
+FDs from listeners and receivers are very specific and suffer from problems
+all other ones under high load do not suffer from. Maybe something specific
+ought to be done for them, if we can guarantee there is no risk of accidental
+reuse (e.g. locate the TG info in the receiver and have a "MT" bit in the
+FD's flags). The risk is always that a close() can result in instant pop-up
+of the same FD on any other thread of the same process.
+
+Observations: right now fdtab[].thread_mask more or less corresponds to a
+declaration of interest, it's very close to meaning "active per thread". It is
+in fact located in the FD while it ought to do nothing there, as it should be
+where the FD is used as it rules accesses to a shared resource that is not
+the FD but what uses it. Indeed, if neither polled_mask nor running_mask have
+a thread's bit, the FD is unknown to that thread and the element using it may
+only be reached from above and not from the FD. As such we ought to have a
+thread_mask on a listener and another one on connections. These ones will
+indicate who uses them. A takeover could then be simplified (atomically set
+exclusivity on the FD's running_mask, upon success, takeover the connection,
+clear the running mask). Probably that the change ought to be performed on
+the connection level first, not the FD level by the way. But running and
+polled are the two relevant elements, one indicates userland knowledge,
+the other one kernel knowledge. For listeners there's no exclusivity so it's
+a bit different but the rule remains the same that we don't have to know
+what threads are *interested* in the FD, only its holder.
+
+Not exact in fact, see FD notes below.
+
+activity
+--------
+There should be one activity array per thread group. The dump should
+simply scan them all since the cumuled values are not very important
+anyway.
+
+applets
+-------
+They use tid_bit only for the task. It looks like the appctx's thread_mask
+is never used (now removed). Furthermore, it looks like the argument is
+*always* tid_bit.
+
+CPU binding
+-----------
+This is going to be tough. It will be needed to detect that threads overlap
+and are not bound (i.e. all threads on same mask). In this case, if the number
+of threads is higher than the number of threads per physical socket, one must
+try hard to evenly spread them among physical sockets (e.g. one thread group
+per physical socket) and start as many threads as needed on each, bound to
+all threads/cores of each socket. If there is a single socket, the same job
+may be done based on L3 caches. Maybe it could always be done based on L3
+caches. The difficulty behind this is the number of sockets to be bound: it
+is not possible to bind several FDs per listener. Maybe with a new bind
+keyword we can imagine to automatically duplicate listeners ? In any case,
+the initially bound cpumap (via taskset) must always be respected, and
+everything should probably start from there.
+
+Frontend binding
+----------------
+We'll have to define a list of threads and thread-groups per frontend.
+Probably that having a group mask and a same thread-mask for each group
+would suffice.
+
+Threads should have two numbers:
+ - the per-process number (e.g. 1..256)
+ - the per-group number (1..64)
+
+The "bind-thread" lines ought to use the following syntax:
+ - bind 45 ## bind to process' thread 45
+ - bind 1/45 ## bind to group 1's thread 45
+ - bind all/45 ## bind to thread 45 in each group
+ - bind 1/all ## bind to all threads in group 1
+ - bind all ## bind to all threads
+ - bind all/all ## bind to all threads in all groups (=all)
+ - bind 1/65 ## rejected
+ - bind 65 ## OK if there are enough
+ - bind 35-45 ## depends. Rejected if it crosses a group boundary.
+
+The global directive "nbthread 28" means 28 total threads for the process. The
+number of groups will sub-divide this. E.g. 4 groups will very likely imply 7
+threads per group. At the beginning, the nbgroup should be manual since it
+implies config adjustments to bind lines.
+
+There should be a trivial way to map a global thread to a group and local ID
+and to do the opposite.
+
+
+Panic handler + watchdog
+------------------------
+Will probably depend on what's done for thread_isolate
+
+Per-thread arrays inside structures
+-----------------------------------
+- listeners have a thr_conn[] array, currently limited to MAX_THREADS. Should
+ we simply bump the limit ?
+- same for servers with idle connections.
+=> doesn't seem very practical.
+- another solution might be to point to dynamically allocated arrays of
+ arrays (e.g. nbthread * nbgroup) or a first level per group and a second
+ per thread.
+=> dynamic allocation based on the global number
+
+Other
+-----
+- what about dynamic thread start/stop (e.g. for containers/VMs) ?
+ E.g. if we decide to start $MANY threads in 4 groups, and only use
+ one, in the end it will not be possible to use less than one thread
+ per group, and at most 64 will be present in each group.
+
+
+FD Notes
+--------
+ - updt_fd_polling() uses thread_mask to figure where to send the update,
+ the local list or a shared list, and which bits to set in update_mask.
+ This could be changed so that it takes the update mask in argument. The
+ call from the poller's fork would just have to broadcast everywhere.
+
+ - pollers use it to figure whether they're concerned or not by the activity
+ update. This looks important as otherwise we could re-enable polling on
+ an FD that changed to another thread.
+
+ - thread_mask being a per-thread active mask looks more exact and is
+ precisely used this way by _update_fd(). In this case using it instead
+ of running_mask to gauge a change or temporarily lock it during a
+ removal could make sense.
+
+ - running should be conditioned by thread. Polled not (since deferred
+ or migrated). In this case testing thread_mask can be enough most of
+ the time, but this requires synchronization that will have to be
+ extended to tgid.. But migration seems a different beast that we shouldn't
+ care about here: if first performed at the higher level it ought to
+ be safe.
+
+In practice the update_mask can be dropped to zero by the first fd_delete()
+as the only authority allowed to fd_delete() is *the* owner, and as soon as
+all running_mask are gone, the FD will be closed, hence removed from all
+pollers. This will be the only way to make sure that update_mask always
+refers to the current tgid.
+
+However, it may happen that a takeover within the same group causes a thread
+to read the update_mask late, while the FD is being wiped by another thread.
+That other thread may close it, causing another thread in another group to
+catch it, and change the tgid and start to update the update_mask. This means
+that it would be possible for a thread entering do_poll() to see the correct
+tgid, then the fd would be closed, reopened and reassigned to another tgid,
+and the thread would see its bit in the update_mask, being confused. Right
+now this should already happen when the update_mask is not cleared, except
+that upon wakeup a migration would be detected and that would be all.
+
+Thus we might need to set the running bit to prevent the FD from migrating
+before reading update_mask, which also implies closing on fd_clr_running() == 0 :-(
+
+Also even fd_update_events() leaves a risk of updating update_mask after
+clearing running, thus affecting the wrong one. Probably that update_mask
+should be updated before clearing running_mask there. Also, how about not
+creating an update on a close ? Not trivial if done before running, unless
+thread_mask==0.
+
+###########################################################
+
+Current state:
+
+
+Mux / takeover / fd_delete() code ||| poller code
+-------------------------------------------------|||---------------------------------------------------
+ \|/
+mux_takeover(): | fd_set_running():
+ if (fd_takeover()<0) | old = {running, thread};
+ return fail; | new = {tid_bit, tid_bit};
+ ... |
+fd_takeover(): | do {
+ atomic_or(running, tid_bit); | if (!(old.thread & tid_bit))
+ old = {running, thread}; | return -1;
+ new = {tid_bit, tid_bit}; | new = { running | tid_bit, old.thread }
+ if (owner != expected) { | } while (!dwcas({running, thread}, &old, &new));
+ atomic_and(runnning, ~tid_bit); |
+ return -1; // fail | fd_clr_running():
+ } | return atomic_and_fetch(running, ~tid_bit);
+ |
+ while (old == {tid_bit, !=0 }) | poll():
+ if (dwcas({running, thread}, &old, &new)) { | if (!owner)
+ atomic_and(runnning, ~tid_bit); | continue;
+ return 0; // success |
+ } | if (!(thread_mask & tid_bit)) {
+ } | epoll_ctl_del();
+ | continue;
+ atomic_and(runnning, ~tid_bit); | }
+ return -1; // fail |
+ | // via fd_update_events()
+fd_delete(): | if (fd_set_running() != -1) {
+ atomic_or(running, tid_bit); | iocb();
+ atomic_store(thread, 0); | if (fd_clr_running() == 0 && !thread_mask)
+ if (fd_clr_running(fd) = 0) | fd_delete_orphan();
+ fd_delete_orphan(); | }
+
+
+The idle_conns_lock prevents the connection from being *picked* and released
+while someone else is reading it. What it does is guarantee that on idle
+connections, the caller of the IOCB will not dereference the task's context
+while the connection is still in the idle list, since it might be picked then
+freed at the same instant by another thread. As soon as the IOCB manages to
+get that lock, it removes the connection from the list so that it cannot be
+taken over anymore. Conversely, the mux's takeover() code runs under that
+lock so that if it frees the connection and task, this will appear atomic
+to the IOCB. The timeout task (which is another entry point for connection
+deletion) does the same. Thus, when coming from the low-level (I/O or timeout):
+ - task always exists, but ctx checked under lock validates; conn removal
+ from list prevents takeover().
+ - t->context is stable, except during changes under takeover lock. So
+ h2_timeout_task may well run on a different thread than h2_io_cb().
+
+Coming from the top:
+ - takeover() done under lock() clears task's ctx and possibly closes the FD
+ (unless some running remains present).
+
+Unlikely but currently possible situations:
+ - multiple pollers (up to N) may have an idle connection's FD being
+ polled, if the connection was passed from thread to thread. The first
+ event on the connection would wake all of them. Most of them would
+ see fdtab[].owner set (the late ones might miss it). All but one would
+ see that their bit is missing from fdtab[].thread_mask and give up.
+ However, just after this test, others might take over the connection,
+ so in practice if terribly unlucky, all but 1 could see their bit in
+ thread_mask just before it gets removed, all of them set their bit
+ in running_mask, and all of them call iocb() (sock_conn_iocb()).
+ Thus all of them dereference the connection and touch the subscriber
+ with no protection, then end up in conn_notify_mux() that will call
+ the mux's wake().
+
+ - multiple pollers (up to N-1) might still be in fd_update_events()
+ manipulating fdtab[].state. The cause is that the "locked" variable
+ is determined by atleast2(thread_mask) but that thread_mask is read
+ at a random instant (i.e. it may be stolen by another one during a
+ takeover) since we don't yet hold running to prevent this from being
+ done. Thus we can arrive here with thread_mask==something_else (1bit),
+ locked==0 and fdtab[].state assigned non-atomically.
+
+ - it looks like nothing prevents h2_release() from being called on a
+ thread (e.g. from the top or task timeout) while sock_conn_iocb()
+ dereferences the connection on another thread. Those killing the
+ connection don't yet consider the fact that it's an FD that others
+ might currently be waking up on.
+
+###################
+
+pb with counter:
+
+users count doesn't say who's using the FD and two users can do the same
+close in turn. The thread_mask should define who's responsible for closing
+the FD, and all those with a bit in it ought to do it.
+
+
+2021-08-25 - update with minimal locking on tgid value
+==========
+
+ - tgid + refcount at once using CAS
+ - idle_conns lock during updates
+ - update:
+ if tgid differs => close happened, thus drop update
+ otherwise normal stuff. Lock tgid until running if needed.
+ - poll report:
+ if tgid differs => closed
+ if thread differs => stop polling (migrated)
+ keep tgid lock until running
+ - test on thread_id:
+ if (xadd(&tgid,65536) != my_tgid) {
+ // was closed
+ sub(&tgid, 65536)
+ return -1
+ }
+ if !(thread_id & tidbit) => migrated/closed
+ set_running()
+ sub(tgid,65536)
+ - note: either fd_insert() or the final close() ought to set
+ polled and update to 0.
+
+2021-09-13 - tid / tgroups etc.
+==========
+
+ - tid currently is the thread's global ID. It's essentially used as an index
+ for arrays. It must be clearly stated that it works this way.
+
+ - tid_bit makes no sense process-wide, so it must be redefined to represent
+ the thread's tid within its group. The name is not much welcome though, but
+ there are 286 of it that are not going to be changed that fast.
+
+ - just like "ti" is the thread_info, we need to have "tg" pointing to the
+ thread_group.
+
+ - other less commonly used elements should be retrieved from ti->xxx. E.g.
+ the thread's local ID.
+
+ - lock debugging must reproduce tgid
+
+ - an offset might be placed in the tgroup so that even with 64 threads max
+ we could have completely separate tid_bits over several groups.
+
+2021-09-15 - bind + listen() + rx
+==========
+
+ - thread_mask (in bind_conf->rx_settings) should become an array of
+ MAX_TGROUP longs.
+ - when parsing "thread 123" or "thread 2/37", the proper bit is set,
+ assuming the array is either a contiguous bitfield or a tgroup array.
+ An option RX_O_THR_PER_GRP or RX_O_THR_PER_PROC is set depending on
+ how the thread num was parsed, so that we reject mixes.
+ - end of parsing: entries translated to the cleanest form (to be determined)
+ - binding: for each socket()/bind()/listen()... just perform one extra dup()
+ for each tgroup and store the multiple FDs into an FD array indexed on
+ MAX_TGROUP. => allows to use one FD per tgroup for the same socket, hence
+ to have multiple entries in all tgroup pollers without requiring the user
+ to duplicate the bind line.
+
+2021-09-15 - global thread masks
+==========
+
+Some global variables currently expect to know about thread IDs and it's
+uncertain what must be done with them:
+ - global_tasks_mask /* Mask of threads with tasks in the global runqueue */
+ => touched under the rq lock. Change it per-group ? What exact use is made ?
+
+ - sleeping_thread_mask /* Threads that are about to sleep in poll() */
+ => seems that it can be made per group
+
+ - all_threads_mask: a bit complicated, derived from nbthread and used with
+ masks and with my_ffsl() to wake threads up. Should probably be per-group
+ but we might miss something for global.
+
+ - stopping_thread_mask: used in combination with all_threads_mask, should
+ move per-group.
+
+ - threads_harmless_mask: indicates all threads that are currently harmless in
+ that they promise not to access a shared resource. Must be made per-group
+ but then we'll likely need a second stage to have the harmless groups mask.
+ threads_idle_mask, threads_sync_mask, threads_want_rdv_mask go with the one
+ above. Maybe the right approach will be to request harmless on a group mask
+ so that we can detect collisions and arbiter them like today, but on top of
+ this it becomes possible to request harmless only on the local group if
+ desired. The subtlety is that requesting harmless at the group level does
+ not mean it's achieved since the requester cannot vouch for the other ones
+ in the same group.
+
+In addition, some variables are related to the global runqueue:
+ __decl_aligned_spinlock(rq_lock); /* spin lock related to run queue */
+ struct eb_root rqueue; /* tree constituting the global run queue, accessed under rq_lock */
+ unsigned int grq_total; /* total number of entries in the global run queue, atomic */
+ static unsigned int global_rqueue_ticks; /* insertion count in the grq, use rq_lock */
+
+And others to the global wait queue:
+ struct eb_root timers; /* sorted timers tree, global, accessed under wq_lock */
+ __decl_aligned_rwlock(wq_lock); /* RW lock related to the wait queue */
+ struct eb_root timers; /* sorted timers tree, global, accessed under wq_lock */
+
+
+2021-09-29 - group designation and masks
+==========
+
+Neither FDs nor tasks will belong to incomplete subsets of threads spanning
+over multiple thread groups. In addition there may be a difference between
+configuration and operation (for FDs). This allows to fix the following rules:
+
+ group mask description
+ 0 0 bind_conf: groups & thread not set. bind to any/all
+ task: it would be nice to mean "run on the same as the caller".
+
+ 0 xxx bind_conf: thread set but not group: thread IDs are global
+ FD/task: group 0, mask xxx
+
+ G>0 0 bind_conf: only group is set: bind to all threads of group G
+ FD/task: mask 0 not permitted (= not owned). May be used to
+ mention "any thread of this group", though already covered by
+ G/xxx like today.
+
+ G>0 xxx bind_conf: Bind to these threads of this group
+ FD/task: group G, mask xxx
+
+It looks like keeping groups starting at zero internally complicates everything
+though. But forcing it to start at 1 might also require that we rescan all tasks
+to replace 0 with 1 upon startup. This would also allow group 0 to be special and
+be used as the default group for any new thread creation, so that group0.count
+would keep the number of unassigned threads. Let's try:
+
+ group mask description
+ 0 0 bind_conf: groups & thread not set. bind to any/all
+ task: "run on the same group & thread as the caller".
+
+ 0 xxx bind_conf: thread set but not group: thread IDs are global
+ FD/task: invalid. Or maybe for a task we could use this to
+ mean "run on current group, thread XXX", which would cover
+ the need for health checks (g/t 0/0 while sleeping, 0/xxx
+ while running) and have wake_expired_tasks() detect 0/0 and
+ wake them up to a random group.
+
+ G>0 0 bind_conf: only group is set: bind to all threads of group G
+ FD/task: mask 0 not permitted (= not owned). May be used to
+ mention "any thread of this group", though already covered by
+ G/xxx like today.
+
+ G>0 xxx bind_conf: Bind to these threads of this group
+ FD/task: group G, mask xxx
+
+With a single group declared in the config, group 0 would implicitly find the
+first one.
+
+
+The problem with the approach above is that a task queued in one group+thread's
+wait queue could very well receive a signal from another thread and/or group,
+and that there is no indication about where the task is queued, nor how to
+dequeue it. Thus it seems that it's up to the application itself to unbind/
+rebind a task. This contradicts the principle of leaving a task waiting in a
+wait queue and waking it anywhere.
+
+Another possibility might be to decide that a task having a defined group but
+a mask of zero is shared and will always be queued into its group's wait queue.
+However, upon expiry, the scheduler would notice the thread-mask 0 and would
+broadcast it to any group.
+
+Right now in the code we have:
+ - 18 calls of task_new(tid_bit)
+ - 18 calls of task_new(MAX_THREADS_MASK)
+ - 2 calls with a single bit
+
+Thus it looks like "task_new_anywhere()", "task_new_on()" and
+"task_new_here()" would be sufficient.