diff options
Diffstat (limited to 'doc/internals/api/layers.txt')
| -rw-r--r-- | doc/internals/api/layers.txt | 190 |
1 files changed, 190 insertions, 0 deletions
diff --git a/doc/internals/api/layers.txt b/doc/internals/api/layers.txt new file mode 100644 index 0000000..b5c35f4 --- /dev/null +++ b/doc/internals/api/layers.txt @@ -0,0 +1,190 @@ +2022-05-27 - Stream layers in HAProxy 2.6 + + +1. Background + +There are streams at plenty of levels in haproxy, essentially due to the +introduction of multiplexed protocols which provide high-level streams on top +of low-level streams, themselves either based on stream-oriented protocols or +datagram-oriented protocols. + +The refactoring of the appctx and muxes that allowed to drop a lot of duplicate +code between 2.5 and 2.6-dev6 raised another concern with some entities like +"conn_stream" that were not specific to connections anymore, "endpoints" that +became entities on their own, and "targets" whose life had been extended to +last all along a connection. + +It was time to rename all such legacy entities introduced in 1.8 and which had +turned particularly confusing over time as their roles evolved. + + +2. Naming principles + +The global renaming of some entities between streams and connections was +articulated around several principles: + + - avoid the confusing use of "context" in shared places. For example, the + endpoint's connection is in "ctx" and nothing makes it obvious that the + endpoint's context is a connection, especially when an applet is there. + + - reserve relative nouns for pointers and not for types. "endpoint", just + like "owner" or "peer" is relative, but when accessed from a different + layer it starts to make no sense at all, or to make one believe it's + something else, particularly with void*. + + - avoid too generic terms that have multiple meanings, or words that are + synonyms in a same place (e.g. "peer" and "remote", or "endpoint" and + "target"). If two synonyms are needed to designate two distinct entities, + there's probably a problem elsewhere, or the problem is poorly defined. + + - make it clearer that all that is manipulated is related to streams. This + particularly important in sample fetch functions for example, which tend + to require low-level access and could be mislead in trying to follow the + wrong chain when trying to get information about a connection. + + - use easily spellable short names that abbreviate unambiguously when used + together in adjacent contexts + + +3. Current state as of 2.6 + +- when a name is required to designate the lower block that starts at the mux + stream or the appctx, it is spoken of as a "stream endpoint", and abbreviated + "se". It's okay because while "endpoint" itself is relative, "stream + endpoint" unequivocally designates one extremity of a stream. If a type is + needed for this in the future (e.g. via obj_type), then the type "stendp" + may be used. Before 2.6-dev6 there was no name for this, it was known as + conn_stream->ctx. + +- the 2.6-dev6 cs_endpoint which preserves the state of a mux stream or an + appctx and abstracts them in front of a conn_stream becomes a "stream + endpoint descriptor", of type "sedesc" and often abbreviated "sd", "sed" + or "ed". Its "target" pointer became "se" as per the rule above. Before + 2.6-dev6, these elements were mixed with others inside conn_stream. From + the appctx it's called "sedesc" (few occurrences hence long name OK). + +- the conn_stream which is always attached to either a stream or a health check + and that is used to reach a mux or an applet becomes a "stream connector" of + type "stconn", generally abbreviated "sc". Its "endp" pointer becomes + "sedesc" as per the rule above, and that one has a back pointer "sc". The + stream uses "scf" and "scb" as the respective front and back pointers to the + stconns. Prior to 2.6-dev6, these parts were split between conn_stream and + stream_interface. + +- the sedesc's "ctx" which is solely used to store the connection as of now, is + renamed "conn" to void any doubt in the context of applets or even muxes. In + the future the connection should be attached to the "se" instead and this + pointer should disappear (or be recycled for anything else). + +The new 2.6 model looks like this: + + +------------------------+ + | stream or health check | + +------------------------+ + ^ \ scf, scb + / \ + | | + \ / + app \ v + +----------+ + | stconn | + +----------+ + ^ \ sedesc + / \ + . . . . | . . . | . . . . . split point (retries etc) + \ / + sc \ v + +----------+ + flags <--| sedesc | : sedesc : + +----------+ ... +----------+ + conn / ^ \ se ^ \ + +------------+ / / \ | \ + | connection |<--' | | ... OR ... | | + +------------+ \ / \ | + mux| ^ |ctx sd \ v : sedesc \ v + | | | +----------------------+ \ # +----------+ svcctx + | | | | mux stream or appctx | | # | appctx |--. + | | | +----------------------+ | # +----------+ | + | | | ^ | / private # : : | + v | | | v > to the # +----------+ | + mux_ops | | +----------------+ \ mux # | svcctx |<-' + | +---->| mux connection | ) # +----------+ + +------ +----------------+ / # + +Stream descriptors may exist in the following modes: + - .conn = NULL, .se = NULL : backend, not connection attempt yet + - .conn = NULL, .se = <appctx> : frontend or backend, applet + - .conn = <conn>, .se = NULL : backend, connection in progress + - .conn = <conn>, .se = <muxs> : frontend or backend, connected + +Notes: + - for historical reasons (connect, forced protocol upgrades, etc), during a + connection setup or a rule-based protocol upgrade, the connection's "ctx" + may temporarily point to the stconn + + +4. Invariants and cardinalities + +Usually a stream is created from an existing stconn from a mux or some applets, +but may also be allocated first by other applets schedulers. After stream_new() +a stream always has exactly one stconn per side (scf, scb), each of which has +one ->sedesc. Each side is initialized with either one or no stream endpoint +attached to the descriptor. + +Both applets and a mux stream always have a stream endpoint descriptor. AS SUCH +IT IS NEVER NECESSARY TO TEST FOR THE EXISTENCE OF THE SEDESC FROM ANY SIDE, IT +ALWAYS EXISTS. This explains why as much as possible it's preferable to use the +sedesc to access flags and statuses from any side, rather than bouncing via the +stconn. + +An applet's app layer is always a stream, which means that there are always +channels accessible above, and there is always an opposite stream connector and +a stream endpoint descriptor. As such, it's always safe for an applet to access +the other side using sc_opposite(). + +When an outgoing connection is in the process of being established, the backend +side sedesc has its ->conn pointer pointing to the pending connection, and no +->se. Once the connection is established and a mux is chosen, it's attached to +the ->se. If an applet is used instead of a mux, the appctx is attached to the +sedesc's ->se and ->conn remains NULL. + +If either side wants to detach from the other, it must allocate a new virgin +sedesc to replace the existing one, and leave the existing one to the endpoint, +since it continues to describe the stream endpoint. The stconn keeps its state +(modulo the updates related to the disconnection). The previous sedesc points +to a NULL stconn. For example, disconnecting from a backend mux will leave the +entities like this: + + +------------------------+ + | stream or health check | + +------------------------+ + ^ \ scf, scb + / \ + | | + \ / + app \ v + +----------+ + | stconn | + +----------+ + ^ \ sedesc + / \ + NULL | | + ^ \ / + sc | / sc \ v + +----------+ / +----------+ + flags <--| sedesc1 | . . . . . | sedesc2 |--> flags + +----------+ / +----------+ + conn / ^ \ se / conn / \ se + +------------+ / / \ | | + | connection |<--' | | v v + +------------+ \ / NULL NULL + mux| ^ |ctx sd \ v + | | | +----------------------+ + | | | | mux stream or appctx | + | | | +----------------------+ + | | | ^ | + v | | | v + mux_ops | | +----------------+ + | +---->| mux connection | + +------ +----------------+ + |