2022-05-27 - Stream layers in HAProxy 2.6 1. Background There are streams at plenty of levels in haproxy, essentially due to the introduction of multiplexed protocols which provide high-level streams on top of low-level streams, themselves either based on stream-oriented protocols or datagram-oriented protocols. The refactoring of the appctx and muxes that allowed to drop a lot of duplicate code between 2.5 and 2.6-dev6 raised another concern with some entities like "conn_stream" that were not specific to connections anymore, "endpoints" that became entities on their own, and "targets" whose life had been extended to last all along a connection. It was time to rename all such legacy entities introduced in 1.8 and which had turned particularly confusing over time as their roles evolved. 2. Naming principles The global renaming of some entities between streams and connections was articulated around several principles: - avoid the confusing use of "context" in shared places. For example, the endpoint's connection is in "ctx" and nothing makes it obvious that the endpoint's context is a connection, especially when an applet is there. - reserve relative nouns for pointers and not for types. "endpoint", just like "owner" or "peer" is relative, but when accessed from a different layer it starts to make no sense at all, or to make one believe it's something else, particularly with void*. - avoid too generic terms that have multiple meanings, or words that are synonyms in a same place (e.g. "peer" and "remote", or "endpoint" and "target"). If two synonyms are needed to designate two distinct entities, there's probably a problem elsewhere, or the problem is poorly defined. - make it clearer that all that is manipulated is related to streams. This particularly important in sample fetch functions for example, which tend to require low-level access and could be mislead in trying to follow the wrong chain when trying to get information about a connection. - use easily spellable short names that abbreviate unambiguously when used together in adjacent contexts 3. Current state as of 2.6 - when a name is required to designate the lower block that starts at the mux stream or the appctx, it is spoken of as a "stream endpoint", and abbreviated "se". It's okay because while "endpoint" itself is relative, "stream endpoint" unequivocally designates one extremity of a stream. If a type is needed for this in the future (e.g. via obj_type), then the type "stendp" may be used. Before 2.6-dev6 there was no name for this, it was known as conn_stream->ctx. - the 2.6-dev6 cs_endpoint which preserves the state of a mux stream or an appctx and abstracts them in front of a conn_stream becomes a "stream endpoint descriptor", of type "sedesc" and often abbreviated "sd", "sed" or "ed". Its "target" pointer became "se" as per the rule above. Before 2.6-dev6, these elements were mixed with others inside conn_stream. From the appctx it's called "sedesc" (few occurrences hence long name OK). - the conn_stream which is always attached to either a stream or a health check and that is used to reach a mux or an applet becomes a "stream connector" of type "stconn", generally abbreviated "sc". Its "endp" pointer becomes "sedesc" as per the rule above, and that one has a back pointer "sc". The stream uses "scf" and "scb" as the respective front and back pointers to the stconns. Prior to 2.6-dev6, these parts were split between conn_stream and stream_interface. - the sedesc's "ctx" which is solely used to store the connection as of now, is renamed "conn" to void any doubt in the context of applets or even muxes. In the future the connection should be attached to the "se" instead and this pointer should disappear (or be recycled for anything else). The new 2.6 model looks like this: +------------------------+ | stream or health check | +------------------------+ ^ \ scf, scb / \ | | \ / app \ v +----------+ | stconn | +----------+ ^ \ sedesc / \ . . . . | . . . | . . . . . split point (retries etc) \ / sc \ v +----------+ flags <--| sedesc | : sedesc : +----------+ ... +----------+ conn / ^ \ se ^ \ +------------+ / / \ | \ | connection |<--' | | ... OR ... | | +------------+ \ / \ | mux| ^ |ctx sd \ v : sedesc \ v | | | +----------------------+ \ # +----------+ svcctx | | | | mux stream or appctx | | # | appctx |--. | | | +----------------------+ | # +----------+ | | | | ^ | / private # : : | v | | | v > to the # +----------+ | mux_ops | | +----------------+ \ mux # | svcctx |<-' | +---->| mux connection | ) # +----------+ +------ +----------------+ / # Stream descriptors may exist in the following modes: - .conn = NULL, .se = NULL : backend, not connection attempt yet - .conn = NULL, .se = : frontend or backend, applet - .conn = , .se = NULL : backend, connection in progress - .conn = , .se = : frontend or backend, connected Notes: - for historical reasons (connect, forced protocol upgrades, etc), during a connection setup or a rule-based protocol upgrade, the connection's "ctx" may temporarily point to the stconn 4. Invariants and cardinalities Usually a stream is created from an existing stconn from a mux or some applets, but may also be allocated first by other applets schedulers. After stream_new() a stream always has exactly one stconn per side (scf, scb), each of which has one ->sedesc. Each side is initialized with either one or no stream endpoint attached to the descriptor. Both applets and a mux stream always have a stream endpoint descriptor. AS SUCH IT IS NEVER NECESSARY TO TEST FOR THE EXISTENCE OF THE SEDESC FROM ANY SIDE, IT ALWAYS EXISTS. This explains why as much as possible it's preferable to use the sedesc to access flags and statuses from any side, rather than bouncing via the stconn. An applet's app layer is always a stream, which means that there are always channels accessible above, and there is always an opposite stream connector and a stream endpoint descriptor. As such, it's always safe for an applet to access the other side using sc_opposite(). When an outgoing connection is in the process of being established, the backend side sedesc has its ->conn pointer pointing to the pending connection, and no ->se. Once the connection is established and a mux is chosen, it's attached to the ->se. If an applet is used instead of a mux, the appctx is attached to the sedesc's ->se and ->conn remains NULL. If either side wants to detach from the other, it must allocate a new virgin sedesc to replace the existing one, and leave the existing one to the endpoint, since it continues to describe the stream endpoint. The stconn keeps its state (modulo the updates related to the disconnection). The previous sedesc points to a NULL stconn. For example, disconnecting from a backend mux will leave the entities like this: +------------------------+ | stream or health check | +------------------------+ ^ \ scf, scb / \ | | \ / app \ v +----------+ | stconn | +----------+ ^ \ sedesc / \ NULL | | ^ \ / sc | / sc \ v +----------+ / +----------+ flags <--| sedesc1 | . . . . . | sedesc2 |--> flags +----------+ / +----------+ conn / ^ \ se / conn / \ se +------------+ / / \ | | | connection |<--' | | v v +------------+ \ / NULL NULL mux| ^ |ctx sd \ v | | | +----------------------+ | | | | mux stream or appctx | | | | +----------------------+ | | | ^ | v | | | v mux_ops | | +----------------+ | +---->| mux connection | +------ +----------------+