1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
|
2022-05-27 - Stream layers in HAProxy 2.6
1. Background
There are streams at plenty of levels in haproxy, essentially due to the
introduction of multiplexed protocols which provide high-level streams on top
of low-level streams, themselves either based on stream-oriented protocols or
datagram-oriented protocols.
The refactoring of the appctx and muxes that allowed to drop a lot of duplicate
code between 2.5 and 2.6-dev6 raised another concern with some entities like
"conn_stream" that were not specific to connections anymore, "endpoints" that
became entities on their own, and "targets" whose life had been extended to
last all along a connection.
It was time to rename all such legacy entities introduced in 1.8 and which had
turned particularly confusing over time as their roles evolved.
2. Naming principles
The global renaming of some entities between streams and connections was
articulated around several principles:
- avoid the confusing use of "context" in shared places. For example, the
endpoint's connection is in "ctx" and nothing makes it obvious that the
endpoint's context is a connection, especially when an applet is there.
- reserve relative nouns for pointers and not for types. "endpoint", just
like "owner" or "peer" is relative, but when accessed from a different
layer it starts to make no sense at all, or to make one believe it's
something else, particularly with void*.
- avoid too generic terms that have multiple meanings, or words that are
synonyms in a same place (e.g. "peer" and "remote", or "endpoint" and
"target"). If two synonyms are needed to designate two distinct entities,
there's probably a problem elsewhere, or the problem is poorly defined.
- make it clearer that all that is manipulated is related to streams. This
particularly important in sample fetch functions for example, which tend
to require low-level access and could be mislead in trying to follow the
wrong chain when trying to get information about a connection.
- use easily spellable short names that abbreviate unambiguously when used
together in adjacent contexts
3. Current state as of 2.6
- when a name is required to designate the lower block that starts at the mux
stream or the appctx, it is spoken of as a "stream endpoint", and abbreviated
"se". It's okay because while "endpoint" itself is relative, "stream
endpoint" unequivocally designates one extremity of a stream. If a type is
needed for this in the future (e.g. via obj_type), then the type "stendp"
may be used. Before 2.6-dev6 there was no name for this, it was known as
conn_stream->ctx.
- the 2.6-dev6 cs_endpoint which preserves the state of a mux stream or an
appctx and abstracts them in front of a conn_stream becomes a "stream
endpoint descriptor", of type "sedesc" and often abbreviated "sd", "sed"
or "ed". Its "target" pointer became "se" as per the rule above. Before
2.6-dev6, these elements were mixed with others inside conn_stream. From
the appctx it's called "sedesc" (few occurrences hence long name OK).
- the conn_stream which is always attached to either a stream or a health check
and that is used to reach a mux or an applet becomes a "stream connector" of
type "stconn", generally abbreviated "sc". Its "endp" pointer becomes
"sedesc" as per the rule above, and that one has a back pointer "sc". The
stream uses "scf" and "scb" as the respective front and back pointers to the
stconns. Prior to 2.6-dev6, these parts were split between conn_stream and
stream_interface.
- the sedesc's "ctx" which is solely used to store the connection as of now, is
renamed "conn" to void any doubt in the context of applets or even muxes. In
the future the connection should be attached to the "se" instead and this
pointer should disappear (or be recycled for anything else).
The new 2.6 model looks like this:
+------------------------+
| stream or health check |
+------------------------+
^ \ scf, scb
/ \
| |
\ /
app \ v
+----------+
| stconn |
+----------+
^ \ sedesc
/ \
. . . . | . . . | . . . . . split point (retries etc)
\ /
sc \ v
+----------+
flags <--| sedesc | : sedesc :
+----------+ ... +----------+
conn / ^ \ se ^ \
+------------+ / / \ | \
| connection |<--' | | ... OR ... | |
+------------+ \ / \ |
mux| ^ |ctx sd \ v : sedesc \ v
| | | +----------------------+ \ # +----------+ svcctx
| | | | mux stream or appctx | | # | appctx |--.
| | | +----------------------+ | # +----------+ |
| | | ^ | / private # : : |
v | | | v > to the # +----------+ |
mux_ops | | +----------------+ \ mux # | svcctx |<-'
| +---->| mux connection | ) # +----------+
+------ +----------------+ / #
Stream descriptors may exist in the following modes:
- .conn = NULL, .se = NULL : backend, not connection attempt yet
- .conn = NULL, .se = <appctx> : frontend or backend, applet
- .conn = <conn>, .se = NULL : backend, connection in progress
- .conn = <conn>, .se = <muxs> : frontend or backend, connected
Notes:
- for historical reasons (connect, forced protocol upgrades, etc), during a
connection setup or a rule-based protocol upgrade, the connection's "ctx"
may temporarily point to the stconn
4. Invariants and cardinalities
Usually a stream is created from an existing stconn from a mux or some applets,
but may also be allocated first by other applets schedulers. After stream_new()
a stream always has exactly one stconn per side (scf, scb), each of which has
one ->sedesc. Each side is initialized with either one or no stream endpoint
attached to the descriptor.
Both applets and a mux stream always have a stream endpoint descriptor. AS SUCH
IT IS NEVER NECESSARY TO TEST FOR THE EXISTENCE OF THE SEDESC FROM ANY SIDE, IT
ALWAYS EXISTS. This explains why as much as possible it's preferable to use the
sedesc to access flags and statuses from any side, rather than bouncing via the
stconn.
An applet's app layer is always a stream, which means that there are always
channels accessible above, and there is always an opposite stream connector and
a stream endpoint descriptor. As such, it's always safe for an applet to access
the other side using sc_opposite().
When an outgoing connection is in the process of being established, the backend
side sedesc has its ->conn pointer pointing to the pending connection, and no
->se. Once the connection is established and a mux is chosen, it's attached to
the ->se. If an applet is used instead of a mux, the appctx is attached to the
sedesc's ->se and ->conn remains NULL.
If either side wants to detach from the other, it must allocate a new virgin
sedesc to replace the existing one, and leave the existing one to the endpoint,
since it continues to describe the stream endpoint. The stconn keeps its state
(modulo the updates related to the disconnection). The previous sedesc points
to a NULL stconn. For example, disconnecting from a backend mux will leave the
entities like this:
+------------------------+
| stream or health check |
+------------------------+
^ \ scf, scb
/ \
| |
\ /
app \ v
+----------+
| stconn |
+----------+
^ \ sedesc
/ \
NULL | |
^ \ /
sc | / sc \ v
+----------+ / +----------+
flags <--| sedesc1 | . . . . . | sedesc2 |--> flags
+----------+ / +----------+
conn / ^ \ se / conn / \ se
+------------+ / / \ | |
| connection |<--' | | v v
+------------+ \ / NULL NULL
mux| ^ |ctx sd \ v
| | | +----------------------+
| | | | mux stream or appctx |
| | | +----------------------+
| | | ^ |
v | | | v
mux_ops | | +----------------+
| +---->| mux connection |
+------ +----------------+
|
