diff options
Diffstat (limited to 'doc/internals')
43 files changed, 11733 insertions, 0 deletions
diff --git a/doc/internals/acl.txt b/doc/internals/acl.txt new file mode 100644 index 0000000..0379331 --- /dev/null +++ b/doc/internals/acl.txt @@ -0,0 +1,82 @@ +2011/12/16 - How ACLs work internally in haproxy - w@1wt.eu + +An ACL is declared by the keyword "acl" followed by a name, followed by a +matching method, followed by one or multiple pattern values : + + acl internal src 127.0.0.0/8 10.0.0.0/8 192.168.0.0/16 + +In the statement above, "internal" is the ACL's name (acl->name), "src" is the +ACL keyword defining the matching method (acl_expr->kw) and the IP addresses +are patterns of type acl_pattern to match against the source address. + +The acl_pattern struct may define one single pattern, a range of values or a +tree of values to match against. The type of the patterns is implied by the +ACL keyword. For instance, the "src" keyword implies IPv4 patterns. + +The line above constitutes an ACL expression (acl_expr). ACL expressions are +formed of a keyword, an optional argument for the keyword, and a list of +patterns (in fact, both a list and a root tree). + +Dynamic values are extracted according to a fetch function defined by the ACL +keyword. This fetch function fills or updates a struct acl_test with all the +extracted information so that a match function can compare it against all the +patterns. The fetch function is called iteratively by the ACL engine until it +reports no more value. This makes sense for instance when checking IP addresses +found in HTTP headers, which can appear multiple times. The acl_test is kept +intact between calls and even holds a context so that the fetch function knows +where to start from for subsequent calls. The match function may also use the +context even though it was not designed for that purpose. + +An ACL is defined only by its name and can be a series of ACL expressions. The +ACL is deemed true when any of its expressions is true. They are evaluated in +the declared order and can involve multiple matching methods. + +So in summary : + + - an ACL is a series of tests to perform on a stream, any of which is enough + to validate the result. + + - each test is defined by an expression associating a keyword and a series of + patterns. + + - a keyword implies several things at once : + - the type of the patterns and how to parse them + - the method to fetch the required information from the stream + - the method to match the fetched information against the patterns + + - a fetch function fills an acl_test struct which is passed to the match + function defined by the keyword + + - the match function tries to match the value in the acl_test against the + pattern list declared in the expression which involved its acl_keyword. + + +ACLs are used by conditional processing rules. A rule generally uses an "if" or +"unless" keyword followed by an ACL condition (acl_cond). This condition is a +series of term suites which are ORed together. Each term suite is a series of +terms which are ANDed together. Terms may be negated before being evaluated in +a suite. A term simply is a pointer to an ACL. + +We could then represent a rule by the following BNF : + + rule = if-cond + | unless-cond + + if-cond (struct acl_cond with ->pol = ACL_COND_IF) + = "if" condition + + unless-cond (struct acl_cond with ->pol = ACL_COND_UNLESS) + = "unless" condition + + condition + = term-suite + | term-suite "||" term-suite + | term-suite "or" term-suite + + term-suite (struct acl_term_suite) + = term + | term term + + term = acl + | "!" acl + diff --git a/doc/internals/api/appctx.txt b/doc/internals/api/appctx.txt new file mode 100644 index 0000000..137ec7b --- /dev/null +++ b/doc/internals/api/appctx.txt @@ -0,0 +1,142 @@ +Instantiation of applet contexts (appctx) in 2.6. + + +1. Background + +Most applets are in fact simplified services that are called by the CLI when a +registered keyword is matched. Some of them only have a ->parse() function +which immediately returns with a final result, while others will return zero +asking for the->io_handler() one to be called till the end. For these ones, a +context is generally needed between calls to know where to restart from. + +Other applets are completely autonomous applets with their init function and +an I/O handler, and these ones also need a persistent context between calls to +the I/O handler. These ones are typically instantiated by "use-service" or by +other means. + +Originally a few integers were provided to keep a trivial state (st0, st1, st2) +and these ones progressively proved insufficient, leading to a "ctx.cli" sub- +context that was allowed to use extra fields of various types. Other applets +preferred to use their own context definition. + +All this resulted in the appctx->ctx to contain a myriad of definitions of +various service contexts, and in some services abusing other services' +definitions by laziness, and others being extended to use their own definition +after having run for a long time on the generic types, some of which were not +noticed and mistakenly used the same storage locations by accident. A massive +cleanup was needed. + + +2. New approach in 2.6 + +In 2.6, there's an "svcctx" pointer that's initialized to NULL before any +instantiation of an applet or of a CLI keyword's function. Applets and keyword +handlers are free to make it point wherever they want, and to find it unaltered +between subsequent calls, including up to the ->release() call. The "st2" state +that was totally abused with random enums is not used anymore and was marked as +deprecated. It's still initialized to zero before the first call though. + +One special area, "svc.storage[]", is large enough to contain any of the +contexts that used to be present under "appctx->ctx". The "svcctx" may be set +to point to this area so that a small structure can be allocated for free and +without requiring error checking. In order to make this easier, a specially +purposed function is provided: "applet_reserve_svcctx()". This function will +require the caller to indicate how large an area it needs, and will return a +pointer to this area after checking that it fits. If it does not, haproxy will +crash. This is purposely done so that it's known during development that if a +small structure doesn't fit, a different approach is required. + +As such, for the vast majority of commands, the process is the following one: + + struct foo_ctx { + int myfield1; + int myfield2; + char *myfield3; + }; + + int io_handler(struct appctx *appctx) + { + struct foo_ctx *ctx = applet_reserve_svcctx(appctx, sizeof(*ctx)); + + if (!ctx->myfield1) { + /* first call */ + ctx->myfield1++; + } + ... + } + +The pointer may be directly accessed from the I/O handler if it's known that it +was already reserved by the init handler or parsing function. Otherwise it's +guaranteed to be NULL so that can also serve as a test for a first call: + + int parse_handler(struct appctx *appctx) + { + struct foo_ctx *ctx = applet_reserve_svcctx(appctx, sizeof(*ctx)); + ctx->myfield1 = 12; + return 0; + } + + int io_handler(struct appctx *appctx) + { + struct foo_ctx *ctx = appctx->svcctx; + + for (; !ctx->myfield1; ctx->myfield1--) { + do_something(); + } + ... + } + +There is no need to free anything because that space is not allocated but just +points to a reserved area. + +If it is too small (its size is APPLET_MAX_SVCCTX bytes), it is preferable to +use it with dynamically allocated structures (pools, malloc, etc). For example: + + int io_handler(struct appctx *appctx) + { + struct foo_ctx *ctx = appctx->svcctx; + + if (!ctx) { + /* first call */ + ctx = pool_alloc(pool_foo_ctx); + if (!ctx) + return 1; + } + ... + } + + void io_release(struct appctx *appctx) + { + pool_free(pool_foo_ctx, appctx->svcctx); + } + +The CLI code itself uses this mechanism for the cli_print_*() functions. Since +these functions are terminal (i.e. not meant to be used in the middle of an I/O +handler as they share the same contextual space), they always reset the svcctx +pointer to place it to the "cli_print_ctx" mapped in ->svc.storage. + + +3. Transition for old code + +A lot of care was taken to make the transition as smooth as possible for +out-of-tree code since that's an API change. A dummy "ctx.cli" struct still +exists in the appctx struct, and it happens to map perfectly to the one set by +cli_print_*, so that if some code uses a mix of both, it will still work. +However, it will build with "deprecated" warnings allowing to spot the +remaining places. It's a good exercise to rename "ctx.cli" in "appctx" and see +if the code still compiles. + +Regarding the "st2" sub-state, it will disappear as well after 2.6, but is +still provided and initialized so that code relying on it will still work even +if it builds with deprecation warnings. The correct approach is to move this +state into the newly defined applet's context, and to stop using the stats +enums STAT_ST_* that often barely match the needs and result in code that is +more complicated than desired (the STAT_ST_* enum values have also been marked +as deprecated). + +The code dealing with "show fd", "show sess" and the peers applet show good +examples of how to convert a registered keyword or an applet. + +All this transition code requires complex layouts that will be removed during +2.7-dev so there is no other long-term option but to update the code (or better +get it merged if it can be useful to other users). diff --git a/doc/internals/api/buffer-api.txt b/doc/internals/api/buffer-api.txt new file mode 100644 index 0000000..ac35300 --- /dev/null +++ b/doc/internals/api/buffer-api.txt @@ -0,0 +1,653 @@ +2018-07-13 - HAProxy Internal Buffer API + + +1. Background + +HAProxy uses a "struct buffer" internally to store data received from external +agents, as well as data to be sent to external agents. These buffers are also +used during data transformation such as compression, header insertion or +defragmentation, and are used to carry intermediary representations between the +various internal layers. They support wrapping at the end, and they carry their +own size information so that in theory it would be possible to use different +buffer sizes in parallel even though this is not currently implemented. + +The format of this structure has evolved over time, to reach a point where it +is convenient and versatile enough to have permitted to make several internal +types converge into a single one (specifically the struct chunk disappeared). + + +2. Representation as of 1.9-dev1 + +The current buffer representation consists in a linear storage area of known +size, with a head position indicating the oldest data, and a total data count +expressed in bytes. The head position, data count and size are expressed as +integers and are positive or null. By convention, the head position is strictly +smaller than the buffer size and the data count is smaller than or equal to the +size, so that wrapping can be resolved with a single subtract. A buffer not +respecting these rules is said to be degenerate. Unless specified otherwise, +the various API functions will adopt an undefined behaviour when passed such a +degenerate buffer. + + Buffer declaration : + + struct buffer { + size_t size; // size of the storage area (wrapping point) + char *area; // start of the storage area + size_t data; // contents length after head + size_t head; // start offset of remaining data relative to area + }; + + + Linear buffer representation : + + area + | + V<--------------------------------------------------------->| size + +-----------+---------------------------------+-------------+ + | |/////////////////////////////////| | + +-----------+---------------------------------+-------------+ + |<--------->|<------------------------------->| + head data ^ + | + tail + + + Wrapping buffer representation : + + area + | + V<--------------------------------------------------------->| size + +---------------+------------------------+------------------+ + |///////////////| |//////////////////| + +---------------+------------------------+------------------+ + |<-------------------------------------->| head + |-------------->| ...data data...|<-----------------| + ^ + | + tail + + +3. Terminology + +Manipulating a buffer just based on a head and a wrapping data count is not +very convenient, so we define a certain number of terms for important elements +characterizing a buffer : + + - origin : pointer to relative position 0 in the storage area. Undefined + when the buffer is not allocated. + + - size : the allocated size of the storage area starting at the origin, + expressed in bytes. A buffer whose size is zero is said not to + be allocated, and its origin in this case is undefined. + + - data : the amount of data the buffer contains, in bytes. It is always + lower than or equal to the buffer's size, hence it is always 0 + for an unallocated buffer. + + - emptiness : a buffer is said to be empty when it contains no data, hence + data == 0. It is possible for such buffers not to be allocated + and to have size == 0 as well. + + - room : the available space in the buffer. This is its size minus data. + + - head : position relative to origin where the oldest data byte is found + (it typically is what send() uses to pick outgoing data). The + head is strictly smaller than the size. + + - tail : position relative to origin where the first spare byte is found + (it typically is what recv() uses to store incoming data). It + is always equal to the buffer's data added to its head modulo + the buffer's size. + + - wrapping : the byte following the last one of the storage area loops back + to position 0. This is called wrapping. The wrapping point is + the first position relative to origin which doesn't belong to + the storage area. There is no wrapping when a buffer is not + allocated. Wrapping requires special care and means that the + regular string manipulation functions are not usable on most + buffers, unless it is known that no wrapping happens. Free + space may wrap as well if the buffer only contains data in the + middle. + + - alignment : a buffer is said to be aligned if its data do not wrap. That + is, its head is strictly before the tail, or the buffer is + empty and the head is null. Aligning a buffer may be required + to use regular string manipulation functions which have no + support for wrapping. + + +A buffer may be in three different states : + - unallocated : size == 0, area == 0 (b_is_null() is true) + - waiting : size == 0, area != 0 + - allocated : size > 0, area > 0 + +It is not permitted to have area == 0 with a non-null size. In addition, the +waiting state may also be used to indicate a read-only buffer which does not +wrap and which must not be freed (e.g. for use with error messages). + +The basic API only covers allocated buffers. Switching to/from the other states +is covered by the management API since it requires specific allocation and free +calls. + + +4. Using buffers + +Buffers are defined in a few files : + - include/common/buf.h : structure definition, and manipulation functions + - include/common/buffer.h : resource management (alloc/free/wait lists) + - include/common/istbuf.h : advanced string manipulation + + +4.1. Basic API + +The basic API is made of the functions which abstract accesses to the buffers +and which help calculating their state, free space or used space. + +====================+==================+======================================= +Function | Arguments/Return | Description +--------------------+------------------+--------------------------------------- +b_is_null() | const buffer *buf| returns true if (and only if) the + | ret: int | buffer is not yet allocated and thus + | | points to a NULL area +--------------------+------------------+--------------------------------------- +b_orig() | const buffer *buf| returns the pointer to the origin of + | ret: char * | the storage, which is the location of + | | byte at offset zero. This is mostly + | | used by functions which handle the + | | wrapping by themselves +--------------------+------------------+--------------------------------------- +b_size() | const buffer *buf| returns the size of the buffer + | ret: size_t | +--------------------+------------------+--------------------------------------- +b_wrap() | const buffer *buf| returns the pointer to the wrapping + | ret: char * | position of the buffer area, which is + | | by definition the first byte not part + | | of the buffer +--------------------+------------------+--------------------------------------- +b_data() | const buffer *buf| returns the number of bytes present in + | ret: size_t | the buffer +--------------------+------------------+--------------------------------------- +b_room() | const buffer *buf| returns the amount of room left in the + | ret: size_t | buffer +--------------------+------------------+--------------------------------------- +b_full() | const buffer *buf| returns true if the buffer is full + | ret: int | +--------------------+------------------+--------------------------------------- +__b_stop() | const buffer *buf| returns a pointer to the byte + | ret: char * | following the end of the buffer, which + | | may be out of the buffer if the buffer + | | ends on the last byte of the area. It + | | is the caller's responsibility to + | | either know that the buffer does not + | | wrap or to check that the result does + | | not wrap +--------------------+------------------+--------------------------------------- +__b_stop_ofs() | const buffer *buf| returns an origin-relative offset + | ret: size_t | pointing to the byte following the end + | | of the buffer, which may be out of the + | | buffer if the buffer ends on the last + | | byte of the area. It's the caller's + | | responsibility to either know that the + | | buffer does not wrap or to check that + | | the result does not wrap +--------------------+------------------+--------------------------------------- +b_stop() | const buffer *buf| returns the pointer to the byte + | ret: char * | following the end of the buffer, which + | | may be out of the buffer if the buffer + | | ends on the last byte of the area +--------------------+------------------+--------------------------------------- +b_stop_ofs() | const buffer *buf| returns an origin-relative offset + | ret: size_t | pointing to the byte following the end + | | of the buffer, which may be out of the + | | buffer if the buffer ends on the last + | | byte of the area +--------------------+------------------+--------------------------------------- +__b_peek() | const buffer *buf| returns a pointer to the data at + | size_t ofs | position <ofs> relative to the head of + | ret: char * | the buffer. Will typically point to + | | input data if called with the amount + | | of output data. It's the caller's + | | responsibility to either know that the + | | buffer does not wrap or to check that + | | the result does not wrap +--------------------+------------------+--------------------------------------- +__b_peek_ofs() | const buffer *buf| returns an origin-relative offset + | size_t ofs | pointing to the data at position <ofs> + | ret: size_t | relative to the head of the + | | buffer. Will typically point to input + | | data if called with the amount of + | | output data. It's the caller's + | | responsibility to either know that the + | | buffer does not wrap or to check that + | | the result does not wrap +--------------------+------------------+--------------------------------------- +b_peek() | const buffer *buf| returns a pointer to the data at + | size_t ofs | position <ofs> relative to the head of + | ret: char * | the buffer. Will typically point to + | | input data if called with the amount + | | of output data. If applying <ofs> to + | | the buffers' head results in a + | | position between <size> and 2*>size>-1 + | | included, a wrapping compensation is + | | applied to the result +--------------------+------------------+--------------------------------------- +b_peek_ofs() | const buffer *buf| returns an origin-relative offset + | size_t ofs | pointing to the data at position <ofs> + | ret: size_t | relative to the head of the + | | buffer. Will typically point to input + | | data if called with the amount of + | | output data. If applying <ofs> to the + | | buffers' head results in a position + | | between <size> and 2*>size>-1 + | | included, a wrapping compensation is + | | applied to the result +--------------------+------------------+--------------------------------------- +__b_head() | const buffer *buf| returns the pointer to the buffer's + | ret: char * | head, which is the location of the + | | next byte to be dequeued. The result + | | is undefined for unallocated buffers +--------------------+------------------+--------------------------------------- +__b_head_ofs() | const buffer *buf| returns an origin-relative offset + | ret: size_t | pointing to the buffer's head, which + | | is the location of the next byte to be + | | dequeued. The result is undefined for + | | unallocated buffers +--------------------+------------------+--------------------------------------- +b_head() | const buffer *buf| returns the pointer to the buffer's + | ret: char * | head, which is the location of the + | | next byte to be dequeued. The result + | | is undefined for unallocated + | | buffers. If applying <ofs> to the + | | buffers' head results in a position + | | between <size> and 2*>size>-1 + | | included, a wrapping compensation is + | | applied to the result +--------------------+------------------+--------------------------------------- +b_head_ofs() | const buffer *buf| returns an origin-relative offset + | ret: size_t | pointing to the buffer's head, which + | | is the location of the next byte to be + | | dequeued. The result is undefined for + | | unallocated buffers. If applying + | | <ofs> to the buffers' head results in + | | a position between <size> and + | | 2*>size>-1 included, a wrapping + | | compensation is applied to the result +--------------------+------------------+--------------------------------------- +__b_tail() | const buffer *buf| returns the pointer to the tail of the + | ret: char * | buffer, which is the location of the + | | first byte where it is possible to + | | enqueue new data. The result is + | | undefined for unallocated buffers +--------------------+------------------+--------------------------------------- +__b_tail_ofs() | const buffer *buf| returns an origin-relative offset + | ret: size_t | pointing to the tail of the buffer, + | | which is the location of the first + | | byte where it is possible to enqueue + | | new data. The result is undefined for + | | unallocated buffers +--------------------+------------------+--------------------------------------- +b_tail() | const buffer *buf| returns the pointer to the tail of the + | ret: char * | buffer, which is the location of the + | | first byte where it is possible to + | | enqueue new data. The result is + | | undefined for unallocated buffers +--------------------+------------------+--------------------------------------- +b_tail_ofs() | const buffer *buf| returns an origin-relative offset + | ret: size_t | pointing to the tail of the buffer, + | | which is the location of the first + | | byte where it is possible to enqueue + | | new data. The result is undefined for + | | unallocated buffers +--------------------+------------------+--------------------------------------- +b_next() | const buffer *buf| for an absolute pointer <p> pointing + | const char *p | to a valid location within buffer <b>, + | ret: char * | returns the absolute pointer to the + | | next byte, which usually is at (p + 1) + | | unless p reaches the wrapping point + | | and wrapping is needed +--------------------+------------------+--------------------------------------- +b_next_ofs() | const buffer *buf| for an origin-relative offset <o> + | size_t o | pointing to a valid location within + | ret: size_t | buffer <b>, returns either the + | | relative offset pointing to the next + | | byte, which usually is at (o + 1) + | | unless o reaches the wrapping point + | | and wrapping is needed +--------------------+------------------+--------------------------------------- +b_dist() | const buffer *buf| returns the distance between two + | const char *from | pointers, taking into account the + | const char *to | ability to wrap around the buffer's + | ret: size_t | end. The operation is not defined if + | | either of the pointers does not belong + | | to the buffer or if their distance is + | | greater than the buffer's size +--------------------+------------------+--------------------------------------- +b_almost_full() | const buffer *buf| returns 1 if the buffer uses at least + | ret: int | 3/4 of its capacity, otherwise + | | zero. Buffers of size zero are + | | considered full +--------------------+------------------+--------------------------------------- +b_space_wraps() | const buffer *buf| returns non-zero only if the buffer's + | ret: int | free space wraps, which means that the + | | buffer contains data that are not + | | touching at least one edge +--------------------+------------------+--------------------------------------- +b_contig_data() | const buffer *buf| returns the amount of data that can + | size_t start | contiguously be read at once starting + | ret: size_t | from a relative offset <start> (which + | | allows to easily pre-compute blocks + | | for memcpy). The start point will + | | typically contain the amount of past + | | data already returned by a previous + | | call to this function +--------------------+------------------+--------------------------------------- +b_contig_space() | const buffer *buf| returns the amount of bytes that can + | ret: size_t | be appended to the buffer at once +--------------------+------------------+--------------------------------------- +b_getblk() | const buffer *buf| gets one full block of data at once + | char *blk | from a buffer, starting from offset + | size_t len | <offset> after the buffer's head, and + | size_t offset | limited to no more than <len> bytes. + | ret: size_t | The caller is responsible for ensuring + | | that neither <offset> nor <offset> + + | | <len> exceed the total number of bytes + | | available in the buffer. Return zero + | | if not enough data was available, in + | | which case blk is left undefined, or + | | the number of bytes read which is + | | equal to the requested size +--------------------+------------------+--------------------------------------- +b_getblk_nc() | const buffer *buf| gets one or two blocks of data at once + | const char **blk1| from a buffer, starting from offset + | size_t *len1 | <ofs> after the beginning of its + | const char **blk2| output, and limited to no more than + | size_t *len2 | <max> bytes. The caller is responsible + | size_t ofs | for ensuring that neither <ofs> nor + | size_t max | <ofs>+<max> exceed the total number of + | ret: int | bytes available in the buffer. Returns + | | 0 if not enough data were available, + | | or the number of blocks filled (1 or + | | 2). <blk1> is always filled before + | | <blk2>. The unused blocks are left + | | undefined, and the buffer is left + | | unaffected. Unused buffers are left in + | | an undefined state +--------------------+------------------+--------------------------------------- +b_reset() | buffer *buf | resets a buffer. The size is not + | ret: void | touched. In practice it resets the + | | head and the data length +--------------------+------------------+--------------------------------------- +b_sub() | buffer *buf | decreases the buffer length by <count> + | size_t count | without touching the head position + | ret: void | (only the tail moves). this may mostly + | | be used to trim pending data before + | | reusing a buffer. The caller is + | | responsible for not removing more than + | | the available data +--------------------+------------------+--------------------------------------- +b_add() | buffer *buf | increase the buffer length by <count> + | size_t count | without touching the head position + | ret: void | (only the tail moves). This is used + | | when adding data at the tail of a + | | buffer. The caller is responsible for + | | not adding more than the available + | | room +--------------------+------------------+--------------------------------------- +b_set_data() | buffer *buf | sets the buffer's length, by adjusting + | size_t len | the buffer's tail only. The caller is + | ret: void | responsible for passing a valid length +--------------------+------------------+--------------------------------------- +b_del() | buffer *buf | deletes <del> bytes at the head of + | size_t del | buffer <b> and updates the head. The + | ret: void | caller is responsible for not removing + | | more than the available data. This is + | | used after sending data from the + | | buffer +--------------------+------------------+--------------------------------------- +b_realign_if_empty()| buffer *buf | realigns a buffer if it's empty, does + | ret: void | nothing otherwise. This is mostly used + | | after b_del() to make an empty + | | buffer's free space contiguous +--------------------+------------------+--------------------------------------- +b_slow_realign() | buffer *buf | realigns a possibly wrapping buffer so + | size_t output | that the part remaining to be parsed + | ret: void | is contiguous and starts at the + | | beginning of the buffer and the + | | already parsed output part ends at the + | | end of the buffer. This provides the + | | best conditions since it allows the + | | largest inputs to be processed at once + | | and ensures that once the output data + | | leaves, the whole buffer is available + | | at once. The number of output bytes + | | supposedly present at the beginning of + | | the buffer and which need to be moved + | | to the end must be passed in <output>. + | | It will effectively make this offset + | | the new wrapping point. A temporary + | | swap area at least as large as b->size + | | must be provided in <swap>. It's up + | | to the caller to ensure <output> is no + | | larger than the difference between the + | | whole buffer's length and its input +--------------------+------------------+--------------------------------------- +b_putchar() | buffer *buf | tries to append char <c> at the end of + | char c | buffer <b>. Supports wrapping. New + | ret: void | data are silently discarded if the + | | buffer is already full +--------------------+------------------+--------------------------------------- +b_putblk() | buffer *buf | tries to append block <blk> at the end + | const char *blk | of buffer <b>. Supports wrapping. Data + | size_t len | are truncated if the buffer is too + | ret: size_t | short or if not enough space is + | | available. It returns the number of + | | bytes really copied +--------------------+------------------+--------------------------------------- +b_move() | buffer *buf | moves block (src,len) left or right + | size_t src | by <shift> bytes, supporting wrapping + | size_t len | and overlapping. + | size_t shift | +--------------------+------------------+--------------------------------------- +b_rep_blk() | buffer *buf | writes the block <blk> at position + | char *pos | <pos> which must be in buffer <b>, and + | char *end | moves the part between <end> and the + | const char *blk | buffer's tail just after the end of + | size_t len | the copy of <blk>. This effectively + | ret: int | replaces the part located between + | | <pos> and <end> with a copy of <blk> + | | of length <len>. The buffer's length + | | is automatically updated. This is used + | | to replace a block with another one + | | inside a buffer. The shift value + | | (positive or negative) is returned. If + | | there's no space left, the move is not + | | done. If <len> is null, the <blk> + | | pointer is allowed to be null, in + | | order to erase a block +--------------------+------------------+--------------------------------------- +b_xfer() | buffer *src | transfers at most <count> bytes from + | buffer *dst | buffer <src> to buffer <dst> and + | size_t cout | returns the number of bytes copied. + | ret: size_t | The bytes are removed from <src> and + | | added to <dst>. The caller guarantees + | | that <count> is <= b_room(dst) +====================+==================+======================================= + + +4.2. String API + +The string API aims at providing both convenient and efficient ways to read and +write to/from buffers using indirect strings (ist). These strings and some +associated functions are defined in ist.h. + +====================+==================+======================================= +Function | Arguments/Return | Description +--------------------+------------------+--------------------------------------- +b_isteq() | const buffer *b | b_isteq() : returns > 0 if the first + | size_t o | <n> characters of buffer <b> starting + | size_t n | at offset <o> relative to the buffer's + | const ist ist | head match <ist>. (empty strings do + | ret: int | match). It is designed to be used with + | | reasonably small strings (it matches a + | | single byte per loop iteration). It is + | | expected to be used with an offset to + | | skip old data. Return value number of + | | matching bytes if >0, not enough bytes + | | or empty string if 0, or non-matching + | | byte found if <0. +--------------------+------------------+--------------------------------------- +b_isteat | struct buffer *b | b_isteat() : "eats" string <ist> from + | const ist ist | the head of buffer <b>. Wrapping data + | ret: ssize_t | is explicitly supported. It matches a + | | single byte per iteration so strings + | | should remain reasonably small. + | | Returns the number of bytes matched + | | and eaten if >0, not enough bytes or + | | matched empty string if 0, or non + | | matching byte found if <0. +--------------------+------------------+--------------------------------------- +b_istput | struct buffer *b | b_istput() : injects string <ist> at + | const ist ist | the tail of output buffer <b> provided + | ret: ssize_t | that it fits. Wrapping is supported. + | | It's designed for small strings as it + | | only writes a single byte per + | | iteration. Returns the number of + | | characters copied (ist.len), 0 if it + | | temporarily does not fit, or -1 if it + | | will never fit. It will only modify + | | the buffer upon success. In all cases, + | | the contents are copied prior to + | | reporting an error, so that the + | | destination at least contains a valid + | | but truncated string. +--------------------+------------------+--------------------------------------- +b_putist | struct buffer *b | b_putist() : tries to copy as much as + | const ist ist | possible of string <ist> into buffer + | ret: size_t | <b> and returns the number of bytes + | | copied (truncation is possible). It + | | uses b_putblk() and is suitable for + | | large blocks. +====================+==================+======================================= + + +4.3. Management API + +The management API makes a distinction between an empty buffer, which by +definition is not allocated but is ready to be allocated at any time, and a +buffer which failed an allocation and is waiting for an available area to be +offered. The functions allow to register on a list to be notified about buffer +availability, to notify others of a number of buffers just released, and to be +and to be notified of buffer availability. All allocations are made through the +standard buffer pools. + +====================+==================+======================================= +Function | Arguments/Return | Description +--------------------+------------------+--------------------------------------- +buffer_almost_full | const buffer *buf| returns true if the buffer is not null + | ret: int | and at least 3/4 of the buffer's space + | | are used. A waiting buffer will match. +--------------------+------------------+--------------------------------------- +b_alloc | buffer *buf | ensures that <buf> is allocated or + | ret: buffer * | allocates a buffer and assigns it to + | | *buf. If no memory is available, (1) + | | is assigned instead with a zero size. + | | The allocated buffer is returned, or + | | NULL in case no memory is available +--------------------+------------------+--------------------------------------- +__b_free | buffer *buf | releases <buf> which must be allocated + | ret: void | and marks it empty +--------------------+------------------+--------------------------------------- +b_free | buffer *buf | releases <buf> only if it is allocated + | ret: void | and marks it empty +--------------------+------------------+--------------------------------------- +offer_buffers() | void *from | offer a buffer currently belonging to + | uint threshold | target <from> to whoever needs + | ret: void | one. Any pointer is valid for <from>, + | | including NULL. Its purpose is to + | | avoid passing a buffer to oneself in + | | case of failed allocations (e.g. need + | | two buffers, get one, fail, release it + | | and wake up self again). In case of + | | normal buffer release where it is + | | expected that the caller is not + | | waiting for a buffer, NULL is fine +====================+==================+======================================= + + +5. Porting code from older versions + +The previous buffer API introduced in 1.5-dev9 (May 2012) used to look like the +following (with the struct renamed to old_buffer here to avoid confusion during +quick lookups at the doc). It's worth noting that the "data" field used to be +part of the struct but with a different type and meaning. It's important to be +careful about potential code making use of &b->data as it will silently compile +but fail. + + Previous buffer declaration : + + struct old_buffer { + char *p; /* buffer's start pointer, separates in and out data */ + unsigned int size; /* buffer size in bytes */ + unsigned int i; /* number of input bytes pending for analysis in the buffer */ + unsigned int o; /* number of out bytes the sender can consume from this buffer */ + char data[0]; /* <size> bytes */ + }; + + Previous linear buffer representation : + + data p + | | + V V + +-----------+--------------------+------------+-------------+ + | |////////////////////|////////////| | + +-----------+--------------------+------------+-------------+ + <---------------------------------------------------------> size + <------------------> <----------> + o i + +There is this correspondence between old and new fields (some will involve a +knowledge of a channel when the output byte count is required) : + + Old | New + --------+---------------------------------------------------- + p | data + head + co_data(channel) // ci_head(channel) + size | size + i | data - co_data(channel) // ci_data(channel) + o | co_data(channel) // channel->output + data | area + --------+----------------------------------------------------- + +Then some common expressions can be mapped like this : + + Old | New + -----------------------+--------------------------------------- + b->data | b_orig(b) + &b->data | b_orig(b) + bi_ptr(b) | ci_head(channel) + bi_end(b) | b_tail(b) + bo_ptr(b) | b_head(b) + bo_end(b) | co_tail(channel) + bi_putblk(b,s,l) | b_putblk(b,s,l) + bo_getblk(b,s,l,o) | b_getblk(b,s,l,o) + bo_getblk_nc(b,s,l,o) | b_getblk_nc(b,s,l,o,0,co_data(channel)) + b->i + b->o | b_data(b) + b->data + b->size | b_wrap(b) + b->i += len | b_add(b, len) + b->i -= len | b_sub(b, len) + b->i = len | b_set_data(b, co_data(channel) + len) + b->o += len | b_add(b, len); channel->output += len + b->o -= len | b_del(b, len); channel->output -= len + -----------------------+--------------------------------------- + +The buffer modification functions are less straightforward and depend a lot on +the context where they are used. It is strongly advised to figure in the list +of functions above what is available based on what is attempted to be done in +the existing code. + +Note that it is very likely that any out-of-tree code relying on buffers will +not use both ->i and ->o but instead will use exclusively ->i on the side +producing data and use exclusively ->o on the side consuming data (such as in a +mux or in an applet). In both cases, it should be assumed that the other side +is always zero and that either ->i or ->o is replaced with ->data, making the +remaining code much simpler (no more code duplication based on the data +direction). diff --git a/doc/internals/api/event_hdl.txt b/doc/internals/api/event_hdl.txt new file mode 100644 index 0000000..72eeff8 --- /dev/null +++ b/doc/internals/api/event_hdl.txt @@ -0,0 +1,1015 @@ + ----------------------------------------- + event_hdl Guide - version 2.8 + ( Last update: 2022-11-14 ) + ------------------------------------------ + +ABSTRACT +-------- + +The event_hdl support is a new feature of HAProxy 2.7. It is a way to easily +handle general events in a simple to maintain fashion, while keeping core code +impact to the bare minimum. + +This document first describes how to use already supported events, +then how to add support for your very own events. + +This feature is quite new for now. The API is not frozen and will be +updated/modified/improved/extended as needed. + +SUMMARY +------- + + 1. event_hdl introduction + 2. How to handle existing events + 2.1 SYNC mode + 2.2 ASYNC mode + 2.2.1 normal version + 2.2.2 task version + 2.3 Advanced features + 2.3.1 sub_mgmt + 2.3.2 subscription external lookups + 2.3.3 subscription ptr + 2.3.4 private_free + 3. How to add support for new events + 3.1 Declaring a new event data structure + 3.2 Publishing an event + 4. Subscription lists + 5. misc/helper functions + + +1. EVENT_HDL INTRODUCTION +----------------------- + +EVENT_HDL provides two complementary APIs, both are implemented +in src/event_hdl.c and include/haproxy/event_hdl(-t).h: + +One API targeting developers that want to register event +handlers that will be notified when specific events occur in the process. +(See section 2.) + +One API targeting developers that want to notify registered handlers about +an event that is happening in the process. +(See section 3.) + +2. HOW TO HANDLE EXISTING EVENTS +--------------------- + +To handle existing events, you must first decide which events you're +interested in. + +event types are defined as follow: + +``` + /* type for storing event subscription type */ + typedef struct event_hdl_sub_type + { + /* up to 256 families, non cumulative, adjust if needed */ + uint8_t family; + /* up to 16 sub types using bitmasks, adjust if needed */ + uint16_t subtype; + } event_hdl_sub_type; +``` + +For an up to date list of already supported events, +please refer to include/haproxy/event_hdl-t.h +At the end of the file you will find existing event types. + +Each event family provides an unique data structure that will +be provided to the event handler (registered to one or more +event subtypes) when such events occur. + +An event handler can subscribe to a single event family type at a time, but +within the family type it can subscribe to multiple event subtypes. + + For example, let's consider the SERVER family type. + + Let's assume it provides the event_hdl_cb_data_server data structure. + + We can register a handler that will be notified for + every SERVER event types using: + EVENT_HDL_SUB_SERVER + + This will include EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_SUB_SERVER_DEL [...] + + But we can also subscribe to a specific subtype only, + for example server deletion: + EVENT_HDL_SUB_SERVER_DEL + + You can even combine multiple SERVER subtypes using + event_hdl_sub_type_add function helper: + event_hdl_sub_type_add(EVENT_HDL_SUB_SERVER_DEL, + EVENT_HDL_SUB_SERVER_ADD) + + (will refer to server deletion as well as server addition) + +Registering a handler comes into multiple flavors: + + SYNC mode: + handler is called in a blocking manner directly from the + thread that publishes the event. + This mode should be used with precaution because it could + slow the caller or cause deadlocks if used improperly. + + Sync mode is useful when you directly depend on data or + state consistency from the caller. + + Sync mode gives you access to unsafe elements in the data structure + provided by the caller (again, see event_hdl-t.h for more details). + The data structure may provide lock hints in the unsafe section + so that you know which locks are already held within the + calling context, hopefully preventing you from relocking + an already locked element and preventing deadlocks. + + ASYNC mode: + handler is called in a non-blocking manner + (in a dedicated tasklet), + thus, the caller (that published the event) is not affected + by the handler. (time wise and data wise) + + This is the safest way to handle events, + but it also comes with a limitation: + + unsafe elements in the data structure provided by + the caller SHOULD be used under NO circumstances. + Indeed, only safe elements are meant to be used + when handling the event in async mode. + + ASYNC mode is declined in 2 different versions: + normal: + handler is simply a function pointer + (same prototype as sync mode), + that is called asynchronously with relevant data + when the event is published. Only difference with + sync mode here is that 'unsafe' data provided + by the data structure may not be used. + task: + handler is a user defined task(let) that uses an event + queue to consume pending events. + This mode is interesting when you need to perform + advanced operations or you need to handle the event + in an already existing task context. + It is a bit more complicated to setup, but really + nothing to worry about, some examples will be + provided later in this document. + +event subscription is performed using the function: + + event_hdl_subscribe(list, event, hdl); + + The function returns 1 in case of success, + and 0 in case of failure (bad arguments, or memory error) + + The function may BUG_ON if used improperly (invalid arguments) + + <list> is either user specified list used to store the + new subscription, or NULL if you want to store the subscription + in the process global list. + + <list> is also asked when publishing an event, + so specifying list could be useful, if, for example, + you only want to subscribe to a specific subscription list + (see this as a scope for example, NULL being full scope, + and specific list being limited scope) + + We will use server events as an example: + + You could register to events for ALL servers by using the + global list (NULL), or only to a specific server events + by using the subscription list dedicated to a single server. + + <event> are the events (family.subtypes) you're subscribing to + + <hdl> contains required handler options, it must be provided using + EVENT_HDL_(TASK_)(A)SYNC() and EVENT_HDL_ID_(TASK_)(A)SYNC() + helper macros. + + See include/haproxy/event_hdl.h or below to know which macro + best suits your needs. + + When registering a handler, you have the ability to provide an + unique ID (using EVENT_HDL_ID_ macro family) that could be used + later to perform lookups on the subscription. + ID is stored as an uint64_t hash that is expected to be computed using + general purpose event_hdl_id inline function provided by event_hdl.h. + Not providing an ID (using EVENT_HDL_ macro family) + results in the subscription being considered as anonymous. + As the name implies, anonymous subscriptions don't support lookups. + +2.1 SYNC MODE +--------------------- + +Example, you want to register a sync handler that will be called when +a new server is added. + +Here is what the handler function will look like: +``` +void my_sync_handler(const struct event_hdl_cb *cb, void *private) +{ + const struct event_hdl_cb_data_server *server = cb->e_data; + + /* using EVENT_HDL_ASSERT_SYNC is a good practice to ensure + * that the function breaks if used in async mode + * (because we will access unsafe data in this function that + * is sync mode only) + */ + EVENT_HDL_ASSERT_SYNC(cb); + printf("I've been called for '%s', private = %p\n", + event_hdl_sub_type_to_string(cb->e_type), private); + printf("server name is '%s'\n", server->safe.name); + + /* here it is safe to use unsafe data */ + printf("server ptr is '%p'\n", server->unsafe.ptr); + + /* from here you have the possibility to manage the subscription + * cb->sub_mgmt->unsub(cb->sub_mgmt); + * // hdl will be removed from the subscription list + */ +} +``` + +Here is how you perform the subscription: + +anonymous subscription: +``` + int private = 10; + + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_SYNC(my_sync_handler, &private, NULL)); +``` + +identified subscription: +``` + int private = 10; + uint64_t id = event_hdl_id("test", "sync"); + + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ID_SYNC(id, + my_sync_handler, + &private, + NULL)); + +``` + +identified subscription where freeing private is required when subscription ends: +(also works for anonymous) +(more on this feature in 2.3.4) +``` + int *private = malloc(sizeof(*private)); + uint64_t id = event_hdl_id("test", "sync_free"); + + BUG_ON(!private); + *private = 10; + + /* passing free as 'private_free' function so that + * private can be freed when unregistering is performed + */ + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ID_SYNC(id, + my_sync_handler, + private, + free)); + + + /* ... */ + + // unregistering the identified hdl + if (event_hdl_lookup_unsubscribe(NULL, id)) { + printf("private will automatically be freed!\n"); + } +``` + +2.2 ASYNC MODE +--------------------- + +As mentioned before, async mode comes in 2 flavors, normal and task. + +2.2.1 NORMAL VERSION +--------------------- + +Normal is meant to be really easy to use, and highly compatible with sync mode. + +(Handler can easily be converted or copy pasted from async to sync mode +and vice versa) + +Quick warning about sync to async handler conversion: + +please always use EVENT_HDL_ASSERT_SYNC whenever you develop a +sync handler that performs unsafe data access. + +This way, if the handler were to be converted or copy pasted as is to +async mode without removing unsafe data accesses, +the handler will forcefully fail to indicate an error so that you +know something has to be fixed in your handler code. + +Back to our async handler, let's say you want to declare an +async handler that will be called when a new server is added. + +Here is what the handler function will look like: +``` +void my_async_handler(const struct event_hdl_cb *cb, void *private) +{ + const struct event_hdl_cb_data_server *server = cb->e_data; + + printf("I've been called for '%s', private = %p\n", + event_hdl_sub_type_to_string(cb->e_type), private); + printf("server name is '%s'\n", server->safe.name); + + /* here it is not safe to use unsafe data */ + + /* from here you have the possibility to manage the subscription + * cb->sub_mgmt->unsub(cb->sub_mgmt); + * // hdl will be removed from the subscription list + */ +} +``` + +Note that it is pretty similar to sync handler, except +for unsafe data access. + +Here is how you declare the subscription: + +anonymous subscription: +``` + int private = 10; + + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ASYNC(my_async_handler, &private, NULL)); +``` + +identified subscription: +``` + int private = 10; + uint64_t id = event_hdl_id("test", "async"); + + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ID_ASYNC(id, + my_async_handler, + &private, + NULL)); + +``` + +identified subscription where freeing private is required when subscription ends: +(also works for anonymous) +``` + int *private = malloc(sizeof(*private)); + uint64_t id = event_hdl_id("test", "async_free"); + + BUG_ON(!private); + *private = 10; + + /* passing free as 'private_free' function so that + * private can be freed when unregistering is performed + */ + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ID_ASYNC(id, + my_async_handler, + private, + free)); + + /* ... */ + + // unregistering the identified hdl + if (event_hdl_lookup_unsubscribe(NULL, id)) { + printf("private will automatically be freed when " + "all pending events referencing private " + "are consumed!\n"); + } +``` + +2.2.2 TASK VERSION +--------------------- + +task version requires a bit more setup, but it's pretty +straightforward actually. + + +First, you need to initialize an event queue that will be used +by event_hdl facility to push you events according to your subscription: + +``` + event_hdl_async_equeue my_q; + + event_hdl_async_equeue_init(&my_q); +``` + + +Then, you need to declare a task(let) (or reuse existing task(let)) + +It is your responsibility to make sure that the task(let) still exists +(is not freed) when calling the subscribe function +(and that the task remains valid as long as the subscription is). + +When a subscription referencing your task is over +(either ended because of list purge, external code or from the handler itself), +you will receive the EVENT_HDL_SUB_END event. +When you receive this event, you must free it as usual and you can safely +assume that the related subscription won't be sending you any more events. + +Here is what your task will look like (involving a single event queue): + +``` +struct task *event_hdl_async_task_my(struct task *task, + void *ctx, unsigned int state) +{ + struct tasklet *tl = (struct tasklet *)task; + event_hdl_async_equeue *queue = ctx; + struct event_hdl_async_event *event; + struct event_hdl_cb_data_server *srv; + uint8_t done = 0; + + while ((event = event_hdl_async_equeue_pop(queue))) + { + if (event_hdl_sub_type_equal(event->type, EVENT_HDL_SUB_END)) { + done = 1; + event_hdl_async_free_event(event); + printf("no more events to come, " + "subscription is over\n"); + break; + } + + srv = event->data; + + printf("task event %s, %d (name = %s)\n", + event_hdl_sub_type_to_string(event->type), + *((int *)event->private), srv->safe.name); + event_hdl_async_free_event(event); + } + + if (done) { + /* our job is done, subscription is over: + * no more events to come + */ + tasklet_free(tl); + return NULL; + } + return task; +} + +``` + +Here is how we would initialize the task event_hdl_async_task_my: +``` + struct tasklet *my_task; + + my_task = tasklet_new(); + BUG_ON(!my_task); + my_task->context = &my_q; // we declared my_q previously in this example + /* we declared event_hdl_async_task_my previously + * in this example + */ + my_task->process = event_hdl_async_task_my; + +``` + +Given our task and our previously initialized event queue, here is how +to perform the subscription: +``` + int test_val = 11; + uint64_t id = event_hdl_id("test", "my_task"); + + /* anonymous variant */ + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ASYNC_TASK(&my_q, + my_task, + &test_val, + NULL)); + /* identified variant */ + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ID_ASYNC_TASK(id, + &my_q, + my_task, + &test_val, + NULL)); +``` + +Note: it is not recommended to perform multiple subscriptions + that share the same event queue or same task(let) (or both) + + That is, having more than one subscription waking a task(let) + and/or feeding the same event queue. + + No check is performed on this when registering, so the API + won't prevent you from doing it. + + If you are going to do this anyway despite this warning: + + In the case you need to stop the task prematurely + (if this is not going to happen please skip this paragraph): + You are responsible for acknowledging the end of every + active subscriptions that refer to your task or + your event queue(s). + And you really don't want a subscription associated with + your task or event queue to keep going when the task + is not active anymore because: + 1: there will be memory leak + (event queue might continue to receive new events) + 2: there is a 100% chance of process crash in case of event + because we will try to wake a task (your task) + that might already be freed. Thus UAF will occur. + +2.3 ADVANCED FEATURES +----------------------- + +We've already covered some of these features in the previous examples. +Here is a documented recap. + + +2.3.1 SUB MGMT +----------------------- + +From an event handler context, either sync or async mode: + You have the ability to directly manage the subscription + that provided the event. + +As of today, these actions are supported: + - Consulting the subscription. + - Modifying the subscription (resubscribing within same family) + - Unregistering the subscription (unsubscribing). + +To do this, consider the following structure: +``` + struct event_hdl_sub_mgmt + { + /* manage subscriptions from event + * this must not be used directly because + * locking might be required + */ + struct event_hdl_sub *this; + /* safe functions than can be used from + * event context (sync and async mode) + */ + struct event_hdl_sub_type (*getsub)(const struct event_hdl_sub_mgmt *); + int (*resub)(const struct event_hdl_sub_mgmt *, struct event_hdl_sub_type); + void (*unsub)(const struct event_hdl_sub_mgmt *); + }; + +``` +A reference to this structure is provided in every handler mode. + +Sync mode and normal async mode (directly from the callback data pointer): +``` + const struct event_hdl_cb *cb; + // cb->sub_mgmt + // cb->sub_mgmt->getsub(cb->sub_mgmt); + // cb->sub_mgmt->unsub(cb->sub_mgmt); +``` + +task and notify async modes (from the event): +``` + struct event_hdl_async_event *event; + // event->sub_mgmt + // event->sub_mgmt.getsub(&event->sub_mgmt); + // event->sub_mgmt.unsub(&event->sub_mgmt); +``` + +2.3.2 SUBSCRIPTION EXTERNAL LOOKUPS +----------------------- + +As you've seen in 2.3.1, managing the subscription directly +from the handler is a possibility. + +But for identified subscriptions, you also have the ability to +perform lookups and management operations on specific subscriptions +within a list based on their ID, anywhere in the code. + +/!\ This feature is not available for anonymous subscriptions /!\ + +Here are the actions already supported: + + - unregistering a subscription (unsubscribing) + - updating a subscription (resubscribing within same family) + - getting a ptr/reference to the subscription + +Those functions are documented in event_hdl.h +(search for EVENT_HDL_LOOKUP section). + +To select a specific subscription, you must provide +the unique identifier (uint64_t hash) that was provided when subscribing. +(using event_hdl_id(scope, name) function) + +Notes: + "id" is only unique within a given subscription list. + + When using event_hdl_id to provide the id: + It is your responsibility to make sure that you "own" + the scope if you rely on name to be "free". + + As ID computation is backed by xxhash hash API, + you should be aware that hash collisions could occur, + but are extremely rare and are thus considered safe + enough for this usage. + (see event_hdl.h for implementation details) + + Please consider ptr based subscription management if + these limitations don't fit your requirements. + +Here are some examples: + +unsubscribing: +``` + /* registering "scope":"name" subscription */ + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ID_SYNC(event_hdl_id("scope", "name"), + my_sync_handler, + NULL, + NULL)); + /* unregistering "scope":"name" subscription */ + event_hdl_lookup_unsubscribe(NULL, event_hdl_id("scope", "name")); +``` + +2.3.3 SUBSCRIPTION PTR +----------------------- + +To manage existing subscriptions from external code, +we already talked about identified subscriptions that +allow lookups within list. + +But there is another way to accomplish this. + +When subscribing, you can use the event_hdl_subscribe_ptr() function +variant (same arguments as event_hdl_subscribe()). + +What this function does, is instead of returning 1 in case of +success and 0 in case of failure: it returns a valid subscription ptr +for success and NULL for failure. + +Returned ptr is guaranteed to remain valid even if subscription +is ended meanwhile because the ptr is internally guarded with a refcount. + +Thus, as long as you don't explicitly unregister the subscription with +event_hdl_unsubscribe() or drop the reference using event_hdl_drop(), +subscription ptr won't be freed. + +This ptr will allow you to use the following subscription +management functions from external code: + + - event_hdl_take() to increment subscription ptr refcount + (automatically incremented when using event_hdl_subscribe_ptr) + - event_hdl_drop() to decrement subscription ptr refcount + - event_hdl_resubscribe() to modify subscription subtype + - event_hdl_unsubscribe() to end the subscription + (refcount will be automatically decremented) + +Here is an example: +``` + struct event_hdl_sub *sub_ptr; + + /* registering a subscription with subscribe_ptr */ + sub_ptr = event_hdl_subscribe_ptr(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_SYNC(my_sync_handler, + NULL, + NULL)); + + /* ... */ + + /* unregistering the subscription */ + event_hdl_unsubscribe(sub_ptr); +``` + +Regarding identified subscriptions that were registered using the non ptr +subscribe function: + +You still have the ability to get a reference to the related subscription +(if it still exists), by using event_hdl_lookup_take(list, id) function. +event_hdl_lookup_take will return a subscription ptr in case of success +and NULL in case of failure. +Returned ptr reference is automatically incremented, so it is safe to use. + +Please don't forget to drop the reference +when holding the ptr is no longer needed. + +Example: +``` + struct event_hdl_sub *sub_ptr = NULL; + + /* registering subscription id "test":"ptr" with normal subscribe */ + if (event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD, + EVENT_HDL_ID_SYNC(event_hdl_id("test", "ptr"), + my_sync_handler, + NULL, + NULL))) { + /* fetch ref to subscription "test":"ptr" */ + sub_ptr = event_hdl_lookup_take(NULL, + event_hdl_id("test", "ptr")); + + /* unregister the subscription using lookup */ + event_hdl_lookup_unsubscribe(NULL, + event_hdl_id("test", "ptr")); + } + + /* ... */ + + /* unregistering the subscription with ptr + * will do nothing because subscription was + * already ended by lookup_unsubscribe, but + * here the catch is that sub_ptr is still + * valid so this won't crash the program + */ + if (sub_ptr) { + event_hdl_unsubscribe(sub_ptr); + /* unsubscribe will also result in subscription + * reference drop, thus subscription will be freed here + * because sub_ptr was the last active reference. + * You must not use sub_ptr anymore past this point + * or UAF could occur + */ + } + +``` + +2.3.4 PRIVATE FREE +----------------------- + +Upon handler subscription, you have the ability to provide +a private data pointer that will be passed to the handler +when subscribed events occur. + +Sometimes this private data pointer will rely on dynamically allocated memory. +And in such cases, you have no way of knowing when +freeing this pointer can be done safely. + +You could be tempted to think that freeing right after performing +the unsubscription could be safe. +But this is not the case, remember we could be dealing with async handlers +that might still consume pending events even though unsubscription +has been performed from external code. + +To deal with this, you may want to provide the private_free +function pointer upon subscription. +This way, private_free function will automatically be called +(with private as argument) when private is no longer be used. + +Example: +First we declare our private free function: +``` +void my_private_free(void *my_private_data) { + /* here we only call free, + * but you could do more sophisticated stuff + */ + free(my_private_data); +} +``` +Then: +``` + char *my_private_data = strdup("this string needs to be freed"); + + BUG_ON(!my_private_data); + + event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_DEL, + EVENT_HDL_ID_ASYNC(event_hdl_id("test", "private"), + my_async_handler, + my_private_data, + my_private_free)); + + /* freeing my_private_data is not required anymore, + * it will be automatically freed by our private free + * function when subscription ends + */ + + /* unregistering "test":"private" subscription */ + event_hdl_lookup_unsubscribe(NULL, event_hdl_id("test", "private")); + + /* my_private_free will be automatically summoned when my_private_data + * is not referenced anymore + */ +``` + +3 HOW TO ADD SUPPORT FOR NEW EVENTS +----------------------- + +Adding support for a new event is pretty straightforward. + +First, you need to declare a new event subtype in event_hdl-t.h file +(bottom of the file). + +You might want to declare a whole new event family, in which case +you declare both the new family and the associated subtypes (if any). + +``` + #define EVENT_HDL_SUB_NEW_FAMILY EVENT_HDL_SUB_FAMILY(4) + #define EVENT_HDL_SUB_NEW_FAMILY_SUBTYPE_1 EVENT_HDL_SUB_TYPE(4,0) +``` + +Then, you need to update the event_hdl_sub_type_map map, +defined in src/event_hdl.c file (top of the file) +to add string to event type and event type to string conversion support. +You just need to add the missing entries corresponding to +the event family / subtypes you've defined. + +Please follow this procedure: + You only added a new subtype to existing family: go to section 3.2 + You added a new family: go to section 3.1 + +3.1 DECLARING A NEW EVENT DATA STRUCTURE +----------------------- + +You have the ability to provide additional data for a given +event family when such events occur. + +Note that it is not mandatory: you could simply declare a new event family +that does not provide any data. +If this is your case, you can skip this section and go to 3.2 section. + +Now, take a look at this event data structure template +(also defined at the top of event_hdl-t.h file): +``` + /* event data struct are defined as followed */ + struct event_hdl_cb_data_template { + struct { + /* safe data can be safely used from both + * sync and async functions + * data consistency is guaranteed + */ + } safe; + struct { + /* unsafe data may only be used from sync functions: + * in async mode, data consistency cannot be guaranteed + * and unsafe data may already be stale, thus using + * it is highly discouraged because it + * could lead to undefined behavior + * (UAF, null dereference...) + */ + } unsafe; + }; +``` + +This structure template allows you to easily create a new event +data structure that can be provided with your new event family. + +You should name it after 'struct event_hdl_cb_data_new_family' so that it is +easy to guess the event family it relates to. + +Indeed, each event data structure is to be associated with an +unique event family type. +For each subtypes within a family type, the associated data structure +should be provided when publishing the event. + +The event data struct declaration should not be performed +directly under event_hdl-t.h file: + + It should be done in the header files of the corresponding + facility that will publish/provide this event. + + Example: struct event_hdl_cb_data_server, provided for the + EVENT_HDL_SUB_SERVER event family, is going to be declared in + include/haproxy/server-t.h file. + + However, in event_hdl-t.h, where you declare event family/subtypes, + you should add comments or links to the file containing the relevant + data struct declaration. This way we make sure all events related + information is centralized in event_hdl-t.h while keeping it clean + and not depending on any additional includes (you are free to + depend on specific data types within your custom event data structure). + +Please make sure that EVENT_HDL_ASYNC_EVENT_DATA (defined in event_hdl-t.h) +is greater than sizeof(event_hdl_cb_data_new_family). + +It is required for async handlers to properly consume event data. + +You are free to adjust EVENT_HDL_ASYNC_EVENT_DATA size if needed. + +If EVENT_HDL_ASYNC_EVENT_DATA is not big enough to store your new +event family struct, a compilation assert triggered by EVENT_HDL_CB_DATA +will occur. In addition to this, an extra runtime BUG_ON will make +sure the condition is met when publishing the event. +The goal here is to force haproxy to fail explicitly so you know that +something must be done on your side. + +3.1 PUBLISHING AN EVENT +----------------------- + +Publishing an event is really simple. +It relies on the event_hdl_publish function. + +The function is defined as follow: +``` + int event_hdl_publish(event_hdl_sub_list *sub_list, + event_hdl_sub_type e_type, + const struct event_hdl_cb_data *data); +``` + +We will ignore sub_list argument for now. +In the examples below, we will use sub_list = NULL. +Go to section 4 for a full picture about this feature. + +<e_type>: the event type that should be published. + All subscriptions referring to this event within + a subscription list context will be notified about the event. +<data>: data provided for the event family of <e_type> + If <e_type>.family does not provide additional data, + data should be set to NULL. + If <e_type>.family does provide additional data, data should be set + using EVENT_HDL_CB_DATA macro. + (see the example below) + +The function returns 1 in case of SUCCESS (handlers successfully notified) +and 0 in case of FAILURE (no handlers notified, because of memory error). + +Event publishing can be performed from anywhere in the code. +(this example does not compile) +``` + struct event_hdl_cb_data_new_family event_data; + + /* first we need to prepare event data + * that will be provided to event handlers + */ + + /* safe data, available from both sync and async contexts */ + event_data.safe.my_custom_data = x; + + /* unsafe data, only available from sync contexts */ + event_data.unsafe.my_unsafe_data = y; + + /* once data is prepared, we can publish the event */ + event_hdl_publish(NULL, + EVENT_HDL_SUB_NEW_FAMILY_SUBTYPE_1, + EVENT_HDL_CB_DATA(&event_data)); + + /* EVENT_HDL_SUB_NEW_FAMILY_SUBTYPE_1 event was + * successfully published in global subscription list + */ +``` + +-------------------------------------------------------------------------------- +|You should know that there is currently a limitation about publish function: | +|The function should not be used from critical places | +|(where the calling frequency is high | +|or where timing sensitivity is high). | +| | +|Because in current implementation, subscription list lookups are not | +|optimized for such uses cases. | +-------------------------------------------------------------------------------- + +4 SUBSCRIPTION LISTS +----------------------- + +As you may already know, EVENT_HDL API main functions rely on +subscription lists. +Providing NULL where subscription list argument is required +allows to use the implicit global subscription list. + +But you can also provide a specific subscription list, example: + subscription list associated with a single entity so that you only + subscribe to events of this single entity + +A subscription list is of type event_hdl_sub_list. +It is defined in event_hdl-t.h + +To make use of this feature, you should know about these 2 functions: + +event_hdl_sub_list_init(list): use this fcn to initialize + a new subscription list. + +Example: +``` + event_hdl_sub_list my_custom_list; + + event_hdl_sub_list_init(&my_custom_list); +``` + +event_hdl_sub_list_destroy(list): use this fcn to destroy + an existing subscription list. + +Example: +``` + event_hdl_sub_list_init(&my_custom_list); +``` + + Using this function will cause all the existing subscriptions + within the provided sub_list to be properly unregistered + and deleted according to their types. + +Now we'll take another quick look at event_hdl_publish() function: + +Remember that the function is defined as follow: +``` + int event_hdl_publish(event_hdl_sub_list *sub_list, + event_hdl_sub_type e_type, + const struct event_hdl_cb_data *data); +``` + +In the previous examples, we used sub_list = NULL. + +if sub_list is NULL: + event will be published in in global list +else + event will be published in user specified sub_list + +5 MISC/HELPER FUNCTIONS +----------------------- + +Don't forget to take a look at MISC/HELPER FUNCTIONS in +include/haproxy/event_hdl.h (end of the file) for a +complete list of helper functions / macros. + +We've already used some, if not the vast majority +in the examples shown in this document. + +This includes, to name a few: + - event types manipulation + - event types comparison + - lookup id computing + - subscriber list management (covered in section 4) + - sync/async handler helpers diff --git a/doc/internals/api/filters.txt b/doc/internals/api/filters.txt new file mode 100644 index 0000000..f1d2f34 --- /dev/null +++ b/doc/internals/api/filters.txt @@ -0,0 +1,1188 @@ + ----------------------------------------- + Filters Guide - version 2.9 + ( Last update: 2021-02-24 ) + ------------------------------------------ + Author : Christopher Faulet + Contact : christopher dot faulet at capflam dot org + + +ABSTRACT +-------- + +The filters support is a new feature of HAProxy 1.7. It is a way to extend +HAProxy without touching its core code and, in certain extent, without knowing +its internals. This feature will ease contributions, reducing impact of +changes. Another advantage will be to simplify HAProxy by replacing some parts +by filters. As we will see, and as an example, the HTTP compression is the first +feature moved in a filter. + +This document describes how to write a filter and what to keep in mind to do +so. It also talks about the known limits and the pitfalls to avoid. + +As said, filters are quite new for now. The API is not freezed and will be +updated/modified/improved/extended as needed. + + + +SUMMARY +------- + + 1. Filters introduction + 2. How to use filters + 3. How to write a new filter + 3.1. API Overview + 3.2. Defining the filter name and its configuration + 3.3. Managing the filter lifecycle + 3.3.1. Dealing with threads + 3.4. Handling the streams activity + 3.5. Analyzing the channels activity + 3.6. Filtering the data exchanged + 4. FAQ + + + +1. FILTERS INTRODUCTION +----------------------- + +First of all, to fully understand how filters work and how to create one, it is +best to know, at least from a distance, what is a proxy (frontend/backend), a +stream and a channel in HAProxy and how these entities are linked to each other. +In doc/internals/api/layers.txt is a good overview of the different layers in +HAProxy and in doc/internals/muxes.pdf is described the flow between the +different muxes. + +Then, to support filters, many callbacks has been added to HAProxy at different +places, mainly around channel analyzers. Their purpose is to allow filters to +be involved in the data processing, from the stream creation/destruction to +the data forwarding. Depending of what it should do, a filter can implement all +or part of these callbacks. For now, existing callbacks are focused on +streams. But future improvements could enlarge filters scope. For instance, it +could be useful to handle events at the connection level. + +In HAProxy configuration file, a filter is declared in a proxy section, except +default. So the configuration corresponding to a filter declaration is attached +to a specific proxy, and will be shared by all its instances. it is opaque from +the HAProxy point of view, this is the filter responsibility to manage it. For +each filter declaration matches a uniq configuration. Several declarations of +the same filter in the same proxy will be handle as different filters by +HAProxy. + +A filter instance is represented by a partially opaque context (or a state) +attached to a stream and passed as arguments to callbacks. Through this context, +filter instances are stateful. Depending the filter is declared in a frontend or +a backend section, its instances will be created, respectively, when a stream is +created or when a backend is selected. Their behaviors will also be +different. Only instances of filters declared in a frontend section will be +aware of the creation and the destruction of the stream, and will take part in +the channels analyzing before the backend is defined. + +It is important to remember the configuration of a filter is shared by all its +instances, while the context of an instance is owned by a uniq stream. + +Filters are designed to be chained. It is possible to declare several filters in +the same proxy section. The declaration order is important because filters will +be called one after the other respecting this order. Frontend and backend +filters are also chained, frontend ones called first. Even if the filters +processing is serialized, each filter will bahave as it was alone (unless it was +developed to be aware of other filters). For all that, some constraints are +imposed to filters, especially when data exchanged between the client and the +server are processed. We will discuss again these constraints when we will tackle +the subject of writing a filter. + + + +2. HOW TO USE FILTERS +--------------------- + +To use a filter, the parameter 'filter' should be used, followed by the filter +name and, optionally, its configuration in the desired listen, frontend or +backend section. For instance : + + listen test + ... + filter trace name TST + ... + + +See doc/configuration.txt for a formal definition of the parameter 'filter'. +Note that additional parameters on the filter line must be parsed by the filter +itself. + +The list of available filters is reported by 'haproxy -vv' : + + $> haproxy -vv + HAProxy version 1.7-dev2-3a1d4a-33 2016/03/21 + Copyright 2000-2016 Willy Tarreau <willy@haproxy.org> + + [...] + + Available filters : + [COMP] compression + [TRACE] trace + + +Multiple filter lines can be used in a proxy section to chain filters. Filters +will be called in the declaration order. + +Some filters can support implicit declarations in certain circumstances +(without the filter line). This is not recommended for new features but are +useful for existing ones moved in a filter, for backward compatibility +reasons. Implicit declarations are supported when there is only one filter used +on a proxy. When several filters are used, explicit declarations are mandatory. +The HTTP compression filter is one of these filters. Alone, using 'compression' +keywords is enough to use it. But when at least a second filter is used, a +filter line must be added. + + # filter line is optional + listen t1 + bind *:80 + compression algo gzip + compression offload + server srv x.x.x.x:80 + + # filter line is mandatory for the compression filter + listen t2 + bind *:81 + filter trace name T2 + filter compression + compression algo gzip + compression offload + server srv x.x.x.x:80 + + + + +3. HOW TO WRITE A NEW FILTER +---------------------------- + +To write a filter, there are 2 header files to explore : + + * include/haproxy/filters-t.h : This is the main header file, containing all + important structures to use. It represents the + filter API. + + * include/haproxy/filters.h : This header file contains helper functions that + may be used. It also contains the internal API + used by HAProxy to handle filters. + +To ease the filters integration, it is better to follow some conventions : + + * Use 'flt_' prefix to name the filter (e.g flt_http_comp or flt_trace). + + * Keep everything related to the filter in a same file. + +The filter 'trace' can be used as a template to write new filter. It is a good +start to see how filters really work. + +3.1 API OVERVIEW +---------------- + +Writing a filter can be summarized to write functions and attach them to the +existing callbacks. Available callbacks are listed in the following structure : + + struct flt_ops { + /* + * Callbacks to manage the filter lifecycle + */ + int (*init) (struct proxy *p, struct flt_conf *fconf); + void (*deinit) (struct proxy *p, struct flt_conf *fconf); + int (*check) (struct proxy *p, struct flt_conf *fconf); + int (*init_per_thread) (struct proxy *p, struct flt_conf *fconf); + void (*deinit_per_thread)(struct proxy *p, struct flt_conf *fconf); + + /* + * Stream callbacks + */ + int (*attach) (struct stream *s, struct filter *f); + int (*stream_start) (struct stream *s, struct filter *f); + int (*stream_set_backend)(struct stream *s, struct filter *f, struct proxy *be); + void (*stream_stop) (struct stream *s, struct filter *f); + void (*detach) (struct stream *s, struct filter *f); + void (*check_timeouts) (struct stream *s, struct filter *f); + + /* + * Channel callbacks + */ + int (*channel_start_analyze)(struct stream *s, struct filter *f, + struct channel *chn); + int (*channel_pre_analyze) (struct stream *s, struct filter *f, + struct channel *chn, + unsigned int an_bit); + int (*channel_post_analyze) (struct stream *s, struct filter *f, + struct channel *chn, + unsigned int an_bit); + int (*channel_end_analyze) (struct stream *s, struct filter *f, + struct channel *chn); + + /* + * HTTP callbacks + */ + int (*http_headers) (struct stream *s, struct filter *f, + struct http_msg *msg); + int (*http_payload) (struct stream *s, struct filter *f, + struct http_msg *msg, unsigned int offset, + unsigned int len); + int (*http_end) (struct stream *s, struct filter *f, + struct http_msg *msg); + + void (*http_reset) (struct stream *s, struct filter *f, + struct http_msg *msg); + void (*http_reply) (struct stream *s, struct filter *f, + short status, + const struct buffer *msg); + + /* + * TCP callbacks + */ + int (*tcp_payload) (struct stream *s, struct filter *f, + struct channel *chn, unsigned int offset, + unsigned int len); + }; + + +We will explain in following parts when these callbacks are called and what they +should do. + +Filters are declared in proxy sections. So each proxy have an ordered list of +filters, possibly empty if no filter is used. When the configuration of a proxy +is parsed, each filter line represents an entry in this list. In the structure +'proxy', the filters configurations are stored in the field 'filter_configs', +each one of type 'struct flt_conf *' : + + /* + * Structure representing the filter configuration, attached to a proxy and + * accessible from a filter when instantiated in a stream + */ + struct flt_conf { + const char *id; /* The filter id */ + struct flt_ops *ops; /* The filter callbacks */ + void *conf; /* The filter configuration */ + struct list list; /* Next filter for the same proxy */ + unsigned int flags; /* FLT_CFG_FL_* */ + }; + + * 'flt_conf.id' is an identifier, defined by the filter. It can be + NULL. HAProxy does not use this field. Filters can use it in log messages or + as a uniq identifier to check multiple declarations. It is the filter + responsibility to free it, if necessary. + + * 'flt_conf.conf' is opaque. It is the internal configuration of a filter, + generally allocated and filled by its parsing function (See § 3.2). It is + the filter responsibility to free it. + + * 'flt_conf.ops' references the callbacks implemented by the filter. This + field must be set during the parsing phase (See § 3.2) and can be refine + during the initialization phase (See § 3.3). If it is dynamically allocated, + it is the filter responsibility to free it. + + * 'flt_conf.flags' is a bitfield to specify the filter capabilities. For now, + only FLT_CFG_FL_HTX may be set when a filter is able to process HTX + streams. If not set, the filter is excluded from the HTTP filtering. + + +The filter configuration is global and shared by all its instances. A filter +instance is created in the context of a stream and attached to this stream. in +the structure 'stream', the field 'strm_flt' is the state of all filter +instances attached to a stream : + + /* + * Structure representing the "global" state of filters attached to a + * stream. + */ + struct strm_flt { + struct list filters; /* List of filters attached to a stream */ + struct filter *current[2]; /* From which filter resume processing, for a specific channel. + * This is used for resumable callbacks only, + * If NULL, we start from the first filter. + * 0: request channel, 1: response channel */ + unsigned short flags; /* STRM_FL_* */ + unsigned char nb_req_data_filters; /* Number of data filters registered on the request channel */ + unsigned char nb_rsp_data_filters; /* Number of data filters registered on the response channel */ + unsigned long long offset[2]; /* gloal offset of input data already filtered for a specific channel + * 0: request channel, 1: response channel */ + }; + + +Filter instances attached to a stream are stored in the field +'strm_flt.filters', each instance is of type 'struct filter *' : + + /* + * Structure representing a filter instance attached to a stream + * + * 2D-Array fields are used to store info per channel. The first index + * stands for the request channel, and the second one for the response + * channel. Especially, <next> and <fwd> are offsets representing amount of + * data that the filter are, respectively, parsed and forwarded on a + * channel. Filters can access these values using FLT_NXT and FLT_FWD + * macros. + */ + struct filter { + struct flt_conf *config; /* the filter's configuration */ + void *ctx; /* The filter context (opaque) */ + unsigned short flags; /* FLT_FL_* */ + unsigned long long offset[2]; /* Offset of input data already filtered for a specific channel + * 0: request channel, 1: response channel */ + unsigned int pre_analyzers; /* bit field indicating analyzers to + * pre-process */ + unsigned int post_analyzers; /* bit field indicating analyzers to + * post-process */ + struct list list; /* Next filter for the same proxy/stream */ + }; + + * 'filter.config' is the filter configuration previously described. All + instances of a filter share it. + + * 'filter.ctx' is an opaque context. It is managed by the filter, so it is its + responsibility to free it. + + * 'filter.pre_analyzers and 'filter.post_analyzers will be described later + (See § 3.5). + + * 'filter.offset' will be described later (See § 3.6). + + +3.2. DEFINING THE FILTER NAME AND ITS CONFIGURATION +--------------------------------------------------- + +During the filter development, the first thing to do is to add it in the +supported filters. To do so, its name must be registered as a valid keyword on +the filter line : + + /* Declare the filter parser for "my_filter" keyword */ + static struct flt_kw_list flt_kws = { "MY_FILTER_SCOPE", { }, { + { "my_filter", parse_my_filter_cfg, NULL /* private data */ }, + { NULL, NULL, NULL }, + } + }; + INITCALL1(STG_REGISTER, flt_register_keywords, &flt_kws); + + +Then the filter internal configuration must be defined. For instance : + + struct my_filter_config { + struct proxy *proxy; + char *name; + /* ... */ + }; + + +All callbacks implemented by the filter must then be declared. Here, a global +variable is used : + + struct flt_ops my_filter_ops { + .init = my_filter_init, + .deinit = my_filter_deinit, + .check = my_filter_config_check, + + /* ... */ + }; + + +Finally, the function to parse the filter configuration must be written, here +'parse_my_filter_cfg'. This function must parse all remaining keywords on the +filter line : + + /* Return -1 on error, else 0 */ + static int + parse_my_filter_cfg(char **args, int *cur_arg, struct proxy *px, + struct flt_conf *flt_conf, char **err, void *private) + { + struct my_filter_config *my_conf; + int pos = *cur_arg; + + /* Allocate the internal configuration used by the filter */ + my_conf = calloc(1, sizeof(*my_conf)); + if (!my_conf) { + memprintf(err, "%s : out of memory", args[*cur_arg]); + return -1; + } + my_conf->proxy = px; + + /* ... */ + + /* Parse all keywords supported by the filter and fill the internal + * configuration */ + pos++; /* Skip the filter name */ + while (*args[pos]) { + if (!strcmp(args[pos], "name")) { + if (!*args[pos + 1]) { + memprintf(err, "'%s' : '%s' option without value", + args[*cur_arg], args[pos]); + goto error; + } + my_conf->name = strdup(args[pos + 1]); + if (!my_conf->name) { + memprintf(err, "%s : out of memory", args[*cur_arg]); + goto error; + } + pos += 2; + } + + /* ... parse other keywords ... */ + } + *cur_arg = pos; + + /* Set callbacks supported by the filter */ + flt_conf->ops = &my_filter_ops; + + /* Last, save the internal configuration */ + flt_conf->conf = my_conf; + return 0; + + error: + if (my_conf->name) + free(my_conf->name); + free(my_conf); + return -1; + } + + +WARNING : In this parsing function, 'flt_conf->ops' must be initialized. All + arguments of the filter line must also be parsed. This is mandatory. + +In the previous example, the filter lne should be read as follows : + + filter my_filter name MY_NAME ... + + +Optionally, by implementing the 'flt_ops.check' callback, an extra set is added +to check the internal configuration of the filter after the parsing phase, when +the HAProxy configuration is fully defined. For instance : + + /* Check configuration of a trace filter for a specified proxy. + * Return 1 on error, else 0. */ + static int + my_filter_config_check(struct proxy *px, struct flt_conf *my_conf) + { + if (px->mode != PR_MODE_HTTP) { + Alert("The filter 'my_filter' cannot be used in non-HTTP mode.\n"); + return 1; + } + + /* ... */ + + return 0; + } + + + +3.3. MANAGING THE FILTER LIFECYCLE +---------------------------------- + +Once the configuration parsed and checked, filters are ready to by used. There +are two main callbacks to manage the filter lifecycle : + + * 'flt_ops.init' : It initializes the filter for a proxy. This callback may be + defined to finish the filter configuration. + + * 'flt_ops.deinit' : It cleans up what the parsing function and the init + callback have done. This callback is useful to release + memory allocated for the filter configuration. + +Here is an example : + + /* Initialize the filter. Returns -1 on error, else 0. */ + static int + my_filter_init(struct proxy *px, struct flt_conf *fconf) + { + struct my_filter_config *my_conf = fconf->conf; + + /* ... */ + + return 0; + } + + /* Free resources allocated by the trace filter. */ + static void + my_filter_deinit(struct proxy *px, struct flt_conf *fconf) + { + struct my_filter_config *my_conf = fconf->conf; + + if (my_conf) { + free(my_conf->name); + /* ... */ + free(my_conf); + } + fconf->conf = NULL; + } + + +3.3.1 DEALING WITH THREADS +-------------------------- + +When HAProxy is compiled with the threads support and started with more that one +thread (global.nbthread > 1), then it is possible to manage the filter per +thread with following callbacks : + + * 'flt_ops.init_per_thread': It initializes the filter for each thread. It + works the same way than 'flt_ops.init' but in the + context of a thread. This callback is called + after the thread creation. + + * 'flt_ops.deinit_per_thread': It cleans up what the init_per_thread callback + have done. It is called in the context of a + thread, before exiting it. + +It is the filter responsibility to deal with concurrency. check, init and deinit +callbacks are called on the main thread. All others are called on a "worker" +thread (not always the same). It is also the filter responsibility to know if +HAProxy is started with more than one thread. If it is started with one thread +(or compiled without the threads support), these callbacks will be silently +ignored (in this case, global.nbthread will be always equal to one). + + +3.4. HANDLING THE STREAMS ACTIVITY +----------------------------------- + +It may be interesting to handle streams activity. For now, there is three +callbacks that should define to do so : + + * 'flt_ops.stream_start' : It is called when a stream is started. This + callback can fail by returning a negative value. It + will be considered as a critical error by HAProxy + which disabled the listener for a short time. + + * 'flt_ops.stream_set_backend' : It is called when a backend is set for a + stream. This callbacks will be called for all + filters attached to a stream (frontend and + backend). Note this callback is not called if + the frontend and the backend are the same. + + * 'flt_ops.stream_stop' : It is called when a stream is stopped. This callback + always succeed. Anyway, it is too late to return an + error. + +For instance : + + /* Called when a stream is created. Returns -1 on error, else 0. */ + static int + my_filter_stream_start(struct stream *s, struct filter *filter) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* ... */ + + return 0; + } + + /* Called when a backend is set for a stream */ + static int + my_filter_stream_set_backend(struct stream *s, struct filter *filter, + struct proxy *be) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* ... */ + + return 0; + } + + /* Called when a stream is destroyed */ + static void + my_filter_stream_stop(struct stream *s, struct filter *filter) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* ... */ + } + + +WARNING : Handling the streams creation and destruction is only possible for + filters defined on proxies with the frontend capability. + +In addition, it is possible to handle creation and destruction of filter +instances using following callbacks: + + * 'flt_ops.attach' : It is called after a filter instance creation, when it is + attached to a stream. This happens when the stream is + started for filters defined on the stream's frontend and + when the backend is set for filters declared on the + stream's backend. It is possible to ignore the filter, if + needed, by returning 0. This could be useful to have + conditional filtering. + + * 'flt_ops.detach' : It is called when a filter instance is detached from a + stream, before its destruction. This happens when the + stream is stopped for filters defined on the stream's + frontend and when the analyze ends for filters defined on + the stream's backend. + +For instance : + + /* Called when a filter instance is created and attach to a stream */ + static int + my_filter_attach(struct stream *s, struct filter *filter) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + if (/* ... */) + return 0; /* Ignore the filter here */ + return 1; + } + + /* Called when a filter instance is detach from a stream, just before its + * destruction */ + static void + my_filter_detach(struct stream *s, struct filter *filter) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* ... */ + } + +Finally, it may be interesting to notify the filter when the stream is woken up +because of an expired timer. This could let a chance to check some internal +timeouts, if any. To do so the following callback must be used : + + * 'flt_opt.check_timeouts' : It is called when a stream is woken up because of + an expired timer. + +For instance : + + /* Called when a stream is woken up because of an expired timer */ + static void + my_filter_check_timeouts(struct stream *s, struct filter *filter) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* ... */ + } + + +3.5. ANALYZING THE CHANNELS ACTIVITY +------------------------------------ + +The main purpose of filters is to take part in the channels analyzing. To do so, +there is 2 callbacks, 'flt_ops.channel_pre_analyze' and +'flt_ops.channel_post_analyze', called respectively before and after each +analyzer attached to a channel, except analyzers responsible for the data +forwarding (TCP or HTTP). Concretely, on the request channel, these callbacks +could be called before following analyzers : + + * tcp_inspect_request (AN_REQ_INSPECT_FE and AN_REQ_INSPECT_BE) + * http_wait_for_request (AN_REQ_WAIT_HTTP) + * http_wait_for_request_body (AN_REQ_HTTP_BODY) + * http_process_req_common (AN_REQ_HTTP_PROCESS_FE) + * process_switching_rules (AN_REQ_SWITCHING_RULES) + * http_process_req_ common (AN_REQ_HTTP_PROCESS_BE) + * http_process_tarpit (AN_REQ_HTTP_TARPIT) + * process_server_rules (AN_REQ_SRV_RULES) + * http_process_request (AN_REQ_HTTP_INNER) + * tcp_persist_rdp_cookie (AN_REQ_PRST_RDP_COOKIE) + * process_sticking_rules (AN_REQ_STICKING_RULES) + +And on the response channel : + + * tcp_inspect_response (AN_RES_INSPECT) + * http_wait_for_response (AN_RES_WAIT_HTTP) + * process_store_rules (AN_RES_STORE_RULES) + * http_process_res_common (AN_RES_HTTP_PROCESS_BE) + +Unlike the other callbacks previously seen before, 'flt_ops.channel_pre_analyze' +can interrupt the stream processing. So a filter can decide to not execute the +analyzer that follows and wait the next iteration. If there are more than one +filter, following ones are skipped. On the next iteration, the filtering resumes +where it was stopped, i.e. on the filter that has previously stopped the +processing. So it is possible for a filter to stop the stream processing on a +specific analyzer for a while before continuing. Moreover, this callback can be +called many times for the same analyzer, until it finishes its processing. For +instance : + + /* Called before a processing happens on a given channel. + * Returns a negative value if an error occurs, 0 if it needs to wait, + * any other value otherwise. */ + static int + my_filter_chn_pre_analyze(struct stream *s, struct filter *filter, + struct channel *chn, unsigned an_bit) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + switch (an_bit) { + case AN_REQ_WAIT_HTTP: + if (/* wait that a condition is verified before continuing */) + return 0; + break; + /* ... * / + } + return 1; + } + + * 'an_bit' is the analyzer id. All analyzers are listed in + 'include/haproxy/channels-t.h'. + + * 'chn' is the channel on which the analyzing is done. It is possible to + determine if it is the request or the response channel by testing if + CF_ISRESP flag is set : + + │ ((chn->flags & CF_ISRESP) == CF_ISRESP) + + +In previous example, the stream processing is blocked before receipt of the HTTP +request until a condition is verified. + +'flt_ops.channel_post_analyze', for its part, is not resumable. It returns a +negative value if an error occurs, any other value otherwise. It is called when +a filterable analyzer finishes its processing, so once for the same analyzer. +For instance : + + /* Called after a processing happens on a given channel. + * Returns a negative value if an error occurs, any other + * value otherwise. */ + static int + my_filter_chn_post_analyze(struct stream *s, struct filter *filter, + struct channel *chn, unsigned an_bit) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + struct http_msg *msg; + + switch (an_bit) { + case AN_REQ_WAIT_HTTP: + if (/* A test on received headers before any other treatment */) { + msg = ((chn->flags & CF_ISRESP) ? &s->txn->rsp : &s->txn->req); + txn->status = 400; + msg->msg_state = HTTP_MSG_ERROR; + http_reply_and_close(s, s->txn->status, http_error_message(s)); + return -1; /* This is an error ! */ + } + break; + /* ... * / + } + return 1; + } + + +Pre and post analyzer callbacks of a filter are not automatically called. They +must be regiesterd explicitly on analyzers, updating the value of +'filter.pre_analyzers' and 'filter.post_analyzers' bit fields. All analyzer bits +are listed in 'include/types/channels.h'. Here is an example : + + static int + my_filter_stream_start(struct stream *s, struct filter *filter) + { + /* ... * / + + /* Register the pre analyzer callback on all request and response + * analyzers */ + filter->pre_analyzers |= (AN_REQ_ALL | AN_RES_ALL) + + /* Register the post analyzer callback of only on AN_REQ_WAIT_HTTP and + * AN_RES_WAIT_HTTP analyzers */ + filter->post_analyzers |= (AN_REQ_WAIT_HTTP | AN_RES_WAIT_HTTP) + + /* ... * / + return 0; + } + + +To surround activity of a filter during the channel analyzing, two new analyzers +has been added : + + * 'flt_start_analyze' (AN_REQ/RES_FLT_START_FE/AN_REQ_RES_FLT_START_BE) : For + a specific filter, this analyzer is called before any call to the + 'channel_analyze' callback. From the filter point of view, it calls the + 'flt_ops.channel_start_analyze' callback. + + * 'flt_end_analyze' (AN_REQ/RES_FLT_END) : For a specific filter, this + analyzer is called when all other analyzers have finished their + processing. From the filter point of view, it calls the + 'flt_ops.channel_end_analyze' callback. + +These analyzers are called only once per streams. + +'flt_ops.channel_start_analyze' and 'flt_ops.channel_end_analyze' callbacks can +interrupt the stream processing, as 'flt_ops.channel_analyze'. Here is an +example : + + /* Called when analyze starts for a given channel + * Returns a negative value if an error occurs, 0 if it needs to wait, + * any other value otherwise. */ + static int + my_filter_chn_start_analyze(struct stream *s, struct filter *filter, + struct channel *chn) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* ... TODO ... */ + + return 1; + } + + /* Called when analyze ends for a given channel + * Returns a negative value if an error occurs, 0 if it needs to wait, + * any other value otherwise. */ + static int + my_filter_chn_end_analyze(struct stream *s, struct filter *filter, + struct channel *chn) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* ... TODO ... */ + + return 1; + } + + +Workflow on channels can be summarized as following : + + FE: Called for filters defined on the stream's frontend + BE: Called for filters defined on the stream's backend + + +------->---------+ + | | | + +----------------------+ | +----------------------+ + | flt_ops.attach (FE) | | | flt_ops.attach (BE) | + +----------------------+ | +----------------------+ + | | | + V | V + +--------------------------+ | +------------------------------------+ + | flt_ops.stream_start (FE)| | | flt_ops.stream_set_backend (FE+BE) | + +--------------------------+ | +------------------------------------+ + | | | + ... | ... + | | | + | ^ | + | --+ | | --+ + +------<----------+ | | +--------<--------+ | + | | | | | | | + V | | | V | | ++-------------------------------+ | | | +-------------------------------+ | | +| flt_start_analyze (FE) +-+ | | | flt_start_analyze (BE) +-+ | +|(flt_ops.channel_start_analyze)| | F | |(flt_ops.channel_start_analyze)| | ++---------------+---------------+ | R | +-------------------------------+ | + | | O | | | + +------<---------+ | N ^ +--------<-------+ | B + | | | T | | | | A ++---------------|------------+ | | E | +---------------|------------+ | | C +|+--------------V-------------+ | | N | |+--------------V-------------+ | | K +||+----------------------------+ | | D | ||+----------------------------+ | | E +|||flt_ops.channel_pre_analyze | | | | |||flt_ops.channel_pre_analyze | | | N +||| V | | | | ||| V | | | D +||| analyzer (FE) +-+ | | ||| analyzer (FE+BE) +-+ | ++|| V | | | +|| V | | + +|flt_ops.channel_post_analyze| | | +|flt_ops.channel_post_analyze| | + +----------------------------+ | | +----------------------------+ | + | --+ | | | + +------------>------------+ ... | + | | + [ data filtering (see below) ] | + | | + ... | + | | + +--------<--------+ | + | | | + V | | + +-------------------------------+ | | + | flt_end_analyze (FE+BE) +-+ | + | (flt_ops.channel_end_analyze) | | + +---------------+---------------+ | + | --+ + V + +----------------------+ + | flt_ops.detach (BE) | + +----------------------+ + | + V + +--------------------------+ + | flt_ops.stream_stop (FE) | + +--------------------------+ + | + V + +----------------------+ + | flt_ops.detach (FE) | + +----------------------+ + | + V + +By zooming on an analyzer box we have: + + ... + | + V + | + +-----------<-----------+ + | | + +-----------------+--------------------+ | + | | | | + | +--------<---------+ | | + | | | | | + | V | | | + | flt_ops.channel_pre_analyze ->-+ | ^ + | | | | + | | | | + | V | | + | analyzer --------->-----+--+ + | | | + | | | + | V | + | flt_ops.channel_post_analyze | + | | | + | | | + +-----------------+--------------------+ + | + V + ... + + + 3.6. FILTERING THE DATA EXCHANGED +----------------------------------- + +WARNING : To fully understand this part, it is important to be aware on how the + buffers work in HAProxy. For the HTTP part, it is also important to + understand how data are parsed and structured, and how the internal + representation, called HTX, works. See doc/internals/buffer-api.txt + and doc/internals/htx-api.txt for details. + +An extended feature of the filters is the data filtering. By default a filter +does not look into data exchanged between the client and the server because it +is expensive. Indeed, instead of forwarding data without any processing, each +byte need to be buffered. + +So, to enable the data filtering on a channel, at any time, in one of previous +callbacks, 'register_data_filter' function must be called. And conversely, to +disable it, 'unregister_data_filter' function must be called. For instance : + + my_filter_http_headers(struct stream *s, struct filter *filter, + struct http_msg *msg) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + + /* 'chn' must be the request channel */ + if (!(msg->chn->flags & CF_ISRESP)) { + struct htx *htx; + struct ist hdr; + struct http_hdr_ctx ctx; + + htx = htxbuf(msg->chn->buf); + + /* Enable the data filtering for the request if 'X-Filter' header + * is set to 'true'. */ + hdr = ist("X-Filter); + ctx.blk = NULL; + if (http_find_header(htx, hdr, &ctx, 0) && + ctx.value.len >= 4 && memcmp(ctx.value.ptr, "true", 4) == 0) + register_data_filter(s, chn, filter); + } + + return 1; + } + +Here, the data filtering is enabled if the HTTP header 'X-Filter' is found and +set to 'true'. + +If several filters are declared, the evaluation order remains the same, +regardless the order of the registrations to the data filtering. Data +registrations must be performed before the data forwarding step. However, a +filter may be unregistered from the data filtering at any time. + +Depending on the stream type, TCP or HTTP, the way to handle data filtering is +different. HTTP data are structured while TCP data are raw. And there are more +callbacks for HTTP streams to fully handle all steps of an HTTP transaction. But +the main part is the same. The data filtering is performed in one callback, +called in loop on input data starting at a specific offset for a given +length. Data analyzed by a filter are considered as forwarded from its point of +view. Because filters are chained, a filter never analyzes more data than its +predecessors. Thus only data analyzed by the last filter are effectively +forwarded. This means, at any time, any filter may choose to not analyze all +available data (available from its point of view), blocking the data forwarding. + +Internally, filters own 2 offsets representing the number of bytes already +analyzed in the available input data, one per channel. There is also an offset +couple at the stream level, in the strm_flt object, representing the total +number of bytes already forwarded. These offsets may be retrieved and updated +using following macros : + + * FLT_OFF(flt, chn) + + * FLT_STRM_OFF(s, chn) + +where 'flt' is the 'struct filter' passed as argument in all callbacks, 's' the +filtered stream and 'chn' is the considered channel. However, there is no reason +for a filter to use these macros or take care of these offsets. + + +3.6.1 FILTERING DATA ON TCP STREAMS +----------------------------------- + +The TCP data filtering for TCP streams is the easy case, because HAProxy do not +parse these data. Data are stored in raw in the buffer. So there is only one +callback to consider: + + * 'flt_ops.tcp_payload : This callback is called when input data are + available. If not defined, all available data will be considered as analyzed + and forwarded from the filter point of view. + +This callback is called only if the filter is registered to analyze TCP +data. Here is an example : + + /* Returns a negative value if an error occurs, else the number of + * consumed bytes. */ + static int + my_filter_tcp_payload(struct stream *s, struct filter *filter, + struct channel *chn, unsigned int offset, + unsigned int len) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + int ret = len; + + /* Do not parse more than 'my_conf->max_parse' bytes at a time */ + if (my_conf->max_parse != 0 && ret > my_conf->max_parse) + ret = my_conf->max_parse; + + /* if available data are not completely parsed, wake up the stream to + * be sure to not freeze it. The best is probably to set a + * chn->analyse_exp timer */ + if (ret != len) + task_wakeup(s->task, TASK_WOKEN_MSG); + return ret; + } + +But it is important to note that tunnelled data of an HTTP stream may also be +filtered via this callback. Tunnelled data are data exchange after an HTTP tunnel +is established between the client and the server, via an HTTP CONNECT or via a +protocol upgrade. In this case, the data are structured. Of course, to do so, +the filter must be able to parse HTX data and must have the FLT_CFG_FL_HTX flag +set. At any time, the IS_HTX_STRM() macros may be used on the stream to know if +it is an HTX stream or a TCP stream. + + +3.6.2 FILTERING DATA ON HTTP STREAMS +------------------------------------ + +The HTTP data filtering is a bit more complex because HAProxy data are +structutred and represented to an internal format, called HTX. So basically +there is the HTTP counterpart to the previous callback : + + * 'flt_ops.http_payload' : This callback is called when input data are + available. If not defined, all available data will be considered as analyzed + and forwarded for the filter. + +But the prototype for this callbacks is slightly different. Instead of having +the channel as parameter, we have the HTTP message (struct http_msg). This +callback is called only if the filter is registered to analyze TCP data. Here is +an example : + + /* Returns a negative value if an error occurs, else the number of + * consumed bytes. */ + static int + my_filter_http_payload(struct stream *s, struct filter *filter, + struct http_msg *msg, unsigned int offset, + unsigned int len) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + struct htx *htx = htxbuf(&msg->chn->buf); + struct htx_ret htxret = htx_find_offset(htx, offset); + struct htx_blk *blk; + + blk = htxret.blk; + offset = htxret.ret; + for (; blk; blk = htx_get_next_blk(blk, htx)) { + enum htx_blk_type type = htx_get_blk_type(blk); + + if (type == HTX_BLK_UNUSED) + continue; + else if (type == HTX_BLK_DATA) { + /* filter data */ + } + else + break; + } + + return len; + } + +In addition, there are two others callbacks : + + * 'flt_ops.http_headers' : This callback is called just before the HTTP body + forwarding and after any processing on the request/response HTTP + headers. When defined, this callback is always called for HTTP streams + (i.e. without needs of a registration on data filtering). + Here is an example : + + + /* Returns a negative value if an error occurs, 0 if it needs to wait, + * any other value otherwise. */ + static int + my_filter_http_headers(struct stream *s, struct filter *filter, + struct http_msg *msg) + { + struct my_filter_config *my_conf = FLT_CONF(filter); + struct htx *htx = htxbuf(&msg->chn->buf); + struct htx_sl *sl = http_get_stline(htx); + int32_t pos; + + for (pos = htx_get_first(htx); pos != -1; pos = htx_get_next(htx, pos)) { + struct htx_blk *blk = htx_get_blk(htx, pos); + enum htx_blk_type type = htx_get_blk_type(blk); + struct ist n, v; + + if (type == HTX_BLK_EOH) + break; + if (type != HTX_BLK_HDR) + continue; + + n = htx_get_blk_name(htx, blk); + v = htx_get_blk_value(htx, blk); + /* Do something on the header name/value */ + } + + return 1; + } + + * 'flt_ops.http_end' : This callback is called when the whole HTTP message was + processed. It may interrupt the stream processing. So, it could be used to + synchronize the HTTP request with the HTTP response, for instance : + + /* Returns a negative value if an error occurs, 0 if it needs to wait, + * any other value otherwise. */ + static int + my_filter_http_end(struct stream *s, struct filter *filter, + struct http_msg *msg) + { + struct my_filter_ctx *my_ctx = filter->ctx; + + + if (!(msg->chn->flags & CF_ISRESP)) /* The request */ + my_ctx->end_of_req = 1; + else /* The response */ + my_ctx->end_of_rsp = 1; + + /* Both the request and the response are finished */ + if (my_ctx->end_of_req == 1 && my_ctx->end_of_rsp == 1) + return 1; + + /* Wait */ + return 0; + } + +Then, to finish, there are 2 informational callbacks : + + * 'flt_ops.http_reset' : This callback is called when an HTTP message is + reset. This happens either when a 1xx informational response is received, or + if we're retrying to send the request to the server after it failed. It + could be useful to reset the filter context before receiving the true + response. + By checking s->txn->status, it is possible to know why this callback is + called. If it's a 1xx, we're called because of an informational + message. Otherwise, it is a L7 retry. + + * 'flt_ops.http_reply' : This callback is called when, at any time, HAProxy + decides to stop the processing on a HTTP message and to send an internal + response to the client. This mainly happens when an error or a redirect + occurs. + + +3.6.3 REWRITING DATA +-------------------- + +The last part, and the trickiest one about the data filtering, is about the data +rewriting. For now, the filter API does not offer a lot of functions to handle +it. There are only functions to notify HAProxy that the data size has changed to +let it update internal state of filters. This is the developer responsibility to +update data itself, i.e. the buffer offsets, using following function : + + * 'flt_update_offsets()' : This function must be called when a filter alter + incoming data. It updates offsets of the stream and of all filters + preceding the calling one. Do not call this function when a filter change + the size of incoming data leads to an undefined behavior. + +A good example of filter changing the data size is the HTTP compression filter. diff --git a/doc/internals/api/htx-api.txt b/doc/internals/api/htx-api.txt new file mode 100644 index 0000000..62b3093 --- /dev/null +++ b/doc/internals/api/htx-api.txt @@ -0,0 +1,570 @@ + ----------------------------------------------- + HTX API + Version 1.1 + ( Last update: 2021-02-24 ) + ----------------------------------------------- + Author : Christopher Faulet + Contact : cfaulet at haproxy dot com + +1. Background + +Historically, HAProxy stored HTTP messages in a raw fashion in buffers, keeping +parsing information separately in a "struct http_msg" owned by the stream. It was +optimized to the data transfer, but not so much for rewrites. It was also HTTP/1 +centered. While it was the only HTTP version supported, it was not a +problem. But with the rise of HTTP/2, it starts to be hard to still use this +representation. + +At the first age of the HTTP/2 in HAProxy, H2 messages were converted into +H1. This was terribly unefficient because it required two parsing passes, a +first one in H2 and a second one in H1, with a conversion in the middle. And of +course, the same was also true in the opposite direction. outgoing H1 messages +had to be converted back in H2 to be sent. Even worse, because the H2->H1 +conversion, only client H2 connections were supported. + +So, to address all these problems, we decided to replace the old raw +representation by a version-agnostic and self-structured internal HTTP +representation, the HTX. As an additional benefit, with this new representation, +the message parsing and its processing are now separated, making all the HTTP +analysis simpler and cleaner. The parsing of HTTP messages is now handled by +the multiplexers (h1 or h2). + + +2. The HTX message + +The HTX is a structure containing useful information about an HTTP message +followed by a contiguous array with some parts of the message. These parts are +called blocks. A block is composed of metadata (htx_blk) and an associated +payload. Blocks' metadata are stored starting from the end of the array while +their payload are stored at the beginning. Blocks' metadata are often simply +called blocks. it is a misuse of language that's simplify explanations. + +Internally, this structure is "hidden" in a buffer. This way, there are few +changes into intermediate layers (stream-interface and channels). They still +manipulate buffers. Only the multiplexer and the stream have to know how data +are really stored. From the HTX perspective, a buffer is just a memory +area. When an HTX message is stored in a buffer, this one appears as full. + + * General view of an HTX message : + + + buffer->area + | + |<------------ buffer->size == buffer->data ----------------------| + | | + | |<------------- Blocks array (htx->size) ------------------>| + V | | + +-----+-----------------+-------------------------+---------------+ + | HTX | PAYLOADS ==> | | <== HTX_BLKs | + +-----+-----------------+-------------------------+---------------+ + | | | | + |<-payloads part->|<----- free space ------>|<-blocks part->| + (htx->data) + + +The blocks part remains linear and sorted. It may be see as an array with +negative indexes. But, instead of using negative indexes, we use positive +positions to identify a block. This position is then converted to an address +relatively to the beginning of the blocks array. + + tail head + | | + V V + .....--+----+-----------------------+------+------+ + | Bn | ... | B1 | B0 | + .....--+----+-----------------------+------+------+ + ^ ^ ^ + Addr of the block Addr of the block Addr of the block + at the position N at the position 1 at the position 0 + + +In the HTX structure, 3 "special" positions are stored : + + - tail : Position of the newest inserted block + - head : Position of the oldest inserted block + - first : Position of the first block to (re)start the analyse + +The blocks part never wrap. If we have no space to allocate a new block and if +there is a hole at the beginning of the blocks part (so at the end of the blocks +array), we move back all blocks. + + + tail head tail head + | | | | + V V V V + ...+--------------+---------+ blocks ...----------+--------------+ + | X== HTX_BLKS | | defrag | <== HTX_BLKS | + ...+--------------+---------+ =====> ...----------+--------------+ + + +The payloads part is a raw space that may wrap. A block's payload must never be +accessed directly. Instead a block must be selected to retrieve the address of +its payload. + + + +------------------------( B0.addr )--------------------------+ + | +-------------------( B1.addr )----------------------+ | + | | +-----------( B2.addr )----------------+ | | + V V V | | | + +-----+----+-------+----+--------+-------------+-------+----+----+----+ + | HTX | P0 | P1 | P2 | ...==> | | <=... | B2 | B1 | B0 | + +-----+----+-------+----+--------+-------------+-------+----+----+----+ + + +Because the payloads part may wrap, there are 2 usable free spaces : + + - The free space in front of the blocks part. This one is used if and only if + the other one was not used yet. + + - The free space at the beginning of the message. Once this one is used, the + other one is never used again, until a message defragmentation. + + + * Linear payloads part : + + + head_addr end_addr tail_addr + | | | + V V V + +-----+--------------------+-------------+--------------------+-------... + | HTX | | PAYLOADS | | HTX_BLKs + +-----+--------------------+-------------+--------------------+-------... + |<-- free space 2 -->| |<-- free space 1 -->| + (used if the other is too small) (used in priority) + + + * Wrapping payloads part : + + + head_addr end_addr tail_addr + | | | + V V V + +-----+----+----------------+--------+----------------+-------+-------... + | HTX | | PAYLOADS part2 | | PAYLOADS part1 | | HTX_BLKs + +-----+----+----------------+--------+----------------+-------+-------... + |<-->| |<------>| |<----->| + unusable free space unusable + free space free space + + +Finally, when the usable free space is not enough to store a new block, unusable +parts may be get back with a full defragmentation. The payloads part is then +realigned at the beginning of the blocks array and the free space becomes +continuous again. + + +3. The HTX blocks + +An HTX block can be as well a start-line as a header, a body part or a +trailer. For all these types of block, a payload is attached to the block. It +can also be a marker, the end-of-headers or end-of-trailers. For these blocks, +there is no payload but it counts for a byte. It is important to not skip it +when data are forwarded. + +As already said, a block is composed of metadata and a payload. Metadata are +stored in the blocks part and are composed of 2 fields : + + - info : It a 32 bits field containing the block's type on 4 bits followed + by the payload length. See below for details. + + - addr : The payload's address, if any, relatively to the beginning the + array used to store part of the HTTP message itself. + + + * Block's info representation : + + 0b 0000 0000 0000 0000 0000 0000 0000 0000 + ---- ------------------------ --------- + type value (1 MB max) name length (header/trailer - 256B max) + ---------------------------------- + data length (256 MB max) + (body, method, path, version, status, reason) + + +Supported types are : + + - 0000 (0) : The request start-line + - 0001 (1) : The response start-line + - 0010 (2) : A header block + - 0011 (3) : The end-of-headers marker + - 0100 (4) : A data block + - 0101 (5) : A trailer block + - 0110 (6) : The end-of-trailers marker + - 1111 (15) : An unused block + +Other types are unused for now and reserved for futur extensions. + +An HTX message is typically composed of following blocks, in this order : + + - a start-line + - zero or more header blocks + - an end-of-headers marker + - zero or more data blocks + - zero or more trailer blocks (optional) + - an end-of-trailers marker (optional but always set if there is at least + one trailer block) + +Only one HTTP request at a time can be stored in an HTX message. For HTTP +response, it is more complicated. Only one "final" response can be stored in an +HTX message. It is a response with status-code 101 or greater or equal to +200. But it may be preceded by several 1xx informational responses. Such +responses are part of the same HTX message. + +When the end of the message is reached a special flag is set on the message +(HTX_FL_EOM). It means no more data are expected for this message, except +tunneled data. But tunneled data will never be mixed with message data to avoid +ambiguities. Thus once the flag marking the end of the message is set, it is +easy to know the message ends. The end is reached if the HTX message is empty or +on the tail HTX block in the HTX message. Once all blocks of the HTX message are +consumed, tunneled data, if any, may be transferred. + + +3.1. The start-line + +Every HTX message starts with a start-line. Its payload is a "struct htx_sl". In +addition to the parts of the HTTP start-line, this structure contains some +information about the represented HTTP message, mainly in the form of flags +(HTX_SL_F_*). For instance, if an HTTP message contains the header +"conten-length", then the flag HTX_SL_F_CLEN is set. + +Each HTTP message has its own start-line. So an HTX request has one and only one +start-line because it must contain only one HTTP request at a time. But an HTX +response may have more than one start-line if the final HTTP response is +precedeed by some 1xx informational responses. + +In HTTP/2, there is no start-line. So the H2 multiplexer must create one when it +converts an H2 message to HTX : + + - For the request, it uses the pseudo headers ":method", ":path" or + ":authority" depending on the method and the hardcoded version "HTTP/2.0". + + - For the response, it used the hardcoded version "HTTP/2.0", the + pseudo-header ":status" and an empty reason. + + +3.2. The headers and trailers + +HTX Headers and trailers are quite similar. Different types are used to simplify +headers processing. But from the HTX point of view, there is no real difference, +except their position in the HTX message. The header blocks always follow an HTX +start-line while trailer blocks come after the data. If there is no data, they +follow the end-of-headers marker. + +Headers and trailers are the only blocks containing a Key/Value payload. The +corresponding end-of marker must always be placed after each group to mark, as +it name suggests, the end. + +In HTTP/1, trailers are only present on chunked messages. But chunked messages +do not always have trailers. In this case, the end-of-trailers block may or may +not be present. Multiplexers must be able to handle both situations. In HTTP/2, +trailers are only present if a HEADERS frame is sent after DATA frames. + + +3.3. The data + +The payload body of an HTTP message is stored as DATA blocks in the HTX +message. For HTTP/1 messages, it is the message body without the chunks +formatting, if any. For HTTP/2, it is the payload of DATA frames. + +The DATA blocks are the only HTX blocks that may be partially processed (copied +or removed). All other types of block must be entirely processed. This means +DATA blocks can be resized. + + +3.4. The end-of markers + +These blocks are used to delimit parts of an HTX message. It exists two +markers : + + - end-of-headers (EOH) + - end-of-trailers (EOT) + +EOH is always present in an HTX message. EOT is optional. + + +4. The HTX API + + +4.1. Get/set HTX message from/to the underlying buffer + +The first thing to do to process an HTX message is to get it from the underlying +buffer. There are 2 functions to do so, the second one relying on the first : + + - htxbuf() returns an HTX message from a buffer. It does not modify the + buffer. It only initialize the HTX message if the buffer is empty. + + - htx_from_buf() uses htxbuf(). But it also updates the underlying buffer so + that it appears as full. + +Both functions return a "zero-sized" HTX message if the buffer is null. This +way, the HTX message is always valid. The first function is the default function +to use. The second one is only useful when some content will be added. For +instance, it used by the HTX analyzers when HAProxy generates a response. Thus, +the buffer is in a right state. + +Once the processing done, if the HTX message has been modified, the underlying +buffer must be also updated, except htx_from_buf() was used _AND_ data was only +added. For all other cases, the function htx_to_buf() must be called. + +Finally, the function htx_reset() may be called at any time to reset an HTX +message. And the function buf_room_for_htx_data() may be called to know if a raw +buffer is full from the HTX perspective. It is used during conversion from/to +the HTX. + + +4.2. Helpers to deal with free space in an HTX message + +Once with an HTX message, following functions may help to process it : + + - htx_used_space() and htx_meta_space() return, respectively, the total + space used in an HTX message and the space used by block's metadata only. + + - htx_free_space() and htx_free_data_space() return, respectively, the total + free space in an HTX message and the free space available for the payload + if a new HTX block is stored (so it is the total free space minus the size + of an HTX block). + + - htx_is_empty() and htx_is_not_empty() are boolean functions to know if an + HTX message is empty or not. + + - htx_get_max_blksz() returns the maximum size available for the payload, + not exceeding a maximum, metadata included. + + - htx_almost_full() should be used to know if an HTX message uses at least + 3/4 of its capacity. + + +4.3. HTX Blocks manipulations + +Once the available sapce in an HTX message is known, the next step is to add HTX +blocks. First of all the function htx_nbblks() returns the number of blocks +allocated in an HTX message. Then, there is an add function per block's type : + + - htx_add_stline() adds a start-line. The type (request or response) and the + flags of the start-line must be provided, as well as its three parts + (method,uri,version or version,status-code,reason). + + - htx_add_header() and htx_add_trailers() are similar. The name and the + value must be provided. The inserted HTX block is returned on success or + NULL if an error occurred. + + - htx_add_endof() must be used to add any end-of marker. The block's type + (EOH or EOT) must be specified. The inserted HTX block is returned on + success or NULL if an error occurred. + + - htx_add_all_headers() and htx_add_all_trailers() add, respectively, a list + of headers and a list of trailers, followed by the appropriate end-of + marker. On success, this marker is returned. Otherwise, NULL is + returned. Note there is no rollback on the HTX message when an error + occurred. Some headers or trailers may have been added. So it is the + caller responsibility to take care of that. + + - htx_add_data() must be used to add a DATA block. Unlike previous + functions, this one returns the number of bytes copied or 0 if nothing was + copied. If possible, the data are appended to the tail block if it is a + DATA block. Only a part of the payload may be copied because this function + will try to limit the message defragmentation and the wrapping of blocks + as far as possible. + + - htx_add_data_atonce() must be used if all data must be added or nothing. + It tries to insert all the payload, this function returns the inserted + block on success. Otherwise it returns NULL. + +When an HTX block is added, it is always the last one (the tail). But, if a +block must be added at a specific place, it is not really handy. 2 functions may +help (others could be added) : + + - htx_add_last_data() adds a DATA block just after all other DATA blocks and + before any trailers and EOT marker. It relies on htx_add_data_atonce(), so + a defragmentation may be performed. + + - htx_move_blk_before() moves a specific block just after another one. Both + blocks must already be in the HTX message and the block to move must + always be placed after the "pivot". + +Once added, there are three functions to update the block's payload : + + - htx_replace_stline() updates a start-line. The HTX block must be passed as + argument. Only string parts of the start-line are updated by this + function. On success, it returns the new start-line. So it is pretty easy + to update its flags. NULL is returned if an error occurred. + + - htx_replace_header() fully replaces a header (its name and its value) by a + new one. The HTX block must be passed a argument, as well as its new name + and its new value. The new header can be smaller or larger than the old + one. This function returns the new HTX block on success, or NULL is an + error occurred. + + - htx_replace_blk_value() replaces a part of a block's payload or its + totality. It works for HEADERS, TRAILERS or DATA blocks. The HTX block + must be provided with the part to remove and the new one. The new part can + be smaller or larger than the old one. This function returns the new HTX + block on success, or NULL is an error occurred. + + - htx_change_blk_value_len() changes the size of the value. It is the caller + responsibility to change the value itself, make sure there is enough space + and update allocated value. This function updates the HTX message + accordingly. + + - htx_set_blk_value_len() changes the size of the value. It is the caller + responsibility to change the value itself, make sure there is enough space + and update allocated value. Unlike the function + htx_change_blk_value_len(), this one does not update the HTX message. So + it should be used with caution. + + - htx_cut_data_blk() removes <n> bytes from the beginning of a DATA + block. The block's start address and its length are adjusted, and the + htx's total data count is updated. This is used to mark that part of some + data were transferred from a DATA block without removing this DATA + block. No sanity check is performed, the caller is responsible for doing + this exclusively on DATA blocks, and never removing more than the block's + size. + + - htx_remove_blk() removes a block from an HTX message. It returns the + following block or NULL if it is the tail block. + +Finally, a block may be removed using the function htx_remove_blk(). This +function returns the block following the one removed or NULL if it is the tail +block. + + +4.4. The HTX start-line + +Unlike other HTX blocks, the start-line is a bit special because its payload is +a structure followed by its three parts : + + +--------+-------+-------+-------+ + | HTX_SL | PART1 | PART2 | PART3 | + +--------+-------+-------+-------+ + +Some macros and functions may help to manipulate these parts : + + - HTX_SL_P{N}_LEN() and HTX_SL_P{N}_PTR() are macros to get the length of a + part and a pointer on it. {N} should be 1, 2 or 3. + + - HTX_SL_REQ_MLEN(), HTX_SL_REQ_ULEN(), HTX_SL_REQ_VLEN(), + HTX_SL_REQ_MPTR(), HTX_SL_REQ_UPTR() and HTX_SL_REQ_VPTR() are macros to + get info about a request start-line. These macros only wrap HTX_SL_P* + ones. + + - HTX_SL_RES_VLEN(), HTX_SL_RES_CLEN(), HTX_SL_RES_RLEN(), + HTX_SL_RES_VPTR(), HTX_SL_RES_CPTR() and HTX_SL_RES_RPTR() are macros to + get info about a response start-line. These macros only wrap HTX_SL_P* + ones. + + - htx_sl_p1(), htx_sl_p2() and htx_sl_p2() are functions to get the ist + corresponding to the right part of a start-line. + + - htx_sl_req_meth(), htx_sl_req_uri() and htx_sl_req_vsn() get the ist + corresponding to the right part of a request start-line. + + - htx_sl_res_vsn(), htx_sl_res_code() and htx_sl_res_reason() get the ist + corresponding to the right part of a response start-line. + + +4.5. Iterate on the HTX message + +To iterate on an HTX message, the first thing to do is to get the HTX block to +start the loop. There are three special blocks in an HTX message that may be +good candidates to start a loop : + + - the head block. It is the oldest inserted block. Multiplexers always start + to consume an HTX message from this block. The function htx_get_head() + returns its position and htx_get_head_blk() returns the blocks itself. In + addition, the function htx_get_head_type() returns its block's type. + + - the tail block. It is the newest inserted block. The function + htx_get_tail() returns its position and htx_get_tail_blk() returns the + blocks itself. In addition, the function htx_get_tail_type() returns its + block's type. + + - the first block. It is the block where to (re)start the analyse. It is + used as start point by HTX analyzers. The function htx_get_first() returns + its position and htx_get_first_blk() returns the blocks itself. In + addition, the function htx_get_first_type() returns its block's type. + +For all these functions, if the HTX message is empty, -1 is returned for the +block's position, NULL instead of a block and HTX_BLK_UNUSED for its type. + +Then to iterate on blocks, foreword or backward : + + - htx_get_prev() and htx_get_next() return, respectively, the position of + the previous block or the next block, given a specific position. Or -1 if + an edge is reached. + + - htx_get_prev_blk() and htx_get_next_blk() return, respectively, the + previous block or the next one, given a specific block. Or NULL if an edge + is reached. + +4.6. Access block content and info + +Following functions may be used to retrieve information about a specific HTX +block : + + - htx_get_blk_pos() returns the position of a block. It must be in the HTX + message. + + - htx_get_blk_ptr() returns a pointer on the payload of a block. + + - htx_get_blk_type() returns the type of a block. + + - htx_get_blksz() returns the payload size of a block + + - htx_get_blk_name() returns the name of a block, only if it is a header or + a trailer. Otherwise, it returns an empty string. + + - htx_get_blk_value() returns the value of a block, depending on its + type. For header and trailer blocks, it is the value field. For markers + (EOH or EOT), an empty string is returned. For other blocks an ist + pointing on the block payload is returned. + + - htx_is_unique_blk() may be used to know if a block is the only one + remaining inside an HTX message, excluding unused blocks. This function is + pretty useful to determine the end of a HTX message, in conjunction with + HTX_FL_EOM flag. + +4.7. Advanced functions + +Some more advanced functions may be used to do complex processing on the HTX +message. These functions are used by HTX analyzers or by multiplexers. + + - htx_truncate() removes all blocks after the one containing a specific + offset relatively to the head block of the HTX message. If the offset is + inside a DATA block, it is truncated. For all other blocks, the removal + starts to the next block. + + - htx_drain() tries to remove a specific amount of bytes of payload. If the + tail block is a DATA block, it may be truncated if necessary. All other + block are removed at once or kept. This function returns a mixed value, + with the first block not removed, or NULL if everything was removed, and + the amount of data drained. + + - htx_xfer_blks() transfers HTX blocks from an HTX message to another, + stopping on the first block of a specified type or when a specific amount + of bytes, including meta-data, was moved. If the tail block is a DATA + block, it may be partially moved. All other block are transferred at once + or kept. This function returns a mixed value, with the last block moved, + or NULL if nothing was moved, and the amount of data transferred. When + HEADERS or TRAILERS blocks must be transferred, this function transfers + all of them. Otherwise, if it is not possible, it triggers an error. It is + the caller responsibility to transfer all headers or trailers at once. + + - htx_append_msg() append an HTX message to another one. All the message is + copied or nothing. So, if an error occurred, a rollback is performed. This + function returns 1 on success and 0 on error. + + - htx_reserve_max_data() Reserves the maximum possible size for an HTX data + block, by extending an existing one or by creating a new one. It returns a + compound result with the HTX block and the position where new data must be + inserted (0 for a new block). If an error occurs or if there is no space + left, NULL is returned instead of a pointer on an HTX block. + + - htx_find_offset() looks for the HTX block containing a specific offset, + starting at the HTX message's head. The function returns the found HTX + block and the position inside this block where the offset is. If the + offset is outside of the HTX message, NULL is returned. + + - htx_defrag() defragments an HTX message. It removes unused blocks and + unwraps the payloads part. A temporary buffer is used to do so. This + function never fails. A referenced block may be provided. If so, the + corresponding new block is returned. Otherwise, NULL is returned. diff --git a/doc/internals/api/initcalls.txt b/doc/internals/api/initcalls.txt new file mode 100644 index 0000000..a341edc --- /dev/null +++ b/doc/internals/api/initcalls.txt @@ -0,0 +1,366 @@ +Initialization stages aka how to get your code initialized at the right moment + + +1. Background + +Originally all subsystems were initialized via a dedicated function call +from the huge main() function. Then some code started to become conditional +or a bit more modular and the #ifdef placed there became a mess, resulting +in init code being moved to function constructors in each subsystem's own +file. Then pools of various things were introduced, starting to make the +whole init sequence more complicated due to some forms of internal +dependencies. Later epoll was introduced, requiring a post-fork callback, +and finally threads arrived also requiring some post-thread init/deinit +and allocation, marking the old architecture's last breath. Finally the +whole thing resulted in lots of init code duplication and was simplified +in 1.9 with the introduction of initcalls and initialization stages. + + +2. New architecture + +The new architecture relies on two layers : + - the registration functions + - the INITCALL macros and initialization stages + +The first ones are mostly used to add a callback to a list. The second ones +are used to specify when to call a function. Both are totally independent, +however they are generally combined via another set consisting in the REGISTER +macros which make some registration functions be called at some specific points +during the init sequence. + + +3. Registration functions + +Registration functions never fail. Or more precisely, if they fail it will only +be on out-of-memory condition, and they will cause the process to immediately +exit. As such they do not return any status and the caller doesn't have to care +about their success. + +All available functions are described below in alphanumeric ordering. Please +make sure to respect this ordering when adding new ones. + +- void hap_register_build_opts(const char *str, int must_free) + + This appends the zero-terminated constant string <str> to the list of known + build options that will be reported on the output of "haproxy -vv". A line + feed character ('\n') will automatically be appended after the string when it + is displayed. The <must_free> argument must be zero, unless the string was + allocated by any malloc-compatible function such as malloc()/calloc()/ + realloc()/strdup() or memprintf(), in which case it's better to pass a + non-null value so that the string is freed upon exit. Note that despite the + function's prototype taking a "const char *", the pointer will actually be + cast and freed. The const char* is here to leave more freedom to use consts + when making such options lists. + +- void hap_register_per_thread_alloc(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called when + threads are started, at the beginning of the polling loop. This is also valid + for the main thread and will be called even if threads are disabled, so that + it is guaranteed that this function will be called in any circumstance. Each + thread will first call all these functions exactly once when it starts. Calls + are serialized by the init_mutex, so that locking is not necessary in these + functions. There is no relation between the thread numbers and the callback + ordering. The function is expected to return non-zero on success, or zero on + failure. A failure will make the process emit a succinct error message and + immediately exit. See also hap_register_per_thread_free() for functions + called after these ones. + +- void hap_register_per_thread_deinit(void (*fct)()); + + This adds a call to function <fct> to the list of functions to be called when + threads are gracefully stopped, at the end of the polling loop. This is also + valid for the main thread and will be called even if threads are disabled, so + that it is guaranteed that this function will be called in any circumstance + if the process experiences a soft stop. Each thread will call this function + exactly once when it stops. However contrary to _alloc() and _init(), the + calls are made without any protection, thus if any shared resource if touched + by the function, the function is responsible for protecting it. The reason + behind this is that such resources are very likely to be still in use in one + other thread and that most of the time the functions will in fact only touch + a refcount or deinitialize their private resources. See also + hap_register_per_thread_free() for functions called after these ones. + +- void hap_register_per_thread_free(void (*fct)()); + + This adds a call to function <fct> to the list of functions to be called when + threads are gracefully stopped, at the end of the polling loop, after all calls + to _deinit() callbacks are done for this thread. This is also valid for the + main thread and will be called even if threads are disabled, so that it is + guaranteed that this function will be called in any circumstance if the + process experiences a soft stop. Each thread will call this function exactly + once when it stops. However contrary to _alloc() and _init(), the calls are + made without any protection, thus if any shared resource if touched by the + function, the function is responsible for protecting it. The reason behind + this is that such resources are very likely to be still in use in one other + thread and that most of the time the functions will in fact only touch a + refcount or deinitialize their private resources. See also + hap_register_per_thread_deinit() for functions called before these ones. + +- void hap_register_per_thread_init(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called when + threads are started, at the beginning of the polling loop, right after the + list of _alloc() functions. This is also valid for the main thread and will + be called even if threads are disabled, so that it is guaranteed that this + function will be called in any circumstance. Each thread will call this + function exactly once when it starts, and calls are serialized by the + init_mutex which is held over all _alloc() and _init() calls, so that locking + is not necessary in these functions. In other words for all threads but the + current one, the sequence of _alloc() and _init() calls will be atomic. There + is no relation between the thread numbers and the callback ordering. The + function is expected to return non-zero on success, or zero on failure. A + failure will make the process emit a succinct error message and immediately + exit. See also hap_register_per_thread_alloc() for functions called before + these ones. + +- void hap_register_pre_check(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called at + the step just before the configuration validity checks. This is useful when you + need to create things like it would have been done during the configuration + parsing and where the initialization should continue in the configuration + check. + It could be used for example to generate a proxy with multiple servers using + the configuration parser itself. At this step the final trash buffers are + allocated. Threads are not yet started so no protection is required. The + function is expected to return non-zero on success, or zero on failure. A + failure will make the process emit a succinct error message and immediately + exit. + +- void hap_register_post_check(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called at + the end of the configuration validity checks, just at the point where the + program either forks or exits depending whether it's called with "-c" or not. + Such calls are suited for memory allocation or internal table pre-computation + that would preferably not be done on the fly to avoid inducing extra time to + a pure configuration check. Threads are not yet started so no protection is + required. The function is expected to return non-zero on success, or zero on + failure. A failure will make the process emit a succinct error message and + immediately exit. + +- void hap_register_post_deinit(void (*fct)()) + + This adds a call to function <fct> to the list of functions to be called when + freeing the global sections at the end of deinit(), after everything is + stopped. The process is single-threaded at this point, thus these functions + are suitable for releasing configuration elements provided that no other + _deinit() function uses them, i.e. only close/release what is strictly + private to the subsystem. Since such functions are mostly only called during + soft stops (reloads) or failed startups, they tend to experience much less + test coverage than others despite being more exposed, and as such a lot of + care must be taken to test them especially when facing partial subsystem + initializations followed by errors. + +- void hap_register_post_proxy_check(int (*fct)(struct proxy *)) + + This adds a call to function <fct> to the list of functions to be called for + each proxy, after the calls to _post_server_check(). This can allow, for + example, to pre-configure default values for an option in a frontend based on + the "bind" lines or something in a backend based on the "server" lines. It's + worth being aware that such a function must be careful not to waste too much + time in order not to significantly slow down configurations with tens of + thousands of backends. The function is expected to return non-zero on + success, or zero on failure. A failure will make the process emit a succinct + error message and immediately exit. + +- void hap_register_post_server_check(int (*fct)(struct server *)) + + This adds a call to function <fct> to the list of functions to be called for + each server, after the call to check_config_validity(). This can allow, for + example, to preset a health state on a server or to allocate a protocol- + specific memory area. It's worth being aware that such a function must be + careful not to waste too much time in order not to significantly slow down + configurations with tens of thousands of servers. The function is expected + to return non-zero on success, or zero on failure. A failure will make the + process emit a succinct error message and immediately exit. + +- void hap_register_proxy_deinit(void (*fct)(struct proxy *)) + + This adds a call to function <fct> to the list of functions to be called when + freeing the resources during deinit(). These functions will be called as part + of the proxy's resource cleanup. Note that some of the proxy's fields will + already have been freed and others not, so such a function must not use any + information from the proxy that is subject to being released. In particular, + all servers have already been deleted. Since such functions are mostly only + called during soft stops (reloads) or failed startups, they tend to + experience much less test coverage than others despite being more exposed, + and as such a lot of care must be taken to test them especially when facing + partial subsystem initializations followed by errors. It's worth mentioning + that too slow functions could have a significant impact on the configuration + check or exit time especially on large configurations. + +- void hap_register_server_deinit(void (*fct)(struct server *)) + + This adds a call to function <fct> to the list of functions to be called when + freeing the resources during deinit(). These functions will be called as part + of the server's resource cleanup. Note that some of the server's fields will + already have been freed and others not, so such a function must not use any + information from the server that is subject to being released. Since such + functions are mostly only called during soft stops (reloads) or failed + startups, they tend to experience much less test coverage than others despite + being more exposed, and as such a lot of care must be taken to test them + especially when facing partial subsystem initializations followed by errors. + It's worth mentioning that too slow functions could have a significant impact + on the configuration check or exit time especially on large configurations. + + +4. Initialization stages + +In order to offer some guarantees, the startup of the program is split into +several stages. Some callbacks can be placed into each of these stages using +an INITCALL macro, with 0 to 3 arguments, respectively called INITCALL0 to +INITCALL3. These macros must be placed anywhere at the top level of a C file, +preferably at the end so that the referenced symbols have already been met, +but it may also be fine to place them right after the callbacks themselves. + +Such callbacks are referenced into small structures containing a pointer to the +function and 3 arguments. NULL replaces unused arguments. The callbacks are +cast to (void (*)(void *, void *, void *)) and the arguments to (void *). + +The first argument to the INITCALL macro is the initialization stage. The +second one is the callback function, and others if any are the arguments. +The init stage must be among the values of the "init_stage" enum, currently, +and in this execution order: + + - STG_PREPARE : used to preset variables, pre-initialize lookup tables and + pre-initialize list heads + - STG_LOCK : used to pre-initialize locks + - STG_REGISTER : used to register static lists such as keywords + - STG_ALLOC : used to allocate the required structures + - STG_POOL : used to create pools + - STG_INIT : used to initialize subsystems + +Each stage is guaranteed that previous stages have successfully completed. This +means that an INITCALL placed at stage STG_INIT is guaranteed that all pools +were already created and will be usable. Conversely, an INITCALL placed at +stage STG_REGISTER must not rely on any field that requires preliminary +allocation nor initialization. A callback cannot rely on other callbacks of the +same stage, as the execution order within a stage is undefined and essentially +depends on the linking order. + +The STG_REGISTER level is made for run-time linking of the various modules that +compose the executable. Keywords, protocols and various other elements that are +local known to each compilation unit can will be appended into common lists at +boot time. This is why this call is placed just before STG_ALLOC. + +Note that trash is needed in various functions. Trash is a pool and is +allocated during STG_POOL, so it's not permitted to use it before STG_INIT, +where it will only use the default size, and may be reallocated later with a +different size. + +Example: register a very early call to init_log() with no argument, and another + call to cli_register_kw(&cli_kws) much later: + + INITCALL0(STG_PREPARE, init_log); + INITCALL1(STG_REGISTER, cli_register_kw, &cli_kws); + +Technically speaking, each call to such a macro adds a distinct local symbol +whose dynamic name involves the line number. These symbols are placed into a +separate section and the beginning and end section pointers are provided by the +linker. When too old a linker is used, a fallback is applied consisting in +placing them into a linked list which is built by a constructor function for +each initcall (this takes more room). + +Due to the symbols internally using the line number, it is very important not +to place more than one INITCALL per line in the source file. + +It is also strongly recommended that functions and referenced arguments are +static symbols local to the source file, unless they are global registration +functions like in the example above with cli_register_kw(), where only the +argument is a local keywords table. + +INITCALLs do not expect the callback function to return anything and as such +do not perform any error check. As such, they are very similar to constructors +offered by the compiler except that they are segmented in stages. It is thus +the responsibility of the called functions to perform their own error checking +and to exit in case of error. This may change in the future. + + +5. REGISTER family of macros + +The association of INITCALLs and registration functions allows to perform some +early dynamic registration of functions to be used anywhere, as well as values +to be added to existing lists without having to manipulate list elements. For +the sake of simplification, these combinations are available as a set of +REGISTER macros which register calls to certain functions at the appropriate +init stage. Such macros must be used at the top level in a file, just like +INITCALL macros. The following macros are currently supported. Please keep them +alphanumerically ordered: + +- REGISTER_BUILD_OPTS(str) + + Adds the constant string <str> to the list of build options. This is done by + registering a call to hap_register_build_opts(str, 0) at stage STG_REGISTER. + The string will not be freed. + +- REGISTER_CONFIG_POSTPARSER(name, parser) + + Adds a call to function <parser> at the end of the config parsing. The + function is called at the very end of check_config_validity() and may be used + to initialize a subsystem based on global settings for example. This is done + by registering a call to cfg_register_postparser(name, parser) at stage + STG_REGISTER. + +- REGISTER_CONFIG_SECTION(name, parse, post) + + Registers a new config section name <name> which will be parsed by function + <parse> (if not null), and with an optional call to function <post> at the + end of the section. Function <parse> must be of type (int (*parse)(const char + *file, int linenum, char **args, int inv)), and returns 0 on success or an + error code among the ERR_* set on failure. The <post> callback takes no + argument and returns a similar error code. This is achieved by registering a + call to cfg_register_section() with the three arguments at stage + STG_REGISTER. + +- REGISTER_PER_THREAD_ALLOC(fct) + + Registers a call to register_per_thread_alloc(fct) at stage STG_REGISTER. + +- REGISTER_PER_THREAD_DEINIT(fct) + + Registers a call to register_per_thread_deinit(fct) at stage STG_REGISTER. + +- REGISTER_PER_THREAD_FREE(fct) + + Registers a call to register_per_thread_free(fct) at stage STG_REGISTER. + +- REGISTER_PER_THREAD_INIT(fct) + + Registers a call to register_per_thread_init(fct) at stage STG_REGISTER. + +- REGISTER_POOL(ptr, name, size) + + Used internally to declare a new pool. This is made by calling function + create_pool_callback() with these arguments at stage STG_POOL. Do not use it + directly, use either DECLARE_POOL() or DECLARE_STATIC_POOL() instead. + +- REGISTER_PRE_CHECK(fct) + + Registers a call to register_pre_check(fct) at stage STG_REGISTER. + +- REGISTER_POST_CHECK(fct) + + Registers a call to register_post_check(fct) at stage STG_REGISTER. + +- REGISTER_POST_DEINIT(fct) + + Registers a call to register_post_deinit(fct) at stage STG_REGISTER. + +- REGISTER_POST_PROXY_CHECK(fct) + + Registers a call to register_post_proxy_check(fct) at stage STG_REGISTER. + +- REGISTER_POST_SERVER_CHECK(fct) + + Registers a call to register_post_server_check(fct) at stage STG_REGISTER. + +- REGISTER_PROXY_DEINIT(fct) + + Registers a call to register_proxy_deinit(fct) at stage STG_REGISTER. + +- REGISTER_SERVER_DEINIT(fct) + + Registers a call to register_server_deinit(fct) at stage STG_REGISTER. + diff --git a/doc/internals/api/ist.txt b/doc/internals/api/ist.txt new file mode 100644 index 0000000..0f118d6 --- /dev/null +++ b/doc/internals/api/ist.txt @@ -0,0 +1,167 @@ +2021-11-08 - Indirect Strings (IST) API + + +1. Background +------------- + +When parsing traffic, most of the standard C string functions are unusable +since they rely on a trailing zero. In addition, for the rare ones that support +a length, we have to constantly maintain both the pointer and the length. But +then, it's easy to come up with complex lengths and offsets calculations all +over the place, rendering the code hard to read and bugs hard to avoid or spot. + +IST provides a solution to this by defining a structure made of exactly two +word size elements, that most C ABIs know how to handle as a register when +used as a function argument or a function's return value. The functions are +inlined to leave a maximum set of opportunities to the compiler or optimization +and expression reduction, and as a result they are often inexpensive to use. It +is important however to keep in mind that all of these are designed for minimal +code size when dealing with short strings (i.e. parsing tokens in protocols), +and they are not optimal for processing large blocks. + + +2. API description +------------------ + +IST are defined like this: + + struct ist { + char *ptr; // pointer to the string's first byte + size_t len; // number of valid bytes starting from ptr + }; + +A string is not set if its ->ptr member is NULL. In this case .len is undefined +and is recommended to be zero. + +Declaring a function returning an IST: + + struct ist produce_ist(int ok) + { + return ok ? IST("OK") : IST("KO"); + } + +Declaring a function consuming an IST: + + void say_ist(struct ist i) + { + write(1, istptr(i), istlen(i)); + } + +Chaining the two: + + void say_ok(int ok) + { + say_ist(produce_ist(ok)); + } + +Notes: + - the arguments are passed as value, not reference, so there's no need for + any "const" in their declaration (except to catch coding mistakes). + Pointers to ist may benefit from being marked "const" however. + + - similarly for the return value, there's no point is marking it "const" as + this would protect the pointer and length, not the data. + + - use ist0() to append a trailing zero to a variable string for use with + printf()'s "%s" format, or for use with functions that work on NUL- + terminated strings, but beware of not doing this with constants. + + - the API provides a starting pointer and current length, but does not + provide an allocated size. It remains up to the caller to know how large + the allocated area is when adding data, though most functions make this + easy. + +The following macros and functions are defined. Those whose name starts with +underscores require special care and must not be used without being certain +they are properly used (typically subject to buffer overflows if misused). Note +that most functions were added over time depending on instant needs, and some +are very close to each other. Many useful functions are still missing and would +deserve being added. + +Below, arguments "i1","i2" are all of type "ist". Arguments "s" are +NUL-terminated strings of type "char*", and "cs" are of type "const char *". +Arguments "c" are of type "char", and "n" are of type size_t. + + IST(cs):ist make constant IST from a NUL-terminated const string + IST_NULL:ist return an unset IST = ist2(NULL,0) + __istappend(i1,c):ist append character <c> at the end of ist <i1> + ist(s):ist return an IST from a nul-terminated string + ist0(i1):char* write a \0 at the end of an IST, return the string + ist2(cs,l):ist return a variable IST from a const string and length + ist2bin(s,i1):ist copy IST into a buffer, return the result + ist2bin_lc(s,i1):ist like ist2bin() but turning turning to lower case + ist2bin_uc(s,i1):ist like ist2bin() but turning turning to upper case + ist2str(s,i1):ist copy IST into a buffer, add NUL and return the result + ist2str_lc(s,i1):ist like ist2str() but turning turning to lower case + ist2str_uc(s,i1):ist like ist2str() but turning turning to upper case + ist_find(i1,c):ist return first occurrence of char <c> in <i1> + ist_find_ctl(i1):char* return pointer to first CTL char in <i1> or NULL + ist_skip(i1,c):ist return first occurrence of char not <c> in <i1> + istadv(i1,n):ist advance the string by <n> characters + istalloc(n):ist return allocated string of zero initial length + istcat(d,s,n):ssize_t copy <s> after <d> for <n> chars max, return len or -1 + istchr(i1,c):char* return pointer to first occurrence of <c> in <i1> + istclear(i1*):size_t return previous size and set size to zero + istcpy(d,s,n):ssize_t copy <s> over <d> for <n> chars max, return len or -1 + istdiff(i1,i2):int return the ordinal difference, like strcmp() + istdup(i1):ist allocate new ist and copy original one into it + istend(i1):char* return pointer to first character after the IST + isteq(i1,i2):int return non-zero if strings are equal + isteqi(i1,i2):int like isteq() but case-insensitive + istfree(i1*) free of allocated <i1>/IST_NULL and set it to IST_NULL + istissame(i1,i2):int return true if pointers and lengths are equal + istist(i1,i2):ist return first occurrence of <i2> in <i1> + istlen(i1):size_t return the length of the IST (number of characters) + istmatch(i1,i2):int return non-zero if i1 starts like i2 (empty OK) + istmatchi(i1,i2):int like istmatch() but case insensitive + istneq(i1,i2,n):int like isteq() but limited to the first <n> chars + istnext(i1):ist return the IST advanced by one character + istnmatch(i1,i2,n):int like istmatch() but limited to the first <n> chars + istpad(s,i1):ist copy IST into a buffer, add a NUL, return the result + istptr(i1):char* return the starting pointer of the IST + istscat(d,s,n):ssize_t same as istcat() but always place a NUL at the end + istscpy(d,s,n):ssize_t same as istcpy() but always place a NUL at the end + istshift(i1*):char return the first character and advance the IST by one + istsplit(i1*,c):ist return part before <c>, make ist start from <c> + iststop(i1,c):ist truncate ist before first occurrence of <c> + isttest(i1):int return true if ist is not NULL, false otherwise + isttrim(i1,n):ist return ist trimmed to no more than <n> characters + istzero(i1,n):ist trim to <n> chars, trailing zero included. + + +3. Quick index by typical C construct or function +------------------------------------------------- + +Some common C constructs may be adjusted to use ist instead. The mapping is not +always one-to-one, but usually the computations on the length part tends to +disappear in the refactoring, allowing to directly chain function calls. The +entries below are hints to figure what function to look for in order to rewrite +some common use cases. + + char* IST equivalent + + strchr() istchr(), ist_find(), iststop() + strstr() istist() + strcpy() istcpy() + strscpy() istscpy() + strlcpy() istscpy() + strcat() istcat() + strscat() istscat() + strlcat() istscat() + strcmp() istdiff() + strdup() istdup() + !strcmp() isteq() + !strncmp() istneq(), istmatch(), istnmatch() + !strcasecmp() isteqi() + !strncasecmp() istneqi(), istmatchi() + strtok() istsplit() + return NULL return IST_NULL + s = malloc() s = istalloc() + free(s); s = NULL istfree(&s) + p != NULL isttest(p) + c = *(p++) c = istshift(p) + *(p++) = c __istappend(p, c) + p += n istadv(p, n) + p + strlen(p) istend(p) + p[max] = 0 isttrim(p, max) + p[max+1] = 0 istzero(p, max) diff --git a/doc/internals/api/layers.txt b/doc/internals/api/layers.txt new file mode 100644 index 0000000..b5c35f4 --- /dev/null +++ b/doc/internals/api/layers.txt @@ -0,0 +1,190 @@ +2022-05-27 - Stream layers in HAProxy 2.6 + + +1. Background + +There are streams at plenty of levels in haproxy, essentially due to the +introduction of multiplexed protocols which provide high-level streams on top +of low-level streams, themselves either based on stream-oriented protocols or +datagram-oriented protocols. + +The refactoring of the appctx and muxes that allowed to drop a lot of duplicate +code between 2.5 and 2.6-dev6 raised another concern with some entities like +"conn_stream" that were not specific to connections anymore, "endpoints" that +became entities on their own, and "targets" whose life had been extended to +last all along a connection. + +It was time to rename all such legacy entities introduced in 1.8 and which had +turned particularly confusing over time as their roles evolved. + + +2. Naming principles + +The global renaming of some entities between streams and connections was +articulated around several principles: + + - avoid the confusing use of "context" in shared places. For example, the + endpoint's connection is in "ctx" and nothing makes it obvious that the + endpoint's context is a connection, especially when an applet is there. + + - reserve relative nouns for pointers and not for types. "endpoint", just + like "owner" or "peer" is relative, but when accessed from a different + layer it starts to make no sense at all, or to make one believe it's + something else, particularly with void*. + + - avoid too generic terms that have multiple meanings, or words that are + synonyms in a same place (e.g. "peer" and "remote", or "endpoint" and + "target"). If two synonyms are needed to designate two distinct entities, + there's probably a problem elsewhere, or the problem is poorly defined. + + - make it clearer that all that is manipulated is related to streams. This + particularly important in sample fetch functions for example, which tend + to require low-level access and could be mislead in trying to follow the + wrong chain when trying to get information about a connection. + + - use easily spellable short names that abbreviate unambiguously when used + together in adjacent contexts + + +3. Current state as of 2.6 + +- when a name is required to designate the lower block that starts at the mux + stream or the appctx, it is spoken of as a "stream endpoint", and abbreviated + "se". It's okay because while "endpoint" itself is relative, "stream + endpoint" unequivocally designates one extremity of a stream. If a type is + needed for this in the future (e.g. via obj_type), then the type "stendp" + may be used. Before 2.6-dev6 there was no name for this, it was known as + conn_stream->ctx. + +- the 2.6-dev6 cs_endpoint which preserves the state of a mux stream or an + appctx and abstracts them in front of a conn_stream becomes a "stream + endpoint descriptor", of type "sedesc" and often abbreviated "sd", "sed" + or "ed". Its "target" pointer became "se" as per the rule above. Before + 2.6-dev6, these elements were mixed with others inside conn_stream. From + the appctx it's called "sedesc" (few occurrences hence long name OK). + +- the conn_stream which is always attached to either a stream or a health check + and that is used to reach a mux or an applet becomes a "stream connector" of + type "stconn", generally abbreviated "sc". Its "endp" pointer becomes + "sedesc" as per the rule above, and that one has a back pointer "sc". The + stream uses "scf" and "scb" as the respective front and back pointers to the + stconns. Prior to 2.6-dev6, these parts were split between conn_stream and + stream_interface. + +- the sedesc's "ctx" which is solely used to store the connection as of now, is + renamed "conn" to void any doubt in the context of applets or even muxes. In + the future the connection should be attached to the "se" instead and this + pointer should disappear (or be recycled for anything else). + +The new 2.6 model looks like this: + + +------------------------+ + | stream or health check | + +------------------------+ + ^ \ scf, scb + / \ + | | + \ / + app \ v + +----------+ + | stconn | + +----------+ + ^ \ sedesc + / \ + . . . . | . . . | . . . . . split point (retries etc) + \ / + sc \ v + +----------+ + flags <--| sedesc | : sedesc : + +----------+ ... +----------+ + conn / ^ \ se ^ \ + +------------+ / / \ | \ + | connection |<--' | | ... OR ... | | + +------------+ \ / \ | + mux| ^ |ctx sd \ v : sedesc \ v + | | | +----------------------+ \ # +----------+ svcctx + | | | | mux stream or appctx | | # | appctx |--. + | | | +----------------------+ | # +----------+ | + | | | ^ | / private # : : | + v | | | v > to the # +----------+ | + mux_ops | | +----------------+ \ mux # | svcctx |<-' + | +---->| mux connection | ) # +----------+ + +------ +----------------+ / # + +Stream descriptors may exist in the following modes: + - .conn = NULL, .se = NULL : backend, not connection attempt yet + - .conn = NULL, .se = <appctx> : frontend or backend, applet + - .conn = <conn>, .se = NULL : backend, connection in progress + - .conn = <conn>, .se = <muxs> : frontend or backend, connected + +Notes: + - for historical reasons (connect, forced protocol upgrades, etc), during a + connection setup or a rule-based protocol upgrade, the connection's "ctx" + may temporarily point to the stconn + + +4. Invariants and cardinalities + +Usually a stream is created from an existing stconn from a mux or some applets, +but may also be allocated first by other applets schedulers. After stream_new() +a stream always has exactly one stconn per side (scf, scb), each of which has +one ->sedesc. Each side is initialized with either one or no stream endpoint +attached to the descriptor. + +Both applets and a mux stream always have a stream endpoint descriptor. AS SUCH +IT IS NEVER NECESSARY TO TEST FOR THE EXISTENCE OF THE SEDESC FROM ANY SIDE, IT +ALWAYS EXISTS. This explains why as much as possible it's preferable to use the +sedesc to access flags and statuses from any side, rather than bouncing via the +stconn. + +An applet's app layer is always a stream, which means that there are always +channels accessible above, and there is always an opposite stream connector and +a stream endpoint descriptor. As such, it's always safe for an applet to access +the other side using sc_opposite(). + +When an outgoing connection is in the process of being established, the backend +side sedesc has its ->conn pointer pointing to the pending connection, and no +->se. Once the connection is established and a mux is chosen, it's attached to +the ->se. If an applet is used instead of a mux, the appctx is attached to the +sedesc's ->se and ->conn remains NULL. + +If either side wants to detach from the other, it must allocate a new virgin +sedesc to replace the existing one, and leave the existing one to the endpoint, +since it continues to describe the stream endpoint. The stconn keeps its state +(modulo the updates related to the disconnection). The previous sedesc points +to a NULL stconn. For example, disconnecting from a backend mux will leave the +entities like this: + + +------------------------+ + | stream or health check | + +------------------------+ + ^ \ scf, scb + / \ + | | + \ / + app \ v + +----------+ + | stconn | + +----------+ + ^ \ sedesc + / \ + NULL | | + ^ \ / + sc | / sc \ v + +----------+ / +----------+ + flags <--| sedesc1 | . . . . . | sedesc2 |--> flags + +----------+ / +----------+ + conn / ^ \ se / conn / \ se + +------------+ / / \ | | + | connection |<--' | | v v + +------------+ \ / NULL NULL + mux| ^ |ctx sd \ v + | | | +----------------------+ + | | | | mux stream or appctx | + | | | +----------------------+ + | | | ^ | + v | | | v + mux_ops | | +----------------+ + | +---->| mux connection | + +------ +----------------+ + diff --git a/doc/internals/api/list.txt b/doc/internals/api/list.txt new file mode 100644 index 0000000..d03cf03 --- /dev/null +++ b/doc/internals/api/list.txt @@ -0,0 +1,195 @@ +2021-11-09 - List API + + +1. Background +------------- + +HAProxy's lists are almost all doubly-linked and circular so that it is always +possible to insert at the beginning, append at the end, scan them in any order +and delete any element without having to scan to search the predecessor nor the +successor. + +A list's head is just a regular list element, and an element always points to +another list element. Such elements only have two pointers, the next and the +previous elements. The object being pointed to is retrieved by subtracting the +list element's offset in its structure from the list element's pointer. This +way there is no need for any separate allocation for the list element, for a +pointer to the object in the list, nor for a pointer to the list element from +the object, as the list is embedded into the object. + +All basic operations are provided, as well as some iterators. Some iterators +are safe for removal of the current element within the loop, others not. In any +case a list cannot be freely modified while iterating over it (e.g. the current +element's successor cannot not be freed if it's saved as the restart point). + +Extreme care is taken nowadays in HAProxy to make sure that no dangling +pointers are left in elements, so it is important to always initialize list +heads and list elements, as well as elements that are removed from a list if +they are not immediately freed, so that their deletion is idempotent. A rule of +thumb is that a list pointer's validity never has to be checked, it is always +valid to dereference it. A lot of complex bugs have been caused in the past by +incorrect list manipulation, such as an element being deleted twice, resulting +in damaging previously adjacent elements' neighbours. This usually has serious +consequences at locations that are totally different from the one of the bug, +and that are only detected much later, so it is required to be particularly +strict on using lists safely. + +The lists are not thread-safe, but mt_lists may be used instead. + + +2. API description +------------------ + +A list is defined like this, both for the list's head, and for any other +element: + + struct list { + struct list *n; /* next */ + struct list *p; /* prev */ + }; + +An empty list points to itself for both pointers. I.e. a list's head is both +its own successor and its own predecessor. This guarantees that insertions +and deletions can be done without any check and that deletion is idempotent. +For this reason and by convention, a detached element ought to be represented +like an empty head. + +Lists are manipulated using a set of macros which are used to initialize, add, +remove, or iterate over elements. Most of these macros are extremely simple and +are not even protected against multiple evaluation, so it is fundamentally +important that the expressions used in the arguments are idempotent and that +the result does not depend on the evaluation order of the arguments. + +Macro Description + +ILH + Initialized List Head : this is a non-NULL, non-empty list element used + to prevent the compiler from moving an empty list head declaration to + BSS, typically when it appears in an array of keywords Without this, + some older versions of gcc tend to trim all the array and cause + corruption. + +LIST_INIT(l) + Initialize the list as an empty list head + +LIST_HEAD_INIT(l) + Return a valid initialized empty list head pointing to this + element. Essentially used with assignments in declarations. + +LIST_INSERT(l, e) + Add an element at the beginning of a list and return it + +LIST_APPEND(l, e) + Add an element at the end of a list and return it + +LIST_SPLICE(n, o) + Add the contents of a list <o> at the beginning of another list <n>. + The old list head remains untouched. + +LIST_SPLICE_END_DETACHED(n, o) + Add the contents of a list whose first element is is <o> and last one + is <o->p> at the end of another list <n>. The old list DOES NOT have + any head here. + +LIST_DELETE(e) + Remove an element from a list and return it. Safe to call on + initialized elements, but will not change the element itself so it is + not idempotent. Consider using LIST_DEL_INIT() instead unless called + immediately after a free(). + +LIST_DEL_INIT(e) + Remove an element from a list, initialize it and return it so that a + subsequent LIST_DELETE() is safe. This is faster than performing a + LIST_DELETE() followed by a LIST_INIT() as pointers are not reloaded. + +LIST_ELEM(l, t, m) + Return a pointer of type <t> to a structure containing a list head + member called <m> at address <l>. Note that <l> can be the result of a + function or macro since it's used only once. + +LIST_ISEMPTY(l) + Check if the list head <l> is empty (=initialized) or not, and return + non-zero only if so. + +LIST_INLIST(e) + Check if the list element <e> was added to a list or not, thus return + true unless the element was initialized. + +LIST_INLIST_ATOMIC(e) + Atomically check if the list element's next pointer points to anything + different from itself, implying the element should be part of a + list. This usually is similar to LIST_INLIST() except that while that + one might be instrumented using debugging code to perform further + consistency checks, the macro below guarantees to always perform a + single atomic test and is safe to use with barriers. + +LIST_NEXT(l, t, m) + Return a pointer of type <t> to a structure following the element which + contains list head <l>, which is known as member <m> in struct <t>. + +LIST_PREV(l, t, m) + Return a pointer of type <t> to a structure preceding the element which + contains list head <l>, which is known as member <m> in struct <t>. + Note that this macro is first undefined as it happened to already exist + on some old OSes. + +list_for_each_entry(i, l, m) + Iterate local variable <i> through a list of items of type "typeof(*i)" + which are linked via a "struct list" member named <m>. A pointer to the + head of the list is passed in <l>. No temporary variable is needed. + Note that <i> must not be modified during the loop. + +list_for_each_entry_from(i, l, m) + Same as list_for_each_entry() but starting from current value of <i> + instead of the list's head. + +list_for_each_entry_from_rev(i, l, m) + Same as list_for_each_entry_rev() but starting from current value of <i> + instead of the list's head. + +list_for_each_entry_rev(i, l, m) + Iterate backwards local variable <i> through a list of items of type + "typeof(*i)" which are linked via a "struct list" member named <m>. A + pointer to the head of the list is passed in <l>. No temporary variable + is needed. Note that <i> must not be modified during the loop. + +list_for_each_entry_safe(i, b, l, m) + Iterate variable <i> through a list of items of type "typeof(*i)" which + are linked via a "struct list" member named <m>. A pointer to the head + of the list is passed in <l>. A temporary backup variable <b> of same + type as <i> is needed so that <i> may safely be deleted if needed. Note + that it is only permitted to delete <i> and no other element during + this operation! + +list_for_each_entry_safe_from(i, b, l, m) + Same as list_for_each_entry_safe() but starting from current value of + <i> instead of the list's head. + +list_for_each_entry_safe_from_rev(i, b, l, m) + Same as list_for_each_entry_safe_rev() but starting from current value + of <i> instead of the list's head. + +list_for_each_entry_safe_rev(i, b, l, m) + Iterate backwards local variable <i> through a list of items of type + "typeof(*i)" which are linked via a "struct list" member named <m>. A + pointer to the head of the list is passed in <l>. A temporary variable + <b> of same type as <i> is needed so that <i> may safely be deleted if + needed. Note that it is only permitted to delete <i> and no other + element during this operation! + +3. Notes +-------- + +- This API is quite old and some macros are missing. For example there's still + no list_first() so it's common to use LIST_ELEM(head->n, ...) instead. Some + older parts of the code also used to rely on list_for_each() followed by a + break to stop on the first element. + +- Some parts were recently renamed because LIST_ADD() used to do what + LIST_INSERT() currently does and was often mistaken with LIST_ADDQ() which is + what LIST_APPEND() now is. As such it is not totally impossible that some + places use a LIST_INSERT() where a LIST_APPEND() would be desired. + +- The structure must not be modified at all (even to add debug info). Some + parts of the code assume that its layout is exactly this one, particularly + the parts ensuring the casting between MT lists and lists. diff --git a/doc/internals/api/pools.txt b/doc/internals/api/pools.txt new file mode 100644 index 0000000..d84fb9d --- /dev/null +++ b/doc/internals/api/pools.txt @@ -0,0 +1,585 @@ +2022-02-24 - Pools structure and API + +1. Background +------------- + +Memory allocation is a complex problem covered by a massive amount of +literature. Memory allocators found in field cover a broad spectrum of +capabilities, performance, fragmentation, efficiency etc. + +The main difficulty of memory allocation comes from finding the optimal chunks +for arbitrary sized requests, that will still preserve a low fragmentation +level. Doing this well is often expensive in CPU usage and/or memory usage. + +In programs like HAProxy that deal with a large number of fixed size objects, +there is no point having to endure all this risk of fragmentation, and the +associated costs (sometimes up to several milliseconds with certain minimalist +allocators) are simply not acceptable. A better approach consists in grouping +frequently used objects by size, knowing that due to the high repetitiveness of +operations, a freed object will immediately be needed for another operation. + +This grouping of objects by size is what is called a pool. Pools are created +for certain frequently allocated objects, are usually merged together when they +are of the same size (or almost the same size), and significantly reduce the +number of calls to the memory allocator. + +With the arrival of threads, pools started to become a bottleneck so they now +implement an optional thread-local lockless cache. Finally with the arrival of +really efficient memory allocator in modern operating systems, the shared part +has also become optional so that it doesn't consume memory if it does not bring +any value. + +In 2.6-dev2, a number of debugging options that used to be configured at build +time only changed to boot-time and can be modified using keywords passed after +"-dM" on the command line, which sets or clears bits in the pool_debugging +variable. The build-time options still affect the default settings however. +Default values may be consulted using "haproxy -dMhelp". + + +2. Principles +------------- + +The pools architecture is selected at build time. The main options are: + + - thread-local caches and process-wide shared pool enabled (1) + + This is the default situation on most operating systems. Each thread has + its own local cache, and when depleted it refills from the process-wide + pool that avoids calling the standard allocator too often. It is possible + to force this mode at build time by setting CONFIG_HAP_GLOBAL_POOLS or at + boot time with "-dMglobal". + + - thread-local caches only are enabled (2) + + This is the situation on operating systems where a fast and modern memory + allocator is detected and when it is estimated that the process-wide shared + pool will not bring any benefit. This detection is automatic at build time, + but may also be forced at build tmie by setting CONFIG_HAP_NO_GLOBAL_POOLS + or at boot time with "-dMno-global". + + - pass-through to the standard allocator (3) + + This is used when one absolutely wants to disable pools and rely on regular + malloc() and free() calls, essentially in order to trace memory allocations + by call points, either internally via DEBUG_MEM_STATS, or externally via + tools such as Valgrind. This mode of operation may be forced at build time + by setting DEBUG_NO_POOLS or at boot time with "-dMno-cache". + + - pass-through to an mmap-based allocator for debugging (4) + + This is used only during deep debugging when trying to detect various + conditions such as use-after-free. In this case each allocated object's + size is rounded up to a multiple of a page size (4096 bytes) and an + integral number of pages is allocated for each object using mmap(), + surrounded by two unaccessible holes that aim to detect some out-of-bounds + accesses. Released objects are instantly freed using munmap() so that any + immediate subsequent access to the memory area crashes the process if the + area had not been reallocated yet. This mode can be enabled at build time + by setting DEBUG_UAF, or at run time by disabling pools and enabling UAF + with "-dMuaf". It tends to consume a lot of memory and not to scale at all + with concurrent calls, that tends to make the system stall. The watchdog + may even trigger on some slow allocations. + +There are no more provisions for running with a shared pool but no thread-local +cache: the shared pool's main goal is to compensate for the expensive calls to +the memory allocator. This gain may be huge on tiny systems using basic +allocators, but the thread-local cache will already achieve this. And on larger +threaded systems, the shared pool's benefit is visible when the underlying +allocator scales poorly, but in this case the shared pool would suffer from +the same limitations without its thread-local cache and wouldn't provide any +benefit. + +Summary of the various operation modes: + + (1) (2) (3) (4) + + User User User User + | | | | + pool_alloc() V V | | + +---------+ +---------+ | | + | Thread | | Thread | | | + | Local | | Local | | | + | Cache | | Cache | | | + +---------+ +---------+ | | + | | | | + pool_refill*() V | | | + +---------+ | | | + | Shared | | | | + | Pool | | | | + +---------+ | | | + | | | | + malloc() V V V | + +---------+ +---------+ +---------+ | + | Library | | Library | | Library | | + +---------+ +---------+ +---------+ | + | | | | + mmap() V V V V + +---------+ +---------+ +---------+ +---------+ + | OS | | OS | | OS | | OS | + +---------+ +---------+ +---------+ +---------+ + +One extra build define, DEBUG_FAIL_ALLOC, is used to enforce random allocation +failure in pool_alloc() by randomly returning NULL, to test that callers +properly handle allocation failures. It may also be enabled at boot time using +"-dMfail". In this case the desired average rate of allocation failures can be +fixed by global setting "tune.fail-alloc" expressed in percent. + +The thread-local caches contain the freshest objects. Its total size amounts to +the number of bytes set in global.tune.pool_cache_size and that may be adjusted +by the "tune.memory.hot-size" global option, which itself defaults to build +time setting CONFIG_HAP_POOL_CACHE_SIZE, which was 1MB before 2.6 and 512kB +after. The aim is to keep hot objects that still fit in the CPU core's private +L2 cache. Once these objects do not fit into the cache anymore, there's no +benefit keeping them local to the thread, so they'd rather be returned to the +shared pool or the main allocator so that any other thread may make use of +them. Under extreme thread contention the cost of accessing shared structures +in the global cache or in malloc() may still be important and it may prove +useful to increase the thread-local cache size. + + +3. Storage in thread-local caches +--------------------------------- + +This section describes how objects are linked in thread local caches. This is +not meant to be a concern for users of the pools API but it can be useful when +inspecting post-mortem dumps or when trying to figure certain size constraints. + +Objects are stored in the local cache using a doubly-linked list. This ensures +that they can be visited by freshness order like a stack, while at the same +time being able to access them from oldest to newest when it is needed to +evict coldest ones first: + + - releasing an object to the cache always puts it on the top. + + - allocating an object from the cache always takes the topmost one, hence the + freshest one. + + - scanning for older objects to evict starts from the bottom, where the + oldest ones are located + +To that end, each thread-local cache keeps a list head in the "list" member of +its "pool_cache_head" descriptor, that links all objects cast to type +"pool_cache_item" via their "by_pool" member. + +Note that the mechanism described above only works for a single pool. When +trying to limit the total cache size to a certain value, all pools included, +there is also a need to arrange all objects from all pools together in the +local caches. For this, each thread_ctx maintains a list head of recently +released objects, all pools included, in its member "pool_lru_head". All items +in a thread-local cache are linked there via their "by_lru" member. + +This means that releasing an object using pool_free() consists in inserting +it at the beginning of two lists: + - the local pool_cache_head's "list" list head + - the thread context's "pool_lru_head" list head + +Allocating an object consists in picking the first entry from the pool's "list" +and deleting its "by_pool" and "by_lru" links. + +Evicting an object consists in scanning the thread context's "pool_lru_head" +backwards and deleting the object's "by_pool" and "by_lru" links. + +Given that entries are both inserted and removed synchronously, we have the +guarantee that the oldest object in the thread's LRU list is always the oldest +object in its pool, and that the next element is the cache's list head. This is +what allows the LRU eviction mechanism to figure what pool an object belongs to +when releasing it. + +Note: + | Since a pool_cache_item has two list entries, on 64-bit systems it will be + | 32-bytes long. This is the smallest size that a pool may be, and any smaller + | size will automatically be rounded up to this size. + +When build option DEBUG_POOL_INTEGRITY is set, or the boot-time option +"-dMintegrity" is passed on the command line, the area of the object between +the two list elements and the end according to pool->size will be filled with +pseudo-random words during pool_put_to_cache(), and these words will be +compared between each other during pool_get_from_cache(), and the process will +crash in case any bit differs, as this would indicate that the memory area was +modified after the free. The pseudo-random pattern is in fact incremented by +(~0)/3 upon each free so that roughly half of the bits change each time and we +maximize the likelihood of detecting a single bit flip in either direction. In +order to avoid an immediate reuse and maximize the time the object spends in +the cache, when this option is set, objects are picked from the cache from the +oldest one instead of the freshest one. This way even late memory corruptions +have a chance to be detected. + +When build option DEBUG_MEMORY_POOLS is set, or the boot-time option "-dMtag" +is passed on the executable's command line, pool objects are allocated with +one extra pointer compared to the requested size, so that the bytes that follow +the memory area point to the pool descriptor itself as long as the object is +allocated via pool_alloc(). Upon releasing via pool_free(), the pointer is +compared and the code will crash in if it differs. This allows to detect both +memory overflows and object released to the wrong pool (code bug resulting from +a copy-paste error typically). + +Thus an object will look like this depending whether it's in the cache or is +currently in use: + + in cache in use + +------------+ +------------+ + <--+ by_pool.p | | N bytes | + | by_pool.n +--> | | + +------------+ |N=16 min on | + <--+ by_lru.p | | 32-bit, | + | by_lru.n +--> | 32 min on | + +------------+ | 64-bit | + : : : : + | N bytes | | | + +------------+ +------------+ \ optional, only if + : (unused) : : pool ptr : > DEBUG_MEMORY_POOLS + +------------+ +------------+ / is set at build time + or -dMtag at boot time + +Right now no provisions are made to return objects aligned on larger boundaries +than those currently covered by malloc() (i.e. two pointers). This need appears +from time to time and the layout above might evolve a little bit if needed. + + +4. Storage in the process-wide shared pool +------------------------------------------ + +In order for the shared pool not to be a contention point in a multi-threaded +environment, objects are allocated from or released to shared pools by clusters +of a few objects at once. The maximum number of objects that may be moved to or +from a shared pool at once is defined by CONFIG_HAP_POOL_CLUSTER_SIZE at build +time, and currently defaults to 8. + +In order to remain scalable, the shared pool has to make some tradeoffs to +limit the number of atomic operations and the duration of any locked operation. +As such, it's composed of a single-linked list of clusters, themselves made of +a single-linked list of objects. + +Clusters and objects are of the same type "pool_item" and are accessed from the +pool's "free_list" member. This member points to the latest pool_item inserted +into the pool by a release operation. And the pool_item's "next" member points +to the next pool_item, which was the one present in the pool's free_list just +before the pool_item was inserted, and the last pool_item in the list simply +has a NULL "next" field. + +The pool_item's "down" pointer points down to the next objects part of the same +cluster, that will be released or allocated at the same time as the first one. +Each of these items also has a NULL "next" field, and are chained by their +respective "down" pointers until the last one is detected by a NULL value. + +This results in the following layout: + + pool pool_item pool_item pool_item + +-----------+ +------+ +------+ +------+ + | free_list +--> | next +--> | next +--> | NULL | + +-----------+ +------+ +------+ +------+ + | down | | NULL | | down | + +--+---+ +------+ +--+---+ + | | + V V + +------+ +------+ + | NULL | | NULL | + +------+ +------+ + | down | | NULL | + +--+---+ +------+ + | + V + +------+ + | NULL | + +------+ + | NULL | + +------+ + +Allocating an entry is only a matter of performing two atomic allocations on +the free_list and reading the pool's "next" value: + + - atomically mark the free_list as being updated by writing a "magic" pointer + - read the first pool_item's "next" field + - atomically replace the free_list with this value + +This results in a fast operation that instantly retrieves a cluster at once. +Then outside of the critical section entries are walked over and inserted into +the local cache one at a time. In order to keep the code simple and efficient, +objects allocated from the shared pool are all placed into the local cache, and +only then the first one is allocated from the cache. This operation is +performed by the dedicated function pool_refill_local_from_shared() which is +called from pool_get_from_cache() when the cache is empty. It means there is an +overhead of two list insert/delete operations for the first object and that +could be avoided at the expense of more complex code in the fast path, but this +is negligible since it only concerns objects that need to be visited anyway. + +Freeing a group of objects consists in performing the operation the other way +around: + + - atomically mark the free_list as being updated by writing a "magic" pointer + - write the free_list value to the to-be-released item's "next" entry + - atomically replace the free_list with the pool_item's pointer + +The cluster will simply have to be prepared before being sent to the shared +pool. The operation of releasing a cluster at once is performed by function +pool_put_to_shared_cache() which is called from pool_evict_last_items() which +itself is responsible for building the clusters. + +Due to the way objects are stored, it is important to try to group objects as +much as possible when releasing them because this is what will condition their +retrieval as groups as well. This is the reason why pool_evict_last_items() +uses the LRU to find a first entry but tries to pick several items at once from +a single cache. Tests have shown that CONFIG_HAP_POOL_CLUSTER_SIZE set to 8 +achieves up to 6-6.5 objects on average per operation, which effectively +divides by as much the average time spent per object by each thread and pushes +the contention point further. + +Also, grouping items in clusters is a property of the process-wide shared pool +and not of the thread-local caches. This means that there is no grouped +operation when not using the shared pool (mode "2" in the diagram above). + + +5. API +------ + +The following functions are public and available for user code: + +struct pool_head *create_pool(char *name, uint size, uint flags) + Create a new pool named <name> for objects of size <size> bytes. Pool + names are truncated to their first 11 characters. Pools of very similar + size will usually be merged if both have set the flag MEM_F_SHARED in + <flags>. When DEBUG_DONT_SHARE_POOLS was set at build time, or + "-dMno-merge" is passed on the executable's command line, the pools + also need to have the exact same name to be merged. In addition, unless + MEM_F_EXACT is set in <flags>, the object size will usually be rounded + up to the size of pointers (16 or 32 bytes). The name that will appear + in the pool upon merging is the name of the first created pool. The + returned pointer is the new (or reused) pool head, or NULL upon error. + Pools created this way must be destroyed using pool_destroy(). + +void *pool_destroy(struct pool_head *pool) + Destroy pool <pool>, that is, all of its unused objects are freed and + the structure is freed as well if the pool didn't have any used objects + anymore. In this case NULL is returned. If some objects remain in use, + the pool is preserved and its pointer is returned. This ought to be + used essentially on exit or in rare situations where some internal + entities that hold pools have to be destroyed. + +void pool_destroy_all(void) + Destroy all pools, without checking which ones still have used entries. + This is only meant for use on exit. + +void *__pool_alloc(struct pool_head *pool, uint flags) + Allocate an entry from the pool <pool>. The allocator will first look + for an object in the thread-local cache if enabled, then in the shared + pool if enabled, then will fall back to the operating system's default + allocator. NULL is returned if the object couldn't be allocated (due to + configured limits or lack of memory). Object allocated this way have to + be released using pool_free(). Like with malloc(), by default the + contents of the returned object are undefined. If memory poisonning is + enabled, the object will be filled with the poisonning byte. If the + global "pool.fail-alloc" setting is non-zero and DEBUG_FAIL_ALLOC is + enabled, a random number generator will be called to randomly return a + NULL. The allocator's behavior may be adjusted using a few flags passed + in <flags>: + - POOL_F_NO_POISON : when set, disables memory poisonning (e.g. when + pointless and expensive, like for buffers) + - POOL_F_MUST_ZERO : when set, the memory area will be zeroed before + being returned, similar to what calloc() does + - POOL_F_NO_FAIL : when set, disables the random allocation failure, + e.g. for use during early init code or critical sections. + +void *pool_alloc(struct pool_head *pool) + This is an exact equivalent of __pool_alloc(pool, 0). It is the regular + way to allocate entries from a pool. + +void *pool_alloc_nocache(struct pool_head *pool) + Allocate an entry from the pool <pool>, bypassing the cache. If shared + pools are enabled, they will be consulted first. Otherwise the object + is allocated using the operating system's default allocator. This is + essentially used during early boot to pre-allocate a number of objects + for pools which require a minimum number of entries to exist. + +void *pool_zalloc(struct pool_head *pool) + This is an exact equivalent of __pool_alloc(pool, POOL_F_MUST_ZERO). + +void pool_free(struct pool_head *pool, void *ptr) + Free an entry allocate from one of the pool_alloc() functions above + from pool <pool>. The object will be placed into the thread-local cache + if enabled, or in the shared pool if enabled, or will be released using + the operating system's default allocator. When a local cache is + enabled, if the local cache size becomes larger than 75% of the maximum + size configured at build time, some objects will be evicted to the + shared pool. Such objects are taken first from the same pool, but if + the total size is really huge, other pools might be checked as well. + Some extra checks enabled at build time may enforce extra checks so + that the process will immediately crash if the object was not allocated + from this pool or experienced an overflow or some memory corruption. + +void pool_flush(struct pool_head *pool) + Free all unused objects from shared pool <pool>. Thread-local caches + are not affected. This is essentially used when running low on memory + or when stopping, in order to release a maximum amount of memory for + the new process. + +void pool_gc(struct pool_head *pool) + Free all unused objects from all pools, but respecting the minimum + number of spare objects required for each of them. Then, for operating + systems which support it, indicate the system that all unused memory + can be released. Thread-local caches are not affected. This operation + differs from pool_flush() in that it is run locklessly, under thread + isolation, and on all pools in a row. It is called by the SIGQUIT + signal handler and upon exit. Note that the obsolete argument <pool> is + not used and the convention is to pass NULL there. + +void dump_pools_to_trash(void) + Dump the current status of all pools into the trash buffer. This is + essentially used by the "show pools" CLI command or the SIGQUIT signal + handler to dump them on stderr. The total report size may not exceed + the size of the trash buffer. If it does, some entries will be missing. + +void dump_pools(void) + Dump the current status of all pools to stderr. This just calls + dump_pools_to_trash() and writes the trash to stderr. + +int pool_total_failures(void) + Report the total number of failed allocations. This is solely used to + report the "PoolFailed" metrics of the "show info" output. The total + is calculated on the fly by summing the number of failures in all pools + and is only meant to be used as an indicator rather than a precise + measure. + +ullong pool_total_allocated(void) + Report the total number of bytes allocated in all pools, for reporting + in the "PoolAlloc_MB" field of the "show info" output. The total is + calculated on the fly by summing the number of allocated bytes in all + pools and is only meant to be used as an indicator rather than a + precise measure. + +ullong pool_total_used(void) + Report the total number of bytes used in all pools, for reporting in + the "PoolUsed_MB" field of the "show info" output. The total is + calculated on the fly by summing the number of used bytes in all pools + and is only meant to be used as an indicator rather than a precise + measure. Note that objects present in caches are accounted as used. + +Some other functions exist and are only used by the pools code itself. While +not strictly forbidden to use outside of this code, it is generally recommended +to avoid touching them in order not to create undesired dependencies that will +complicate maintenance. + +A few macros exist to ease the declaration of pools: + +DECLARE_POOL(ptr, name, size) + Placed at the top level of a file, this declares a global memory pool + as variable <ptr>, name <name> and size <size> bytes per element. This + is made via a call to REGISTER_POOL() and by assigning the resulting + pointer to variable <ptr>. <ptr> will be created of type "struct + pool_head *". If the pool needs to be visible outside of the function + (which is likely), it will also need to be declared somewhere as + "extern struct pool_head *<ptr>;". It is recommended to place such + declarations very early in the source file so that the variable is + already known to all subsequent functions which may use it. + +DECLARE_STATIC_POOL(ptr, name, size) + Placed at the top level of a file, this declares a static memory pool + as variable <ptr>, name <name> and size <size> bytes per element. This + is made via a call to REGISTER_POOL() and by assigning the resulting + pointer to local variable <ptr>. <ptr> will be created of type "static + struct pool_head *". It is recommended to place such declarations very + early in the source file so that the variable is already known to all + subsequent functions which may use it. + + +6. Build options +---------------- + +A number of build-time defines allow to tune the pools behavior. All of them +have to be enabled using "-Dxxx" or "-Dxxx=yyy" in the makefile's DEBUG +variable. + +DEBUG_NO_POOLS + When this is set, pools are entirely disabled, and allocations are made + using malloc() instead. This is not recommended for production but may + be useful for tracing allocations. It corresponds to "-dMno-cache" at + boot time. + +DEBUG_MEMORY_POOLS + When this is set, an extra pointer is allocated at the end of each + object to reference the pool the object was allocated from and detect + buffer overflows. Then, pool_free() will provoke a crash in case it + detects an anomaly (pointer at the end not matching the pool). It + corresponds to "-dMtag" at boot time. + +DEBUG_FAIL_ALLOC + When enabled, a global setting "tune.fail-alloc" may be set to a non- + zero value representing a percentage of memory allocations that will be + made to fail in order to stress the calling code. It corresponds to + "-dMfail" at boot time. + +DEBUG_DONT_SHARE_POOLS + When enabled, pools of similar sizes are not merged unless the have the + exact same name. It corresponds to "-dMno-merge" at boot time. + +DEBUG_UAF + When enabled, pools are disabled and all allocations and releases pass + through mmap() and munmap(). The memory usage significantly inflates + and the performance degrades, but this allows to detect a lot of + use-after-free conditions by crashing the program at the first abnormal + access. This should not be used in production. It corresponds to + boot-time options "-dMuaf". Caching is disabled but may be re-enabled + using "-dMcache". + +DEBUG_POOL_INTEGRITY + When enabled, objects picked from the cache are checked for corruption + by comparing their contents against a pattern that was placed when they + were inserted into the cache. Objects are also allocated in the reverse + order, from the oldest one to the most recent, so as to maximize the + ability to detect such a corruption. The goal is to detect writes after + free (or possibly hardware memory corruptions). Contrary to DEBUG_UAF + this cannot detect reads after free, but may possibly detect later + corruptions and will not consume extra memory. The CPU usage will + increase a bit due to the cost of filling/checking the area and for the + preference for cold cache instead of hot cache, though not as much as + with DEBUG_UAF. This option is meant to be usable in production. It + corresponds to boot-time options "-dMcold-first,integrity". + +DEBUG_POOL_TRACING + When enabled, the callers of pool_alloc() and pool_free() will be + recorded into an extra memory area placed after the end of the object. + This may only be required by developers who want to get a few more + hints about code paths involved in some crashes, but will serve no + purpose outside of this. It remains compatible (and completes well) + DEBUG_POOL_INTEGRITY above. Such information become meaningless once + the objects leave the thread-local cache. It corresponds to boot-time + option "-dMcaller". + +DEBUG_MEM_STATS + When enabled, all malloc/calloc/realloc/strdup/free calls are accounted + for per call place (file+line number), and may be displayed or reset on + the CLI using "debug dev memstats". This is essentially used to detect + potential leaks or abnormal usages. When pools are enabled (default), + such calls are rare and the output will mostly contain calls induced by + libraries. When pools are disabled, about all calls to pool_alloc() and + pool_free() will also appear since they will be remapped to standard + functions. + +CONFIG_HAP_GLOBAL_POOLS + When enabled, process-wide shared pools will be forcefully enabled even + if not considered useful on the platform. The default is to let haproxy + decide based on the OS and C library. It corresponds to boot-time + option "-dMglobal". + +CONFIG_HAP_NO_GLOBAL_POOLS + When enabled, process-wide shared pools will be forcefully disabled + even if considered useful on the platform. The default is to let + haproxy decide based on the OS and C library. It corresponds to + boot-time option "-dMno-global". + +CONFIG_HAP_POOL_CACHE_SIZE + This allows one to define the default size of the per-thread cache, in + bytes. The default value is 512 kB (524288). Smaller values will use + less memory at the expense of a possibly higher CPU usage when using + many threads. Higher values will give diminishing returns on + performance while using much more memory. Usually there is no benefit + in using more than a per-core L2 cache size. It would be better not to + set this value lower than a few times the size of a buffer (bufsize, + defaults to 16 kB). In addition, keep in mind that this option may be + changed at runtime using "tune.memory.hot-size". + +CONFIG_HAP_POOL_CLUSTER_SIZE + This allows one to define the maximum number of objects that will be + groupped together in an allocation from the shared pool. Values 4 to 8 + have experimentally shown good results with 16 threads. On systems with + more cores or loosely coupled caches exhibiting slow atomic operations, + it could possibly make sense to slightly increase this value. diff --git a/doc/internals/api/scheduler.txt b/doc/internals/api/scheduler.txt new file mode 100644 index 0000000..dd1ad5f --- /dev/null +++ b/doc/internals/api/scheduler.txt @@ -0,0 +1,228 @@ +2021-11-17 - Scheduler API + + +1. Background +------------- + +The scheduler relies on two major parts: + - the wait queue or timers queue, which contains an ordered tree of the next + timers to expire + + - the run queue, which contains tasks that were already woken up and are + waiting for a CPU slot to execute. + +There are two types of schedulable objects in HAProxy: + - tasks: they contain one timer and can be in the run queue without leaving + their place in the timers queue. + + - tasklets: they do not have the timers part and are either sleeping or + running. + +Both the timers queue and run queue in fact exist both shared between all +threads and per-thread. A task or tasklet may only be queued in a single of +each at a time. The thread-local queues are not thread-safe while the shared +ones are. This means that it is only permitted to manipulate an object which +is in the local queue or in a shared queue, but then after locking it. As such +tasks and tasklets are usually pinned to threads and do not move, or only in +very specific ways not detailed here. + +In case of doubt, keep in mind that it's not permitted to manipulate another +thread's private task or tasklet, and that any task held by another thread +might vanish while it's being looked at. + +Internally a large part of the task and tasklet struct is shared between +the two types, which reduces code duplication and eases the preservation +of fairness in the run queue by interleaving all of them. As such, some +fields or flags may not always be relevant to tasklets and may be ignored. + + +Tasklets do not use a thread mask but use a thread ID instead, to which they +are bound. If the thread ID is negative, the tasklet is not bound but may only +be run on the calling thread. + + +2. API +------ + +There are few functions exposed by the scheduler. A few more ones are in fact +accessible but if not documented there they'd rather be avoided or used only +when absolutely certain they're suitable, as some have delicate corner cases. +In doubt, checking the sched.pdf diagram may help. + +int total_run_queues() + Return the approximate number of tasks in run queues. This is racy + and a bit inaccurate as it iterates over all queues, but it is + sufficient for stats reporting. + +int task_in_rq(t) + Return non-zero if the designated task is in the run queue (i.e. it was + already woken up). + +int task_in_wq(t) + Return non-zero if the designated task is in the timers queue (i.e. it + has a valid timeout and will eventually expire). + +int thread_has_tasks() + Return non-zero if the current thread has some work to be done in the + run queue. This is used to decide whether or not to sleep in poll(). + +void task_wakeup(t, f) + Will make sure task <t> will wake up, that is, will execute at least + once after the start of the function is called. The task flags <f> will + be ORed on the task's state, among TASK_WOKEN_* flags exclusively. In + multi-threaded environments it is safe to wake up another thread's task + and even if the thread is sleeping it will be woken up. Users have to + keep in mind that a task running on another thread might very well + finish and go back to sleep before the function returns. It is + permitted to wake the current task up, in which case it will be + scheduled to run another time after it returns to the scheduler. + +struct task *task_unlink_wq(t) + Remove the task from the timers queue if it was in it, and return it. + It may only be done for the local thread, or for a shared thread that + might be in the shared queue. It must not be done for another thread's + task. + +void task_queue(t) + Place or update task <t> into the timers queue, where it may already + be, scheduling it for an expiration at date t->expire. If t->expire is + infinite, nothing is done, so it's safe to call this function without + prior checking the expiration date. It is only valid to call this + function for local tasks or for shared tasks who have the calling + thread in their thread mask. + +void task_set_thread(t, id) + Change task <t>'s thread ID to new value <id>. This may only be + performed by the task itself while running. This is only used to let a + task voluntarily migrate to another thread. Thread id -1 is used to + indicate "any thread". It's ignored and replaced by zero when threads + are disabled. + +void tasklet_wakeup(tl) + Make sure that tasklet <tl> will wake up, that is, will execute at + least once. The tasklet will run on its assigned thread, or on any + thread if its TID is negative. + +void tasklet_wakeup_on(tl, thr) + Make sure that tasklet <tl> will wake up on thread <thr>, that is, will + execute at least once. The designated thread may only differ from the + calling one if the tasklet is already configured to run on another + thread, and it is not permitted to self-assign a tasklet if its tid is + negative, as it may already be scheduled to run somewhere else. Just in + case, only use tasklet_wakeup() which will pick the tasklet's assigned + thread ID. + +struct tasklet *tasklet_new() + Allocate a new tasklet and set it to run by default on the calling + thread. The caller may change its tid to another one before using it. + The new tasklet is returned. + +struct task *task_new_anywhere() + Allocate a new task to run on any thread, and return the task, or NULL + in case of allocation issue. Note that such tasks will be marked as + shared and will go through the locked queues, thus their activity will + be heavier than for other ones. See also task_new_here(). + +struct task *task_new_here() + Allocate a new task to run on the calling thread, and return the task, + or NULL in case of allocation issue. + +struct task *task_new_on(t) + Allocate a new task to run on thread <t>, and return the task, or NULL + in case of allocation issue. + +void task_destroy(t) + Destroy this task. The task will be unlinked from any timers queue, + and either immediately freed, or asynchronously killed if currently + running. This may only be done by one of the threads this task is + allowed to run on. Developers must not forget that the task's memory + area is not always immediately freed, and that certain misuses could + only have effect later down the chain (e.g. use-after-free). + +void tasklet_free() + Free this tasklet, which must not be running, so that may only be + called by the thread responsible for the tasklet, typically the + tasklet's process() function itself. + +void task_schedule(t, d) + Schedule task <t> to run no later than date <d>. If the task is already + running, or scheduled for an earlier instant, nothing is done. If the + task was not in queued or was scheduled to run later, its timer entry + will be updated. This function assumes that it will never be called + with a timer in the past nor with TICK_ETERNITY. Only one of the + threads assigned to the task may call this function. + +The task's ->process() function receives the following arguments: + + - struct task *t: a pointer to the task itself. It is always valid. + + - void *ctx : a copy of the task's ->context pointer at the moment + the ->process() function was called by the scheduler. A + function must use this and not task->context, because + task->context might possibly be changed by another thread. + For instance, the muxes' takeover() function do this. + + - uint state : a copy of the task's ->state field at the moment the + ->process() function was executed. A function must use + this and not task->state as the latter misses the wakeup + reasons and may constantly change during execution along + concurrent wakeups (threads or signals). + +The possible state flags to use during a call to task_wakeup() or seen by the +task being called are the following; they're automatically cleaned from the +state field before the call to ->process() + + - TASK_WOKEN_INIT each creation of a task causes a first wakeup with this + flag set. Applications should not set it themselves. + + - TASK_WOKEN_TIMER this indicates the task's expire date was reached in the + timers queue. Applications should not set it themselves. + + - TASK_WOKEN_IO indicates the wake-up happened due to I/O activity. Now + that all low-level I/O processing happens on tasklets, + this notion of I/O is now application-defined (for + example stream-interfaces use it to notify the stream). + + - TASK_WOKEN_SIGNAL indicates that a signal the task was subscribed to was + received. Applications should not set it themselves. + + - TASK_WOKEN_MSG any application-defined wake-up reason, usually for + inter-task communication (e.g filters vs streams). + + - TASK_WOKEN_RES a resource the task was waiting for was finally made + available, allowing the task to continue its work. This + is essentially used by buffers and queues. Applications + may carefully use it for their own purpose if they're + certain not to rely on existing ones. + + - TASK_WOKEN_OTHER any other application-defined wake-up reason. + + +In addition, a few persistent flags may be observed or manipulated by the +application, both for tasks and tasklets: + + - TASK_SELF_WAKING when set, indicates that this task was found waking + itself up, and its class will change to bulk processing. + If this behavior is under control temporarily expected, + and it is not expected to happen again, it may make + sense to reset this flag from the ->process() function + itself. + + - TASK_HEAVY when set, indicates that this task does so heavy + processing that it will become mandatory to give back + control to I/Os otherwise big latencies might occur. It + may be set by an application that expects something + heavy to happen (tens to hundreds of microseconds), and + reset once finished. An example of user is the TLS stack + which sets it when an imminent crypto operation is + expected. + + - TASK_F_USR1 This is the first application-defined persistent flag. + It is always zero unless the application changes it. An + example of use cases is the I/O handler for backend + connections, to mention whether the connection is safe + to use or might have recently been migrated. + +Finally, when built with -DDEBUG_TASK, an extra sub-structure "debug" is added +to both tasks and tasklets to note the code locations of the last two calls to +task_wakeup() and tasklet_wakeup(). diff --git a/doc/internals/body-parsing.txt b/doc/internals/body-parsing.txt new file mode 100644 index 0000000..be209af --- /dev/null +++ b/doc/internals/body-parsing.txt @@ -0,0 +1,165 @@ +2014/04/16 - Pointer assignments during processing of the HTTP body + +In HAProxy, a struct http_msg is a descriptor for an HTTP message, which stores +the state of an HTTP parser at any given instant, relative to a buffer which +contains part of the message being inspected. + +Currently, an http_msg holds a few pointers and offsets to some important +locations in a message depending on the state the parser is in. Some of these +pointers and offsets may move when data are inserted into or removed from the +buffer, others won't move. + +An important point is that the state of the parser only translates what the +parser is reading, and not at all what is being done on the message (eg: +forwarding). + +For an HTTP message <msg> and a buffer <buf>, we have the following elements +to work with : + + +Buffer : +-------- + +buf.size : the allocated size of the buffer. A message cannot be larger than + this size. In general, a message will even be smaller because the + size is almost always reduced by global.maxrewrite bytes. + +buf.data : memory area containing the part of the message being worked on. This + area is exactly <buf.size> bytes long. It should be seen as a sliding + window over the message, but in terms of implementation, it's closer + to a wrapping window. For ease of processing, new messages (requests + or responses) are aligned to the beginning of the buffer so that they + never wrap and common string processing functions can be used. + +buf.p : memory pointer (char *) to the beginning of the buffer as the parser + understands it. It commonly refers to the first character of an HTTP + request or response, but during forwarding, it can point to other + locations. This pointer always points to a location in <buf.data>. + +buf.i : number of bytes after <buf.p> that are available in the buffer. If + <buf.p + buf.i> exceeds <buf.data + buf.size>, then the pending data + wrap at the end of the buffer and continue at <buf.data>. + +buf.o : number of bytes already processed before <buf.p> that are pending + for departure. These bytes may leave at any instant once a connection + is established. These ones may wrap before <buf.data> to start before + <buf.data + buf.size>. + +It's common to call the part between buf.p and buf.p+buf.i the input buffer, and +the part between buf.p-buf.o and buf.p the output buffer. This design permits +efficient forwarding without copies. As a result, forwarding one byte from the +input buffer to the output buffer only consists in : + - incrementing buf.p + - incrementing buf.o + - decrementing buf.i + + +Message : +--------- +Unless stated otherwise, all values are relative to <buf.p>, and are always +comprised between 0 and <buf.i>. These values are relative offsets and they do +not need to take wrapping into account, they are used as if the buffer was an +infinite length sliding window. The buffer management functions handle the +wrapping automatically. + +msg.next : points to the next byte to inspect. This offset is automatically + adjusted when inserting/removing some headers. In data states, it is + automatically adjusted to the number of bytes already inspected. + +msg.sov : start of value. First character of the header's value in the header + states, start of the body in the data states. Strictly positive + values indicate that headers were not forwarded yet (<buf.p> is + before the start of the body), and null or negative values are seen + after headers are forwarded (<buf.p> is at or past the start of the + body). The value stops changing when data start to leave the buffer + (in order to avoid integer overflows). So the maximum possible range + is -<buf.size> to +<buf.size>. This offset is automatically adjusted + when inserting or removing some headers. It is useful to rewind the + request buffer to the beginning of the body at any phase. The + response buffer does not really use it since it is immediately + forwarded to the client. + +msg.sol : start of line. Points to the beginning of the current header line + while parsing headers. It is cleared to zero in the BODY state, + and contains exactly the number of bytes comprising the preceding + chunk size in the DATA state (which can be zero), so that the sum of + msg.sov + msg.sol always points to the beginning of data for all + states starting with DATA. For chunked encoded messages, this sum + always corresponds to the beginning of the current chunk of data as + it appears in the buffer, or to be more precise, it corresponds to + the first of the remaining bytes of chunked data to be inspected. In + TRAILERS state, it contains the length of the last parsed part of + the trailer headers. + +msg.eoh : end of headers. Points to the CRLF (or LF) preceding the body and + marking the end of headers. It is where new headers are appended. + This offset is automatically adjusted when inserting/removing some + headers. It always contains the size of the headers excluding the + trailing CRLF even after headers have been forwarded. + +msg.eol : end of line. Points to the CRLF or LF of the current header line + being inspected during the various header states. In data states, it + holds the trailing CRLF length (1 or 2) so that msg.eoh + msg.eol + always equals the exact header length. It is not affected during data + states nor by forwarding. + +The beginning of the message headers can always be found this way even after +headers or data have been forwarded, provided that everything is still present +in the buffer : + + headers = buf.p + msg->sov - msg->eoh - msg->eol + + +Message length : +---------------- +msg.chunk_len : amount of bytes of the current chunk or total message body + remaining to be inspected after msg.next. It is automatically + incremented when parsing a chunk size, and decremented as data + are forwarded. + +msg.body_len : total message body length, for logging. Equals Content-Length + when used, otherwise is the sum of all correctly parsed chunks. + + +Message state : +--------------- +msg.msg_state contains the current parser state, one of HTTP_MSG_*. The state +indicates what byte is expected at msg->next. + +HTTP_MSG_BODY : all headers have been parsed, parsing of body has not + started yet. + +HTTP_MSG_100_SENT : parsing of body has started. If a 100-Continue was needed + it has already been sent. + +HTTP_MSG_DATA : some bytes are remaining for either the whole body when + the message size is determined by Content-Length, or for + the current chunk in chunked-encoded mode. + +HTTP_MSG_CHUNK_CRLF : msg->next points to the CRLF after the current data chunk. + +HTTP_MSG_TRAILERS : msg->next points to the beginning of a possibly empty + trailer line after the final empty chunk. + +HTTP_MSG_DONE : all the Content-Length data has been inspected, or the + final CRLF after trailers has been met. + + +Message forwarding : +-------------------- +Forwarding part of a message consists in advancing buf.p up to the point where +it points to the byte following the last one to be forwarded. This can be done +inline if enough bytes are present in the buffer, or in multiple steps if more +buffers need to be forwarded (possibly including splicing). Thus by definition, +after a block has been scheduled for being forwarded, msg->next and msg->sov +must be reset. + +The communication channel between the producer and the consumer holds a counter +of extra bytes remaining to be forwarded directly without consulting analysers, +after buf.p. This counter is called to_forward. It commonly holds the advertised +chunk length or content-length that does not fit in the buffer. For example, if +2000 bytes are to be forwarded, and 10 bytes are present after buf.p as reported +by buf.i, then both buf.o and buf.p will advance by 10, buf.i will be reset, and +to_forward will be set to 1990 so that in total, 2000 bytes will be forwarded. +At the end of the forwarding, buf.p will point to the first byte to be inspected +after the 2000 forwarded bytes. diff --git a/doc/internals/connect-status.txt b/doc/internals/connect-status.txt new file mode 100644 index 0000000..70bbcc5 --- /dev/null +++ b/doc/internals/connect-status.txt @@ -0,0 +1,28 @@ +Normally, we should use getsockopt(fd, SOL_SOCKET, SO_ERROR) on a pending +connect() to detect whether the connection correctly established or not. + +Unfortunately, getsockopt() does not report the status of a pending connection, +which means that it returns 0 if the connection is still pending. This has to +be expected, because as the name implies it, it only returns errors. + +With the speculative I/O, a new problem was introduced : if we pretend the +socket was indicated as ready and we go to the socket's write() function, +a pending connection will then inevitably be identified as established. + +In fact, there are solutions to this issue : + + - send() returns -EAGAIN if it cannot write, so that as long as there are + pending data in the buffer, we'll be informed about the status of the + connection + + - connect() on an already pending connection will return -1 with errno set to + one of the following values : + - EALREADY : connection already in progress + - EISCONN : connection already established + - anything else will indicate an error. + +=> So instead of using getsockopt() on a pending connection with no data, we + will switch to connect(). This implies that the connection address must be + known within the socket's write() function. + + diff --git a/doc/internals/connection-header.txt b/doc/internals/connection-header.txt new file mode 100644 index 0000000..b74cea0 --- /dev/null +++ b/doc/internals/connection-header.txt @@ -0,0 +1,196 @@ +2010/01/16 - Connection header adjustments depending on the transaction mode. + + +HTTP transactions supports 5 possible modes : + + WANT_TUN : default, nothing changed + WANT_TUN + httpclose : headers set for close in both dirs + WANT_KAL : keep-alive desired in both dirs + WANT_SCL : want close with the server and KA with the client + WANT_CLO : want close on both sides. + +When only WANT_TUN is set, nothing is changed nor analysed, so for commodity +below, we'll refer to WANT_TUN+httpclose as WANT_TUN. + +The mode is adjusted in 3 steps : + - configuration sets initial mode + - request headers set required request mode + - response headers set the final mode + + +1) Adjusting the initial mode via the configuration + + option httpclose => TUN + option http-keep-alive => KAL + option http-server-close => SCL + option forceclose => CLO + +Note that option httpclose combined with any other option is equivalent to +forceclose. + + +2) Adjusting the request mode once the request is parsed + +If we cannot determine the body length from the headers, we set the mode to CLO +but later we'll switch to tunnel mode once forwarding the body. That way, all +parties are informed of the correct mode. + +Depending on the request version and request Connection header, we may have to +adjust the current transaction mode and to update the connection header. + +mode req_ver req_hdr new_mode hdr_change +TUN 1.0 - TUN - +TUN 1.0 ka TUN del_ka +TUN 1.0 close TUN del_close +TUN 1.0 both TUN del_ka, del_close + +TUN 1.1 - TUN add_close +TUN 1.1 ka TUN del_ka, add_close +TUN 1.1 close TUN - +TUN 1.1 both TUN del_ka + +KAL 1.0 - CLO - +KAL 1.0 ka KAL - +KAL 1.0 close CLO del_close +KAL 1.0 both CLO del_ka, del_close + +KAL 1.1 - KAL - +KAL 1.1 ka KAL del_ka +KAL 1.1 close CLO - +KAL 1.1 both CLO del_ka + +SCL 1.0 - CLO - +SCL 1.0 ka SCL del_ka +SCL 1.0 close CLO del_close +SCL 1.0 both CLO del_ka, del_close + +SCL 1.1 - SCL add_close +SCL 1.1 ka SCL del_ka, add_close +SCL 1.1 close CLO - +SCL 1.1 both CLO del_ka + +CLO 1.0 - CLO - +CLO 1.0 ka CLO del_ka +CLO 1.0 close CLO del_close +CLO 1.0 both CLO del_ka, del_close + +CLO 1.1 - CLO add_close +CLO 1.1 ka CLO del_ka, add_close +CLO 1.1 close CLO - +CLO 1.1 both CLO del_ka + +=> Summary: + - KAL and SCL are only possible with the same requests : + - 1.0 + ka + - 1.1 + ka or nothing + + - CLO is assumed for any non-TUN request which contains at least a close + header, as well as for any 1.0 request without a keep-alive header. + + - del_ka is set whenever we want a CLO or SCL or TUN and req contains a KA, + or when the req is 1.1 and contains a KA. + + - del_close is set whenever a 1.0 request contains a close. + + - add_close is set whenever a 1.1 request must be switched to TUN, SCL, CLO + and did not have a close hdr. + +Note that the request processing is performed in two passes, one with the +frontend's config and a second one with the backend's config. It is only +possible to "raise" the mode between them, so during the second pass, we have +no reason to re-add a header that we previously removed. As an exception, the +TUN mode is converted to CLO once combined because in fact it's an httpclose +option set on a TUN mode connection : + + BE (2) + | TUN KAL SCL CLO + ----+----+----+----+---- + TUN | TUN CLO CLO CLO + + + KAL | CLO KAL SCL CLO + FE + + (1) SCL | CLO SCL SCL CLO + + + CLO | CLO CLO CLO CLO + + +3) Adjusting the final mode once the response is parsed + +This part becomes trickier. It is possible that the server responds with a +version that the client does not necessarily understand. Obviously, 1.1 clients +are asusmed to understand 1.0 responses. The problematic case is a 1.0 client +receiving a 1.1 response without any Connection header. Some 1.0 clients might +know that in 1.1 this means "keep-alive" while others might ignore the version +and assume a "close". Since we know the version on both sides, we may have to +adjust some responses to remove any ambiguous case. That's the reason why the +following table considers both the request and the response version. If the +response length cannot be determined, we switch to CLO mode. + +mode res_ver res_hdr req_ver new_mode hdr_change +TUN 1.0 - any TUN - +TUN 1.0 ka any TUN del_ka +TUN 1.0 close any TUN del_close +TUN 1.0 both any TUN del_ka, del_close + +TUN 1.1 - any TUN add_close +TUN 1.1 ka any TUN del_ka, add_close +TUN 1.1 close any TUN - +TUN 1.1 both any TUN del_ka + +KAL 1.0 - any SCL add_ka +KAL 1.0 ka any KAL - +KAL 1.0 close any SCL del_close, add_ka +KAL 1.0 both any SCL del_close + +KAL 1.1 - 1.0 KAL add_ka +KAL 1.1 - 1.1 KAL - +KAL 1.1 ka 1.0 KAL - +KAL 1.1 ka 1.1 KAL del_ka +KAL 1.1 close 1.0 SCL del_close, add_ka +KAL 1.1 close 1.1 SCL del_close +KAL 1.1 both 1.0 SCL del_close +KAL 1.1 both 1.1 SCL del_ka, del_close + +SCL 1.0 - any SCL add_ka +SCL 1.0 ka any SCL - +SCL 1.0 close any SCL del_close, add_ka +SCL 1.0 both any SCL del_close + +SCL 1.1 - 1.0 SCL add_ka +SCL 1.1 - 1.1 SCL - +SCL 1.1 ka 1.0 SCL - +SCL 1.1 ka 1.1 SCL del_ka +SCL 1.1 close 1.0 SCL del_close, add_ka +SCL 1.1 close 1.1 SCL del_close +SCL 1.1 both 1.0 SCL del_close +SCL 1.1 both 1.1 SCL del_ka, del_close + +CLO 1.0 - any CLO - +CLO 1.0 ka any CLO del_ka +CLO 1.0 close any CLO del_close +CLO 1.0 both any CLO del_ka, del_close + +CLO 1.1 - any CLO add_close +CLO 1.1 ka any CLO del_ka, add_close +CLO 1.1 close any CLO - +CLO 1.1 both any CLO del_ka + +=> in summary : + - the header operations do not depend on the initial mode, they only depend + on versions and current connection header(s). + + - both CLO and TUN modes work similarly, they need to set a close mode on the + response. A 1.1 response will exclusively need the close header, while a 1.0 + response will have it removed. Any keep-alive header is always removed when + found. + + - a KAL request where the server wants to close turns into an SCL response so + that we release the server but still maintain the connection to the client. + + - the KAL and SCL modes work the same way as we need to set keep-alive on the + response. So a 1.0 response will only have the keep-alive header with any + close header removed. A 1.1 response will have the keep-alive header added + for 1.0 requests and the close header removed for all requests. + +Note that the SCL and CLO modes will automatically cause the server connection +to be closed at the end of the data transfer. diff --git a/doc/internals/connection-scale.txt b/doc/internals/connection-scale.txt new file mode 100644 index 0000000..7c3d902 --- /dev/null +++ b/doc/internals/connection-scale.txt @@ -0,0 +1,44 @@ +Problème des connexions simultanées avec un backend + +Pour chaque serveur, 3 cas possibles : + + - pas de limite (par défaut) + - limite statique (maxconn) + - limite dynamique (maxconn/(ratio de px->conn), avec minconn) + +On a donc besoin d'une limite sur le proxy dans le cas de la limite +dynamique, afin de fixer un seuil et un ratio. Ce qui compte, c'est +le point après lequel on passe d'un régime linéaire à un régime +saturé. + +On a donc 3 phases : + + - régime minimal (0..srv->minconn) + - régime linéaire (srv->minconn..srv->maxconn) + - régime saturé (srv->maxconn..) + +Le minconn pourrait aussi ressortir du serveur ? +En pratique, on veut : + - un max par serveur + - un seuil global auquel les serveurs appliquent le max + - un seuil minimal en-dessous duquel le nb de conn est + maintenu. Cette limite a un sens par serveur (jamais moins de X conns) + mais aussi en global (pas la peine de faire du dynamique en dessous de + X conns à répartir). La difficulté en global, c'est de savoir comment + on calcule le nombre min associé à chaque serveur, vu que c'est un ratio + défini à partir du max. + +Ca revient à peu près à la même chose que de faire 2 états : + + - régime linéaire avec un offset (srv->minconn..srv->maxconn) + - régime saturé (srv->maxconn..) + +Sauf que dans ce cas, le min et le max sont bien par serveur, et le seuil est +global et correspond à la limite de connexions au-delà de laquel on veut +tourner à plein régime sur l'ensemble des serveurs. On peut donc parler de +passage en mode "full", "saturated", "optimal". On peut également parler de +la fin de la partie "scalable", "dynamique". + +=> fullconn 1000 par exemple ? + + diff --git a/doc/internals/fd-migration.txt b/doc/internals/fd-migration.txt new file mode 100644 index 0000000..aaddad3 --- /dev/null +++ b/doc/internals/fd-migration.txt @@ -0,0 +1,138 @@ +2021-07-30 - File descriptor migration between threads + +An FD migration may happen on any idle connection that experiences a takeover() +operation by another thread. In this case the acting thread becomes the owner +of the connection (and FD) while previous one(s) need to forget about it. + +File descriptor migration between threads is a fairly complex operation because +it is required to maintain a durable consistency between the pollers states and +the haproxy's desired state. Indeed, very often the FD is registered within one +thread's poller and that thread might be waiting in the system, so there is no +way to synchronously update it. This is where thread_mask, polled_mask and per +thread updates are used: + + - a thread knows if it's allowed to manipulate an FD by looking at its bit in + the FD's thread_mask ; + + - each thread knows if it was polling an FD by looking at its bit in the + polled_mask field ; a recent migration is usually indicated by a bit being + present in polled_mask and absent from thread_mask. + + - other threads know whether it's safe to take over an FD by looking at the + running mask: if it contains any other thread's bit, then other threads are + using it and it's not safe to take it over. + + - sleeping threads are notified about the need to update their polling via + local or global updates to the FD. Each thread has its own local update + list and its own bit in the update_mask to know whether there are pending + updates for it. This allows to reconverge polling with the desired state + at the last instant before polling. + +While the description above could be seen as "progressive" (it technically is) +in that there is always a transition and convergence period in a migrated FD's +life, functionally speaking it's perfectly atomic thanks to the running bit and +to the per-thread idle connections lock: no takeover is permitted without +holding the idle_conns lock, and takeover may only happen by atomically picking +a connection from the list that is also protected by this lock. In practice, an +FD is never taken over by itself, but always in the context of a connection, +and by atomically removing a connection from an idle list, it is possible to +guarantee that a connection will not be picked, hence that its FD will not be +taken over. + +same thread as list! + +The possible entry points to a race to use a file descriptor are the following +ones, with their respective sequences: + + 1) takeover: requested by conn_backend_get() on behalf of connect_server() + - take the idle_conns_lock, protecting against a parallel access from the + I/O tasklet or timeout task + - pick the first connection from the list + - attempt an fd_takeover() on this connection's fd. Usually it works, + unless a late wakeup of the owning thread shows up in the FD's running + mask. The operation is performed in fd_takeover() using a DWCAS which + tries to switch both running and thread_mask to the caller's tid_bit. A + concurrent bit in running is enough to make it fail. This guarantees + another thread does not wakeup from I/O in the middle of the takeover. + In case of conflict, this FD is skipped and the attempt is tried again + with the next connection. + - resets the task/tasklet contexts to NULL, as a signal that they are not + allowed to run anymore. The tasks retrieve their execution context from + the scheduler in the arguments, but will check the tasks' context from + the structure under the lock to detect this possible change, and abort. + - at this point the takeover succeeded, the idle_conns_lock is released and + the connection and its FD are now owned by the caller + + 2) poll report: happens on late rx, shutdown or error on idle conns + - fd_set_running() is called to atomically set the running_mask and check + that the caller's tid_bit is still present in the thread_mask. Upon + failure the caller arranges itself to stop reporting that FD (e.g. by + immediate removal or by an asynchronous update). Upon success, it's + guaranteed that any concurrent fd_takeover() will fail the DWCAS and that + another connection will need to be picked instead. + - FD's state is possibly updated + - the iocb is called if needed (almost always) + - if the iocb didn't kill the connection, release the bit from running_mask + making the connection possibly available to a subsequent fd_takeover(). + + 3) I/O tasklet, timeout task: timeout or subscribed wakeup + - start by taking the idle_conns_lock, ensuring no takeover() will pick the + same connection from this point. + - check the task/tasklet's context to verify that no recently completed + takeover() stole the connection. If it's NULL, the connection was lost, + the lock is released and the task/tasklet killed. Otherwise it is + guaranteed that no other thread may use that connection (current takeover + candidates are waiting on the lock, previous owners waking from poll() + lost their bit in the thread_mask and will not touch the FD). + - the connection is removed from the idle conns list. From this point on, + no other thread will even find it there nor even try fd_takeover() on it. + - the idle_conns_lock is now released, the connection is protected and its + FD is not reachable by other threads anymore. + - the task does what it has to do + - if the connection is still usable (i.e. not upon timeout), it's inserted + again into the idle conns list, meaning it may instantly be taken over + by a competing thread. + + 4) wake() callback: happens on last user after xfers (may free() the conn) + - the connection is still owned by the caller, it's still subscribed to + polling but the connection is idle thus inactive. Errors or shutdowns + may be reported late, via sock_conn_iocb() and conn_notify_mux(), thus + the running bit is set (i.e. a concurrent fd_takeover() will fail). + - if the connection is in the list, the idle_conns_lock is grabbed, the + connection is removed from the list, and the lock is released. + - mux->wake() is called + - if the connection previously was in the list, it's reinserted under the + idle_conns_lock. + + +With the DWCAS removal between running_mask & thread_mask: + +fd_takeover: + 1 if (!CAS(&running_mask, 0, tid_bit)) + 2 return fail; + 3 atomic_store(&thread_mask, tid_bit); + 4 atomic_and(&running_mask, ~tid_bit); + +poller: + 1 do { + 2 /* read consistent running_mask & thread_mask */ + 3 do { + 4 run = atomic_load(&running_mask); + 5 thr = atomic_load(&thread_mask); + 6 } while (run & ~thr); + 7 + 8 if (!(thr & tid_bit)) { + 9 /* takeover has started */ + 10 goto disable_fd; + 11 } + 12 } while (!CAS(&running_mask, run, run | tid_bit)); + +fd_delete: + 1 atomic_or(&running_mask, tid_bit); + 2 atomic_store(&thread_mask, 0); + 3 atomic_and(&running_mask, ~tid_bit); + +The loop in poller:3-6 is used to make sure the thread_mask we read matches +the last updated running_mask. If nobody can give up on fd_takeover(), it +might even be possible to spin on thread_mask only. Late pollers will not +set running anymore with this. diff --git a/doc/internals/hashing.txt b/doc/internals/hashing.txt new file mode 100644 index 0000000..260b6af --- /dev/null +++ b/doc/internals/hashing.txt @@ -0,0 +1,83 @@ +2013/11/20 - How hashing works internally in haproxy - maddalab@gmail.com + +This document describes how HAProxy implements hashing both map-based and +consistent hashing, both prior to versions 1.5 and the motivation and tests +that were done when providing additional options starting in version 2.4 + +A note on hashing in general, hash functions strive to have little +correlation between input and output. The heart of a hash function is its +mixing step. The behavior of the mixing step largely determines whether the +hash function is collision-resistant. Hash functions that are collision +resistant are more likely to have an even distribution of load. + +The purpose of the mixing function is to spread the effect of each message +bit throughout all the bits of the internal state. Ideally every bit in the +hash state is affected by every bit in the message. And we want to do that +as quickly as possible simply for the sake of program performance. A +function is said to satisfy the strict avalanche criterion if, whenever a +single input bit is complemented (toggled between 0 and 1), each of the +output bits should change with a probability of one half for an arbitrary +selection of the remaining input bits. + +To guard against a combination of hash function and input that results in +high rate of collisions, haproxy implements an avalanche algorithm on the +result of the hashing function. In all versions 1.4 and prior avalanche is +always applied when using the consistent hashing directive. It is intended +to provide quite a good distribution for little input variations. The result +is quite suited to fit over a 32-bit space with enough variations so that +a randomly picked number falls equally before any server position, which is +ideal for consistently hashed backends, a common use case for caches. + +In all versions 1.4 and prior HAProxy implements the SDBM hashing function. +However tests show that alternatives to SDBM have a better cache +distribution on different hashing criteria. Additional tests involving +alternatives for hash input and an option to trigger avalanche, we found +different algorithms perform better on different criteria. DJB2 performs +well when hashing ascii text and is a good choice when hashing on host +header. Other alternatives perform better on numbers and are a good choice +when using source ip. The results also vary by use of the avalanche flag. + +The results of the testing can be found under the tests folder. Here is +a summary of the discussion on the results on 1 input criteria and the +methodology used to generate the results. + +A note of the setup when validating the results independently, one +would want to avoid backend server counts that may skew the results. As +an example with DJB2 avoid 33 servers. Please see the implementations of +the hashing function, which can be found in the links under references. + +The following was the set up used + +(a) hash-type consistent/map-based +(b) avalanche on/off +(c) balanche host(hdr) +(d) 3 criteria for inputs + - ~ 10K requests, including duplicates + - ~ 46K requests, unique requests from 1 MM requests were obtained + - ~ 250K requests, including duplicates +(e) 17 servers in backend, all servers were assigned the same weight + +Result of the hashing were obtained across the server via monitoring log +files for haproxy. Population Standard deviation was used to evaluate the +efficacy of the hashing algorithm. Lower standard deviation, indicates +a better distribution of load across the backends. + +On 10K requests, when using consistent hashing with avalanche on host +headers, DJB2 significantly out performs SDBM. Std dev on SDBM was 48.95 +and DJB2 was 26.29. This relationship is inverted with avalanche disabled, +however DJB2 with avalanche enabled out performs SDBM with avalanche +disabled. + +On map-based hashing SDBM out performs DJB2 irrespective of the avalanche +option. SDBM without avalanche is marginally better than with avalanche. +DJB2 performs significantly worse with avalanche enabled. + +Summary: The results of the testing indicate that there isn't a hashing +algorithm that can be applied across all input criteria. It is necessary +to support alternatives to SDBM, which is generally the best option, with +algorithms that are better for different inputs. Avalanche is not always +applicable and may result in less smooth distribution. + +References: +Mixing Functions/Avalanche: https://papa.bretmulvey.com/post/124027987928/hash-functions +Hash Functions: http://www.cse.yorku.ca/~oz/hash.html diff --git a/doc/internals/list.fig b/doc/internals/list.fig new file mode 100644 index 0000000..aeb1f1d --- /dev/null +++ b/doc/internals/list.fig @@ -0,0 +1,599 @@ +#FIG 3.2 Produced by xfig version 2.4 +Landscape +Center +Metric +A4 +119.50 +Single +-2 +1200 2 +6 3960 3420 4320 4230 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 4230 3860 4005 3860 +2 2 0 2 0 2 53 0 20 0.000 0 0 -1 0 0 5 + 4005 3510 4230 3510 4230 4185 4005 4185 4005 3510 +4 1 0 50 0 14 10 0.0000 4 105 105 4120 4062 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 4118 3735 N\001 +-6 +6 4185 5580 4545 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 4455 6020 4230 6020 +2 2 0 2 0 4 53 0 20 0.000 0 0 -1 0 0 5 + 4230 5670 4455 5670 4455 6345 4230 6345 4230 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 4345 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 4343 5895 N\001 +-6 +6 4905 5445 5445 6525 +6 4905 5445 5445 6525 +6 5085 5580 5445 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 5355 6020 5130 6020 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 5130 5670 5355 5670 5355 6345 5130 6345 5130 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 5245 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 5243 5895 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 5355 5670 4905 5670 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 4905 6345 5355 6345 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 4905 5445 5355 5445 5355 6525 4905 6525 4905 5445 +4 1 0 50 0 14 12 0.0000 4 120 120 5040 6075 L\001 +-6 +-6 +6 5805 5445 6345 6525 +6 5805 5445 6345 6525 +6 5985 5580 6345 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 6255 6020 6030 6020 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 6030 5670 6255 5670 6255 6345 6030 6345 6030 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 6145 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 6143 5895 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 6255 5670 5805 5670 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 5805 6345 6255 6345 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 5805 5445 6255 5445 6255 6525 5805 6525 5805 5445 +4 1 0 50 0 14 12 0.0000 4 120 120 5940 6075 L\001 +-6 +-6 +6 6705 5445 7245 6525 +6 6705 5445 7245 6525 +6 6885 5580 7245 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 7155 6020 6930 6020 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 6930 5670 7155 5670 7155 6345 6930 6345 6930 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 7045 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 7043 5895 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 7155 5670 6705 5670 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 6705 6345 7155 6345 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 6705 5445 7155 5445 7155 6525 6705 6525 6705 5445 +4 1 0 50 0 14 12 0.0000 4 120 120 6840 6075 L\001 +-6 +-6 +6 450 5580 810 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 720 6020 495 6020 +2 2 0 2 0 4 53 0 20 0.000 0 0 -1 0 0 5 + 495 5670 720 5670 720 6345 495 6345 495 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 610 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 608 5895 N\001 +-6 +6 1170 5445 1710 6525 +6 1170 5445 1710 6525 +6 1350 5580 1710 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 1620 6020 1395 6020 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 1395 5670 1620 5670 1620 6345 1395 6345 1395 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 1510 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 1508 5895 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 1620 5670 1170 5670 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 1170 6345 1620 6345 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 1170 5445 1620 5445 1620 6525 1170 6525 1170 5445 +4 1 0 50 0 14 12 0.0000 4 120 120 1305 6075 L\001 +-6 +-6 +6 2070 5445 2610 6525 +6 2070 5445 2610 6525 +6 2250 5580 2610 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 2520 6020 2295 6020 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 2295 5670 2520 5670 2520 6345 2295 6345 2295 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 2410 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 2408 5895 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 2520 5670 2070 5670 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 2070 6345 2520 6345 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 2070 5445 2520 5445 2520 6525 2070 6525 2070 5445 +4 1 0 50 0 14 12 0.0000 4 120 120 2205 6075 L\001 +-6 +-6 +6 2970 5445 3510 6525 +6 2970 5445 3510 6525 +6 3150 5580 3510 6390 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 3420 6020 3195 6020 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 3195 5670 3420 5670 3420 6345 3195 6345 3195 5670 +4 1 0 50 0 14 10 0.0000 4 105 105 3310 6222 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 3308 5895 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 3420 5670 2970 5670 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 2970 6345 3420 6345 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 2970 5445 3420 5445 3420 6525 2970 6525 2970 5445 +4 1 0 50 0 14 12 0.0000 4 120 120 3105 6075 L\001 +-6 +-6 +6 720 3420 1080 4230 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 990 3860 765 3860 +2 2 0 2 0 2 53 0 20 0.000 0 0 -1 0 0 5 + 765 3510 990 3510 990 4185 765 4185 765 3510 +4 1 0 50 0 14 10 0.0000 4 105 105 880 4062 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 878 3735 N\001 +-6 +6 2700 3420 3060 4230 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 2970 3860 2745 3860 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 2745 3510 2970 3510 2970 4185 2745 4185 2745 3510 +4 1 0 50 0 14 10 0.0000 4 105 105 2860 4062 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 2858 3735 N\001 +-6 +6 1620 3465 1935 4230 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 1890 3860 1665 3860 +2 2 0 2 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 1665 3510 1890 3510 1890 4185 1665 4185 1665 3510 +4 1 0 50 0 14 10 0.0000 4 105 105 1780 4062 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 1778 3735 N\001 +-6 +6 10485 3330 11025 4410 +6 10665 3465 11025 4275 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 10935 3905 10710 3905 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 10710 3555 10935 3555 10935 4230 10710 4230 10710 3555 +4 1 0 50 0 14 10 0.0000 4 105 105 10825 4107 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 10823 3780 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 10935 3555 10485 3555 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 10485 4230 10935 4230 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 10485 3330 10935 3330 10935 4410 10485 4410 10485 3330 +4 1 0 50 0 14 12 0.0000 4 120 120 10620 3960 L\001 +-6 +6 7110 3105 7650 4185 +6 7110 3105 7650 4185 +6 7290 3240 7650 4050 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 7560 3680 7335 3680 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 7335 3330 7560 3330 7560 4005 7335 4005 7335 3330 +4 1 0 50 0 14 10 0.0000 4 105 105 7450 3882 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 7448 3555 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 7560 3330 7110 3330 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 7110 4005 7560 4005 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 7110 3105 7560 3105 7560 4185 7110 4185 7110 3105 +4 1 0 50 0 14 12 0.0000 4 120 120 7245 3735 L\001 +-6 +-6 +6 8010 3105 8550 4185 +6 8010 3105 8550 4185 +6 8190 3240 8550 4050 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 8460 3680 8235 3680 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 8235 3330 8460 3330 8460 4005 8235 4005 8235 3330 +4 1 0 50 0 14 10 0.0000 4 105 105 8350 3882 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 8348 3555 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 8460 3330 8010 3330 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 8010 4005 8460 4005 +2 2 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 8010 3105 8460 3105 8460 4185 8010 4185 8010 3105 +4 1 0 50 0 14 12 0.0000 4 120 120 8145 3735 L\001 +-6 +-6 +6 9315 990 12195 2160 +6 9675 1080 10035 1890 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 9945 1520 9720 1520 +2 2 0 2 0 2 53 0 20 0.000 0 0 -1 0 0 5 + 9720 1170 9945 1170 9945 1845 9720 1845 9720 1170 +4 1 0 50 0 14 10 0.0000 4 105 105 9835 1722 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 9833 1395 N\001 +-6 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 10935 1520 10710 1520 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 11925 1520 11700 1520 +2 2 0 2 0 7 52 0 20 0.000 0 0 -1 0 0 5 + 10710 1170 10935 1170 10935 1845 10710 1845 10710 1170 +2 2 0 2 0 6 52 0 20 0.000 0 0 -1 0 0 5 + 11700 1170 11925 1170 11925 1845 11700 1845 11700 1170 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 9945 1350 10665 1350 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 10935 1350 11655 1350 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8 + 1 1 1.00 60.00 120.00 + 11925 1350 12105 1350 12195 1350 12195 990 9315 990 9315 1350 + 9495 1350 9675 1350 + 0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8 + 1 1 1.00 60.00 120.00 + 9675 1710 9495 1710 9315 1710 9405 2160 12195 2160 12195 1710 + 12105 1710 11925 1710 + 0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 11655 1710 10935 1710 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 10665 1710 9945 1710 + 0.000 0.000 +4 1 0 50 0 14 10 0.0000 4 105 105 10825 1722 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 10823 1395 N\001 +4 1 0 50 0 14 10 0.0000 4 105 105 11815 1722 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 11813 1395 N\001 +-6 +6 6345 1080 6705 1890 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 6615 1520 6390 1520 +2 2 0 2 0 2 53 0 20 0.000 0 0 -1 0 0 5 + 6390 1170 6615 1170 6615 1845 6390 1845 6390 1170 +4 1 0 50 0 14 10 0.0000 4 105 105 6505 1722 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 6503 1395 N\001 +-6 +6 7335 1080 7695 1890 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 7605 1520 7380 1520 +2 2 0 2 0 6 52 0 20 0.000 0 0 -1 0 0 5 + 7380 1170 7605 1170 7605 1845 7380 1845 7380 1170 +4 1 0 50 0 14 10 0.0000 4 105 105 7495 1722 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 7493 1395 N\001 +-6 +6 8325 1080 8685 1890 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 8595 1520 8370 1520 +2 2 0 2 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 8370 1170 8595 1170 8595 1845 8370 1845 8370 1170 +4 1 0 50 0 14 10 0.0000 4 105 105 8485 1722 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 8483 1395 N\001 +-6 +6 3870 1215 4185 1980 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 4140 1610 3915 1610 +2 2 0 2 0 2 53 0 20 0.000 0 0 -1 0 0 5 + 3915 1260 4140 1260 4140 1935 3915 1935 3915 1260 +4 1 0 50 0 14 10 0.0000 4 105 105 4030 1812 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 4028 1485 N\001 +-6 +6 4770 1215 5085 1980 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 5040 1610 4815 1610 +2 2 0 2 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 4815 1260 5040 1260 5040 1935 4815 1935 4815 1260 +4 1 0 50 0 14 10 0.0000 4 105 105 4930 1812 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 4928 1485 N\001 +-6 +6 2205 990 2925 2160 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 2655 1610 2430 1610 +2 2 0 2 0 2 53 0 20 0.000 0 0 -1 0 0 5 + 2430 1260 2655 1260 2655 1935 2430 1935 2430 1260 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 6 + 1 1 1.00 60.00 120.00 + 2655 1440 2880 1440 2880 1035 2205 1035 2205 1440 2430 1440 + 0.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 6 + 1 1 1.00 60.00 120.00 + 2655 1755 2880 1755 2880 2160 2205 2160 2205 1755 2430 1755 + 0.000 1.000 1.000 1.000 1.000 0.000 +4 1 0 50 0 14 10 0.0000 4 105 105 2545 1812 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 2543 1485 N\001 +-6 +6 525 1350 1455 1830 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 540 1590 1440 1590 +2 2 0 2 0 7 50 0 -1 0.000 0 0 -1 0 0 5 + 540 1365 1440 1365 1440 1815 540 1815 540 1365 +4 1 0 50 0 14 10 0.0000 4 105 735 990 1545 list *N\001 +4 1 0 50 0 14 10 0.0000 4 105 735 990 1770 list *P\001 +-6 +6 4815 3420 5175 4230 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 5085 3860 4860 3860 +2 2 0 2 0 7 53 0 20 0.000 0 0 -1 0 0 5 + 4860 3510 5085 3510 5085 4185 4860 4185 4860 3510 +4 1 0 50 0 14 10 0.0000 4 105 105 4975 4062 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 4973 3735 N\001 +-6 +6 5715 3285 6390 4410 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 6165 3860 5940 3860 +2 2 0 2 0 6 53 0 20 0.000 0 0 -1 0 0 5 + 5940 3510 6165 3510 6165 4185 5940 4185 5940 3510 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 6 + 1 1 1.00 60.00 120.00 + 6165 3690 6390 3690 6390 3285 5715 3285 5715 3690 5940 3690 + 0.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 6 + 1 1 1.00 60.00 120.00 + 6165 4005 6390 4005 6390 4410 5715 4410 5715 4005 5940 4005 + 0.000 1.000 1.000 1.000 1.000 0.000 +4 1 0 50 0 14 10 0.0000 4 105 105 6055 4062 P\001 +4 1 0 50 0 14 10 0.0000 4 105 105 6053 3735 N\001 +-6 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 4050 4725 7605 4725 7605 6840 4050 6840 4050 4725 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 315 4725 3870 4725 3870 6840 315 6840 315 4725 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 3150 4500 315 4500 315 2475 3150 2475 3150 4500 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 6660 2475 8910 2475 8910 4500 6660 4500 6660 2475 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 10035 3375 10485 3330 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 10080 3735 10485 3555 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 9135 2475 12285 2475 12285 4500 9135 4500 9135 2475 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 9270 270 12285 270 12285 2250 9270 2250 9270 270 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 5760 270 9045 270 9045 2250 5760 2250 5760 270 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 3465 270 5535 270 5535 2250 3465 2250 3465 270 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 1845 270 3240 270 3240 2250 1845 2250 1845 270 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 315 270 1620 270 1620 2250 315 2250 315 270 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 3330 2475 6435 2475 6435 4500 3330 4500 3330 2475 +2 4 0 1 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 12285 6840 12285 4725 7785 4725 7785 6840 12285 6840 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 4230 3690 4860 3690 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 4860 4050 4230 4050 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 3960 4050 3780 4050 3600 4050 3600 4410 5580 4410 5580 4050 + 5130 4050 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 6261 5805 6711 5670 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 4461 5805 4911 5670 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 5358 5805 5808 5670 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 6705 6210 6255 6210 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 5805 6210 5355 6210 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 4905 6210 4455 6210 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 4320 6345 4320 6525 4320 6750 7470 6750 7470 6480 7470 6210 + 7155 6210 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8 + 1 1 1.00 60.00 120.00 + 7155 5850 7335 5850 7470 5850 7470 5355 7470 5085 4590 5085 + 4590 5355 4860 5625 + 0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 2526 5805 2976 5670 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 726 5805 1176 5670 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 1623 5805 2073 5670 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 2970 6210 2520 6210 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 2070 6210 1620 6210 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 1170 6210 720 6210 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 585 6345 585 6525 585 6750 3735 6750 3735 6480 3735 6210 + 3420 6210 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8 + 1 1 1.00 60.00 120.00 + 3420 5850 3600 5850 3735 5850 3735 5355 3735 5085 585 5085 + 585 5265 585 5670 + 0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 990 3690 1620 3690 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 1620 4050 990 4050 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 1890 3690 2340 3690 2340 3240 360 3240 360 3690 540 3690 + 720 3690 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 720 4050 540 4050 360 4050 360 4410 2340 4410 2340 4050 + 1890 4050 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 7560 3465 8010 3330 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 7560 3915 8010 3375 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 8460 3465 8775 3465 8820 3060 8730 2745 6750 2745 6705 3330 + 7110 3330 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8 + 1 1 1.00 60.00 120.00 + 8460 3870 8820 3870 8820 4230 8640 4365 6930 4365 6750 4230 + 6705 3510 7065 3375 + 0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 6615 1350 7335 1350 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 7605 1350 8325 1350 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8 + 1 1 1.00 60.00 120.00 + 8595 1350 8775 1350 8865 1350 8865 990 5985 990 5985 1350 + 6165 1350 6345 1350 + 0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8 + 1 1 1.00 60.00 120.00 + 6345 1710 6165 1710 5985 1710 6075 2160 8865 2160 8865 1710 + 8775 1710 8595 1710 + 0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 8325 1710 7605 1710 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 7335 1710 6615 1710 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 4140 1440 4770 1440 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 4770 1800 4140 1800 + 0.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 5040 1440 5490 1440 5490 990 3510 990 3510 1440 3690 1440 + 3870 1440 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 3870 1800 3690 1800 3510 1800 3510 2160 5490 2160 5490 1800 + 5040 1800 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 0 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 5130 3690 5580 3690 5580 3240 3600 3240 3600 3690 3780 3690 + 3960 3690 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +4 1 0 50 0 14 10 0.0000 4 135 3780 5805 4950 Asymmetrical list starting at R(red)\001 +4 1 0 50 0 14 10 0.0000 4 135 4095 10215 4950 Symmetrical lists vs Asymmetrical lists\001 +4 1 0 50 0 12 10 0.0000 4 135 525 5130 5355 foo_0\001 +4 1 0 50 0 12 10 0.0000 4 135 525 6030 5355 foo_1\001 +4 1 0 50 0 12 10 0.0000 4 135 525 6930 5355 foo_2\001 +4 1 0 50 0 14 10 0.0000 4 135 3675 2070 4950 Symmetrical list starting at R(red)\001 +4 1 0 50 0 12 10 0.0000 4 135 525 3195 5355 foo_2\001 +4 1 0 50 0 12 10 0.0000 4 135 525 2295 5355 foo_1\001 +4 1 0 50 0 12 10 0.0000 4 135 525 1395 5355 foo_0\001 +4 1 0 50 0 12 10 0.0000 4 105 315 9855 3420 foo\001 +4 1 0 50 0 12 10 0.0000 4 105 105 9990 3825 E\001 +4 1 0 50 0 14 10 0.0000 4 135 1680 7785 2655 Linking elements\001 +4 1 0 50 0 12 10 0.0000 4 135 525 8235 3015 foo_1\001 +4 1 0 50 0 12 10 0.0000 4 135 525 7335 3015 foo_0\001 +4 1 0 50 0 14 10 0.0000 4 105 1470 2565 675 struct list *G\001 +4 1 0 50 0 14 10 0.0000 4 135 1470 2565 495 LIST_INIT(G):G\001 +4 1 0 50 0 14 10 0.0000 4 135 1890 4500 495 LIST_INSERT(G,W):W\001 +4 1 0 50 0 14 10 0.0000 4 135 3360 10665 2700 foo=LIST_ELEM(E, struct foo*, L)\001 +4 1 0 50 0 14 10 0.0000 4 135 1890 4500 675 LIST_APPEND(G,W):W\001 +4 1 0 50 0 14 10 0.0000 4 135 1890 7425 540 LIST_INSERT(G,Y):Y\001 +4 1 0 50 0 14 10 0.0000 4 135 1890 10755 540 LIST_APPEND(G,Y):Y\001 +4 1 0 50 0 14 10 0.0000 4 135 1680 1755 2790 LIST_DELETE(Y):Y\001 +4 1 0 50 0 14 10 0.0000 4 135 1890 4905 2745 LIST_DEL_INIT(Y):Y\001 +4 1 0 50 0 12 9 0.0000 4 120 2880 10665 2925 Returns a pointer to struct foo*\001 +4 1 0 50 0 12 9 0.0000 4 120 2790 10665 3105 containing header E as member L\001 +4 1 0 50 0 12 9 0.0000 4 120 2700 10755 810 adds Y at the queue (before G)\001 +4 1 0 50 0 12 9 0.0000 4 120 3060 7425 810 adds Y(yellow) just after G(green)\001 +4 1 0 50 0 12 9 0.0000 4 120 1170 4500 855 adds W(white)\001 +4 1 0 50 0 12 9 0.0000 4 105 1080 2565 900 Terminates G\001 +4 1 0 50 0 12 9 0.0000 4 90 540 990 855 N=next\001 +4 1 0 50 0 12 9 0.0000 4 105 540 990 1080 P=prev\001 +4 1 0 50 0 12 9 0.0000 4 120 2610 1755 3060 unlinks and returns Y(yellow)\001 +4 1 0 50 0 12 9 0.0000 4 120 2610 4905 3060 unlinks, inits, and returns Y\001 +4 0 0 50 0 12 8 0.0000 4 105 2175 7875 5265 - both are empty if R->P == R\001 +4 0 0 50 0 12 8 0.0000 4 90 2175 7875 5490 - last element has R->P == &L\001 +4 0 0 50 0 12 8 0.0000 4 105 3150 7875 5715 - FOREACH_ITEM(it, R, end, struct foo*, L)\001 +4 0 0 50 0 12 8 0.0000 4 105 3300 7875 5940 iterates <it> through foo{0,1,2} and stops\001 +4 0 0 50 0 12 8 0.0000 4 105 3900 7875 6165 - FOREACH_ITEM_SAFE(it, bck, R, end, struct foo*, L)\001 +4 0 0 50 0 12 8 0.0000 4 105 3750 7875 6390 does the same except that <bck> allows to delete\001 +4 0 0 50 0 12 8 0.0000 4 105 1950 7875 6570 any node, including <it>\001 +4 1 0 50 0 14 11 0.0000 4 135 1155 945 585 struct list\001 diff --git a/doc/internals/list.png b/doc/internals/list.png Binary files differnew file mode 100644 index 0000000..ec41a6b --- /dev/null +++ b/doc/internals/list.png diff --git a/doc/internals/listener-states.fig b/doc/internals/listener-states.fig new file mode 100644 index 0000000..863e7f5 --- /dev/null +++ b/doc/internals/listener-states.fig @@ -0,0 +1,150 @@ +#FIG 3.2 Produced by xfig version 2.3 +Portrait +Center +Metric +A4 +300.00 +Single +-2 +1200 2 +0 32 #ff60e0 +0 33 #ff8020 +0 34 #56c5ff +0 35 #55d941 +0 36 #f8e010 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 900 450 495 225 900 450 1395 450 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 2700 450 495 225 2700 450 3195 450 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 4500 450 495 225 4500 450 4995 450 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 900 3465 495 225 900 3465 1395 3465 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 2700 2475 495 225 2700 2475 3195 2475 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 3645 1575 495 225 3645 1575 4140 1575 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 4500 2475 495 225 4500 2475 4995 2475 +1 1 0 3 0 7 51 -1 20 0.000 1 0.0000 2700 3471 495 225 2700 3471 3195 3471 +2 1 1 3 1 7 52 -1 -1 8.000 1 0 -1 0 0 2 + 270 1980 5355 1350 +2 2 0 2 32 32 52 -1 20 0.000 1 0 -1 0 0 5 + 2070 3060 3330 3060 3330 3870 2070 3870 2070 3060 +2 3 0 1 33 33 53 -1 20 0.000 1 0 -1 0 0 5 + 2070 990 5130 990 5130 2880 2070 2880 2070 990 +2 2 0 2 35 35 52 -1 20 0.000 1 0 -1 0 0 5 + 270 90 5130 90 5130 855 270 855 270 90 +2 2 0 2 36 36 52 -1 20 0.000 1 0 -1 0 0 5 + 270 3060 1530 3060 1530 3870 270 3870 270 3060 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 1395 450 2250 450 + 0.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 3195 450 4050 450 + 0.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 4095 1665 4455 2025 4500 2250 + 0.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 3195 3510 3600 3465 4140 2655 + 0.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 4410 2250 4365 2070 4050 1710 + 0.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 945 3240 936 2142 2961 1917 3240 1710 + 0.000 1.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 3195 1665 2835 1845 855 2115 855 3240 + 0.000 1.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 5 + 1 1 1.00 60.00 120.00 + 990 3690 1035 3960 2880 4050 4365 3915 4410 2700 + 0.000 1.000 1.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 2700 2700 2700 3240 + 0.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 4095 2610 3600 3375 3150 3420 + 0.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 5 + 1 1 1.00 60.00 120.00 + 4500 2700 4455 4005 2655 4140 945 4005 900 3690 + 0.000 1.000 1.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 2205 2520 1395 2745 1260 2970 1125 3240 + 0.000 1.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 3510 1800 3330 2025 3330 2835 2970 3285 + 0.000 1.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 1170 3285 1305 3015 1485 2790 2250 2610 + 0.000 1.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 2205 3420 1710 3420 1395 3465 + 0.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 1395 3510 1800 3510 2205 3465 + 0.000 1.000 0.000 +3 0 0 3 0 7 51 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 2925 2295 3060 1980 3330 1755 + 0.000 1.000 0.000 +3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 4500 675 4455 990 3960 1395 + 0.000 1.000 0.000 +4 1 0 50 -1 18 10 0.0000 4 120 375 900 450 NEW\001 +4 1 0 50 -1 18 10 0.0000 4 120 315 2700 450 INIT\001 +4 1 0 50 -1 18 10 0.0000 4 120 810 4500 450 ASSIGNED\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 900 630 0\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 2700 630 1\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 4500 630 2\001 +4 1 0 50 -1 16 7 0.0000 4 120 420 1755 405 create()\001 +4 2 0 50 -1 16 7 0.0000 4 120 660 1215 2160 enable() &&\001 +4 2 0 50 -1 16 7 0.0000 4 90 540 1080 2295 !maxconn\001 +4 2 1 51 -1 16 7 1.5708 4 105 600 5355 1485 transitions\001 +4 0 1 51 -1 16 7 1.5708 4 105 600 5355 1260 transitions\001 +4 2 1 51 -1 16 7 1.5708 4 105 795 5265 1485 multi-threaded\001 +4 0 1 51 -1 16 7 1.5708 4 120 870 5265 1260 single-threaded\001 +4 0 0 52 -1 17 7 0.0000 4 90 345 315 765 no FD\001 +4 0 0 52 -1 17 7 0.0000 4 135 315 315 3825 polled\001 +4 1 0 50 -1 18 10 0.0000 4 120 555 900 3465 READY\001 +4 0 0 50 -1 16 7 0.0000 4 120 255 1170 3825 full()\001 +4 2 0 50 -1 16 7 0.0000 4 90 540 2205 3375 !maxconn\001 +4 2 0 50 -1 16 7 0.0000 4 105 675 2295 3240 resume() &&\001 +4 0 0 50 -1 16 7 0.0000 4 105 405 1395 3645 pause()\001 +4 0 0 52 -1 17 7 0.0000 4 135 585 2115 3825 shut(sock)\001 +4 2 0 50 -1 16 7 0.0000 4 120 480 4320 2205 disable()\001 +4 2 0 50 -1 16 7 0.0000 4 105 405 4005 2655 pause()\001 +4 0 0 50 -1 16 7 0.0000 4 105 465 4545 2835 resume()\001 +4 2 0 50 -1 16 7 0.0000 4 120 480 2925 2160 disable()\001 +4 0 0 50 -1 16 7 0.0000 4 105 405 3465 1980 pause()\001 +4 0 0 50 -1 16 7 0.0000 4 120 660 4230 1710 enable() &&\001 +4 0 0 50 -1 16 7 0.0000 4 75 510 4320 1845 maxconn\001 +4 2 0 50 -1 16 7 0.0000 4 105 405 2655 2835 pause()\001 +4 0 0 50 -1 16 7 0.0000 4 105 675 3375 3555 resume() &&\001 +4 0 0 50 -1 16 7 0.0000 4 75 510 3375 3645 maxconn\001 +4 0 0 50 -1 16 7 0.0000 4 120 480 1080 2655 disable()\001 +4 2 0 50 -1 16 7 0.0000 4 105 465 2160 2475 resume()\001 +4 1 0 50 -1 16 7 0.0000 4 120 330 3555 405 .add()\001 +4 0 0 50 -1 16 7 0.0000 4 120 375 4545 810 .bind()\001 +4 0 0 52 -1 17 7 0.0000 4 135 1080 2115 1125 FD ready, not polled\001 +4 0 0 50 -1 16 7 0.0000 4 120 315 1305 3240 limit()\001 +4 1 0 50 -1 18 10 0.0000 4 120 630 2700 2475 LIMITED\001 +4 1 0 50 -1 18 10 0.0000 4 120 555 3645 1575 LISTEN\001 +4 1 0 50 -1 18 10 0.0000 4 120 375 4500 2475 FULL\001 +4 1 0 50 -1 18 10 0.0000 4 120 630 2700 3465 PAUSED\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 2700 3645 3\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 2700 2655 7\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 4500 2655 6\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 900 3645 5\001 +4 1 1 50 -1 16 10 0.0000 4 120 90 3645 1755 4\001 diff --git a/doc/internals/listener-states.png b/doc/internals/listener-states.png Binary files differnew file mode 100644 index 0000000..8757a12 --- /dev/null +++ b/doc/internals/listener-states.png diff --git a/doc/internals/lua_socket.fig b/doc/internals/lua_socket.fig new file mode 100644 index 0000000..7da3294 --- /dev/null +++ b/doc/internals/lua_socket.fig @@ -0,0 +1,113 @@ +#FIG 3.2 Produced by xfig version 1.8 +Landscape +Center +Metric +A4 +100.00 +Single +-2 +1200 2 +6 1125 2745 2565 3555 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 1125 2745 2565 2745 2565 3555 1125 3555 1125 2745 +4 0 0 50 -1 16 12 0.0000 4 180 1080 1215 3195 lua_State *T\001 +4 0 0 50 -1 18 12 0.0000 4 150 990 1215 2925 struct hlua\001 +4 0 0 50 -1 16 12 0.0000 4 195 1245 1215 3465 stop_list *stop\001 +-6 +6 7560 4365 10620 5265 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 7650 4635 10530 4635 10530 5175 7650 5175 7650 4635 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 7560 4365 10620 4365 10620 5265 7560 5265 7560 4365 +4 0 0 50 -1 18 12 0.0000 4 195 2565 7740 4815 struct stream_interface si[0]\001 +4 0 0 50 -1 16 12 0.0000 4 195 1725 7740 5085 enum obj_type *end\001 +4 0 0 50 -1 18 12 0.0000 4 150 1215 7650 4545 struct stream\001 +-6 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 225 4500 2745 4500 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 225 5040 2745 5040 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 225 4770 2745 4770 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 1935 5715 7740 6705 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 2520 3420 3600 4095 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 225 4230 2745 4230 2745 7020 225 7020 225 4230 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 225 6300 2745 6300 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 225 6660 2745 6660 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 1035 2205 2655 2205 2655 3645 1035 3645 1035 2205 +2 1 1 4 4 7 500 -1 -1 4.000 0 0 -1 0 0 2 + 4860 1935 4860 9225 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 7695 6435 5760 4410 +2 2 0 1 0 7 50 -1 20 0.000 0 0 -1 0 0 5 + 3600 3915 6075 3915 6075 4410 3600 4410 3600 3915 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 9450 5040 9225 5670 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 4 + 7740 6300 7695 6345 7695 6525 7740 6570 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 7560 5670 9765 5670 9765 7200 7560 7200 7560 5670 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 7650 5940 9675 5940 9675 7110 7650 7110 7650 5940 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 315 5310 2655 5310 2655 6165 315 6165 315 5310 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 7830 6840 2565 5580 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 7740 6705 9540 6705 9540 6930 7740 6930 7740 6705 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 405 5580 2565 5580 2565 5805 405 5805 405 5580 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 5 + 1 1 1.00 60.00 120.00 + 1215 3105 765 3330 720 3555 765 3915 810 4230 + 0.000 1.000 1.000 1.000 0.000 +3 0 1 1 13 7 50 -1 -1 1.000 0 1 0 3 + 5 1 1.00 60.00 120.00 + 675 7020 675 7785 900 8104 + 0.000 1.000 0.000 +3 0 1 1 13 7 50 -1 -1 1.000 0 1 0 2 + 5 1 1.00 60.00 120.00 + 7740 7200 7740 8100 + 0.000 0.000 +3 0 1 1 13 7 50 -1 -1 1.000 0 1 0 3 + 5 1 1.00 60.00 120.00 + 7605 7200 7605 8865 7740 9000 + 0.000 1.000 0.000 +4 0 0 50 -1 18 12 0.0000 4 150 885 315 4410 stack Lua\001 +4 0 0 50 -1 16 12 0.0000 4 195 1140 315 4680 stack entry 0\001 +4 0 0 50 -1 16 12 0.0000 4 195 1140 315 4950 stack entry 1\001 +4 0 0 50 -1 16 12 0.0000 4 195 1140 315 5220 stack entry 2\001 +4 0 0 50 -1 18 12 0.0000 4 195 1695 405 5490 struct hlua_socket\001 +4 0 0 50 -1 16 12 0.0000 4 195 1140 315 6570 stack entry 3\001 +4 0 0 50 -1 16 12 0.0000 4 195 1140 315 6930 stack entry 4\001 +4 1 12 50 -1 12 9 5.6723 4 135 540 3150 3735 (list)\001 +4 0 0 50 -1 18 12 0.0000 4 150 1305 1125 2430 struct session\001 +4 0 0 50 -1 16 12 0.0000 4 150 1440 1125 2655 struct task *task\001 +4 0 0 50 -1 12 12 0.0000 4 165 1560 990 8100 hlua_tcp_gc()\001 +4 0 0 50 -1 16 12 0.0000 4 195 2430 990 8295 Called just before the object\001 +4 0 0 50 -1 16 12 0.0000 4 195 840 990 8535 garbaging\001 +4 1 12 50 -1 12 9 5.5327 4 135 540 6390 4905 (list)\001 +4 0 0 50 -1 18 12 0.0000 4 195 2205 3690 4095 struct hlua_socket_com\001 +4 0 0 50 -1 16 12 0.0000 4 150 1440 3690 4320 struct task *task\001 +4 0 0 50 -1 18 12 0.0000 4 195 1200 7650 5850 struct appctx\001 +4 0 0 50 -1 18 12 0.0000 4 150 1110 7740 6120 struct <lua>\001 +4 0 0 50 -1 16 12 0.0000 4 195 1620 7740 6615 struct hlua_tcp *wr\001 +4 0 0 50 -1 16 12 0.0000 4 195 1590 7740 6390 struct hlua_tcp *rd\001 +4 0 0 50 -1 12 12 0.0000 4 165 2160 7875 9000 hlua_tcp_release()\001 +4 0 0 50 -1 16 12 0.0000 4 195 3150 7875 9195 Called when the applet is destroyed.\001 +4 0 0 50 -1 12 12 0.0000 4 165 2400 7875 8100 update_tcp_handler()\001 +4 0 0 50 -1 16 12 0.0000 4 195 2640 7875 8295 Called on each change on the \001 +4 0 0 50 -1 16 12 0.0000 4 195 1830 7875 8535 tcp connection state.\001 +4 0 0 50 -1 16 12 0.0000 4 150 1350 495 5760 struct xref *xref\001 +4 0 0 50 -1 16 12 0.0000 4 150 1350 7830 6885 struct xref *xref\001 diff --git a/doc/internals/lua_socket.pdf b/doc/internals/lua_socket.pdf Binary files differnew file mode 100644 index 0000000..e3b80ee --- /dev/null +++ b/doc/internals/lua_socket.pdf diff --git a/doc/internals/muxes.fig b/doc/internals/muxes.fig new file mode 100644 index 0000000..babdd55 --- /dev/null +++ b/doc/internals/muxes.fig @@ -0,0 +1,401 @@ +#FIG 3.2 Produced by xfig version 3.2.8b +Landscape +Center +Inches +Letter +100.00 +Single +-1 +1200 2 +0 32 #bbf2e2 +0 33 #a7ceb3 +0 34 #dae8fc +0 35 #458dba +0 36 #ffe6cc +0 37 #e9b000 +0 38 #1a1a1a +0 39 #8e8e8e +0 40 #ffc1e7 +6 4200 8700 4800 9825 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 4261 9751 4261 8751 4761 8751 4761 9751 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 4761 9751 4761 8751 4261 8751 4261 9751 4761 9751 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 4261 8850 4761 8850 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 4261 8925 4761 8925 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 4261 9000 4761 9000 +-6 +6 1425 3525 2025 4650 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 1486 4576 1486 3576 1986 3576 1986 4576 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 1986 4576 1986 3576 1486 3576 1486 4576 1986 4576 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 1486 3675 1986 3675 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 1486 3750 1986 3750 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 1486 3825 1986 3825 +-6 +6 3225 3525 3825 4650 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 3286 4576 3286 3576 3786 3576 3786 4576 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 3786 4576 3786 3576 3286 3576 3286 4576 3786 4576 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 3286 3675 3786 3675 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 3286 3750 3786 3750 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 3286 3825 3786 3825 +-6 +6 5025 3525 5625 4650 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 5086 4576 5086 3576 5586 3576 5586 4576 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 5586 4576 5586 3576 5086 3576 5086 4576 5586 4576 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 5086 3675 5586 3675 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 5086 3750 5586 3750 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 5086 3825 5586 3825 +-6 +6 6900 3525 7500 4650 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 6961 4576 6961 3576 7461 3576 7461 4576 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 7461 4576 7461 3576 6961 3576 6961 4576 7461 4576 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 6961 3675 7461 3675 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 6961 3750 7461 3750 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 6961 3825 7461 3825 +-6 +6 11925 10725 13875 11475 +2 4 0 3 0 35 50 -1 20 0.000 1 0 7 0 0 5 + 13800 11400 12000 11400 12000 10800 13800 10800 13800 11400 +4 1 0 49 -1 4 18 0.0000 4 285 1335 12900 11175 Transport\001 +-6 +6 6600 1200 10050 1800 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 6692 1261 9959 1261 9959 1761 6692 1761 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 6692 1761 9959 1761 9959 1261 6692 1261 6692 1761 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9750 1261 9750 1761 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9525 1261 9525 1761 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9300 1261 9300 1761 +4 1 0 46 -1 4 16 0.0000 4 210 1605 8025 1575 channel buf\001 +-6 +6 12375 8100 12900 8700 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 12600 8161 12600 8661 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 12425 8161 12825 8161 12825 8661 12425 8661 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 12425 8661 12825 8661 12825 8161 12425 8161 12425 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 12675 8161 12675 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 12750 8161 12750 8661 +-6 +6 11700 8100 12225 8700 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 11925 8161 11925 8661 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 11750 8161 12150 8161 12150 8661 11750 8661 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 11750 8661 12150 8661 12150 8161 11750 8161 11750 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 12000 8161 12000 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 12075 8161 12075 8661 +-6 +6 11025 8100 11550 8700 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 11250 8161 11250 8661 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 11075 8161 11475 8161 11475 8661 11075 8661 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 11075 8661 11475 8661 11475 8161 11075 8161 11075 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 11325 8161 11325 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 11400 8161 11400 8661 +-6 +6 10350 8100 10875 8700 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 10575 8161 10575 8661 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 10400 8161 10800 8161 10800 8661 10400 8661 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 10400 8661 10800 8661 10800 8161 10400 8161 10400 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 10650 8161 10650 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 10725 8161 10725 8661 +-6 +6 13050 8100 13575 8700 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 13275 8161 13275 8661 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 13100 8161 13500 8161 13500 8661 13100 8661 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 13100 8661 13500 8661 13500 8161 13100 8161 13100 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 13350 8161 13350 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 13425 8161 13425 8661 +-6 +6 13725 8100 14250 8700 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 13950 8161 13950 8661 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 13775 8161 14175 8161 14175 8661 13775 8661 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 13775 8661 14175 8661 14175 8161 13775 8161 13775 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 14025 8161 14025 8661 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 14100 8161 14100 8661 +-6 +6 11100 11700 13050 12150 +1 1 0 4 20 40 49 -1 20 0.000 1 0.0000 11400 11925 225 150 11400 11925 11625 12075 +4 0 0 49 -1 4 12 0.0000 4 165 960 11850 12000 I/O tasklet\001 +-6 +6 11100 12300 11700 12600 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 11357 12331 11357 12581 +2 1 0 4 35 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 11157 12331 11614 12331 11614 12581 11157 12581 +2 3 0 0 -1 34 49 -1 20 0.000 0 0 -1 0 0 5 + 11157 12581 11614 12581 11614 12331 11157 12331 11157 12581 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 11443 12331 11443 12581 +2 1 0 2 35 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 11529 12331 11529 12581 +-6 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 10725 5700 75 75 10725 5700 10800 5700 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 12750 5700 75 75 12750 5700 12825 5700 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 13875 5700 75 75 13875 5700 13950 5700 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 11700 5700 75 75 11700 5700 11775 5700 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 2925 6750 75 75 2925 6750 3000 6750 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 4950 6750 75 75 4950 6750 5025 6750 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 6075 6750 75 75 6075 6750 6150 6750 +1 3 0 3 0 0 49 -1 20 0.000 1 0.0000 3900 6750 75 75 3900 6750 3975 6750 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 9525 4140 583 250 9525 4140 10108 3890 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 11341 4140 583 250 11341 4140 11924 3890 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 13154 4140 583 250 13154 4140 13737 3890 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 15033 4140 583 250 15033 4140 15616 3890 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 7182 5173 583 250 7182 5173 7765 4923 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 3507 5173 583 250 3507 5173 4090 4923 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 1719 5173 583 250 1719 5173 2302 4923 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 5325 5175 583 250 5325 5175 5908 4925 +1 1 0 4 10 11 45 -1 20 0.000 1 0.0000 4488 8082 612 250 4488 8082 5100 8082 +1 1 0 4 10 11 49 -1 20 0.000 1 0.0000 12333 7025 417 250 12333 7025 12750 7025 +1 1 0 4 20 40 49 -1 20 0.000 1 0.0000 12392 9240 808 210 12392 9240 13200 9240 +1 1 0 4 20 40 49 -1 20 0.000 1 0.0000 3167 9240 808 210 3167 9240 3975 9240 +1 1 0 4 37 36 49 -1 20 0.000 1 0.0000 1800 11925 225 150 1800 11925 2025 12075 +1 1 0 4 10 11 45 -1 20 0.000 1 0.0000 6600 11925 225 150 6600 11925 6825 12075 +1 1 0 4 20 40 49 -1 20 0.000 1 0.0000 8400 600 900 210 8400 600 9300 600 +2 1 1 1 0 7 49 -1 -1 4.000 1 0 -1 0 0 2 + 2550 3300 2550 6150 +2 1 1 1 0 7 49 -1 -1 4.000 1 0 -1 0 0 2 + 4500 3300 4500 6150 +2 1 1 1 0 7 49 -1 -1 4.000 1 0 -1 0 0 2 + 6300 3300 6300 6150 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 600 8025 600 12225 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 600 3150 600 1800 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 600 1500 600 150 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 1 4 + 1 1 1.00 90.00 180.00 + 1 1 1.00 90.00 180.00 + 3000 3300 3000 1425 3675 600 7500 600 +2 3 0 4 33 32 50 -1 20 0.000 0 0 -1 0 0 5 + 900 3300 900 9900 8100 9900 8100 3300 900 3300 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 0 4 + 1 1 1.00 90.00 180.00 + 3525 3525 3525 2625 4500 1500 6750 1500 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 0 3 + 1 1 1.00 90.00 180.00 + 11295 4425 11295 4725 11700 5625 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 0 3 + 1 1 1.00 90.00 180.00 + 9495 4425 9495 4725 10695 5700 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 0 3 + 1 1 1.00 90.00 180.00 + 13163 4425 13163 4725 12788 5625 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 0 3 + 1 1 1.00 90.00 180.00 + 15013 4427 15013 4725 13888 5702 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 9525 3525 9525 3825 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 13125 3525 13125 3825 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 15000 3525 15000 3825 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 5 + 1 1 1.00 90.00 180.00 + 12300 7275 12300 7725 9975 7725 9975 8400 10425 8400 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 0 3 + 1 1 1.00 90.00 180.00 + 11775 5850 12300 6450 12300 6825 +2 1 1 3 0 7 49 -1 -1 8.000 1 0 -1 0 0 3 + 11475 6150 13200 6150 13200 6825 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 0 1 3 + 1 1 1.00 90.00 180.00 + 3975 6900 4500 7650 4500 7875 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 0 1 3 + 1 1 1.00 90.00 180.00 + 3495 5475 3495 5775 3900 6675 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 0 1 3 + 1 1 1.00 90.00 180.00 + 1695 5475 1695 5775 2895 6750 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 0 1 3 + 1 1 1.00 90.00 180.00 + 7213 5477 7213 5775 6088 6752 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 1725 4875 1725 4575 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 3525 4875 3525 4575 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 5325 4875 5325 4575 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 7200 4875 7200 4575 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 0 1 2 + 1 1 1.00 90.00 180.00 + 4500 8325 4500 8721 +2 1 1 3 0 7 49 -1 -1 8.000 1 0 -1 0 0 3 + 3225 7875 3225 7350 4725 7350 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 1 2 + 1 1 1.00 90.00 180.00 + 1 1 1.00 90.00 180.00 + 3900 10800 3225 9450 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 4500 10800 4500 9750 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 1 1 3 + 1 1 1.00 90.00 180.00 + 1 1 1.00 90.00 180.00 + 12375 10800 12375 9750 12375 9450 +2 1 1 1 0 7 49 -1 -1 4.000 1 0 -1 0 0 2 + 12225 3300 12225 5025 +2 1 1 1 0 7 49 -1 -1 4.000 1 0 -1 0 0 2 + 10425 3300 10425 5025 +2 1 1 1 0 7 49 -1 -1 4.000 1 0 -1 0 0 2 + 14025 3300 14025 5025 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 4 + 1 1 1.00 90.00 180.00 + 9975 1500 10800 1500 11325 2100 11325 3825 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 1 4 + 1 1 1.00 90.00 180.00 + 1 1 1.00 90.00 180.00 + 9300 600 11175 600 11775 1275 11775 3300 +2 3 0 4 33 32 50 -1 20 0.000 0 0 -1 0 0 5 + 8700 3300 8700 9900 15900 9900 15900 3300 8700 3300 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 0 1 5 + 1 1 1.00 90.00 180.00 + 13200 10800 13200 10200 14625 9750 14625 8400 14175 8400 +2 1 0 3 0 7 49 -1 -1 0.000 1 0 -1 0 1 3 + 1 1 1.00 90.00 180.00 + 5325 5475 5325 5775 4950 6675 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 600 5400 600 3300 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 0 0 2 + 600 7800 600 5700 +2 4 0 3 0 35 50 -1 20 0.000 1 0 7 0 0 5 + 5400 11400 3600 11400 3600 10800 5400 10800 5400 11400 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 12150 8400 12450 8400 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 11475 8400 11775 8400 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 10800 8400 11100 8400 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 12825 8400 13125 8400 +2 1 0 3 0 7 49 -1 -1 8.000 1 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 13500 8400 13800 8400 +2 4 0 3 0 35 50 -1 20 0.000 1 0 7 0 0 5 + 2100 12600 1575 12600 1575 12300 2100 12300 2100 12600 +2 4 0 3 33 32 50 -1 20 0.000 1 0 7 0 0 5 + 6900 12600 6375 12600 6375 12300 6900 12300 6900 12600 +4 1 0 49 -1 4 14 1.5708 4 225 1335 450 825 application\001 +4 0 0 49 -1 4 12 1.5708 4 180 2595 2850 3225 mux->subscribe(SUB_RECV)\001 +4 1 0 46 -1 4 16 1.5708 4 210 645 3600 4200 rxbuf\001 +4 1 0 46 -1 4 16 1.5708 4 210 615 4575 9375 dbuf\001 +4 1 0 49 -1 4 16 0.0000 4 210 600 12300 7125 MUX\001 +4 1 0 44 -1 4 16 0.0000 4 210 945 4500 8175 DEMUX\001 +4 2 0 49 -1 4 12 0.0000 4 150 915 3600 8100 Stream ID\001 +4 0 0 49 -1 4 12 0.0000 4 150 915 12825 7125 Stream ID\001 +4 2 0 49 -1 4 12 0.0000 4 180 1635 3300 10125 tasklet_wakeup()\001 +4 2 0 49 -1 4 12 0.0000 4 180 1635 12150 10125 tasklet_wakeup()\001 +4 2 0 49 -1 4 12 0.0000 4 180 1470 11175 3150 mux->snd_buf()\001 +4 0 0 49 -1 4 12 0.0000 4 180 1425 3675 3225 mux->rcv_buf()\001 +4 0 0 49 -1 4 12 0.0000 4 180 1920 13425 10575 xprt->snd_buf(mbuf)\001 +4 0 0 49 -1 4 12 0.0000 4 180 1830 4725 10500 xprt->rcv_buf(dbuf)\001 +4 1 0 49 -1 4 12 0.0000 4 150 3105 8400 2100 HTX contents when mode==HTTP\001 +4 2 0 49 -1 4 12 0.0000 4 180 1635 7500 450 tasklet_wakeup()\001 +4 0 0 49 -1 4 12 0.0000 4 180 1635 9300 450 tasklet_wakeup()\001 +4 1 38 48 -1 4 12 0.0000 4 150 750 9534 4200 encode\001 +4 1 38 48 -1 4 12 0.0000 4 150 750 11325 4200 encode\001 +4 1 38 48 -1 4 12 0.0000 4 150 750 13134 4200 encode\001 +4 1 38 48 -1 4 12 0.0000 4 150 750 15009 4200 encode\001 +4 1 38 48 -1 4 12 0.0000 4 150 765 1725 5250 decode\001 +4 1 38 48 -1 4 12 0.0000 4 150 765 3525 5250 decode\001 +4 1 38 48 -1 4 12 0.0000 4 150 765 5325 5250 decode\001 +4 1 38 48 -1 4 12 0.0000 4 150 765 7200 5250 decode\001 +4 1 38 48 -1 4 12 0.0000 4 180 1035 12375 9300 mux_io_cb\001 +4 0 0 49 -1 4 12 1.5708 4 180 2580 12075 3225 mux->subscribe(SUB_SEND)\001 +4 1 0 49 -1 4 14 1.5708 4 180 1425 450 4500 mux streams\001 +4 1 0 49 -1 4 14 1.5708 4 135 1980 450 6750 mux=conn->mux\001 +4 1 0 49 -1 4 18 0.0000 4 285 1335 4500 11175 Transport\001 +4 1 0 46 -1 4 16 0.0000 4 210 690 14625 8175 mbuf\001 +4 1 38 48 -1 4 12 0.0000 4 180 1035 3159 9300 mux_io_cb\001 +4 0 0 49 -1 4 12 0.0000 4 195 2805 2250 12000 encoding/decoding function\001 +4 0 0 49 -1 4 12 0.0000 4 180 1365 2250 12525 transport layer\001 +4 0 0 49 -1 4 12 0.0000 4 180 2445 7050 12525 multiplexer (MUX/DEMUX)\001 +4 0 0 49 -1 4 12 0.0000 4 195 2655 7050 12000 general processing function\001 +4 0 0 49 -1 4 12 0.0000 4 180 2820 11775 12525 stream buffer (byte-level FIFO)\001 +4 2 0 49 -1 4 12 0.0000 4 180 2550 3675 10725 xprt->subscribe(SUB_RECV)\001 +4 2 0 49 -1 4 12 0.0000 4 180 2535 12225 10725 xprt->subscribe(SUB_SEND)\001 +4 1 0 49 -1 4 14 1.5708 4 180 780 450 2550 stconn\001 +4 1 0 49 -1 4 12 1.5708 4 195 2010 900 1125 (eg: checks, streams)\001 +4 1 0 49 -1 4 14 1.5708 4 180 3720 450 10125 connection = sc->sedesc->conn\001 +4 0 0 49 -1 4 12 0.0000 4 150 600 12225 225 Notes:\001 +4 0 0 49 -1 4 12 0.0000 4 180 2220 12975 675 snd_buf() will move the\001 +4 0 0 49 -1 4 12 0.0000 4 180 2310 12975 975 buffer (zero-copy) when\001 +4 0 0 49 -1 4 12 0.0000 4 180 2310 12975 1275 the destination is empty.\001 +4 0 0 49 -1 4 12 0.0000 4 180 2220 12825 1650 - the application is also\001 +4 0 0 49 -1 4 12 0.0000 4 180 2700 12975 2250 is sc->app with sc->app_ops\001 +4 0 0 49 -1 4 12 0.0000 4 180 2490 12825 2550 - transport layers (xprt) are\001 +4 0 0 49 -1 4 12 0.0000 4 180 2250 12975 2775 stackable. conn->xprt is\001 +4 0 0 49 -1 4 12 0.0000 4 180 1635 12975 3000 the topmost one.\001 +4 0 0 49 -1 4 12 0.0000 4 180 2400 12975 1950 called the app layer and\001 +4 0 0 49 -1 4 12 0.0000 4 180 1995 12825 375 - mux->rcv_buf() and\001 +4 1 38 48 -1 4 12 0.0000 4 180 1440 8409 657 sc_conn_io_cb\001 diff --git a/doc/internals/muxes.pdf b/doc/internals/muxes.pdf Binary files differnew file mode 100644 index 0000000..54f8cc7 --- /dev/null +++ b/doc/internals/muxes.pdf diff --git a/doc/internals/muxes.png b/doc/internals/muxes.png Binary files differnew file mode 100644 index 0000000..a58f42f --- /dev/null +++ b/doc/internals/muxes.png diff --git a/doc/internals/muxes.svg b/doc/internals/muxes.svg new file mode 100644 index 0000000..3feaa4d --- /dev/null +++ b/doc/internals/muxes.svg @@ -0,0 +1,911 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Creator: fig2dev Version 3.2.8b --> +<!-- CreationDate: 2022-05-27 11:37:43 --> +<!-- Magnification: 1 --> +<svg xmlns="http://www.w3.org/2000/svg" + xmlns:xlink="http://www.w3.org/1999/xlink" + width="942pt" height="755pt" + viewBox="254 60 15690 12573"> +<g fill="none"> +<!-- Line --> +<rect x="12000" y="10800" width="1800" height="600" rx="105" fill="#458dba" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<polygon points=" 900,3300 900,9900 8100,9900 8100,3300" fill="#bbf2e2" + stroke="#a7ceb3" stroke-width="45px"/> +<!-- Line --> +<polygon points=" 8700,3300 8700,9900 15900,9900 15900,3300" fill="#bbf2e2" + stroke="#a7ceb3" stroke-width="45px"/> +<!-- Line --> +<rect x="3600" y="10800" width="1800" height="600" rx="105" fill="#458dba" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<rect x="1575" y="12300" width="525" height="300" rx="105" fill="#458dba" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<rect x="6375" y="12300" width="525" height="300" rx="105" fill="#bbf2e2" + stroke="#a7ceb3" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<polygon points=" 4761,9751 4761,8751 4261,8751 4261,9751" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 1986,4576 1986,3576 1486,3576 1486,4576" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 3786,4576 3786,3576 3286,3576 3286,4576" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 5586,4576 5586,3576 5086,3576 5086,4576" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 7461,4576 7461,3576 6961,3576 6961,4576" fill="#dae8fc"/> +<!-- Text --> +<text xml:space="preserve" x="12900" y="11175" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="216" text-anchor="middle">Transport</text> +<!-- Line --> +<polygon points=" 6692,1761 9959,1761 9959,1261 6692,1261" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 12425,8661 12825,8661 12825,8161 12425,8161" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 11750,8661 12150,8661 12150,8161 11750,8161" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 11075,8661 11475,8661 11475,8161 11075,8161" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 10400,8661 10800,8661 10800,8161 10400,8161" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 13100,8661 13500,8661 13500,8161 13100,8161" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 13775,8661 14175,8661 14175,8161 13775,8161" fill="#dae8fc"/> +<!-- Ellipse --> +<ellipse cx="11400" cy="11925" rx="225" ry="150" fill="#ffc1e7" + stroke="#d10000" stroke-width="45px"/> +<!-- Text --> +<text xml:space="preserve" x="11850" y="12000" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">I/O tasklet</text> +<!-- Line --> +<polygon points=" 11157,12581 11614,12581 11614,12331 11157,12331" fill="#dae8fc"/> +<!-- Circle --> +<circle cx="10725" cy="5700" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Circle --> +<circle cx="12750" cy="5700" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Circle --> +<circle cx="13875" cy="5700" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Circle --> +<circle cx="11700" cy="5700" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Circle --> +<circle cx="2925" cy="6750" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Circle --> +<circle cx="4950" cy="6750" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Circle --> +<circle cx="6075" cy="6750" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Circle --> +<circle cx="3900" cy="6750" r="75" fill="#000000" + stroke="#000000" stroke-width="30px"/> +<!-- Ellipse --> +<ellipse cx="9525" cy="4140" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="11341" cy="4140" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="13154" cy="4140" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="15033" cy="4140" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="7182" cy="5173" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="3507" cy="5173" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="1719" cy="5173" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="5325" cy="5175" rx="583" ry="250" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="12333" cy="7025" rx="417" ry="250" fill="#87cfff" + stroke="#0000d1" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="12392" cy="9240" rx="808" ry="210" fill="#ffc1e7" + stroke="#d10000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="3167" cy="9240" rx="808" ry="210" fill="#ffc1e7" + stroke="#d10000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="1800" cy="11925" rx="225" ry="150" fill="#ffe6cc" + stroke="#e9b000" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="8400" cy="600" rx="900" ry="210" fill="#ffc1e7" + stroke="#d10000" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 2550,3300 2550,6150" + stroke="#000000" stroke-width="8px" stroke-linejoin="round" stroke-dasharray="40 40"/> +<!-- Line --> +<polyline points=" 4500,3300 4500,6150" + stroke="#000000" stroke-width="8px" stroke-linejoin="round" stroke-dasharray="40 40"/> +<!-- Line --> +<polyline points=" 6300,3300 6300,6150" + stroke="#000000" stroke-width="8px" stroke-linejoin="round" stroke-dasharray="40 40"/> +<!-- Line --> +<defs> +<clipPath id="cp0"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 645,12029 555,12029 582,12243 618,12243z"/> +</clipPath> +</defs> +<polyline points=" 600,8025 600,12225" clip-path="url(#cp0)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 600,12225 --> +<polygon points=" 555,12029 600,12209 645,12029 555,12029" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp1"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 555,1996 645,1996 618,1782 582,1782z"/> +</clipPath> +</defs> +<polyline points=" 600,3150 600,1800" clip-path="url(#cp1)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 600,1800 --> +<polygon points=" 645,1996 600,1816 555,1996 645,1996" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp2"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 555,346 645,346 618,132 582,132z"/> +</clipPath> +</defs> +<polyline points=" 600,1500 600,150" clip-path="url(#cp2)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 600,150 --> +<polygon points=" 645,346 600,166 555,346 645,346" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp3"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 7304,555 7304,645 7518,618 7518,582z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 3000,3300 3000,1425 3675,600 7500,600" clip-path="url(#cp3)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7500,600 --> +<polygon points=" 7304,645 7484,600 7304,555 7304,645" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Backward arrow to point 3000,3300 --> +<polygon points=" 2955,3104 3000,3284 3045,3104 2955,3104" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp4"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 6554,1455 6554,1545 6768,1518 6768,1482z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 3525,3525 3525,2625 4500,1500 6750,1500" clip-path="url(#cp4)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 6750,1500 --> +<polygon points=" 6554,1545 6734,1500 6554,1455 6554,1545" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp5"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 11661,5428 11578,5465 11691,5649 11724,5634z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 11295,4425 11295,4725 11700,5625" clip-path="url(#cp5)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 11700,5625 --> +<polygon points=" 11578,5465 11693,5610 11661,5428 11578,5465" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp6"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 10571,5541 10514,5611 10698,5725 10720,5697z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 9495,4425 9495,4725 10695,5700" clip-path="url(#cp6)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 10695,5700 --> +<polygon points=" 10514,5611 10682,5690 10571,5541 10514,5611" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp7"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12905,5461 12822,5427 12764,5635 12798,5649z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 13163,4425 13163,4725 12788,5625" clip-path="url(#cp7)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 12788,5625 --> +<polygon points=" 12822,5427 12794,5610 12905,5461 12822,5427" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp8"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 14066,5607 14007,5539 13863,5700 13886,5727z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 15013,4427 15013,4725 13888,5702" clip-path="url(#cp8)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 13888,5702 --> +<polygon points=" 14007,5539 13900,5691 14066,5607 14007,5539" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp9"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 9555,3689 9495,3689 9507,3843 9543,3843z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 9525,3525 9525,3825" clip-path="url(#cp9)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 9525,3825 --> +<polygon points=" 9495,3689 9525,3809 9555,3689 9495,3689" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp10"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 13155,3689 13095,3689 13107,3843 13143,3843z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 13125,3525 13125,3825" clip-path="url(#cp10)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 13125,3825 --> +<polygon points=" 13095,3689 13125,3809 13155,3689 13095,3689" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp11"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 15030,3689 14970,3689 14982,3843 15018,3843z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 15000,3525 15000,3825" clip-path="url(#cp11)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 15000,3825 --> +<polygon points=" 14970,3689 15000,3809 15030,3689 14970,3689" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp12"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 10229,8355 10229,8445 10443,8418 10443,8382z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 12300,7275 12300,7725 9975,7725 9975,8400 10425,8400" clip-path="url(#cp12)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 10425,8400 --> +<polygon points=" 10229,8445 10409,8400 10229,8355 10229,8445" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp13"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12345,6629 12255,6629 12282,6843 12318,6843z + M 3045,3104 2955,3104 2982,3318 3018,3318z"/> +</clipPath> +</defs> +<polyline points=" 11775,5850 12300,6450 12300,6825" clip-path="url(#cp13)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 12300,6825 --> +<polygon points=" 12255,6629 12300,6809 12345,6629 12255,6629" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<polyline points=" 11475,6150 13200,6150 13200,6825" + stroke="#000000" stroke-width="30px" stroke-linejoin="round" stroke-dasharray="80 80"/> +<!-- Line --> +<defs> +<clipPath id="cp14"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12345,6629 12255,6629 12282,6843 12318,6843z + M 4051,7087 4124,7035 3979,6875 3950,6896z"/> +</clipPath> +</defs> +<polyline points=" 3975,6900 4500,7650 4500,7875" clip-path="url(#cp14)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Backward arrow to point 3975,6900 --> +<polygon points=" 4124,7035 3984,6913 4051,7087 4124,7035" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp15"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12345,6629 12255,6629 12282,6843 12318,6843z + M 3450,5671 3540,5671 3513,5457 3477,5457z"/> +</clipPath> +</defs> +<polyline points=" 3495,5475 3495,5775 3900,6675" clip-path="url(#cp15)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Backward arrow to point 3495,5475 --> +<polygon points=" 3540,5671 3495,5491 3450,5671 3540,5671" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp16"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12345,6629 12255,6629 12282,6843 12318,6843z + M 1650,5671 1740,5671 1713,5457 1677,5457z"/> +</clipPath> +</defs> +<polyline points=" 1695,5475 1695,5775 2895,6750" clip-path="url(#cp16)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Backward arrow to point 1695,5475 --> +<polygon points=" 1740,5671 1695,5491 1650,5671 1740,5671" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp17"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12345,6629 12255,6629 12282,6843 12318,6843z + M 7168,5673 7258,5673 7231,5459 7195,5459z"/> +</clipPath> +</defs> +<polyline points=" 7213,5477 7213,5775 6088,6752" clip-path="url(#cp17)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Backward arrow to point 7213,5477 --> +<polygon points=" 7258,5673 7213,5493 7168,5673 7258,5673" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp18"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 1695,4711 1755,4711 1743,4557 1707,4557z + M 7168,5673 7258,5673 7231,5459 7195,5459z"/> +</clipPath> +</defs> +<polyline points=" 1725,4875 1725,4575" clip-path="url(#cp18)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 1725,4575 --> +<polygon points=" 1755,4711 1725,4591 1695,4711 1755,4711" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp19"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 3495,4711 3555,4711 3543,4557 3507,4557z + M 7168,5673 7258,5673 7231,5459 7195,5459z"/> +</clipPath> +</defs> +<polyline points=" 3525,4875 3525,4575" clip-path="url(#cp19)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 3525,4575 --> +<polygon points=" 3555,4711 3525,4591 3495,4711 3555,4711" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp20"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 5295,4711 5355,4711 5343,4557 5307,4557z + M 7168,5673 7258,5673 7231,5459 7195,5459z"/> +</clipPath> +</defs> +<polyline points=" 5325,4875 5325,4575" clip-path="url(#cp20)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 5325,4575 --> +<polygon points=" 5355,4711 5325,4591 5295,4711 5355,4711" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp21"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 7170,4711 7230,4711 7218,4557 7182,4557z + M 7168,5673 7258,5673 7231,5459 7195,5459z"/> +</clipPath> +</defs> +<polyline points=" 7200,4875 7200,4575" clip-path="url(#cp21)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7200,4575 --> +<polygon points=" 7230,4711 7200,4591 7170,4711 7230,4711" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp22"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 7170,4711 7230,4711 7218,4557 7182,4557z + M 4455,8521 4545,8521 4518,8307 4482,8307z"/> +</clipPath> +</defs> +<polyline points=" 4500,8325 4500,8721" clip-path="url(#cp22)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Backward arrow to point 4500,8325 --> +<polygon points=" 4545,8521 4500,8341 4455,8521 4545,8521" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<polyline points=" 3225,7875 3225,7350 4725,7350" + stroke="#000000" stroke-width="30px" stroke-linejoin="round" stroke-dasharray="80 80"/> +<!-- Line --> +<defs> +<clipPath id="cp23"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 3272,9646 3353,9605 3233,9426 3201,9442z + M 3853,10604 3772,10645 3892,10824 3924,10808z"/> +</clipPath> +</defs> +<polyline points=" 3900,10800 3225,9450" clip-path="url(#cp23)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 3225,9450 --> +<polygon points=" 3353,9605 3232,9464 3272,9646 3353,9605" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Backward arrow to point 3900,10800 --> +<polygon points=" 3772,10645 3893,10786 3853,10604 3772,10645" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp24"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 4455,9946 4545,9946 4518,9732 4482,9732z + M 3853,10604 3772,10645 3892,10824 3924,10808z"/> +</clipPath> +</defs> +<polyline points=" 4500,10800 4500,9750" clip-path="url(#cp24)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 4500,9750 --> +<polygon points=" 4545,9946 4500,9766 4455,9946 4545,9946" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp25"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12330,9646 12420,9646 12393,9432 12357,9432z + M 12420,10604 12330,10604 12357,10818 12393,10818z"/> +</clipPath> +</defs> +<polyline points=" 12375,10800 12375,9750 12375,9450" clip-path="url(#cp25)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 12375,9450 --> +<polygon points=" 12420,9646 12375,9466 12330,9646 12420,9646" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Backward arrow to point 12375,10800 --> +<polygon points=" 12330,10604 12375,10784 12420,10604 12330,10604" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<polyline points=" 12225,3300 12225,5025" + stroke="#000000" stroke-width="8px" stroke-linejoin="round" stroke-dasharray="40 40"/> +<!-- Line --> +<polyline points=" 10425,3300 10425,5025" + stroke="#000000" stroke-width="8px" stroke-linejoin="round" stroke-dasharray="40 40"/> +<!-- Line --> +<polyline points=" 14025,3300 14025,5025" + stroke="#000000" stroke-width="8px" stroke-linejoin="round" stroke-dasharray="40 40"/> +<!-- Line --> +<defs> +<clipPath id="cp26"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 11370,3629 11280,3629 11307,3843 11343,3843z + M 12420,10604 12330,10604 12357,10818 12393,10818z"/> +</clipPath> +</defs> +<polyline points=" 9975,1500 10800,1500 11325,2100 11325,3825" clip-path="url(#cp26)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 11325,3825 --> +<polygon points=" 11280,3629 11325,3809 11370,3629 11280,3629" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp27"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 11820,3104 11730,3104 11757,3318 11793,3318z + M 9496,645 9496,555 9282,582 9282,618z"/> +</clipPath> +</defs> +<polyline points=" 9300,600 11175,600 11775,1275 11775,3300" clip-path="url(#cp27)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 11775,3300 --> +<polygon points=" 11730,3104 11775,3284 11820,3104 11730,3104" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Backward arrow to point 9300,600 --> +<polygon points=" 9496,555 9316,600 9496,645 9496,555" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp28"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 11820,3104 11730,3104 11757,3318 11793,3318z + M 13245,10604 13155,10604 13182,10818 13218,10818z"/> +</clipPath> +</defs> +<polyline points=" 13200,10800 13200,10200 14625,9750 14625,8400 14175,8400" clip-path="url(#cp28)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Backward arrow to point 13200,10800 --> +<polygon points=" 13155,10604 13200,10784 13245,10604 13155,10604" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp29"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 11820,3104 11730,3104 11757,3318 11793,3318z + M 5280,5671 5370,5671 5343,5457 5307,5457z"/> +</clipPath> +</defs> +<polyline points=" 5325,5475 5325,5775 4950,6675" clip-path="url(#cp29)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Backward arrow to point 5325,5475 --> +<polygon points=" 5370,5671 5325,5491 5280,5671 5370,5671" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp30"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 555,3496 645,3496 618,3282 582,3282z + M 5280,5671 5370,5671 5343,5457 5307,5457z"/> +</clipPath> +</defs> +<polyline points=" 600,5400 600,3300" clip-path="url(#cp30)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 600,3300 --> +<polygon points=" 645,3496 600,3316 555,3496 645,3496" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<polyline points=" 600,7800 600,5700" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<defs> +<clipPath id="cp31"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12314,8370 12314,8430 12468,8418 12468,8382z + M 5280,5671 5370,5671 5343,5457 5307,5457z"/> +</clipPath> +</defs> +<polyline points=" 12150,8400 12450,8400" clip-path="url(#cp31)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 12450,8400 --> +<polygon points=" 12314,8430 12434,8400 12314,8370 12314,8430" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp32"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 11639,8370 11639,8430 11793,8418 11793,8382z + M 5280,5671 5370,5671 5343,5457 5307,5457z"/> +</clipPath> +</defs> +<polyline points=" 11475,8400 11775,8400" clip-path="url(#cp32)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 11775,8400 --> +<polygon points=" 11639,8430 11759,8400 11639,8370 11639,8430" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp33"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 10964,8370 10964,8430 11118,8418 11118,8382z + M 5280,5671 5370,5671 5343,5457 5307,5457z"/> +</clipPath> +</defs> +<polyline points=" 10800,8400 11100,8400" clip-path="url(#cp33)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 11100,8400 --> +<polygon points=" 10964,8430 11084,8400 10964,8370 10964,8430" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp34"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 12989,8370 12989,8430 13143,8418 13143,8382z + M 5280,5671 5370,5671 5343,5457 5307,5457z"/> +</clipPath> +</defs> +<polyline points=" 12825,8400 13125,8400" clip-path="url(#cp34)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 13125,8400 --> +<polygon points=" 12989,8430 13109,8400 12989,8370 12989,8430" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp35"> + <path clip-rule="evenodd" d="M 254,60 H 15944 V 12633 H 254 z + M 13664,8370 13664,8430 13818,8418 13818,8382z + M 5280,5671 5370,5671 5343,5457 5307,5457z"/> +</clipPath> +</defs> +<polyline points=" 13500,8400 13800,8400" clip-path="url(#cp35)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 13800,8400 --> +<polygon points=" 13664,8430 13784,8400 13664,8370 13664,8430" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Text --> +<g transform="translate(450,825) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="168" text-anchor="middle">application</text> +</g><!-- Text --> +<g transform="translate(2850,3225) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">mux->subscribe(SUB_RECV)</text> +</g><!-- Text --> +<text xml:space="preserve" x="12300" y="7125" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">MUX</text> +<!-- Text --> +<text xml:space="preserve" x="3600" y="8100" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="end">Stream ID</text> +<!-- Text --> +<text xml:space="preserve" x="12825" y="7125" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">Stream ID</text> +<!-- Text --> +<text xml:space="preserve" x="3300" y="10125" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="end">tasklet_wakeup()</text> +<!-- Text --> +<text xml:space="preserve" x="12150" y="10125" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="end">tasklet_wakeup()</text> +<!-- Text --> +<text xml:space="preserve" x="11175" y="3150" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="end">mux->snd_buf()</text> +<!-- Text --> +<text xml:space="preserve" x="3675" y="3225" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">mux->rcv_buf()</text> +<!-- Text --> +<text xml:space="preserve" x="13425" y="10575" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">xprt->snd_buf(mbuf)</text> +<!-- Text --> +<text xml:space="preserve" x="4725" y="10500" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">xprt->rcv_buf(dbuf)</text> +<!-- Text --> +<text xml:space="preserve" x="8400" y="2100" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">HTX contents when mode==HTTP</text> +<!-- Text --> +<text xml:space="preserve" x="7500" y="450" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="end">tasklet_wakeup()</text> +<!-- Text --> +<text xml:space="preserve" x="9300" y="450" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">tasklet_wakeup()</text> +<!-- Text --> +<g transform="translate(12075,3225) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">mux->subscribe(SUB_SEND)</text> +</g><!-- Text --> +<g transform="translate(450,4500) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="168" text-anchor="middle">mux streams</text> +</g><!-- Text --> +<g transform="translate(450,6750) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="168" text-anchor="middle">mux=conn->mux</text> +</g><!-- Text --> +<text xml:space="preserve" x="4500" y="11175" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="216" text-anchor="middle">Transport</text> +<!-- Text --> +<text xml:space="preserve" x="2250" y="12000" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">encoding/decoding function</text> +<!-- Text --> +<text xml:space="preserve" x="2250" y="12525" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">transport layer</text> +<!-- Text --> +<text xml:space="preserve" x="7050" y="12525" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">multiplexer (MUX/DEMUX)</text> +<!-- Text --> +<text xml:space="preserve" x="7050" y="12000" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">general processing function</text> +<!-- Text --> +<text xml:space="preserve" x="11775" y="12525" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">stream buffer (byte-level FIFO)</text> +<!-- Text --> +<text xml:space="preserve" x="3675" y="10725" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="end">xprt->subscribe(SUB_RECV)</text> +<!-- Text --> +<text xml:space="preserve" x="12225" y="10725" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="end">xprt->subscribe(SUB_SEND)</text> +<!-- Text --> +<g transform="translate(450,2550) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="168" text-anchor="middle">stconn</text> +</g><!-- Text --> +<g transform="translate(900,1125) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">(eg: checks, streams)</text> +</g><!-- Text --> +<g transform="translate(450,10125) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="168" text-anchor="middle">connection = sc->sedesc->conn</text> +</g><!-- Text --> +<text xml:space="preserve" x="12225" y="225" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">Notes:</text> +<!-- Text --> +<text xml:space="preserve" x="12975" y="675" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">snd_buf() will move the</text> +<!-- Text --> +<text xml:space="preserve" x="12975" y="975" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">buffer (zero-copy) when</text> +<!-- Text --> +<text xml:space="preserve" x="12975" y="1275" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">the destination is empty.</text> +<!-- Text --> +<text xml:space="preserve" x="12825" y="1650" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">- the application is also</text> +<!-- Text --> +<text xml:space="preserve" x="12975" y="2250" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">is sc->app with sc->app_ops</text> +<!-- Text --> +<text xml:space="preserve" x="12825" y="2550" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">- transport layers (xprt) are</text> +<!-- Text --> +<text xml:space="preserve" x="12975" y="2775" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">stackable. conn->xprt is</text> +<!-- Text --> +<text xml:space="preserve" x="12975" y="3000" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">the topmost one.</text> +<!-- Text --> +<text xml:space="preserve" x="12975" y="1950" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">called the app layer and</text> +<!-- Text --> +<text xml:space="preserve" x="12825" y="375" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="start">- mux->rcv_buf() and</text> +<!-- Line --> +<polyline points=" 4261,9751 4261,8751 4761,8751 4761,9751" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 1486,4576 1486,3576 1986,3576 1986,4576" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 3286,4576 3286,3576 3786,3576 3786,4576" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 5086,4576 5086,3576 5586,3576 5586,4576" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 6961,4576 6961,3576 7461,3576 7461,4576" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 6692,1261 9959,1261 9959,1761 6692,1761" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 12425,8161 12825,8161 12825,8661 12425,8661" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 11750,8161 12150,8161 12150,8661 11750,8661" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 11075,8161 11475,8161 11475,8661 11075,8661" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 10400,8161 10800,8161 10800,8661 10400,8661" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 13100,8161 13500,8161 13500,8661 13100,8661" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 13775,8161 14175,8161 14175,8661 13775,8661" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 11157,12331 11614,12331 11614,12581 11157,12581" + stroke="#458dba" stroke-width="45px"/> +<!-- Text --> +<text xml:space="preserve" x="9534" y="4200" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">encode</text> +<!-- Text --> +<text xml:space="preserve" x="11325" y="4200" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">encode</text> +<!-- Text --> +<text xml:space="preserve" x="13134" y="4200" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">encode</text> +<!-- Text --> +<text xml:space="preserve" x="15009" y="4200" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">encode</text> +<!-- Text --> +<text xml:space="preserve" x="1725" y="5250" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">decode</text> +<!-- Text --> +<text xml:space="preserve" x="3525" y="5250" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">decode</text> +<!-- Text --> +<text xml:space="preserve" x="5325" y="5250" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">decode</text> +<!-- Text --> +<text xml:space="preserve" x="7200" y="5250" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">decode</text> +<!-- Text --> +<text xml:space="preserve" x="12375" y="9300" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">mux_io_cb</text> +<!-- Text --> +<text xml:space="preserve" x="3159" y="9300" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">mux_io_cb</text> +<!-- Text --> +<text xml:space="preserve" x="8409" y="657" fill="#1a1a1a" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="144" text-anchor="middle">sc_conn_io_cb</text> +<!-- Line --> +<polyline points=" 4261,8850 4761,8850" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 4261,8925 4761,8925" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 4261,9000 4761,9000" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 1486,3675 1986,3675" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 1486,3750 1986,3750" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 1486,3825 1986,3825" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 3286,3675 3786,3675" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 3286,3750 3786,3750" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 3286,3825 3786,3825" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 5086,3675 5586,3675" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 5086,3750 5586,3750" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 5086,3825 5586,3825" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 6961,3675 7461,3675" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 6961,3750 7461,3750" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 6961,3825 7461,3825" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9750,1261 9750,1761" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9525,1261 9525,1761" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9300,1261 9300,1761" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 12600,8161 12600,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 12675,8161 12675,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 12750,8161 12750,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 11925,8161 11925,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 12000,8161 12000,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 12075,8161 12075,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 11250,8161 11250,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 11325,8161 11325,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 11400,8161 11400,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 10575,8161 10575,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 10650,8161 10650,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 10725,8161 10725,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 13275,8161 13275,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 13350,8161 13350,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 13425,8161 13425,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 13950,8161 13950,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 14025,8161 14025,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 14100,8161 14100,8661" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 11357,12331 11357,12581" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 11443,12331 11443,12581" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 11529,12331 11529,12581" + stroke="#458dba" stroke-width="15px"/> +<!-- Text --> +<text xml:space="preserve" x="8025" y="1575" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">channel buf</text> +<!-- Text --> +<g transform="translate(3600,4200) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">rxbuf</text> +</g><!-- Text --> +<g transform="translate(4575,9375) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">dbuf</text> +</g><!-- Text --> +<text xml:space="preserve" x="14625" y="8175" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">mbuf</text> +<!-- Ellipse --> +<ellipse cx="4488" cy="8082" rx="612" ry="250" fill="#87cfff" + stroke="#0000d1" stroke-width="45px"/> +<!-- Ellipse --> +<ellipse cx="6600" cy="11925" rx="225" ry="150" fill="#87cfff" + stroke="#0000d1" stroke-width="45px"/> +<!-- Text --> +<text xml:space="preserve" x="4500" y="8175" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">DEMUX</text> +</g> +</svg> diff --git a/doc/internals/notes-layers.txt b/doc/internals/notes-layers.txt new file mode 100644 index 0000000..541c125 --- /dev/null +++ b/doc/internals/notes-layers.txt @@ -0,0 +1,330 @@ +2018-02-21 - Layering in haproxy 1.9 +------------------------------------ + +2 main zones : + - application : reads from conn_streams, writes to conn_streams, often uses + streams + + - connection : receives data from the network, presented into buffers + available via conn_streams, sends data to the network + + +The connection zone contains multiple layers which behave independently in each +direction. The Rx direction is activated upon callbacks from the lower layers. +The Tx direction is activated recursively from the upper layers. Between every +two layers there may be a buffer, in each direction. When a buffer is full +either in Tx or Rx direction, this direction is paused from the network layer +and the location where the congestion is encountered. Upon end of congestion +(cs_recv() from the upper layer, of sendto() at the lower layers), a +tasklet_wakeup() is performed on the blocked layer so that suspended operations +can be resumed. In this case, the Rx side restarts propagating data upwards +from the lowest blocked level, while the Tx side restarts propagating data +downwards from the highest blocked level. Proceeding like this ensures that +information known to the producer may always be used to tailor the buffer sizes +or decide of a strategy to best aggregate data. Additionally, each time a layer +is crossed without transformation, it becomes possible to send without copying. + +The Rx side notifies the application of data readiness using a wakeup or a +callback. The Tx side notifies the application of room availability once data +have been moved resulting in the uppermost buffer having some free space. + +When crossing a mux downwards, it is possible that the sender is not allowed to +access the buffer because it is not yet its turn. It is not a problem, the data +remains in the conn_stream's buffer (or the stream one) and will be restarted +once the mux is ready to consume these data. + + + cs_recv() -------. cs_send() + ^ +--------> |||||| -------------+ ^ + | | -------' | | stream + --|----------|-------------------------------|-------|------------------- + | | V | connection + data .---. | | room + ready! |---| |---| available! + |---| |---| + |---| |---| + | | '---' + ^ +------------+-------+ | + | | ^ | / + / V | V / + / recvfrom() | sendto() | + -------------|----------------|--------------|--------------------------- + | | poll! V kernel + + +The cs_recv() function should act on pointers to buffer pointers, so that the +callee may decide to pass its own buffer directly by simply swapping pointers. +Similarly for cs_send() it is desirable to let the callee steal the buffer by +swapping the pointers. This way it remains possible to implement zero-copy +forwarding. + +Some operation flags will be needed on cs_recv() : + - RECV_ZERO_COPY : refuse to merge new data into the current buffer if it + will result in a data copy (ie the buffer is not empty), unless no more + than XXX bytes have to be copied (eg: copying 2 cache lines may be cheaper + than waiting and playing with pointers) + + - RECV_AT_ONCE : only perform the operation if it will result in the source + buffer to become empty at the end of the operation so that no two buffers + remain allocated at the end. It will most of the time result in either a + small read or a zero-copy operation. + + - RECV_PEEK : retrieve a copy of pending data without removing these data + from the source buffer. Maybe an alternate solution could consist in + finding the pointer to the source buffer and accessing these data directly, + except that it might be less interesting for the long term, thread-wise. + + - RECV_MIN : receive minimum X bytes (or less with a shutdown), or fail. + This should help various protocol parsers which need to receive a complete + frame before proceeding. + + - RECV_ENOUGH : no more data expected after this read if it's of the + requested size, thus no need to re-enable receiving on the lower layers. + + - RECV_ONE_SHOT : perform a single read without re-enabling reading on the + lower layers, like we currently do when receiving an HTTP/1 request. Like + RECV_ENOUGH where any size is enough. Probably that the two could be merged + (eg: by having a MIN argument like RECV_MIN). + + +Some operation flags will be needed on cs_send() : + - SEND_ZERO_COPY : refuse to merge the presented data with existing data and + prefer to wait for current data to leave and try again, unless the consumer + considers the amount of data acceptable for a copy. + + - SEND_AT_ONCE : only perform the operation if it will result in the source + buffer to become empty at the end of the operation so that no two buffers + remain allocated at the end. It will most of the time result in either a + small write or a zero-copy operation. + + +Both operations should return a composite status : + - number of bytes transferred + - status flags (shutr, shutw, reset, empty, full, ...) + + +2018-07-23 - Update after merging rxbuf +--------------------------------------- + +It becomes visible that the mux will not always be welcome to decode incoming +data because it will sometimes imply extra memory copies and/or usage for no +benefit. + +Ideally, when when a stream is instantiated based on incoming data, these +incoming data should be passed and the upper layers called, but it should then +be up these upper layers to peek more data in certain circumstances. Typically +if the pending connection data are larger than what is expected to be passed +above, it means some data may cause head-of-line blocking (HOL) to other +streams, and needs to be pushed up through the layers to let other streams +continue to work. Similarly very large H2 data frames after header frames +should probably not be passed as they may require copies that could be avoided +if passed later. However if the decoded frame fits into the conn_stream's +buffer, there is an opportunity to use a single buffer for the conn_stream +and the channel. The H2 demux could set a blocking flag indicating it's waiting +for the upper stream to take over demuxing. This flag would be purged once the +upper stream would start reading, or when extra data come and change the +conditions. + +Forcing structured headers and raw data to coexist within a single buffer is +quite challenging for many code parts. For example it's perfectly possible to +see a fragmented buffer containing series of headers, then a small data chunk +that was received at the same time, then a few other headers added by request +processing, then another data block received afterwards, then possibly yet +another header added by option http-send-name-header, and yet another data +block. This causes some pain for compression which still needs to know where +compressed and uncompressed data start/stop. It also makes it very difficult +to account the exact bytes to pass through the various layers. + +One solution consists in thinking about buffers using 3 representations : + + - a structured message, which is used for the internal HTTP representation. + This message may only be atomically processed. It has no clear byte count, + it's a message. + + - a raw stream, consisting in sequences of bytes. That's typically what + happens in data sequences or in tunnel. + + - a pipe, which contains data to be forwarded, and that haproxy cannot have + access to. + +The processing efficiency decreases with the higher complexity above, but the +capabilities increase. The structured message can contain anything including +serialized data blocks to be processed or forwarded. The raw stream contains +data blocks to be processed or forwarded. The pipe only contains data blocks +to be forwarded. The the latter ones are only an optimization of the former +ones. + +Thus ideally a channel should have access to all such 3 storage areas at once, +depending on the use case : + (1) a structured message, + (2) a raw stream, + (3) a pipe + +Right now a channel only has (2) and (3) but after the native HTTP rework, it +will only have (1) and (3). Placing a raw stream exclusively in (1) comes with +some performance drawbacks which are not easily recovered, and with some quite +difficult management still involving the reserve to ensure that a data block +doesn't prevent headers from being appended. But during header processing, the +payload may be necessary so we cannot decide to drop this option. + +A long-term approach would consist in ensuring that a single channel may have +access to all 3 representations at once, and to enumerate priority rules to +define how they interact together. That's exactly what is currently being done +with the pipe and the raw buffer right now. Doing so would also save the need +for storing payload in the structured message and void the requirement for the +reserve. But it would cost more memory to process POST data and server +responses. Thus an intermediary step consists in keeping this model in mind but +not implementing everything yet. + +Short term proposal : a channel has access to a buffer and a pipe. A non-empty +buffer is either in structured message format OR raw stream format. Only the +channel knows. However a structured buffer MAY contain raw data in a properly +formatted way (using the envelope defined by the structured message format). + +By default, when a demux writes to a CS rxbuf, it will try to use the lowest +possible level for what is being done (i.e. splice if possible, otherwise raw +stream, otherwise structured message). If the buffer already contains a +structured message, then this format is exclusive. From this point the MUX has +two options : either encode the incoming data to match the structured message +format, or refrain from receiving into the CS's rxbuf and wait until the upper +layer request those data. + +This opens a simplified option which could be suited even for the long term : + - cs_recv() will take one or two flags to indicate if a buffer already + contains a structured message or not ; the upper layer knows it. + + - cs_recv() will take two flags to indicate what the upper layer is willing + to take : + - structured message only + - raw stream only + - any of them + + From this point the mux can decide to either pass anything or refrain from + doing so. + + - the demux stores the knowledge it has from the contents into some CS flags + to indicate whether or not some structured message are still available, and + whether or not some raw data are still available. Thus the caller knows + whether or not extra data are available. + + - when the demux works on its own, it refrains from passing structured data + to a non-empty buffer, unless these data are causing trouble to other + streams (HOL). + + - when a demux has to encapsulate raw data into a structured message, it will + always have to respect a configured reserve so that extra header processing + can be done on the structured message inside the buffer, regardless of the + supposed available room. In addition, the upper layer may indicate using an + extra recv() flag whether it wants the demux to defragment serialized data + (for example by moving trailing headers apart) or if it's not necessary. + This flag will be set by the stream interface if compression is required or + if the http-buffer-request option is set for example. Probably that using + to_forward==0 is a stronger indication that the reserve must be respected. + + - cs_recv() and cs_send() when fed with a message, should not return byte + counts but message counts (i.e. 0 or 1). This implies that a single call to + either of these functions cannot mix raw data and structured messages at + the same time. + +At this point it looks like the conn_stream will have some encapsulation work +to do for the payload if it needs to be encapsulated into a message. This +further magnifies the importance of *not* decoding DATA frames into the CS's +rxbuf until really needed. + +The CS will probably need to hold indication of what is available at the mux +level, not only in the CS. Eg: we know that payload is still available. + +Using these elements, it should be possible to ensure that full header frames +may be received without enforcing any reserve, that too large frames that do +not fit will be detected because they return 0 message and indicate that such +a message is still pending, and that data availability is correctly detected +(later we may expect that the stream-interface allocates a larger or second +buffer to place the payload). + +Regarding the ability for the channel to forward data, it looks like having a +new function "cs_xfer(src_cs, dst_cs, count)" could be very productive in +optimizing the forwarding to make use of splicing when available. It is not yet +totally clear whether it will split into "cs_xfer_in(src_cs, pipe, count)" +followed by "cs_xfer_out(dst_cs, pipe, count)" or anything different, and it +still needs to be studied. The general idea seems to be that the receiver might +have to call the sender directly once they agree on how to transfer data (pipe +or buffer). If the transfer is incomplete, the cs_xfer() return value and/or +flags will indicate the current situation (src empty, dst full, etc) so that +the caller may register for notifications on the appropriate event and wait to +be called again to continue. + +Short term implementation : + 1) add new CS flags to qualify what the buffer contains and what we expect + to read into it; + + 2) set these flags to pretend we have a structured message when receiving + headers (after all, H1 is an atomic header as well) and see what it + implies for the code; for H1 it's unclear whether it makes sense to try + to set it without the H1 mux. + + 3) use these flags to refrain from sending DATA frames after HEADERS frames + in H2. + + 4) flush the flags at the stream interface layer when performing a cs_send(). + + 5) use the flags to enforce receipt of data only when necessary + +We should be able to end up with sequential receipt in H2 modelling what is +needed for other protocols without interfering with the native H1 devs. + + +2018-08-17 - Considerations after killing cs_recv() +--------------------------------------------------- + +With the ongoing reorganisation of the I/O layers, it's visible that cs_recv() +will have to transfer data between the cs' rxbuf and the channel's buffer while +not being aware of the data format. Moreover, in case there's no data there, it +needs to recursively call the mux's rcv_buf() to trigger a decoding, while this +function is sometimes replaced with cs_recv(). All this shows that cs_recv() is +in fact needed while data are pushed upstream from the lower layers, and is not +suitable for the "pull" mode. Thus it was decided to remove this function and +put its code back into h2_rcv_buf(). The H1 mux's rcv_buf() already couldn't be +replaced with cs_recv() since it is the only one knowing about the buffer's +format. + +This opportunity simplified something : if the cs's rxbuf is only read by the +mux's rcv_buf() method, then it doesn't need to be located into the CS and is +well placed into the mux's representation of the stream. This has an important +impact for H2 as it offers more freedom to the mux to allocate/free/reallocate +this buffer, and it ensures the mux always has access to it. + +Furthermore, the conn_stream's txbuf experienced the same fate. Indeed, the H1 +mux has already uncovered the difficulty related to the channel shutting down +on output, with data stuck into the CS's txbuf. Since the CS is tightly coupled +to the stream and the stream can close immediately once its buffers are empty, +it required a way to support orphaned CS with pending data in their txbuf. This +is something that the H2 mux already has to deal with, by carefully leaving the +data in the channel's buffer. But due to the snd_buf() call being top-down, it +is always possible to push the stream's data via the mux's snd_buf() call +without requiring a CS txbuf anymore. Thus the txbuf (when needed) is only +implemented in the mux and attached to the mux's representation of the stream, +and doing so allows to immediately release the channel once the data are safe +in the mux's buffer. + +This is an important change which clarifies the roles and responsibilities of +each layer in the chain : when receiving data from a mux, it's the mux's +responsibility to make sure it can correctly decode the incoming data and to +buffer the possible excess of data it cannot pass to the requester. This means +that decoding an H2 frame, which is not retryable since it has an impact on the +HPACK decompression context, and which cannot be reordered for the same reason, +simply needs to be performed to the H2 stream's rxbuf which will then be passed +to the stream when this one calls h2_rcv_buf(), even if it reads one byte at a +time. Similarly when calling h2_snd_buf(), it's the mux's responsibility to +read as much as it needs to be able to restart later, possibly by buffering +some data into a local buffer. And it's only once all the output data has been +consumed by snd_buf() that the stream is free to disappear. + +This model presents the nice benefit of being infinitely stackable and solving +the last identified showstoppers to move towards a structured message internal +representation, as it will give full power to the rcv_buf() and snd_buf() to +process what they need. + +For now the conn_stream's flags indicating whether a shutdown has been seen in +any direction or if an end of stream was seen will remain in the conn_stream, +though it's likely that some of them will move to the mux's representation of +the stream after structured messages are implemented. diff --git a/doc/internals/notes-poll-connect.txt b/doc/internals/notes-poll-connect.txt new file mode 100644 index 0000000..5cb0885 --- /dev/null +++ b/doc/internals/notes-poll-connect.txt @@ -0,0 +1,93 @@ +2022-11-17 - Tests involving poll() return states upon a pending connect(). + +- connect() to a closed port returns OUT and HUP: + + $ dev/poll/poll -v -l clo -c pol + #### BEGIN #### + cmd #1 stp #1: clo(l=3): ret=0 + cmd #2 stp #0: con(c=4): ret=-1 (Connection refused) + cmd #2 stp #1: pol(c=4): ret=1 ev=0x14 (OUT HUP) + #### END #### + +=> with HUP we *know* the connection failed, since we never asked for a + SHUTW before connecting. It is indeed an error as can be seen with + connect() returning -1 ECONNREFUSED. + +- connect() to a port that does close(accept()) does return IN and RDHUP: + + $ dev/poll/poll -v -s clo -c pol + #### BEGIN #### + cmd #1 stp #0: con(c=4): ret=0 + cmd #1 stp #0: acc(l=3): ret=5 + cmd #1 stp #1: clo(s=5): ret=0 + cmd #2 stp #1: pol(c=4): ret=1 ev=0x2005 (IN OUT RDHUP) + #### END #### + +=> here there's no HUP, only RDHUP because the FIN is pending in the + socket buffers, waiting to be read. + +- for a HUP to happen after a connect() to a valid port, one would have to + perform a shutw() on the client, which is normally not the case, indicating + that HUP is reliable here: + + $ dev/poll/poll -v -s clo -c shw,pol + #### BEGIN #### + cmd #1 stp #0: con(c=4): ret=0 + cmd #1 stp #0: acc(l=3): ret=5 + cmd #1 stp #1: clo(s=5): ret=0 + cmd #2 stp #1: shw(c=4): ret=0 + cmd #2 stp #2: pol(c=4): ret=1 ev=0x2015 (IN OUT HUP RDHUP) + #### END #### + +- one case that may happen is when sending a request and immediately shutw() + (which leaves a TIME_WAIT so not recommended): + + $ dev/poll/poll -v -c snd,shw -s clo -c pol,rcv,pol + #### BEGIN #### + cmd #1 stp #0: con(c=4): ret=0 + cmd #1 stp #1: snd(c=4): ret=3 + cmd #1 stp #2: shw(c=4): ret=0 + cmd #2 stp #0: acc(l=3): ret=5 + cmd #2 stp #1: clo(s=5): ret=0 + cmd #3 stp #1: pol(c=4): ret=1 ev=0x201d (IN OUT ERR HUP RDHUP) + cmd #3 stp #2: rcv(c=4): ret=-1 (Connection reset by peer) + cmd #3 stp #3: pol(c=4): ret=1 ev=0x2015 (IN OUT HUP RDHUP) + #### END #### + +=> here it's impossible to know from the client whether the server consumed the + data or not, which is normal since a close on the server causes an RST to be + emitted for the data in flight, hence the ERR here. It's also worth noting + that once POLL_ERR is consumed by recv() it disappears. + +- for the server, sending a shutw() before closing here delivers an ACK in time + that prevents the RST from being sent, thus connect() is not notified (but if + the server has too much to send, it will truncate and emit an RST): + + $ dev/poll/poll -v -c snd,shw -s shw,clo -c pol,rcv,pol + #### BEGIN #### + cmd #1 stp #0: con(c=4): ret=0 + cmd #1 stp #1: snd(c=4): ret=3 + cmd #1 stp #2: shw(c=4): ret=0 + cmd #2 stp #0: acc(l=3): ret=5 + cmd #2 stp #1: shw(s=5): ret=0 + cmd #2 stp #2: clo(s=5): ret=0 + cmd #3 stp #1: pol(c=4): ret=1 ev=0x2015 (IN OUT HUP RDHUP) + cmd #3 stp #2: rcv(c=4): ret=0 + cmd #3 stp #3: pol(c=4): ret=1 ev=0x2015 (IN OUT HUP RDHUP) + #### END #### + +- if the server sends a response, disables lingering and closes with RST, it is + possible to get HUP and ERR at the same time during the connect() phase, and + recv() can still receive the pending response: + + $ dev/poll/poll -v -s snd,lin,clo -c pol,rcv,pol + #### BEGIN #### + cmd #1 stp #0: con(c=4): ret=0 + cmd #1 stp #0: acc(l=3): ret=5 + cmd #1 stp #1: snd(s=5): ret=3 + cmd #1 stp #2: lin(s=5): ret=0 + cmd #1 stp #3: clo(s=5): ret=0 + cmd #2 stp #1: pol(c=4): ret=1 ev=0x201d (IN OUT ERR HUP RDHUP) + cmd #2 stp #2: rcv(c=4): ret=3 + cmd #2 stp #3: pol(c=4): ret=1 ev=0x201d (IN OUT ERR HUP RDHUP) + #### END #### diff --git a/doc/internals/notes-pollhup.txt b/doc/internals/notes-pollhup.txt new file mode 100644 index 0000000..ced332b --- /dev/null +++ b/doc/internals/notes-pollhup.txt @@ -0,0 +1,281 @@ +tcp mode 8001->8008 + + +Remote test: +============ + +willy@up1:~$ echo bar | ncat -lp8008 +willy@wtap:haproxy$ echo foo | ncat 127.1 8001 + +17:09:53.663154 epoll_wait(3, [{EPOLLIN, {u32=5, u64=5}}], 200, 1000) = 1 +17:09:54.582146 accept4(5, {sa_family=AF_INET, sin_port=htons(33378), sin_addr=inet_addr("127.0.0.1")}, [128->16], SOCK_NONBLOCK) = 8 +17:09:54.582299 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:09:54.582527 accept4(5, 0x7ffc4a8bf330, [128], SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable) +17:09:54.582655 recvfrom(8, "foo\n", 15360, 0, NULL, NULL) = 4 +17:09:54.582727 recvfrom(8, "", 15356, 0, NULL, NULL) = 0 +17:09:54.582827 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 +17:09:54.582878 setsockopt(9, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:09:54.582897 connect(9, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("10.0.3.82")}, 16) = -1 EINPROGRESS (Operation now in progress) +17:09:54.582941 sendto(9, "foo\n", 4, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily unavailable) +17:09:54.582968 epoll_ctl(3, EPOLL_CTL_ADD, 9, {EPOLLOUT, {u32=9, u64=9}}) = 0 +17:09:54.582997 epoll_wait(3, [{EPOLLOUT, {u32=9, u64=9}}], 200, 1000) = 1 +17:09:54.583686 connect(9, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("10.0.3.82")}, 16) = 0 +17:09:54.583706 sendto(9, "foo\n", 4, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 4 +17:09:54.583733 recvfrom(9, 0x19c2300, 15360, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +17:09:54.583755 shutdown(9, SHUT_WR) = 0 +17:09:54.583775 epoll_ctl(3, EPOLL_CTL_MOD, 9, {EPOLLIN|EPOLLRDHUP, {u32=9, u64=9}}) = 0 +17:09:54.583802 epoll_wait(3, [{EPOLLIN, {u32=9, u64=9}}], 200, 1000) = 1 +17:09:54.584672 recvfrom(9, "bar\n", 16384, 0, NULL, NULL) = 4 +17:09:54.584713 recvfrom(9, "", 16380, 0, NULL, NULL) = 0 +17:09:54.584743 sendto(8, "bar\n", 4, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE, NULL, 0) = 4 +17:09:54.584819 epoll_wait(3, [], 200, 0) = 0 +17:09:54.584901 epoll_wait(3, [], 200, 1000) = 0 + + +Notes: + - we had data available to try the connect() (see first attempt), despite + this during the retry we sent the connect again! + + - why do we wait before sending the shutw to the server if we already know + it's needed ? missing CF_SHUTW_NOW ? Missing request forwarding ? Missing + auto-close ? + + - response didn't feature HUP nor RDHUP + + +Local: + +17:15:43.010786 accept4(5, {sa_family=AF_INET, sin_port=htons(33506), sin_addr=inet_addr("127.0.0.1")}, [128->16], SOCK_NONBLOCK) = 8 +17:15:43.011013 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:15:43.011181 accept4(5, 0x7ffcd9092cd0, [128], SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable) +17:15:43.011231 recvfrom(8, "foo\n", 15360, 0, NULL, NULL) = 4 +17:15:43.011296 recvfrom(8, "", 15356, 0, NULL, NULL) = 0 +17:15:43.011318 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 +17:15:43.011340 setsockopt(9, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:15:43.011353 connect(9, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) +17:15:43.011395 sendto(9, "foo\n", 4, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 4 +17:15:43.011425 shutdown(9, SHUT_WR) = 0 +17:15:43.011459 recvfrom(9, "bar\n", 16384, 0, NULL, NULL) = 4 +17:15:43.011491 recvfrom(9, "", 16380, 0, NULL, NULL) = 0 +17:15:43.011525 sendto(8, "bar\n", 4, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE, NULL, 0) = 4 +17:15:43.011584 epoll_wait(3, [], 200, 0) = 0 + +Notes: + - the shutdown() was properly done right after the sendto(), proving that + CF_SHUTW_NOW and auto-close were present. Maybe difference is sync vs async + send. + + +Local with delay before closing client: + +17:18:17.155349 epoll_wait(3, [{EPOLLIN, {u32=5, u64=5}}], 200, 1000) = 1 +17:18:17.727327 accept4(5, {sa_family=AF_INET, sin_port=htons(33568), sin_addr=inet_addr("127.0.0.1")}, [128->16], SOCK_NONBLOCK) = 8 +17:18:17.727553 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:18:17.727661 accept4(5, 0x7fff4eb9a0b0, [128], SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable) +17:18:17.727798 recvfrom(8, 0xbda300, 15360, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +17:18:17.727830 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 +17:18:17.727858 setsockopt(9, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:18:17.727877 connect(9, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) +17:18:17.727923 epoll_ctl(3, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLRDHUP, {u32=8, u64=8}}) = 0 +17:18:17.727945 epoll_ctl(3, EPOLL_CTL_ADD, 9, {EPOLLOUT, {u32=9, u64=9}}) = 0 +17:18:17.727989 epoll_wait(3, [{EPOLLOUT, {u32=9, u64=9}}], 200, 1000) = 1 +17:18:17.728010 connect(9, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 +17:18:17.728027 recvfrom(9, "bar\n", 15360, 0, NULL, NULL) = 4 +17:18:17.728055 recvfrom(9, 0xbd62f4, 15356, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +17:18:17.728073 sendto(8, "bar\n", 4, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 4 +17:18:17.728104 epoll_ctl(3, EPOLL_CTL_MOD, 9, {EPOLLIN|EPOLLRDHUP, {u32=9, u64=9}}) = 0 +17:18:17.728127 epoll_wait(3, [], 200, 1000) = 0 +17:18:18.729411 epoll_wait(3, [], 200, 1000) = 0 +17:18:19.730654 epoll_wait(3, [{EPOLLIN|EPOLLRDHUP, {u32=8, u64=8}}], 200, 1000) = 1 +17:18:20.299268 recvfrom(8, "", 16384, 0, NULL, NULL) = 0 +17:18:20.299336 epoll_ctl(3, EPOLL_CTL_DEL, 8, 0x7ff3a969f7d0) = 0 +17:18:20.299379 epoll_wait(3, [], 200, 0) = 0 +17:18:20.299401 shutdown(9, SHUT_WR) = 0 +17:18:20.299523 epoll_wait(3, [{EPOLLIN|EPOLLHUP|EPOLLRDHUP, {u32=9, u64=9}}], 200, 1000) = 1 +17:18:20.299678 recvfrom(9, "", 16384, 0, NULL, NULL) = 0 +17:18:20.299761 epoll_wait(3, [], 200, 0) = 0 + +Notes: server sent the response in two parts ("bar" then EOF) just due to +netcat's implementation. The second epoll_wait() caught it. + +Here we clearly see that : + - read0 alone returns EPOLLIN|EPOLLRDHUP + - read0 after shutw returns EPOLLIN|EPOLLRDHUP|EPOLLHUP + => difference indeed is "cannot write" + + +Local with a delay before closing the server: + +17:30:32.527157 epoll_wait(3, [{EPOLLIN, {u32=5, u64=5}}], 200, 1000) = 1 +17:30:33.216827 accept4(5, {sa_family=AF_INET, sin_port=htons(33908), sin_addr=inet_addr("127.0.0.1")}, [128->16], SOCK_NONBLOCK) = 8 +17:30:33.216957 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:30:33.216984 accept4(5, 0x7ffc1a1fb0c0, [128], SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable) +17:30:33.217071 recvfrom(8, "GET / HTTP/1.0\r\n\r\n\n", 15360, 0, NULL, NULL) = 19 +17:30:33.217115 recvfrom(8, "", 15341, 0, NULL, NULL) = 0 +17:30:33.217135 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 +17:30:33.217176 setsockopt(9, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +17:30:33.217190 connect(9, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) +17:30:33.217233 sendto(9, "GET / HTTP/1.0\r\n\r\n\n", 19, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 19 +17:30:33.217272 shutdown(9, SHUT_WR) = 0 +17:30:33.217318 recvfrom(9, 0x109b2f0, 16384, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +17:30:33.217332 epoll_ctl(3, EPOLL_CTL_ADD, 9, {EPOLLIN|EPOLLRDHUP, {u32=9, u64=9}}) = 0 +17:30:33.217355 epoll_wait(3, [{EPOLLIN|EPOLLHUP|EPOLLRDHUP, {u32=9, u64=9}}], 200, 1000) = 1 +17:30:33.217377 recvfrom(9, "HTTP/1.0 200\r\nContent-length: 0\r\nX-req: size=19, time=0 ms\r\nX-rsp: id=dummy, code=200, cache=1, size=0, time=0 ms (0 real)\r\n\r\n", 16384, 0, NULL, NULL) = 126 +17:30:33.217395 close(9) = 0 +17:30:33.217411 sendto(8, "HTTP/1.0 200\r\nContent-length: 0\r\nX-req: size=19, time=0 ms\r\nX-rsp: id=dummy, code=200, cache=1, size=0, time=0 ms (0 real)\r\n\r\n", 126, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE, NULL, 0) = 126 +17:30:33.217464 close(8) = 0 +17:30:33.217496 epoll_wait(3, [], 200, 0) = 0 + + +Notes: + - RDHUP is properly present while some data remain pending. + - HUP is present since RDHUP + shutw + +It could be concluded that HUP indicates RDHUP+shutw and in no way indicates +the ability to avoid reading. + +Below HUP|ERR|OUT are reported on connection failures, thus WITHOUT read: + +accept4(5, {sa_family=AF_INET, sin_port=htons(39080), sin_addr=inet_addr("127.0.0.1")}, [128->16], SOCK_NONBLOCK) = 8 +setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +accept4(5, 0x7ffffba55730, [128], SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable) +recvfrom(8, "foo\n", 15360, 0, NULL, NULL) = 4 +recvfrom(8, 0x7f634dcfeff4, 15356, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 +fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 +setsockopt(9, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +connect(9, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("10.0.3.82")}, 16) = -1 EINPROGRESS (Operation now in progress) +sendto(9, "foo\n", 4, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily unavailable) +epoll_ctl(3, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLRDHUP, {u32=8, u64=8}}) = 0 +epoll_ctl(3, EPOLL_CTL_ADD, 9, {EPOLLOUT, {u32=9, u64=9}}) = 0 +epoll_wait(3, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=9, u64=9}}], 200, 1000) = 1 +getsockopt(9, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 +recvfrom(9, "", 15360, 0, NULL, NULL) = 0 +close(9) = 0 + + +On a failed connect attempt immediately followed by a failed recv (all flags +set), we can see this: + +socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 +fcntl(8, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 +setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 +connect(8, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("10.0.3.82")}, 16) = -1 EINPROGRESS (Operation now in progress) +recvfrom(8, 0x1084a20, 16384, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +epoll_ctl(3, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=8, u64=8}}) = 0 +epoll_wait(3, [{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP|EPOLLRDHUP, {u32=8, u64=8}}], 200, 1000) = 1 +connect(8, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("10.0.3.82")}, 16) = -1 ECONNREFUSED (Connection refused) +close(8) = 0 + +=> all flags are reported in case of error. + +It's also interesting to note that POLLOUT is still reported after a shutw, +and no send error is ever reported after shutw: + + shutdown(4, SHUT_WR) = 0 + poll([{fd=4, events=POLLIN|POLLOUT|POLLRDHUP}], 1, 0) = 1 ([{fd=4, revents=POLLOUT}]) + +and: + shutdown(4, SHUT_WR) = 0 + sendto(5, "foo", 3, MSG_NOSIGNAL, NULL, 0) = 3 + poll([{fd=4, events=POLLIN|POLLOUT|POLLRDHUP}], 1, 0) = 1 ([{fd=4, revents=POLLIN|POLLOUT}]) + +and: + shutdown(4, SHUT_WR) = 0 + sendto(4, "bar", 3, MSG_NOSIGNAL, NULL, 0) = -1 EPIPE (Broken pipe) + poll([{fd=4, events=POLLIN|POLLOUT|POLLRDHUP}], 1, 0) = 1 ([{fd=4, revents=POLLOUT}]) + + +POLLOUT is still reported after a SHUTWR: + +socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 +setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 +bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 +listen(3, 1000) = 0 +getsockname(3, {sa_family=AF_INET, sin_port=htons(34729), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0 +socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 4 +connect(4, {sa_family=AF_INET, sin_port=htons(34729), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 +accept(3, 0x7ffcd6a68300, [0->16]) = 5 +fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0 +brk(NULL) = 0xc4e000 +brk(0xc6f000) = 0xc6f000 +write(1, "\n", 1 +) = 1 +shutdown(4, SHUT_WR) = 0 +poll([{fd=4, events=POLLIN|POLLOUT|POLLRDHUP}], 1, 0) = 1 ([{fd=4, revents=POLLOUT}]) +write(1, "ret=1 ev={fd:4 ev:4}\n", 21ret=1 ev={fd:4 ev:4} +) = 21 +close(5) = 0 +close(4) = 0 +close(3) = 0 + +Performing a write() on it reports a SIGPIPE: + +shutdown(4, SHUT_WR) = 0 +sendto(4, "bar", 3, MSG_NOSIGNAL, NULL, 0) = -1 EPIPE (Broken pipe) +poll([{fd=4, events=POLLIN|POLLOUT|POLLRDHUP}], 1, 0) = 1 ([{fd=4, revents=POLLOUT}]) + + +On SHUT_RD we see POLLIN|POLLOUT|POLLRDHUP (there's no data pending here) : +shutdown(4, SHUT_RD) = 0 +poll([{fd=4, events=POLLIN|POLLOUT|POLLRDHUP}], 1, 0) = 1 ([{fd=4, revents=POLLIN|POLLOUT|POLLRDHUP}]) + + +What is observed in the end : + - POLLOUT is always reported for anything SHUT_WR even if it would cause a broken pipe, including listeners if they're also SHUT_RD + - POLLHUP is always reported for anything SHUT_WR + having a SHUT_RD pending with or without anything to read, including listeners + - POLLIN is always reported for anything to read or a pending zero + - POLLIN is NOT reported for SHUT_RD listeners, even with pending connections, only OUT+HUP are reported + - POLLIN and POLLRDHUP are always reported after a SHUTR + - POLLERR also enables IN,OUT,HUP,RHUP + + + + + + + +Currently there's a bit of an issue with connect() being too impatient to read: + +16:26:06.818521 connect(9, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) +16:26:06.818558 recvfrom(9, 0x1db9400, 16320, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +16:26:06.818571 epoll_ctl(3, EPOLL_CTL_ADD, 9, {EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=9, u64=9}}) = 0 +16:26:06.818588 epoll_wait(3, [{EPOLLOUT, {u32=9, u64=9}}], 200, 1000) = 1 +16:26:06.818603 connect(9, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 +16:26:06.818617 sendto(9, "GET /?s=10k HTTP/1.1\r\nhost: 127.0.0.1:4445\r\nuser-agent: curl/7.54.1\r\naccept: */*\r\n\r\n", 84, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 84 +16:26:06.818660 epoll_ctl(3, EPOLL_CTL_MOD, 9, {EPOLLIN|EPOLLRDHUP, {u32=9, u64=9}}) = 0 +16:26:06.818696 epoll_wait(3, [{EPOLLIN, {u32=9, u64=9}}], 200, 1000) = 1 +16:26:06.818711 recvfrom(9, "HTTP/1.1 200\r\nContent-length: 10240\r\nX-req: size=84, time=0 ms\r\nX-rsp: id=dummy, code=200, cache=1, size=10240, time=0 ms (0 real)\r\n\r\n89.123456789.12345678\n.123456789.123456789.123456789.123456789.123"..., 16320, 0, NULL, NULL) = 10374 +16:26:06.818735 recvfrom(9, 0x1dd75f6, 5946, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) +16:26:06.818790 epoll_ctl(3, EPOLL_CTL_DEL, 9, 0x7ffa818fd7d0) = 0 +16:26:06.818804 epoll_wait(3, [], 200, 0) = 0 + + + + +This one shows that the error is not definitive, it disappears once it's +been signaled, then only shut remains! Also it's a proof that an error +may well be reported after a shutw, so the r/w error may not be merged +with a shutw since it may appear after an deliberate shutw. + +$ ./contrib/debug/poll -v -c snd,shw -s pol,rcv,pol,rcv,pol,snd,lin,clo -c pol,rcv,pol,rcv,pol,rcv,pol +#### BEGIN #### +cmd #1 stp #1: do_snd(4): ret=3 +cmd #1 stp #2: do_shw(4): ret=0 +cmd #2 stp #0: do_acc(3): ret=5 +cmd #2 stp #1: do_pol(5): ret=1 ev=0x2005 (IN OUT RDHUP) +cmd #2 stp #2: do_rcv(5): ret=3 +cmd #2 stp #3: do_pol(5): ret=1 ev=0x2005 (IN OUT RDHUP) +cmd #2 stp #4: do_rcv(5): ret=0 +cmd #2 stp #5: do_pol(5): ret=1 ev=0x2005 (IN OUT RDHUP) +cmd #2 stp #6: do_snd(5): ret=3 +cmd #2 stp #7: do_lin(5): ret=0 +cmd #2 stp #8: do_clo(5): ret=0 +cmd #3 stp #1: do_pol(4): ret=1 ev=0x201d (IN OUT ERR HUP RDHUP) +cmd #3 stp #2: do_rcv(4): ret=3 +cmd #3 stp #3: do_pol(4): ret=1 ev=0x201d (IN OUT ERR HUP RDHUP) +cmd #3 stp #4: do_rcv(4): ret=-1 (Connection reset by peer) +cmd #3 stp #5: do_pol(4): ret=1 ev=0x2015 (IN OUT HUP RDHUP) +cmd #3 stp #6: do_rcv(4): ret=0 +cmd #3 stp #7: do_pol(4): ret=1 ev=0x2015 (IN OUT HUP RDHUP) +#### END #### diff --git a/doc/internals/notes-polling.txt b/doc/internals/notes-polling.txt new file mode 100644 index 0000000..e7741a6 --- /dev/null +++ b/doc/internals/notes-polling.txt @@ -0,0 +1,192 @@ +2019-09-03 + +u8 fd.state; +u8 fd.ev; + + +ev = one of : + #define FD_POLL_IN 0x01 + #define FD_POLL_PRI 0x02 + #define FD_POLL_OUT 0x04 + #define FD_POLL_ERR 0x08 + #define FD_POLL_HUP 0x10 + +Could we instead have : + + FD_WAIT_IN 0x01 + FD_WAIT_OUT 0x02 + FD_WAIT_PRI 0x04 + FD_SEEN_HUP 0x08 + FD_SEEN_HUP 0x10 + FD_WAIT_CON 0x20 <<= shouldn't this be in the connection itself in fact ? + +=> not needed, covered by the state instead. + +What is missing though is : + - FD_DATA_PENDING -- overlaps with READY_R, OK if passed by pollers only + - FD_EOI_PENDING + - FD_ERR_PENDING + - FD_EOI + - FD_SHW + - FD_ERR + +fd_update_events() could do that : + + if ((fd_data_pending|fd_eoi_pending|fd_err_pending) && !(fd_err|fd_eoi)) + may_recv() + + if (fd_send_ok && !(fd_err|fd_shw)) + may_send() + + if (fd_err) + wake() + +the poller could do that : + HUP+OUT => always indicates a failed connect(), it should not lack ERR. Is this err_pending ? + + ERR HUP OUT IN + 0 0 0 0 => nothing + 0 0 0 1 => FD_DATA_PENDING + 0 0 1 0 => FD_SEND_OK + 0 0 1 1 => FD_DATA_PENDING|FD_SEND_OK + 0 1 0 0 => FD_EOI (|FD_SHW) + 0 1 0 1 => FD_DATA_PENDING|FD_EOI_PENDING (|FD_SHW) + 0 1 1 0 => FD_EOI |FD_ERR (|FD_SHW) + 0 1 1 1 => FD_EOI_PENDING (|FD_ERR_PENDING) |FD_DATA_PENDING (|FD_SHW) + 1 X 0 0 => FD_ERR | FD_EOI (|FD_SHW) + 1 X X 1 => FD_ERR_PENDING | FD_EOI_PENDING | FD_DATA_PENDING (|FD_SHW) + 1 X 1 0 => FD_ERR | FD_EOI (|FD_SHW) + + OUT+HUP,OUT+HUP+ERR => FD_ERR + +This reorders to: + + IN ERR HUP OUT + 0 0 0 0 => nothing + 0 0 0 1 => FD_SEND_OK + 0 0 1 0 => FD_EOI (|FD_SHW) + + 0 X 1 1 => FD_ERR | FD_EOI (|FD_SHW) + 0 1 X 0 => FD_ERR | FD_EOI (|FD_SHW) + 0 1 X 1 => FD_ERR | FD_EOI (|FD_SHW) + + 1 0 0 0 => FD_DATA_PENDING + 1 0 0 1 => FD_DATA_PENDING|FD_SEND_OK + 1 0 1 0 => FD_DATA_PENDING|FD_EOI_PENDING (|FD_SHW) + 1 0 1 1 => FD_EOI_PENDING (|FD_ERR_PENDING) |FD_DATA_PENDING (|FD_SHW) + 1 1 X X => FD_ERR_PENDING | FD_EOI_PENDING | FD_DATA_PENDING (|FD_SHW) + +Regarding "|SHW", it's normally useless since it will already have been done, +except on connect() error where this indicates there's no need for SHW. + +FD_EOI and FD_SHW could be part of the state (FD_EV_SHUT_R, FD_EV_SHUT_W). +Then all states having these bit and another one would be transient and need +to resync. We could then have "fd_shut_recv" and "fd_shut_send" to turn these +states. + +The FD's ev then only needs to update EOI_PENDING, ERR_PENDING, ERR, DATA_PENDING. +With this said, these are not exactly polling states either, as err/eoi/shw are +orthogonal to the other states and are required to update them so that the polling +state really is DISABLED in the end. So we need more of an operational status for +the FD containing EOI_PENDING, EOI, ERR_PENDING, ERR, SHW, CLO?. These could be +classified in 3 categories: read:(OPEN, EOI_PENDING, EOI); write:(OPEN,SHW), +ctrl:(OPEN,ERR_PENDING,ERR,CLO). That would be 2 bits for R, 1 for W, 2 for ctrl +or total 5 vs 6 for individual ones, but would be harder to manipulate. + +Proposal: + - rename fdtab[].state to "polling_state" + - rename fdtab[].ev to "status" + +Note: POLLHUP is also reported is a listen() socket has gone in shutdown() +TEMPORARILY! Thus we may not always consider this as a final error. + + +Work hypothesis: + +SHUT RDY ACT + 0 0 0 => disabled + 0 0 1 => active + 0 1 0 => stopped + 0 1 1 => ready + 1 0 0 => final shut + 1 0 1 => shut pending without data + 1 1 0 => shut pending, stopped + 1 1 1 => shut pending + +PB: we can land into final shut if one thread disables the FD while another + one that was waiting on it reports it as shut. Theorically it should be + implicitly ready though, since reported. But if no data is reported, it + will be reportedly shut only. And no event will be reported then. This + might still make sense since it's not active, thus we don't want events. + But it will not be enabled later either in this case so the shut really + risks not to be properly reported. The issue is that there's no difference + between a shut coming from the bottom and a shut coming from the top, and + we need an event to report activity here. Or we may consider that a poller + never leaves a final shut by itself (100) and always reports it as + shut+stop (thus ready) if it was not active. Alternately, if active is + disabled, shut should possibly be ignored, then a poller cannot report + shut. But shut+stopped seems the most suitable as it corresponds to + disabled->stopped transition. + +Now let's add ERR. ERR necessarily implies SHUT as there doesn't seem to be a +valid case of ERR pending without shut pending. + +ERR SHUT RDY ACT + 0 0 0 0 => disabled + 0 0 0 1 => active + 0 0 1 0 => stopped + 0 0 1 1 => ready + + 0 1 0 0 => final shut, no error + 0 1 0 1 => shut pending without data + 0 1 1 0 => shut pending, stopped + 0 1 1 1 => shut pending + + 1 0 X X => invalid + + 1 1 0 0 => final shut, error encountered + 1 1 0 1 => error pending without data + 1 1 1 0 => error pending after data, stopped + 1 1 1 1 => error pending + +So the algorithm for the poller is: + - if (shutdown_pending or error) reported and ACT==0, + report SHUT|RDY or SHUT|ERR|RDY + +For read handlers : + - if (!(flags & (RDY|ACT))) + return + - if (ready) + try_to_read + - if (err) + report error + - if (shut) + read0 + +For write handlers: + - if (!(flags & (RDY|ACT))) + return + - if (err||shut) + report error + - if (ready) + try_to_write + +For listeners: + - if (!(flags & (RDY|ACT))) + return + - if (err||shut) + pause + - if (ready) + try_to_accept + +Kqueue reports events differently, it says EV_EOF() on READ or WRITE, that +we currently map to FD_POLL_HUP and FD_POLL_ERR. Thus kqueue reports only +POLLRDHUP and not POLLHUP, so for now a direct mapping of POLLHUP to +FD_POLL_HUP does NOT imply write closed with kqueue while it does for others. + +Other approach, use the {RD,WR}_{ERR,SHUT,RDY} flags to build a composite +status in each poller and pass this to fd_update_events(). We normally +have enough to be precise, and this latter will rework the events. + +FIXME: Normally on KQUEUE we're supposed to look at kev[].fflags to get the error +on EV_EOF() on read or write. diff --git a/doc/internals/pattern.dia b/doc/internals/pattern.dia Binary files differnew file mode 100644 index 0000000..3d13215 --- /dev/null +++ b/doc/internals/pattern.dia diff --git a/doc/internals/pattern.pdf b/doc/internals/pattern.pdf Binary files differnew file mode 100644 index 0000000..a8d8bc9 --- /dev/null +++ b/doc/internals/pattern.pdf diff --git a/doc/internals/polling-states.fig b/doc/internals/polling-states.fig new file mode 100644 index 0000000..3b2c782 --- /dev/null +++ b/doc/internals/polling-states.fig @@ -0,0 +1,59 @@ +#FIG 3.2 Produced by xfig version 2.3 +Portrait +Center +Metric +A4 +100.00 +Single +-2 +1200 2 +2 1 0 1 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 1125 1350 1125 1800 +2 1 0 1 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 1125 2250 1125 2700 +2 1 0 1 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 1125 3150 1125 3600 +2 1 0 1 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 1575 1800 1575 1350 +2 1 0 1 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 1575 3600 1575 3150 +2 4 0 1 0 7 51 -1 20 0.000 0 0 7 0 0 5 + 1800 1350 900 1350 900 900 1800 900 1800 1350 +2 4 0 1 0 7 51 -1 20 0.000 0 0 7 0 0 5 + 1800 2250 900 2250 900 1800 1800 1800 1800 2250 +2 4 0 1 0 7 51 -1 20 0.000 0 0 7 0 0 5 + 1800 4050 900 4050 900 3600 1800 3600 1800 4050 +2 4 0 1 0 7 51 -1 20 0.000 0 0 7 0 0 5 + 1800 3150 900 3150 900 2700 1800 2700 1800 3150 +2 1 0 1 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 1350 450 1350 900 +2 1 0 1 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 1 1 1.00 90.00 180.00 + 1575 2700 1575 2250 +4 2 0 50 -1 16 8 0.0000 4 105 270 1080 1485 want\001 +4 2 0 50 -1 16 8 0.0000 4 120 255 1035 3285 stop\001 +4 0 0 50 -1 16 8 0.0000 4 105 270 1665 3510 want\001 +4 1 0 50 -1 16 10 0.0000 4 120 735 1350 1080 STOPPED\001 +4 1 0 50 -1 16 10 0.0000 4 120 795 1350 3780 DISABLED\001 +4 1 0 50 -1 16 10 0.0000 4 120 555 1350 2880 ACTIVE\001 +4 1 0 50 -1 16 10 0.0000 4 120 540 1350 1980 READY\001 +4 0 0 50 -1 16 8 0.0000 4 90 210 1665 2565 may\001 +4 2 0 50 -1 16 8 0.0000 4 105 240 1035 2430 cant\001 +4 1 0 50 -1 16 8 0.0000 4 120 240 1350 1260 R,!A\001 +4 1 0 50 -1 16 8 0.0000 4 120 210 1350 2160 R,A\001 +4 1 0 50 -1 16 8 0.0000 4 120 240 1350 3060 !R,A\001 +4 1 0 50 -1 16 8 0.0000 4 120 270 1350 3960 !R,!A\001 +4 0 0 50 -1 16 8 0.0000 4 120 255 1665 1710 stop\001 +4 0 0 50 -1 16 10 0.0000 4 150 855 2520 1125 R=ready flag\001 +4 0 0 50 -1 16 10 0.0000 4 150 885 2520 1290 A=active flag\001 +4 0 0 50 -1 16 10 0.0000 4 150 1365 2520 2475 fd_want sets A flag\001 +4 0 0 50 -1 16 10 0.0000 4 150 1440 2520 2640 fd_stop clears A flag\001 +4 0 0 50 -1 16 10 0.0000 4 150 1905 2520 3300 update() updates the poller.\001 +4 0 0 50 -1 16 10 0.0000 4 150 2190 2520 2970 fd_cant clears R flag (EAGAIN)\001 +4 0 0 50 -1 16 10 0.0000 4 150 2115 2520 3135 fd_rdy sets R flag (poll return)\001 diff --git a/doc/internals/sched.fig b/doc/internals/sched.fig new file mode 100644 index 0000000..4134420 --- /dev/null +++ b/doc/internals/sched.fig @@ -0,0 +1,748 @@ +#FIG 3.2 Produced by xfig version 2.4 +Landscape +Center +Metric +A4 +150.00 +Single +-2 +1200 2 +0 32 #c5ebe1 +0 33 #86c8a2 +0 34 #ffebac +0 35 #cbb366 +0 36 #c7b696 +0 37 #effbff +0 38 #dfcba6 +0 39 #414141 +0 40 #aeaaae +0 41 #595559 +0 42 #414141 +0 43 #868286 +0 44 #bec3be +0 45 #868286 +0 46 #bec3be +0 47 #dfe3df +0 48 #8e8e8e +0 49 #8e8e8e +0 50 #414141 +0 51 #868286 +0 52 #bec3be +0 53 #dfe3df +0 54 #414141 +0 55 #868286 +0 56 #bec3be +0 57 #dfe3df +0 58 #868286 +0 59 #bec3be +0 60 #dfe3df +0 61 #c7b696 +0 62 #effbff +0 63 #dfcba6 +0 64 #c7b696 +0 65 #effbff +0 66 #dfcba6 +0 67 #aeaaae +0 68 #595559 +0 69 #8e8e8e +0 70 #414141 +0 71 #868286 +0 72 #bec3be +0 73 #dfe3df +0 74 #414141 +0 75 #868286 +0 76 #bec3be +0 77 #dfe3df +0 78 #868286 +0 79 #bec3be +0 80 #dfe3df +0 81 #414141 +0 82 #868286 +0 83 #bec3be +0 84 #414141 +0 85 #bec3be +0 86 #dfe3df +0 87 #414141 +0 88 #868286 +0 89 #bec3be +0 90 #8e8e8e +0 91 #414141 +0 92 #868286 +0 93 #bec3be +0 94 #dfe3df +0 95 #414141 +0 96 #868286 +0 97 #bec3be +0 98 #dfe3df +0 99 #bebebe +0 100 #515151 +0 101 #e7e3e7 +0 102 #000049 +0 103 #797979 +0 104 #303430 +0 105 #414541 +0 106 #414141 +0 107 #868286 +0 108 #bec3be +0 109 #dfe3df +0 110 #cfcfcf +0 111 #cfcfcf +0 112 #cfcfcf +0 113 #cfcfcf +0 114 #cfcfcf +0 115 #cfcfcf +0 116 #cfcfcf +0 117 #cfcfcf +0 118 #cfcfcf +0 119 #cfcfcf +0 120 #cfcfcf +0 121 #cfcfcf +0 122 #cfcfcf +0 123 #cfcfcf +0 124 #cfcfcf +0 125 #cfcfcf +0 126 #cfcfcf +0 127 #cfcfcf +0 128 #cfcfcf +0 129 #cfcfcf +0 130 #cfcfcf +0 131 #cfcfcf +0 132 #cfcfcf +0 133 #cfcfcf +0 134 #cfcfcf +0 135 #cfcfcf +0 136 #cfcfcf +0 137 #cfcfcf +0 138 #cfcfcf +0 139 #cfcfcf +0 140 #cfcfcf +0 141 #cfcfcf +0 142 #cfcfcf +0 143 #cfcfcf +0 144 #cfcfcf +0 145 #cfcfcf +0 146 #cfcfcf +0 147 #cfcfcf +0 148 #cfcfcf +0 149 #cfcfcf +0 150 #c7c3c7 +0 151 #868286 +0 152 #bec3be +0 153 #dfe3df +0 154 #8e8e8e +0 155 #8e8e8e +0 156 #494549 +0 157 #868686 +0 158 #c7c7c7 +0 159 #e7e7e7 +0 160 #f7f7f7 +0 161 #9e9e9e +0 162 #717571 +0 163 #aeaaae +0 164 #494549 +0 165 #aeaaae +0 166 #595559 +0 167 #bec3be +0 168 #dfe3df +0 169 #494549 +0 170 #616561 +0 171 #494549 +0 172 #868286 +0 173 #bec3be +0 174 #dfe3df +0 175 #bec3be +0 176 #dfe3df +0 177 #c7b696 +0 178 #effbff +0 179 #dfcba6 +0 180 #414141 +0 181 #868286 +0 182 #bec3be +0 183 #dfe3df +0 184 #8e8e8e +0 185 #aeaaae +0 186 #595559 +0 187 #414141 +0 188 #868286 +0 189 #bec3be +0 190 #868286 +0 191 #bec3be +0 192 #dfe3df +0 193 #8e8e8e +0 194 #8e8e8e +0 195 #414141 +0 196 #868286 +0 197 #bec3be +0 198 #dfe3df +0 199 #414141 +0 200 #868286 +0 201 #bec3be +0 202 #dfe3df +0 203 #868286 +0 204 #bec3be +0 205 #dfe3df +0 206 #c7b696 +0 207 #effbff +0 208 #dfcba6 +0 209 #c7b696 +0 210 #effbff +0 211 #dfcba6 +0 212 #aeaaae +0 213 #595559 +0 214 #8e8e8e +0 215 #414141 +0 216 #868286 +0 217 #bec3be +0 218 #dfe3df +0 219 #414141 +0 220 #868286 +0 221 #bec3be +0 222 #dfe3df +0 223 #868286 +0 224 #bec3be +0 225 #dfe3df +0 226 #414141 +0 227 #868286 +0 228 #bec3be +0 229 #414141 +0 230 #bec3be +0 231 #dfe3df +0 232 #414141 +0 233 #868286 +0 234 #bec3be +0 235 #8e8e8e +0 236 #414141 +0 237 #868286 +0 238 #bec3be +0 239 #dfe3df +0 240 #414141 +0 241 #868286 +0 242 #bec3be +0 243 #dfe3df +0 244 #414141 +0 245 #868286 +0 246 #bec3be +0 247 #dfe3df +0 248 #868286 +0 249 #bec3be +0 250 #dfe3df +0 251 #8e8e8e +0 252 #8e8e8e +0 253 #494549 +0 254 #aeaaae +0 255 #494549 +0 256 #aeaaae +0 257 #595559 +0 258 #bec3be +0 259 #dfe3df +0 260 #494549 +0 261 #616561 +0 262 #494549 +0 263 #868286 +0 264 #bec3be +0 265 #dfe3df +0 266 #bec3be +0 267 #dfe3df +0 268 #dfe3ef +0 269 #96969e +0 270 #d7dbd7 +0 271 #9ea2b6 +0 272 #9e0000 +0 273 #efefef +0 274 #86aeff +0 275 #7171ff +0 276 #bbf2e2 +0 277 #a7ceb3 +0 278 #dae8fc +0 279 #458dba +0 280 #ffe6cc +0 281 #e9b000 +0 282 #1a1a1a +0 283 #ffc1e7 +0 284 #009ed7 +0 285 #006d9e +0 286 #00719e +0 287 #9e9a9e +0 288 #000000 +0 289 #595959 +0 290 #006596 +0 291 #00a6d7 +0 292 #b6b6b6 +0 293 #8edbef +0 294 #00699e +0 295 #595d59 +0 296 #69d3e7 +0 297 #a6e3ef +0 298 #9ec7d7 +0 299 #aeb2ae +0 300 #00b6df +0 301 #00aed7 +0 302 #797d79 +0 303 #00a2d7 +0 304 #303030 +0 305 #006996 +0 306 #086d9e +0 307 #86b6cf +0 308 #f7fbf7 +0 309 #9ec3d7 +0 310 #ffff96 +0 311 #ff600a +5 1 0 2 0 7 50 -1 -1 0.000 0 0 1 0 11301.000 3060.000 11205 3825 10530 3060 11205 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 11289.000 3060.000 11385 3825 12060 3060 11385 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 11293.750 3060.000 10890 3105 11700 3060 10890 3015 + 2 1 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 0 1 0 7611.000 3060.000 7515 3825 6840 3060 7515 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 7599.000 3060.000 7695 3825 8370 3060 7695 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 7603.750 3060.000 7200 3105 8010 3060 7200 3015 + 2 1 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 0 1 0 4956.000 3060.000 4860 3825 4185 3060 4860 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 4944.000 3060.000 5040 3825 5715 3060 5040 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 4948.750 3060.000 4545 3105 5355 3060 4545 3015 + 2 1 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 0 1 0 1266.000 3060.000 1170 3825 495 3060 1170 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 1254.000 3060.000 1350 3825 2025 3060 1350 2295 + 0 0 1.00 60.00 120.00 +5 1 0 2 0 7 50 -1 -1 0.000 0 1 0 1 1258.750 3060.000 855 3105 1665 3060 855 3015 + 2 1 1.00 60.00 120.00 +6 10606 2371 11985 3749 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11768 3060 11970 3060 11967 3119 11959 3177 11946 3234 11929 3291 + 11907 3345 11879 3397 11704 3296 11723 3259 11738 3222 11751 3182 + 11760 3142 11765 3101 11768 3060 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11704 3296 11879 3397 11848 3447 11812 3494 11772 3537 11729 3577 + 11682 3613 11633 3644 11531 3469 11566 3447 11599 3422 11628 3393 + 11657 3364 11682 3331 11704 3296 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11531 3469 11633 3644 11580 3672 11526 3694 11469 3711 11412 3724 + 11354 3732 11295 3734 11295 3532 11336 3530 11377 3525 11417 3516 + 11457 3503 11494 3488 11531 3469 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11295 3532 11295 3734 11236 3732 11178 3724 11121 3711 11064 3694 + 11010 3672 10958 3644 11059 3469 11096 3488 11133 3503 11173 3516 + 11213 3525 11254 3530 11295 3532 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11059 3469 10958 3644 10908 3613 10861 3577 10818 3537 10778 3494 + 10742 3447 10711 3398 10886 3296 10908 3331 10933 3364 10962 3393 + 10991 3422 11024 3447 11059 3469 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 10886 3296 10711 3398 10683 3345 10661 3291 10644 3234 10631 3177 + 10623 3119 10621 3060 10823 3060 10825 3101 10830 3142 10839 3182 + 10852 3222 10867 3259 10886 3296 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 10823 3060 10621 3060 10623 3001 10631 2943 10644 2886 10661 2829 + 10683 2775 10711 2723 10886 2824 10867 2861 10852 2898 10839 2938 + 10830 2978 10825 3019 10823 3060 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 10886 2824 10711 2723 10742 2673 10778 2626 10818 2583 10861 2543 + 10908 2507 10958 2476 11059 2651 11024 2673 10991 2698 10962 2727 + 10933 2756 10908 2789 10886 2824 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11059 2651 10958 2476 11010 2448 11064 2426 11121 2409 11178 2396 + 11236 2388 11295 2386 11295 2588 11254 2590 11213 2595 11173 2604 + 11133 2617 11096 2632 11059 2651 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11295 2588 11295 2386 11354 2388 11412 2396 11469 2409 11526 2426 + 11580 2448 11632 2476 11531 2651 11494 2632 11457 2617 11417 2604 + 11377 2595 11336 2590 11295 2588 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11531 2651 11632 2476 11682 2507 11729 2543 11772 2583 11812 2626 + 11848 2673 11879 2723 11704 2824 11682 2789 11657 2756 11628 2727 + 11599 2698 11566 2673 11531 2651 +2 3 0 2 0 31 50 -1 20 0.000 0 0 -1 0 0 15 + 11704 2824 11879 2723 11907 2775 11929 2829 11946 2886 11959 2943 + 11967 3001 11969 3060 11767 3060 11765 3019 11760 2978 11751 2938 + 11738 2898 11723 2861 11704 2824 +-6 +6 4261 2371 5640 3749 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 5423 3060 5625 3060 5622 3119 5614 3177 5601 3234 5584 3291 + 5562 3345 5534 3397 5359 3296 5378 3259 5393 3222 5406 3182 + 5415 3142 5420 3101 5423 3060 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 5359 3296 5534 3397 5503 3447 5467 3494 5427 3537 5384 3577 + 5337 3613 5288 3644 5186 3469 5221 3447 5254 3422 5283 3393 + 5312 3364 5337 3331 5359 3296 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 5186 3469 5288 3644 5235 3672 5181 3694 5124 3711 5067 3724 + 5009 3732 4950 3734 4950 3532 4991 3530 5032 3525 5072 3516 + 5112 3503 5149 3488 5186 3469 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 4950 3532 4950 3734 4891 3732 4833 3724 4776 3711 4719 3694 + 4665 3672 4613 3644 4714 3469 4751 3488 4788 3503 4828 3516 + 4868 3525 4909 3530 4950 3532 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 4714 3469 4613 3644 4563 3613 4516 3577 4473 3537 4433 3494 + 4397 3447 4366 3398 4541 3296 4563 3331 4588 3364 4617 3393 + 4646 3422 4679 3447 4714 3469 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 4541 3296 4366 3398 4338 3345 4316 3291 4299 3234 4286 3177 + 4278 3119 4276 3060 4478 3060 4480 3101 4485 3142 4494 3182 + 4507 3222 4522 3259 4541 3296 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 4478 3060 4276 3060 4278 3001 4286 2943 4299 2886 4316 2829 + 4338 2775 4366 2723 4541 2824 4522 2861 4507 2898 4494 2938 + 4485 2978 4480 3019 4478 3060 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 4541 2824 4366 2723 4397 2673 4433 2626 4473 2583 4516 2543 + 4563 2507 4613 2476 4714 2651 4679 2673 4646 2698 4617 2727 + 4588 2756 4563 2789 4541 2824 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 4714 2651 4613 2476 4665 2448 4719 2426 4776 2409 4833 2396 + 4891 2388 4950 2386 4950 2588 4909 2590 4868 2595 4828 2604 + 4788 2617 4751 2632 4714 2651 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 4950 2588 4950 2386 5009 2388 5067 2396 5124 2409 5181 2426 + 5235 2448 5287 2476 5186 2651 5149 2632 5112 2617 5072 2604 + 5032 2595 4991 2590 4950 2588 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 5186 2651 5287 2476 5337 2507 5384 2543 5427 2583 5467 2626 + 5503 2673 5534 2723 5359 2824 5337 2789 5312 2756 5283 2727 + 5254 2698 5221 2673 5186 2651 +2 3 0 2 0 13 50 -1 20 0.000 0 0 -1 0 0 15 + 5359 2824 5534 2723 5562 2775 5584 2829 5601 2886 5614 2943 + 5622 3001 5624 3060 5422 3060 5420 3019 5415 2978 5406 2938 + 5393 2898 5378 2861 5359 2824 +-6 +6 2250 4815 3960 5265 +1 1 0 3 8 11 52 -1 20 0.000 1 0.0000 3105 5049 810 171 3105 5049 3915 5049 +4 1 0 50 -1 6 10 0.0000 4 150 1125 3105 5130 Most Urgent\001 +-6 +6 6916 2371 8295 3749 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 8078 3060 8280 3060 8277 3119 8269 3177 8256 3234 8239 3291 + 8217 3345 8189 3397 8014 3296 8033 3259 8048 3222 8061 3182 + 8070 3142 8075 3101 8078 3060 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 8014 3296 8189 3397 8158 3447 8122 3494 8082 3537 8039 3577 + 7992 3613 7943 3644 7841 3469 7876 3447 7909 3422 7938 3393 + 7967 3364 7992 3331 8014 3296 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7841 3469 7943 3644 7890 3672 7836 3694 7779 3711 7722 3724 + 7664 3732 7605 3734 7605 3532 7646 3530 7687 3525 7727 3516 + 7767 3503 7804 3488 7841 3469 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7605 3532 7605 3734 7546 3732 7488 3724 7431 3711 7374 3694 + 7320 3672 7268 3644 7369 3469 7406 3488 7443 3503 7483 3516 + 7523 3525 7564 3530 7605 3532 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7369 3469 7268 3644 7218 3613 7171 3577 7128 3537 7088 3494 + 7052 3447 7021 3398 7196 3296 7218 3331 7243 3364 7272 3393 + 7301 3422 7334 3447 7369 3469 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7196 3296 7021 3398 6993 3345 6971 3291 6954 3234 6941 3177 + 6933 3119 6931 3060 7133 3060 7135 3101 7140 3142 7149 3182 + 7162 3222 7177 3259 7196 3296 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7133 3060 6931 3060 6933 3001 6941 2943 6954 2886 6971 2829 + 6993 2775 7021 2723 7196 2824 7177 2861 7162 2898 7149 2938 + 7140 2978 7135 3019 7133 3060 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7196 2824 7021 2723 7052 2673 7088 2626 7128 2583 7171 2543 + 7218 2507 7268 2476 7369 2651 7334 2673 7301 2698 7272 2727 + 7243 2756 7218 2789 7196 2824 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7369 2651 7268 2476 7320 2448 7374 2426 7431 2409 7488 2396 + 7546 2388 7605 2386 7605 2588 7564 2590 7523 2595 7483 2604 + 7443 2617 7406 2632 7369 2651 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7605 2588 7605 2386 7664 2388 7722 2396 7779 2409 7836 2426 + 7890 2448 7942 2476 7841 2651 7804 2632 7767 2617 7727 2604 + 7687 2595 7646 2590 7605 2588 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 7841 2651 7942 2476 7992 2507 8039 2543 8082 2583 8122 2626 + 8158 2673 8189 2723 8014 2824 7992 2789 7967 2756 7938 2727 + 7909 2698 7876 2673 7841 2651 +2 3 0 2 0 31 50 -1 43 0.000 0 0 -1 0 0 15 + 8014 2824 8189 2723 8217 2775 8239 2829 8256 2886 8269 2943 + 8277 3001 8279 3060 8077 3060 8075 3019 8070 2978 8061 2938 + 8048 2898 8033 2861 8014 2824 +-6 +6 571 2371 1950 3749 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1733 3060 1935 3060 1932 3119 1924 3177 1911 3234 1894 3291 + 1872 3345 1844 3397 1669 3296 1688 3259 1703 3222 1716 3182 + 1725 3142 1730 3101 1733 3060 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1669 3296 1844 3397 1813 3447 1777 3494 1737 3537 1694 3577 + 1647 3613 1598 3644 1496 3469 1531 3447 1564 3422 1593 3393 + 1622 3364 1647 3331 1669 3296 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1496 3469 1598 3644 1545 3672 1491 3694 1434 3711 1377 3724 + 1319 3732 1260 3734 1260 3532 1301 3530 1342 3525 1382 3516 + 1422 3503 1459 3488 1496 3469 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1260 3532 1260 3734 1201 3732 1143 3724 1086 3711 1029 3694 + 975 3672 923 3644 1024 3469 1061 3488 1098 3503 1138 3516 + 1178 3525 1219 3530 1260 3532 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1024 3469 923 3644 873 3613 826 3577 783 3537 743 3494 + 707 3447 676 3398 851 3296 873 3331 898 3364 927 3393 + 956 3422 989 3447 1024 3469 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 851 3296 676 3398 648 3345 626 3291 609 3234 596 3177 + 588 3119 586 3060 788 3060 790 3101 795 3142 804 3182 + 817 3222 832 3259 851 3296 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 788 3060 586 3060 588 3001 596 2943 609 2886 626 2829 + 648 2775 676 2723 851 2824 832 2861 817 2898 804 2938 + 795 2978 790 3019 788 3060 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 851 2824 676 2723 707 2673 743 2626 783 2583 826 2543 + 873 2507 923 2476 1024 2651 989 2673 956 2698 927 2727 + 898 2756 873 2789 851 2824 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1024 2651 923 2476 975 2448 1029 2426 1086 2409 1143 2396 + 1201 2388 1260 2386 1260 2588 1219 2590 1178 2595 1138 2604 + 1098 2617 1061 2632 1024 2651 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1260 2588 1260 2386 1319 2388 1377 2396 1434 2409 1491 2426 + 1545 2448 1597 2476 1496 2651 1459 2632 1422 2617 1382 2604 + 1342 2595 1301 2590 1260 2588 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1496 2651 1597 2476 1647 2507 1694 2543 1737 2583 1777 2626 + 1813 2673 1844 2723 1669 2824 1647 2789 1622 2756 1593 2727 + 1564 2698 1531 2673 1496 2651 +2 3 0 2 0 13 50 -1 43 0.000 0 0 -1 0 0 15 + 1669 2824 1844 2723 1872 2775 1894 2829 1911 2886 1924 2943 + 1932 3001 1934 3060 1732 3060 1730 3019 1725 2978 1716 2938 + 1703 2898 1688 2861 1669 2824 +-6 +6 1800 1845 2520 2385 +4 1 0 50 -1 6 10 0.0000 4 120 570 2160 1980 Global\001 +4 1 0 50 -1 6 10 0.0000 4 120 495 2160 2160 tasks\001 +4 1 0 50 -1 6 10 0.0000 4 150 705 2160 2340 (locked)\001 +-6 +6 3960 1935 4500 2250 +4 1 0 50 -1 6 10 0.0000 4 120 495 4230 2250 tasks\001 +4 1 0 50 -1 6 10 0.0000 4 120 465 4230 2070 Local\001 +-6 +6 8190 1845 8910 2385 +4 1 0 50 -1 6 10 0.0000 4 150 705 8550 2340 (locked)\001 +4 1 0 50 -1 6 10 0.0000 4 120 585 8550 2160 timers\001 +4 1 0 50 -1 6 10 0.0000 4 120 570 8550 1980 Global\001 +-6 +6 10215 1935 10845 2250 +4 1 0 50 -1 6 10 0.0000 4 120 585 10530 2250 timers\001 +4 1 0 50 -1 6 10 0.0000 4 120 465 10530 2070 Local\001 +-6 +6 2430 945 3735 1530 +1 1 0 3 20 29 52 -1 20 0.000 1 0.0000 3083 1180 607 170 3083 1180 3690 1350 +4 1 0 50 -1 6 10 0.0000 4 120 615 3105 1260 Local ?\001 +4 0 0 50 -1 6 9 0.0000 4 105 315 3375 1530 Yes\001 +4 2 0 50 -1 6 9 0.0000 4 105 225 2790 1530 No\001 +-6 +6 8775 945 10080 1530 +1 1 0 3 20 29 52 -1 20 0.000 1 0.0000 9428 1180 607 170 9428 1180 10035 1350 +4 1 0 50 -1 6 10 0.0000 4 120 615 9450 1260 Local ?\001 +4 0 0 50 -1 6 9 0.0000 4 105 315 9720 1530 Yes\001 +4 2 0 50 -1 6 9 0.0000 4 105 225 9135 1530 No\001 +-6 +6 7200 6345 9810 6885 +2 1 0 4 279 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 7234 6398 9776 6398 9776 6838 7234 6838 +2 3 0 0 -1 278 49 -1 20 0.000 0 0 -1 0 0 5 + 7234 6838 9776 6838 9776 6398 7234 6398 7234 6838 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9613 6398 9613 6838 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9438 6398 9438 6838 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9264 6398 9264 6838 +4 1 0 46 -1 4 16 0.0000 4 210 1620 8460 6705 TL_URGENT\001 +-6 +6 4140 7830 4545 9045 +1 1 0 3 20 29 52 -1 20 0.000 1 1.5708 4330 8437 607 170 4330 8437 4500 7830 +4 1 0 50 -1 6 10 1.5708 4 120 585 4410 8415 Class?\001 +-6 +1 1 0 3 8 11 52 -1 20 0.000 1 0.0000 9450 5049 540 171 9450 5049 9990 5049 +1 1 0 3 20 29 52 -1 20 0.000 1 1.5708 2440 7672 607 170 2440 7672 2610 7065 +1 1 0 3 8 11 52 -1 20 0.000 1 1.5708 10755 7695 810 171 10755 7695 10755 6885 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 7605 3870 7605 4185 9270 4545 9270 4905 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 11301 3870 11301 4185 9636 4545 9636 4905 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 9630 1395 9626 1591 11291 1800 11295 2295 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 9270 1395 9270 1575 7605 1800 7605 2295 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 2 1 1.00 90.00 180.00 + 9450 360 9450 1035 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 1260 3870 1260 4185 2925 4545 2925 4905 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 4956 3870 4956 4185 3291 4545 3291 4905 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 2 1 1.00 90.00 180.00 + 3105 360 3105 1035 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 3285 1395 3285 1575 4950 1845 4950 2385 +2 1 0 3 22 7 54 -1 -1 0.000 1 0 -1 0 0 2 + 9180 5535 9000 5805 +2 1 0 5 13 7 54 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 120.00 240.00 + 3105 5220 3105 5850 3105 7200 7200 7200 +2 1 0 5 22 7 54 -1 -1 0.000 1 0 -1 1 0 5 + 2 1 1.00 120.00 240.00 + 9450 5220 9450 5670 6300 5670 6300 1215 3690 1170 +2 1 0 3 13 7 54 -1 -1 0.000 1 0 -1 0 0 2 + 3195 5535 3015 5805 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 2925 1395 2925 1575 1260 1845 1260 2385 +2 2 0 3 35 34 100 -1 20 0.000 1 0 -1 0 0 5 + 6570 720 12330 720 12330 5400 6570 5400 6570 720 +2 2 0 3 33 32 100 -1 20 0.000 1 0 -1 0 0 5 + 270 720 6030 720 6030 5400 270 5400 270 720 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 2 1 1.00 90.00 180.00 + 315 7650 2250 7650 +2 1 0 5 4 7 54 -1 -1 0.000 1 0 -1 1 0 2 + 2 1 1.00 120.00 240.00 + 10890 7695 12285 7695 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 3 + 2 1 1.00 90.00 180.00 + 4455 8775 4725 8910 7200 8910 +2 1 0 4 279 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 7234 7118 9776 7118 9776 7558 7234 7558 +2 3 0 0 -1 278 49 -1 20 0.000 0 0 -1 0 0 5 + 7234 7558 9776 7558 9776 7118 7234 7118 7234 7558 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9613 7118 9613 7558 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9438 7118 9438 7558 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9264 7118 9264 7558 +2 3 0 0 -1 278 49 -1 20 0.000 0 0 -1 0 0 5 + 7234 8278 9776 8278 9776 7838 7234 7838 7234 8278 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9613 7838 9613 8278 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9438 7838 9438 8278 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9264 7838 9264 8278 +2 1 0 4 279 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 7234 8558 9776 8558 9776 8998 7234 8998 +2 3 0 0 -1 278 49 -1 20 0.000 0 0 -1 0 0 5 + 7234 8998 9776 8998 9776 8558 7234 8558 7234 8998 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9613 8558 9613 8998 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9438 8558 9438 8998 +2 1 0 2 279 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 9264 8558 9264 8998 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 2 + 2 1 1.00 90.00 180.00 + 6075 6480 7200 6480 +2 1 0 3 0 7 50 -1 -1 0.000 1 0 -1 1 0 3 + 2 1 1.00 90.00 180.00 + 2610 7830 3195 8415 4140 8415 +2 1 0 4 45 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 4166 6398 6094 6398 6094 6838 4166 6838 +2 1 0 2 45 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 5923 6398 5923 6838 +2 1 0 2 45 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 5748 6398 5748 6838 +2 1 0 2 45 -1 47 -1 -1 0.000 0 0 -1 0 0 2 + 5574 6398 5574 6838 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 2610 7515 2925 6660 3645 6660 4140 6660 +2 3 0 0 277 276 49 -1 43 0.000 0 0 -1 0 0 5 + 4166 6838 6094 6838 6094 6398 4166 6398 4166 6838 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 3 + 2 1 1.00 90.00 180.00 + 9765 8775 10350 8775 10665 8280 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 3 + 2 1 1.00 90.00 180.00 + 9765 8055 10305 8055 10620 7875 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 3 + 2 1 1.00 90.00 180.00 + 9806 6605 10350 6615 10665 7155 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 3 + 2 1 1.00 90.00 180.00 + 9720 7335 10350 7335 10620 7560 +2 1 1 5 4 7 57 -1 -1 12.000 1 0 -1 1 0 2 + 2 1 1.00 120.00 240.00 + 9900 6165 9900 9450 +2 1 0 2 0 7 54 -1 -1 0.000 1 0 -1 0 0 2 + 10080 7245 9990 7425 +2 1 0 2 0 7 54 -1 -1 0.000 1 0 -1 0 0 2 + 10080 7965 9990 8145 +2 1 0 2 0 7 54 -1 -1 0.000 1 0 -1 0 0 2 + 10080 8685 9990 8865 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 4 + 2 1 1.00 90.00 180.00 + 4500 8550 6255 8550 6705 8190 7200 8190 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 5 + 2 1 1.00 90.00 180.00 + 4500 8280 4725 8100 6435 8100 6750 7470 7200 7470 +2 1 0 3 0 7 54 -1 -1 0.000 1 0 -1 1 0 5 + 2 1 1.00 90.00 180.00 + 4455 8055 4635 7740 6390 7740 6750 6750 7200 6750 +2 1 0 4 279 -1 48 -1 -1 0.000 0 0 -1 0 0 4 + 7234 7838 9776 7838 9776 8278 7234 8278 +2 1 0 2 0 7 54 -1 -1 0.000 1 0 -1 0 0 2 + 10080 6525 9990 6705 +2 2 0 3 43 47 100 -1 20 0.000 1 0 -1 0 0 5 + 1935 5985 11070 5985 11070 9585 1935 9585 1935 5985 +4 1 0 50 -1 4 9 1.5708 4 135 315 12240 3060 past\001 +4 1 0 50 -1 4 9 1.5708 4 120 465 10440 3060 future\001 +4 1 0 50 -1 4 9 1.5708 4 135 315 8550 3060 past\001 +4 1 0 50 -1 4 9 1.5708 4 120 465 6750 3060 future\001 +4 1 0 50 -1 6 10 0.0000 4 120 600 9450 5130 Oldest\001 +4 1 0 50 -1 4 9 1.5708 4 105 540 405 3060 newest\001 +4 1 0 50 -1 4 9 1.5708 4 120 450 2205 3060 oldest\001 +4 1 0 50 -1 4 9 1.5708 4 105 540 4095 3060 newest\001 +4 1 0 50 -1 4 9 1.5708 4 120 450 5895 3060 oldest\001 +4 0 0 50 -1 14 10 0.0000 4 135 1470 9135 5850 runqueue-depth\001 +4 0 0 50 -1 14 10 0.0000 4 135 1470 3195 5715 runqueue-depth\001 +4 1 0 50 -1 6 12 0.0000 4 165 1320 9450 3600 Time-based\001 +4 1 0 50 -1 6 12 0.0000 4 195 1395 9450 3780 Wait queues\001 +4 0 0 50 -1 6 12 0.0000 4 195 1050 9000 4005 - 1 global\001 +4 0 0 50 -1 6 12 0.0000 4 195 1605 9000 4185 - 1 per thread\001 +4 1 0 50 -1 6 12 0.0000 4 195 1650 3105 3600 Priority-based\001 +4 1 0 50 -1 6 12 0.0000 4 180 1365 3105 3780 Run queues\001 +4 0 0 50 -1 6 12 0.0000 4 195 1050 2655 4005 - 1 global\001 +4 0 0 50 -1 6 12 0.0000 4 195 1605 2655 4185 - 1 per thread\001 +4 0 0 50 -1 14 10 0.0000 4 135 1365 3240 585 task_wakeup()\001 +4 0 0 50 -1 14 10 0.0000 4 135 1575 9585 630 task_schedule()\001 +4 0 0 50 -1 14 10 0.0000 4 135 1260 9585 450 task_queue()\001 +4 0 0 50 -1 14 10 0.0000 4 135 1680 315 7560 tasklet_wakeup()\001 +4 2 0 50 -1 14 10 0.0000 4 135 1260 12285 7515 t->process()\001 +4 2 4 50 -1 6 12 0.0000 4 150 525 12285 7335 Run!\001 +4 1 0 46 -1 4 16 0.0000 4 210 1695 8460 7425 TL_NORMAL\001 +4 1 0 46 -1 4 16 0.0000 4 210 1200 8460 8145 TL_BULK\001 +4 1 0 46 -1 4 16 0.0000 4 210 1425 8460 8865 TL_HEAVY\001 +4 1 0 46 -1 4 16 0.0000 4 195 1095 4950 6705 SHARED\001 +4 0 0 50 -1 6 9 0.0000 4 105 345 10035 7515 37%\001 +4 0 0 50 -1 6 9 0.0000 4 105 210 10080 8955 =1\001 +4 1 0 50 -1 4 10 0.0000 4 150 2280 5085 6255 (accessed using atomic ops)\001 +4 0 0 50 -1 6 9 0.0000 4 105 345 10035 6795 50%\001 +4 0 0 50 -1 6 9 0.0000 4 105 345 10035 8235 13%\001 +4 2 0 50 -1 6 9 1.5708 4 105 315 2745 8100 Yes\001 +4 1 0 50 -1 6 10 1.5708 4 120 615 2520 7650 Local ?\001 +4 0 0 50 -1 6 9 1.5708 4 105 225 2700 7110 No\001 +4 0 0 50 -1 14 10 0.0000 4 135 1680 4725 8460 TASK_SELF_WAKING\001 +4 0 0 50 -1 14 10 0.0000 4 135 1050 4725 8820 TASK_HEAVY\001 +4 0 0 50 -1 4 10 0.0000 4 165 675 4725 8010 (default)\001 +4 0 0 50 -1 4 10 0.0000 4 150 1290 4725 7650 In I/O or signals\001 +4 1 0 50 -1 6 10 1.5708 4 150 1125 10815 7695 Most Urgent\001 +4 0 4 50 -1 6 10 0.0000 4 120 480 9990 6480 order\001 +4 0 4 50 -1 6 10 0.0000 4 120 420 9990 6300 Scan\001 +4 1 0 50 -1 6 12 0.0000 4 195 9075 6030 9450 5 class-based tasklet queues per thread (one accessible from remote threads)\001 diff --git a/doc/internals/sched.pdf b/doc/internals/sched.pdf Binary files differnew file mode 100644 index 0000000..d1ce3de --- /dev/null +++ b/doc/internals/sched.pdf diff --git a/doc/internals/sched.png b/doc/internals/sched.png Binary files differnew file mode 100644 index 0000000..65c97a1 --- /dev/null +++ b/doc/internals/sched.png diff --git a/doc/internals/sched.svg b/doc/internals/sched.svg new file mode 100644 index 0000000..0fa329a --- /dev/null +++ b/doc/internals/sched.svg @@ -0,0 +1,1204 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Creator: fig2dev Version 3.2.7b --> +<!-- CreationDate: 2021-02-26 17:49:00 --> +<!-- Magnification: 1.57 --> +<svg xmlns="http://www.w3.org/2000/svg" + xmlns:xlink="http://www.w3.org/1999/xlink" + width="1146pt" height="878pt" + viewBox="237 327 12126 9291"> +<g fill="none"> +<!-- Line --> +<rect x="6570" y="720" width="5760" height="4680" fill="#ffebac" + stroke="#cbb366" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<rect x="270" y="720" width="5760" height="4680" fill="#c5ebe1" + stroke="#86c8a2" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<rect x="1935" y="5985" width="9135" height="3600" fill="#dfe3df" + stroke="#868286" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<defs> +<clipPath id="cp0"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 9960,9130 9900,9190 9840,9130 9867,9483 9933,9483z"/> +</clipPath> +</defs> +<polyline points=" 9900,6165 9900,9450" clip-path="url(#cp0)" + stroke="#ff0000" stroke-width="60px" stroke-linejoin="round" stroke-dasharray="120 120"/> +<!-- Forward arrow to point 9900,9450 --> +<polygon points=" 9840,9130 9900,9430 9960,9130 9900,9190 9840,9130" + stroke="#ff0000" stroke-width="8px" stroke-miterlimit="8" fill="#ff0000"/> +<!-- Line --> +<polyline points=" 9180,5535 9000,5805" + stroke="#b000b0" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<defs> +<clipPath id="cp1"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 6880,7140 6940,7200 6880,7260 7233,7233 7233,7167z"/> +</clipPath> +</defs> +<polyline points=" 3105,5220 3105,5850 3105,7200 7200,7200" clip-path="url(#cp1)" + stroke="#00b000" stroke-width="60px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7200,7200 --> +<polygon points=" 6880,7260 7180,7200 6880,7140 6940,7200 6880,7260" + stroke="#00b000" stroke-width="8px" stroke-miterlimit="8" fill="#00b000"/> +<!-- Line --> +<defs> +<clipPath id="cp2"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 4009,1236 3950,1174 4011,1116 3658,1136 3656,1202z"/> +</clipPath> +</defs> +<polyline points=" 9450,5220 9450,5670 6300,5670 6300,1215 3690,1170" clip-path="url(#cp2)" + stroke="#b000b0" stroke-width="60px" stroke-linejoin="round"/> +<!-- Forward arrow to point 3690,1170 --> +<polygon points=" 4011,1116 3710,1170 4009,1236 3950,1174 4011,1116" + stroke="#b000b0" stroke-width="8px" stroke-miterlimit="8" fill="#b000b0"/> +<!-- Line --> +<polyline points=" 3195,5535 3015,5805" + stroke="#00b000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Line --> +<defs> +<clipPath id="cp3"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 11965,7635 12025,7695 11965,7755 12318,7728 12318,7662z"/> +</clipPath> +</defs> +<polyline points=" 10890,7695 12285,7695" clip-path="url(#cp3)" + stroke="#ff0000" stroke-width="60px" stroke-linejoin="round"/> +<!-- Forward arrow to point 12285,7695 --> +<polygon points=" 11965,7755 12265,7695 11965,7635 12025,7695 11965,7755" + stroke="#ff0000" stroke-width="8px" stroke-miterlimit="8" fill="#ff0000"/> +<!-- Line --> +<defs> +<clipPath id="cp4"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 6955,8865 7000,8910 6955,8955 7218,8928 7218,8892z"/> +</clipPath> +</defs> +<polyline points=" 4455,8775 4725,8910 7200,8910" clip-path="url(#cp4)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7200,8910 --> +<polygon points=" 6955,8955 7180,8910 6955,8865 7000,8910 6955,8955" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp5"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 3895,6615 3940,6660 3895,6705 4158,6678 4158,6642z"/> +</clipPath> +</defs> +<polyline points=" 2610,7515 2925,6660 3645,6660 4140,6660" clip-path="url(#cp5)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 4140,6660 --> +<polygon points=" 3895,6705 4120,6660 3895,6615 3940,6660 3895,6705" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp6"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 10496,8462 10558,8449 10572,8511 10690,8274 10659,8255z"/> +</clipPath> +</defs> +<polyline points=" 9765,8775 10350,8775 10665,8280" clip-path="url(#cp6)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 10665,8280 --> +<polygon points=" 10572,8511 10654,8297 10496,8462 10558,8449 10572,8511" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp7"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 10385,7957 10446,7974 10430,8036 10645,7882 10627,7850z"/> +</clipPath> +</defs> +<polyline points=" 9765,8055 10305,8055 10620,7875" clip-path="url(#cp7)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 10620,7875 --> +<polygon points=" 10430,8036 10603,7885 10385,7957 10446,7974 10430,8036" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp8"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 10580,6921 10564,6982 10503,6966 10659,7180 10690,7161z"/> +</clipPath> +</defs> +<polyline points=" 9806,6605 10350,6615 10665,7155" clip-path="url(#cp8)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 10665,7155 --> +<polygon points=" 10503,6966 10655,7138 10580,6921 10564,6982 10503,6966" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp9"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 10461,7369 10466,7432 10403,7438 10622,7585 10645,7558z"/> +</clipPath> +</defs> +<polyline points=" 9720,7335 10350,7335 10620,7560" clip-path="url(#cp9)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 10620,7560 --> +<polygon points=" 10403,7438 10605,7547 10461,7369 10466,7432 10403,7438" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<polyline points=" 10080,7245 9990,7425" + stroke="#000000" stroke-width="15px" stroke-linejoin="round"/> +<!-- Line --> +<polyline points=" 10080,7965 9990,8145" + stroke="#000000" stroke-width="15px" stroke-linejoin="round"/> +<!-- Line --> +<polyline points=" 10080,8685 9990,8865" + stroke="#000000" stroke-width="15px" stroke-linejoin="round"/> +<!-- Line --> +<defs> +<clipPath id="cp10"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 6955,8145 7000,8190 6955,8235 7218,8208 7218,8172z"/> +</clipPath> +</defs> +<polyline points=" 4500,8550 6255,8550 6705,8190 7200,8190" clip-path="url(#cp10)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7200,8190 --> +<polygon points=" 6955,8235 7180,8190 6955,8145 7000,8190 6955,8235" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp11"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 6955,7425 7000,7470 6955,7515 7218,7488 7218,7452z"/> +</clipPath> +</defs> +<polyline points=" 4500,8280 4725,8100 6435,8100 6750,7470 7200,7470" clip-path="url(#cp11)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7200,7470 --> +<polygon points=" 6955,7515 7180,7470 6955,7425 7000,7470 6955,7515" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp12"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 6955,6705 7000,6750 6955,6795 7218,6768 7218,6732z"/> +</clipPath> +</defs> +<polyline points=" 4455,8055 4635,7740 6390,7740 6750,6750 7200,6750" clip-path="url(#cp12)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7200,6750 --> +<polygon points=" 6955,6795 7180,6750 6955,6705 7000,6750 6955,6795" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<polyline points=" 10080,6525 9990,6705" + stroke="#000000" stroke-width="15px" stroke-linejoin="round"/> +<!-- Ellipse --> +<ellipse cx="3105" cy="5049" rx="810" ry="171" fill="#87cfff" + stroke="#00008f" stroke-width="30px"/> +<!-- Ellipse --> +<ellipse cx="3083" cy="1180" rx="607" ry="170" fill="#ffbfbf" + stroke="#d10000" stroke-width="30px"/> +<!-- Ellipse --> +<ellipse cx="9428" cy="1180" rx="607" ry="170" fill="#ffbfbf" + stroke="#d10000" stroke-width="30px"/> +<!-- Ellipse --> +<ellipse transform="translate(4330,8437) rotate(-90)" rx="607" ry="170" fill="#ffbfbf" + stroke="#d10000" stroke-width="30px"/> +<!-- Ellipse --> +<ellipse cx="9450" cy="5049" rx="540" ry="171" fill="#87cfff" + stroke="#00008f" stroke-width="30px"/> +<!-- Ellipse --> +<ellipse transform="translate(2440,7672) rotate(-90)" rx="607" ry="170" fill="#ffbfbf" + stroke="#d10000" stroke-width="30px"/> +<!-- Ellipse --> +<ellipse transform="translate(10755,7695) rotate(-90)" rx="810" ry="171" fill="#87cfff" + stroke="#00008f" stroke-width="30px"/> +<!-- Line --> +<polygon points=" 11768,3060 11970,3060 11967,3119 11959,3177 11946,3234 11929,3291 11907,3345 + 11879,3397 11704,3296 11723,3259 11738,3222 11751,3182 11760,3142 11765,3101 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11704,3296 11879,3397 11848,3447 11812,3494 11772,3537 11729,3577 11682,3613 + 11633,3644 11531,3469 11566,3447 11599,3422 11628,3393 11657,3364 11682,3331 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11531,3469 11633,3644 11580,3672 11526,3694 11469,3711 11412,3724 11354,3732 + 11295,3734 11295,3532 11336,3530 11377,3525 11417,3516 11457,3503 11494,3488 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11295,3532 11295,3734 11236,3732 11178,3724 11121,3711 11064,3694 11010,3672 + 10958,3644 11059,3469 11096,3488 11133,3503 11173,3516 11213,3525 11254,3530 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11059,3469 10958,3644 10908,3613 10861,3577 10818,3537 10778,3494 10742,3447 + 10711,3398 10886,3296 10908,3331 10933,3364 10962,3393 10991,3422 11024,3447 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 10886,3296 10711,3398 10683,3345 10661,3291 10644,3234 10631,3177 10623,3119 + 10621,3060 10823,3060 10825,3101 10830,3142 10839,3182 10852,3222 10867,3259 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 10823,3060 10621,3060 10623,3001 10631,2943 10644,2886 10661,2829 10683,2775 + 10711,2723 10886,2824 10867,2861 10852,2898 10839,2938 10830,2978 10825,3019 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 10886,2824 10711,2723 10742,2673 10778,2626 10818,2583 10861,2543 10908,2507 + 10958,2476 11059,2651 11024,2673 10991,2698 10962,2727 10933,2756 10908,2789 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11059,2651 10958,2476 11010,2448 11064,2426 11121,2409 11178,2396 11236,2388 + 11295,2386 11295,2588 11254,2590 11213,2595 11173,2604 11133,2617 11096,2632 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11295,2588 11295,2386 11354,2388 11412,2396 11469,2409 11526,2426 11580,2448 + 11632,2476 11531,2651 11494,2632 11457,2617 11417,2604 11377,2595 11336,2590 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11531,2651 11632,2476 11682,2507 11729,2543 11772,2583 11812,2626 11848,2673 + 11879,2723 11704,2824 11682,2789 11657,2756 11628,2727 11599,2698 11566,2673 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 11704,2824 11879,2723 11907,2775 11929,2829 11946,2886 11959,2943 11967,3001 + 11969,3060 11767,3060 11765,3019 11760,2978 11751,2938 11738,2898 11723,2861 +" fill="#ffd600" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 5423,3060 5625,3060 5622,3119 5614,3177 5601,3234 5584,3291 5562,3345 5534,3397 + 5359,3296 5378,3259 5393,3222 5406,3182 5415,3142 5420,3101" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 5359,3296 5534,3397 5503,3447 5467,3494 5427,3537 5384,3577 5337,3613 5288,3644 + 5186,3469 5221,3447 5254,3422 5283,3393 5312,3364 5337,3331" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 5186,3469 5288,3644 5235,3672 5181,3694 5124,3711 5067,3724 5009,3732 4950,3734 + 4950,3532 4991,3530 5032,3525 5072,3516 5112,3503 5149,3488" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 4950,3532 4950,3734 4891,3732 4833,3724 4776,3711 4719,3694 4665,3672 4613,3644 + 4714,3469 4751,3488 4788,3503 4828,3516 4868,3525 4909,3530" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 4714,3469 4613,3644 4563,3613 4516,3577 4473,3537 4433,3494 4397,3447 4366,3398 + 4541,3296 4563,3331 4588,3364 4617,3393 4646,3422 4679,3447" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 4541,3296 4366,3398 4338,3345 4316,3291 4299,3234 4286,3177 4278,3119 4276,3060 + 4478,3060 4480,3101 4485,3142 4494,3182 4507,3222 4522,3259" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 4478,3060 4276,3060 4278,3001 4286,2943 4299,2886 4316,2829 4338,2775 4366,2723 + 4541,2824 4522,2861 4507,2898 4494,2938 4485,2978 4480,3019" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 4541,2824 4366,2723 4397,2673 4433,2626 4473,2583 4516,2543 4563,2507 4613,2476 + 4714,2651 4679,2673 4646,2698 4617,2727 4588,2756 4563,2789" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 4714,2651 4613,2476 4665,2448 4719,2426 4776,2409 4833,2396 4891,2388 4950,2386 + 4950,2588 4909,2590 4868,2595 4828,2604 4788,2617 4751,2632" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 4950,2588 4950,2386 5009,2388 5067,2396 5124,2409 5181,2426 5235,2448 5287,2476 + 5186,2651 5149,2632 5112,2617 5072,2604 5032,2595 4991,2590" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 5186,2651 5287,2476 5337,2507 5384,2543 5427,2583 5467,2626 5503,2673 5534,2723 + 5359,2824 5337,2789 5312,2756 5283,2727 5254,2698 5221,2673" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<polygon points=" 5359,2824 5534,2723 5562,2775 5584,2829 5601,2886 5614,2943 5622,3001 5624,3060 + 5422,3060 5420,3019 5415,2978 5406,2938 5393,2898 5378,2861" fill="#00b000" + stroke="#000000" stroke-width="15px"/> +<!-- Text --> +<text xml:space="preserve" x="3105" y="5130" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Most Urgent</text> +<!-- Line --> +<defs> +<polygon points=" 8078,3060 8280,3060 8277,3119 8269,3177 8256,3234 8239,3291 8217,3345 8189,3397 + 8014,3296 8033,3259 8048,3222 8061,3182 8070,3142 8075,3101" id="p0"/> +<pattern id="tile0" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p0" fill="#ffd600"/> +<use xlink:href="#p0" fill="url(#tile0)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 8014,3296 8189,3397 8158,3447 8122,3494 8082,3537 8039,3577 7992,3613 7943,3644 + 7841,3469 7876,3447 7909,3422 7938,3393 7967,3364 7992,3331" id="p1"/> +<pattern id="tile1" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p1" fill="#ffd600"/> +<use xlink:href="#p1" fill="url(#tile1)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7841,3469 7943,3644 7890,3672 7836,3694 7779,3711 7722,3724 7664,3732 7605,3734 + 7605,3532 7646,3530 7687,3525 7727,3516 7767,3503 7804,3488" id="p2"/> +<pattern id="tile2" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p2" fill="#ffd600"/> +<use xlink:href="#p2" fill="url(#tile2)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7605,3532 7605,3734 7546,3732 7488,3724 7431,3711 7374,3694 7320,3672 7268,3644 + 7369,3469 7406,3488 7443,3503 7483,3516 7523,3525 7564,3530" id="p3"/> +<pattern id="tile3" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p3" fill="#ffd600"/> +<use xlink:href="#p3" fill="url(#tile3)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7369,3469 7268,3644 7218,3613 7171,3577 7128,3537 7088,3494 7052,3447 7021,3398 + 7196,3296 7218,3331 7243,3364 7272,3393 7301,3422 7334,3447" id="p4"/> +<pattern id="tile4" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p4" fill="#ffd600"/> +<use xlink:href="#p4" fill="url(#tile4)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7196,3296 7021,3398 6993,3345 6971,3291 6954,3234 6941,3177 6933,3119 6931,3060 + 7133,3060 7135,3101 7140,3142 7149,3182 7162,3222 7177,3259" id="p5"/> +<pattern id="tile5" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p5" fill="#ffd600"/> +<use xlink:href="#p5" fill="url(#tile5)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7133,3060 6931,3060 6933,3001 6941,2943 6954,2886 6971,2829 6993,2775 7021,2723 + 7196,2824 7177,2861 7162,2898 7149,2938 7140,2978 7135,3019" id="p6"/> +<pattern id="tile6" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p6" fill="#ffd600"/> +<use xlink:href="#p6" fill="url(#tile6)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7196,2824 7021,2723 7052,2673 7088,2626 7128,2583 7171,2543 7218,2507 7268,2476 + 7369,2651 7334,2673 7301,2698 7272,2727 7243,2756 7218,2789" id="p7"/> +<pattern id="tile7" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p7" fill="#ffd600"/> +<use xlink:href="#p7" fill="url(#tile7)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7369,2651 7268,2476 7320,2448 7374,2426 7431,2409 7488,2396 7546,2388 7605,2386 + 7605,2588 7564,2590 7523,2595 7483,2604 7443,2617 7406,2632" id="p8"/> +<pattern id="tile8" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p8" fill="#ffd600"/> +<use xlink:href="#p8" fill="url(#tile8)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7605,2588 7605,2386 7664,2388 7722,2396 7779,2409 7836,2426 7890,2448 7942,2476 + 7841,2651 7804,2632 7767,2617 7727,2604 7687,2595 7646,2590" id="p9"/> +<pattern id="tile9" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p9" fill="#ffd600"/> +<use xlink:href="#p9" fill="url(#tile9)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 7841,2651 7942,2476 7992,2507 8039,2543 8082,2583 8122,2626 8158,2673 8189,2723 + 8014,2824 7992,2789 7967,2756 7938,2727 7909,2698 7876,2673" id="p10"/> +<pattern id="tile10" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p10" fill="#ffd600"/> +<use xlink:href="#p10" fill="url(#tile10)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 8014,2824 8189,2723 8217,2775 8239,2829 8256,2886 8269,2943 8277,3001 8279,3060 + 8077,3060 8075,3019 8070,2978 8061,2938 8048,2898 8033,2861" id="p11"/> +<pattern id="tile11" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p11" fill="#ffd600"/> +<use xlink:href="#p11" fill="url(#tile11)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1733,3060 1935,3060 1932,3119 1924,3177 1911,3234 1894,3291 1872,3345 1844,3397 + 1669,3296 1688,3259 1703,3222 1716,3182 1725,3142 1730,3101" id="p12"/> +<pattern id="tile12" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p12" fill="#00b000"/> +<use xlink:href="#p12" fill="url(#tile12)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1669,3296 1844,3397 1813,3447 1777,3494 1737,3537 1694,3577 1647,3613 1598,3644 + 1496,3469 1531,3447 1564,3422 1593,3393 1622,3364 1647,3331" id="p13"/> +<pattern id="tile13" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p13" fill="#00b000"/> +<use xlink:href="#p13" fill="url(#tile13)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1496,3469 1598,3644 1545,3672 1491,3694 1434,3711 1377,3724 1319,3732 1260,3734 + 1260,3532 1301,3530 1342,3525 1382,3516 1422,3503 1459,3488" id="p14"/> +<pattern id="tile14" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p14" fill="#00b000"/> +<use xlink:href="#p14" fill="url(#tile14)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1260,3532 1260,3734 1201,3732 1143,3724 1086,3711 1029,3694 975,3672 923,3644 + 1024,3469 1061,3488 1098,3503 1138,3516 1178,3525 1219,3530" id="p15"/> +<pattern id="tile15" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p15" fill="#00b000"/> +<use xlink:href="#p15" fill="url(#tile15)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1024,3469 923,3644 873,3613 826,3577 783,3537 743,3494 707,3447 676,3398 851,3296 + 873,3331 898,3364 927,3393 956,3422 989,3447" id="p16"/> +<pattern id="tile16" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p16" fill="#00b000"/> +<use xlink:href="#p16" fill="url(#tile16)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 851,3296 676,3398 648,3345 626,3291 609,3234 596,3177 588,3119 586,3060 788,3060 + 790,3101 795,3142 804,3182 817,3222 832,3259" id="p17"/> +<pattern id="tile17" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p17" fill="#00b000"/> +<use xlink:href="#p17" fill="url(#tile17)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 788,3060 586,3060 588,3001 596,2943 609,2886 626,2829 648,2775 676,2723 851,2824 + 832,2861 817,2898 804,2938 795,2978 790,3019" id="p18"/> +<pattern id="tile18" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p18" fill="#00b000"/> +<use xlink:href="#p18" fill="url(#tile18)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 851,2824 676,2723 707,2673 743,2626 783,2583 826,2543 873,2507 923,2476 1024,2651 + 989,2673 956,2698 927,2727 898,2756 873,2789" id="p19"/> +<pattern id="tile19" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p19" fill="#00b000"/> +<use xlink:href="#p19" fill="url(#tile19)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1024,2651 923,2476 975,2448 1029,2426 1086,2409 1143,2396 1201,2388 1260,2386 + 1260,2588 1219,2590 1178,2595 1138,2604 1098,2617 1061,2632" id="p20"/> +<pattern id="tile20" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p20" fill="#00b000"/> +<use xlink:href="#p20" fill="url(#tile20)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1260,2588 1260,2386 1319,2388 1377,2396 1434,2409 1491,2426 1545,2448 1597,2476 + 1496,2651 1459,2632 1422,2617 1382,2604 1342,2595 1301,2590" id="p21"/> +<pattern id="tile21" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p21" fill="#00b000"/> +<use xlink:href="#p21" fill="url(#tile21)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1496,2651 1597,2476 1647,2507 1694,2543 1737,2583 1777,2626 1813,2673 1844,2723 + 1669,2824 1647,2789 1622,2756 1593,2727 1564,2698 1531,2673" id="p22"/> +<pattern id="tile22" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p22" fill="#00b000"/> +<use xlink:href="#p22" fill="url(#tile22)" + stroke="#000000" stroke-width="15px"/> +<!-- Line --> +<defs> +<polygon points=" 1669,2824 1844,2723 1872,2775 1894,2829 1911,2886 1924,2943 1932,3001 1934,3060 + 1732,3060 1730,3019 1725,2978 1716,2938 1703,2898 1688,2861" id="p23"/> +<pattern id="tile23" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#000000" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p23" fill="#00b000"/> +<use xlink:href="#p23" fill="url(#tile23)" + stroke="#000000" stroke-width="15px"/> +<!-- Text --> +<text xml:space="preserve" x="2160" y="1980" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Global</text> +<!-- Text --> +<text xml:space="preserve" x="2160" y="2160" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">tasks</text> +<!-- Text --> +<text xml:space="preserve" x="2160" y="2340" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">(locked)</text> +<!-- Text --> +<text xml:space="preserve" x="4230" y="2250" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">tasks</text> +<!-- Text --> +<text xml:space="preserve" x="4230" y="2070" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Local</text> +<!-- Text --> +<text xml:space="preserve" x="8550" y="2340" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">(locked)</text> +<!-- Text --> +<text xml:space="preserve" x="8550" y="2160" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">timers</text> +<!-- Text --> +<text xml:space="preserve" x="8550" y="1980" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Global</text> +<!-- Text --> +<text xml:space="preserve" x="10530" y="2250" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">timers</text> +<!-- Text --> +<text xml:space="preserve" x="10530" y="2070" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Local</text> +<!-- Text --> +<text xml:space="preserve" x="3105" y="1260" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Local ?</text> +<!-- Text --> +<text xml:space="preserve" x="3375" y="1530" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="start">Yes</text> +<!-- Text --> +<text xml:space="preserve" x="2790" y="1530" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="end">No</text> +<!-- Text --> +<text xml:space="preserve" x="9450" y="1260" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Local ?</text> +<!-- Text --> +<text xml:space="preserve" x="9720" y="1530" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="start">Yes</text> +<!-- Text --> +<text xml:space="preserve" x="9135" y="1530" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="end">No</text> +<!-- Text --> +<g transform="translate(4410,8415) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Class?</text> +</g><!-- Arc --> +<defs> +<clipPath id="cp13"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 11065,2294 11189,2298 11078,2353 11217,2303 11213,2283z"/> +</clipPath> +</defs> +<path d="M 11205,3825 A 771 771 0 0 1 11205 2295" clip-path="url(#cp13)" + stroke="#000000" stroke-width="15px"/> +<!-- Forward arrow to point 11205,2295 --> +<polyline points=" 11065,2294 11189,2298 11078,2353" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp14"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 11065,2294 11189,2298 11078,2353 11217,2303 11213,2283z + M 11525,3826 11401,3822 11512,3767 11373,3817 11377,3837z"/> +</clipPath> +</defs> +<path d="M 11385,3825 A 771 771 0 0 0 11385 2295" clip-path="url(#cp14)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 11385,3825 --> +<polyline points=" 11525,3826 11401,3822 11512,3767" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp15"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 11065,2294 11189,2298 11078,2353 11217,2303 11213,2283z + M 10908,3277 10928,3240 10965,3260 10897,3092 10877,3098z"/> +</clipPath> +</defs> +<path d="M 10890,3105 A 406 406 0 1 0 10890 3015" clip-path="url(#cp15)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 10890,3105 --> +<polygon points=" 10965,3260 10895,3124 10908,3277 10928,3240 10965,3260" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Arc --> +<defs> +<clipPath id="cp16"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 7375,2294 7499,2298 7388,2353 7527,2303 7523,2283z + M 10908,3277 10928,3240 10965,3260 10897,3092 10877,3098z"/> +</clipPath> +</defs> +<path d="M 7515,3825 A 771 771 0 0 1 7515 2295" clip-path="url(#cp16)" + stroke="#000000" stroke-width="15px"/> +<!-- Forward arrow to point 7515,2295 --> +<polyline points=" 7375,2294 7499,2298 7388,2353" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp17"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 7375,2294 7499,2298 7388,2353 7527,2303 7523,2283z + M 7835,3826 7711,3822 7822,3767 7683,3817 7687,3837z"/> +</clipPath> +</defs> +<path d="M 7695,3825 A 771 771 0 0 0 7695 2295" clip-path="url(#cp17)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 7695,3825 --> +<polyline points=" 7835,3826 7711,3822 7822,3767" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp18"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 7375,2294 7499,2298 7388,2353 7527,2303 7523,2283z + M 7218,3277 7238,3240 7275,3260 7207,3092 7187,3098z"/> +</clipPath> +</defs> +<path d="M 7200,3105 A 406 406 0 1 0 7200 3015" clip-path="url(#cp18)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 7200,3105 --> +<polygon points=" 7275,3260 7205,3124 7218,3277 7238,3240 7275,3260" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Arc --> +<defs> +<clipPath id="cp19"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 4720,2294 4844,2298 4733,2353 4872,2303 4868,2283z + M 7218,3277 7238,3240 7275,3260 7207,3092 7187,3098z"/> +</clipPath> +</defs> +<path d="M 4860,3825 A 771 771 0 0 1 4860 2295" clip-path="url(#cp19)" + stroke="#000000" stroke-width="15px"/> +<!-- Forward arrow to point 4860,2295 --> +<polyline points=" 4720,2294 4844,2298 4733,2353" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp20"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 4720,2294 4844,2298 4733,2353 4872,2303 4868,2283z + M 5180,3826 5056,3822 5167,3767 5028,3817 5032,3837z"/> +</clipPath> +</defs> +<path d="M 5040,3825 A 771 771 0 0 0 5040 2295" clip-path="url(#cp20)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 5040,3825 --> +<polyline points=" 5180,3826 5056,3822 5167,3767" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp21"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 4720,2294 4844,2298 4733,2353 4872,2303 4868,2283z + M 4563,3277 4583,3240 4620,3260 4552,3092 4532,3098z"/> +</clipPath> +</defs> +<path d="M 4545,3105 A 406 406 0 1 0 4545 3015" clip-path="url(#cp21)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 4545,3105 --> +<polygon points=" 4620,3260 4550,3124 4563,3277 4583,3240 4620,3260" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Arc --> +<defs> +<clipPath id="cp22"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 1030,2294 1154,2298 1043,2353 1182,2303 1178,2283z + M 4563,3277 4583,3240 4620,3260 4552,3092 4532,3098z"/> +</clipPath> +</defs> +<path d="M 1170,3825 A 771 771 0 0 1 1170 2295" clip-path="url(#cp22)" + stroke="#000000" stroke-width="15px"/> +<!-- Forward arrow to point 1170,2295 --> +<polyline points=" 1030,2294 1154,2298 1043,2353" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp23"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 1030,2294 1154,2298 1043,2353 1182,2303 1178,2283z + M 1490,3826 1366,3822 1477,3767 1338,3817 1342,3837z"/> +</clipPath> +</defs> +<path d="M 1350,3825 A 771 771 0 0 0 1350 2295" clip-path="url(#cp23)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 1350,3825 --> +<polyline points=" 1490,3826 1366,3822 1477,3767" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8"/> +<!-- Arc --> +<defs> +<clipPath id="cp24"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 1030,2294 1154,2298 1043,2353 1182,2303 1178,2283z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<path d="M 855,3105 A 406 406 0 1 0 855 3015" clip-path="url(#cp24)" + stroke="#000000" stroke-width="15px"/> +<!-- Backward arrow to point 855,3105 --> +<polygon points=" 930,3260 860,3124 873,3277 893,3240 930,3260" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp25"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 9315,4660 9270,4705 9225,4660 9252,4923 9288,4923z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 7605,3870 7605,4185 9270,4545 9270,4905" clip-path="url(#cp25)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 9270,4905 --> +<polygon points=" 9225,4660 9270,4885 9315,4660 9270,4705 9225,4660" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp26"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 9681,4660 9636,4705 9591,4660 9618,4923 9654,4923z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 11301,3870 11301,4185 9636,4545 9636,4905" clip-path="url(#cp26)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 9636,4905 --> +<polygon points=" 9591,4660 9636,4885 9681,4660 9636,4705 9591,4660" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp27"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 11338,2050 11293,2095 11248,2051 11277,2313 11313,2313z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 9630,1395 9626,1591 11291,1800 11295,2295" clip-path="url(#cp27)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 11295,2295 --> +<polygon points=" 11248,2051 11295,2275 11338,2050 11293,2095 11248,2051" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp28"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 7650,2050 7605,2095 7560,2050 7587,2313 7623,2313z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 9270,1395 9270,1575 7605,1800 7605,2295" clip-path="url(#cp28)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7605,2295 --> +<polygon points=" 7560,2050 7605,2275 7650,2050 7605,2095 7560,2050" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp29"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 9495,790 9450,835 9405,790 9432,1053 9468,1053z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 9450,360 9450,1035" clip-path="url(#cp29)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 9450,1035 --> +<polygon points=" 9405,790 9450,1015 9495,790 9450,835 9405,790" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp30"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 2970,4660 2925,4705 2880,4660 2907,4923 2943,4923z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 1260,3870 1260,4185 2925,4545 2925,4905" clip-path="url(#cp30)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 2925,4905 --> +<polygon points=" 2880,4660 2925,4885 2970,4660 2925,4705 2880,4660" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp31"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 3336,4660 3291,4705 3246,4660 3273,4923 3309,4923z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 4956,3870 4956,4185 3291,4545 3291,4905" clip-path="url(#cp31)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 3291,4905 --> +<polygon points=" 3246,4660 3291,4885 3336,4660 3291,4705 3246,4660" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp32"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 3150,790 3105,835 3060,790 3087,1053 3123,1053z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 3105,360 3105,1035" clip-path="url(#cp32)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 3105,1035 --> +<polygon points=" 3060,790 3105,1015 3150,790 3105,835 3060,790" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp33"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 4995,2140 4950,2185 4905,2140 4932,2403 4968,2403z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 3285,1395 3285,1575 4950,1845 4950,2385" clip-path="url(#cp33)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 4950,2385 --> +<polygon points=" 4905,2140 4950,2365 4995,2140 4950,2185 4905,2140" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp34"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 1305,2140 1260,2185 1215,2140 1242,2403 1278,2403z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 2925,1395 2925,1575 1260,1845 1260,2385" clip-path="url(#cp34)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 1260,2385 --> +<polygon points=" 1215,2140 1260,2365 1305,2140 1260,2185 1215,2140" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp35"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 2005,7605 2050,7650 2005,7695 2268,7668 2268,7632z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 315,7650 2250,7650" clip-path="url(#cp35)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 2250,7650 --> +<polygon points=" 2005,7695 2230,7650 2005,7605 2050,7650 2005,7695" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp36"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 6955,6435 7000,6480 6955,6525 7218,6498 7218,6462z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 6075,6480 7200,6480" clip-path="url(#cp36)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 7200,6480 --> +<polygon points=" 6955,6525 7180,6480 6955,6435 7000,6480 6955,6525" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Line --> +<defs> +<clipPath id="cp37"> + <path clip-rule="evenodd" d="M 237,327 H 12363 V 9618 H 237 z + M 3895,8370 3940,8415 3895,8460 4158,8433 4158,8397z + M 873,3277 893,3240 930,3260 862,3092 842,3098z"/> +</clipPath> +</defs> +<polyline points=" 2610,7830 3195,8415 4140,8415" clip-path="url(#cp37)" + stroke="#000000" stroke-width="30px" stroke-linejoin="round"/> +<!-- Forward arrow to point 4140,8415 --> +<polygon points=" 3895,8460 4120,8415 3895,8370 3940,8415 3895,8460" + stroke="#000000" stroke-width="8px" stroke-miterlimit="8" fill="#000000"/> +<!-- Text --> +<g transform="translate(12240,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">past</text> +</g><!-- Text --> +<g transform="translate(10440,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">future</text> +</g><!-- Text --> +<g transform="translate(8550,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">past</text> +</g><!-- Text --> +<g transform="translate(6750,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">future</text> +</g><!-- Text --> +<text xml:space="preserve" x="9450" y="5130" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Oldest</text> +<!-- Text --> +<g transform="translate(405,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">newest</text> +</g><!-- Text --> +<g transform="translate(2205,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">oldest</text> +</g><!-- Text --> +<g transform="translate(4095,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">newest</text> +</g><!-- Text --> +<g transform="translate(5895,3060) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="108" text-anchor="middle">oldest</text> +</g><!-- Text --> +<text xml:space="preserve" x="9135" y="5850" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">runqueue-depth</text> +<!-- Text --> +<text xml:space="preserve" x="3195" y="5715" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">runqueue-depth</text> +<!-- Text --> +<text xml:space="preserve" x="9450" y="3600" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="middle">Time-based</text> +<!-- Text --> +<text xml:space="preserve" x="9450" y="3780" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="middle">Wait queues</text> +<!-- Text --> +<text xml:space="preserve" x="9000" y="4005" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="start">- 1 global</text> +<!-- Text --> +<text xml:space="preserve" x="9000" y="4185" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="start">- 1 per thread</text> +<!-- Text --> +<text xml:space="preserve" x="3105" y="3600" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="middle">Priority-based</text> +<!-- Text --> +<text xml:space="preserve" x="3105" y="3780" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="middle">Run queues</text> +<!-- Text --> +<text xml:space="preserve" x="2655" y="4005" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="start">- 1 global</text> +<!-- Text --> +<text xml:space="preserve" x="2655" y="4185" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="start">- 1 per thread</text> +<!-- Text --> +<text xml:space="preserve" x="3240" y="585" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">task_wakeup()</text> +<!-- Text --> +<text xml:space="preserve" x="9585" y="630" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">task_schedule()</text> +<!-- Text --> +<text xml:space="preserve" x="9585" y="450" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">task_queue()</text> +<!-- Text --> +<text xml:space="preserve" x="315" y="7560" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">tasklet_wakeup()</text> +<!-- Text --> +<text xml:space="preserve" x="12285" y="7515" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="end">t->process()</text> +<!-- Text --> +<text xml:space="preserve" x="12285" y="7335" fill="#ff0000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="end">Run!</text> +<!-- Text --> +<text xml:space="preserve" x="10035" y="7515" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="start">37%</text> +<!-- Text --> +<text xml:space="preserve" x="10080" y="8955" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="start">=1</text> +<!-- Text --> +<text xml:space="preserve" x="5085" y="6255" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="120" text-anchor="middle">(accessed using atomic ops)</text> +<!-- Text --> +<text xml:space="preserve" x="10035" y="6795" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="start">50%</text> +<!-- Text --> +<text xml:space="preserve" x="10035" y="8235" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="start">13%</text> +<!-- Text --> +<g transform="translate(2745,8100) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="end">Yes</text> +</g><!-- Text --> +<g transform="translate(2520,7650) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Local ?</text> +</g><!-- Text --> +<g transform="translate(2700,7110) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="108" text-anchor="start">No</text> +</g><!-- Text --> +<text xml:space="preserve" x="4725" y="8460" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">TASK_SELF_WAKING</text> +<!-- Text --> +<text xml:space="preserve" x="4725" y="8820" fill="#000000" font-family="Courier" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">TASK_HEAVY</text> +<!-- Text --> +<text xml:space="preserve" x="4725" y="8010" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="120" text-anchor="start">(default)</text> +<!-- Text --> +<text xml:space="preserve" x="4725" y="7650" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="120" text-anchor="start">In I/O or signals</text> +<!-- Text --> +<g transform="translate(10815,7695) rotate(-90)" > +<text xml:space="preserve" x="0" y="0" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="middle">Most Urgent</text> +</g><!-- Text --> +<text xml:space="preserve" x="9990" y="6480" fill="#ff0000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">order</text> +<!-- Text --> +<text xml:space="preserve" x="9990" y="6300" fill="#ff0000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="120" text-anchor="start">Scan</text> +<!-- Text --> +<text xml:space="preserve" x="6030" y="9450" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="bold" font-size="144" text-anchor="middle">5 class-based tasklet queues per thread (one accessible from remote threads)</text> +<!-- Line --> +<polygon points=" 7234,6838 9776,6838 9776,6398 7234,6398" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 7234,7558 9776,7558 9776,7118 7234,7118" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 7234,8278 9776,8278 9776,7838 7234,7838" fill="#dae8fc"/> +<!-- Line --> +<polygon points=" 7234,8998 9776,8998 9776,8558 7234,8558" fill="#dae8fc"/> +<!-- Line --> +<defs> +<polygon points=" 4166,6838 6094,6838 6094,6398 4166,6398" id="p24"/> +<pattern id="tile24" patternUnits="userSpaceOnUse" + x="0" y="0" width="134" height="67"> +<g stroke-width="7.5" stroke="#a7ceb3" fill="none"> +<path d="M -7,30 73,70 M 61,-3 141,37 M -7,37 73,-3 M 61,70 141,30"/> +</g> +</pattern> +</defs> +<use xlink:href="#p24" fill="#bbf2e2"/> +<use xlink:href="#p24" fill="url(#tile24)"/> +<!-- Line --> +<polyline points=" 7234,6398 9776,6398 9776,6838 7234,6838" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 7234,7118 9776,7118 9776,7558 7234,7558" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 7234,8558 9776,8558 9776,8998 7234,8998" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 4166,6398 6094,6398 6094,6838 4166,6838" + stroke="#868286" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 7234,7838 9776,7838 9776,8278 7234,8278" + stroke="#458dba" stroke-width="45px"/> +<!-- Line --> +<polyline points=" 9613,6398 9613,6838" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9438,6398 9438,6838" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9264,6398 9264,6838" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9613,7118 9613,7558" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9438,7118 9438,7558" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9264,7118 9264,7558" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9613,7838 9613,8278" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9438,7838 9438,8278" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9264,7838 9264,8278" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9613,8558 9613,8998" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9438,8558 9438,8998" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 9264,8558 9264,8998" + stroke="#458dba" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 5923,6398 5923,6838" + stroke="#868286" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 5748,6398 5748,6838" + stroke="#868286" stroke-width="15px"/> +<!-- Line --> +<polyline points=" 5574,6398 5574,6838" + stroke="#868286" stroke-width="15px"/> +<!-- Text --> +<text xml:space="preserve" x="8460" y="6705" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">TL_URGENT</text> +<!-- Text --> +<text xml:space="preserve" x="8460" y="7425" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">TL_NORMAL</text> +<!-- Text --> +<text xml:space="preserve" x="8460" y="8145" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">TL_BULK</text> +<!-- Text --> +<text xml:space="preserve" x="8460" y="8865" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">TL_HEAVY</text> +<!-- Text --> +<text xml:space="preserve" x="4950" y="6705" fill="#000000" font-family="AvantGarde" font-style="normal" font-weight="normal" font-size="192" text-anchor="middle">SHARED</text> +</g> +</svg> diff --git a/doc/internals/ssl_cert.dia b/doc/internals/ssl_cert.dia Binary files differnew file mode 100644 index 0000000..52496a1 --- /dev/null +++ b/doc/internals/ssl_cert.dia diff --git a/doc/internals/stats-v2.txt b/doc/internals/stats-v2.txt new file mode 100644 index 0000000..7d2ae76 --- /dev/null +++ b/doc/internals/stats-v2.txt @@ -0,0 +1,8 @@ + + Qcur Qmax Scur Smax Slim Scum Fin Fout Bin Bout Ereq Econ Ersp Sts Wght Act Bck EChk Down +Frontend - - X maxX Y totX I O I O Q - - - - - - - - +Server X maxX X maxX Y totX I O I O - C R S W A B E D +Server X maxX X maxX Y totX I O I O - C R S W A B E D +Server X maxX X maxX Y totX I O I O - C R S W A B E D +Backend X maxX X maxX Y totX I O I O - C R S totW totA totB totE totD + diff --git a/doc/internals/stconn-close.txt b/doc/internals/stconn-close.txt new file mode 100644 index 0000000..fe1ddca --- /dev/null +++ b/doc/internals/stconn-close.txt @@ -0,0 +1,74 @@ +2023-05-23 - closing states on the stream endpoint descriptor + +This document deals with the current flags on the SE desc: + + - SE_FL_ERR_PENDING: an error was met while sending, but some incoming data + might still be pending. This flag will be promoted to SE_FL_ERROR when the + SE_FL_EOI or SE_FL_EOS flags are set via the standard API (se_fl_set()). + + - SE_FL_ERROR ("ERR"): an error was met, last data were received if any, and no + more progress will happen. + + - SE_FL_EOI ("EOI"): the end of the input message was seen, without implying + an end of the connection nor the end of event reporting for this stream. For + example and end of HTTP request or response will set EOI, after which it's + still possible (in case of a request) to bring an abort or error. Said + differently, the expected end of the message was seen. + + - SE_FL_EOS ("EOS"): the definitive end of the input data was detected. It may + result from an error, an abort, a connection shutdown, and no more receive + events will be reported. + +The different muxes (H1,H2,H3) can face slightly different situations due to +the nature, properties, and limitations of their underlying protocols, and will +set these 3 flags to best translate the lower layer's situation and report it +to the upper layer: + + +-----------+----------------------------------------------------------------- + |ERR EOS EOI| Description per mux + +-----------+----------------------------------------------------------------- + | 0 0 0 | all: transfer still in progress + +-----------+----------------------------------------------------------------- + | 0 0 1 | H1: end of message reached. + | | H2: "ES" flag seen on a frame. + | | H3: not set + +-----------+----------------------------------------------------------------- + | 0 1 0 | H1: not set (*1) + | | H2: not set (*2) + | | H3: RST received before FIN (client stops uploading) + +-----------+----------------------------------------------------------------- + | 0 1 1 | H1: end of message + read0, such as close response or aborted + | | request + | | H2: not set (*2) + | | H3: end of message reached (any subsequent RSTs are ignored) + +-----------+----------------------------------------------------------------- + | 1 0 0 | all: could be used to report a protocol error (ex: invalid chunk + | | encoding, forbidden response header seen from a server). + +-----------+----------------------------------------------------------------- + | 1 0 1 | all: could be used to report an internal error or a downstream + | | protocol error, such as a forbidden header in an HTX block + | | coming from the stream layer or the impossibility to encode + | | a message. Seems unused right now. + +-----------+----------------------------------------------------------------- + | 1 1 0 | H1: truncated client input data before response, or truncated + | | response from the server + | | H2: RST or read0 received before end of input message + | | H3: RST + STOP_SENDING before FIN + +-----------+----------------------------------------------------------------- + | 1 1 1 | H1: error face while sending after end of input message + | | H2: RST or read0 received after end of input message + | | H3: STOP_SENDING received after a frame with FIN + +-----------+----------------------------------------------------------------- + +*1: EOS alone is currently not set by H1, however this situation could best + describe an H1 upload that was interrupted by the client while receiving + an early response, a reused persistent server connection that delivered a + read0 immediately after the request was sent, or a truncated server + response (or possibly one in close mode when no C-L was advertised). Right + now these situations are always accompanied with an ERR flag in addition to + the EOS one. + +*2: H2 doesn't set EOS without ERR because currently the only ways to close a + stream in H2 are by resetting the stream (which conveys an error) or + closing the connection (which renders it unusable in both directions and + prevents from sending as well). diff --git a/doc/internals/stream-sock-states.fig b/doc/internals/stream-sock-states.fig new file mode 100644 index 0000000..79131e5 --- /dev/null +++ b/doc/internals/stream-sock-states.fig @@ -0,0 +1,535 @@ +#FIG 3.2 Produced by xfig version 2.0 +Portrait +Center +Metric +A4 +100.00 +Single +-2 +1200 2 +0 32 #8e8e8e +6 2295 1260 2430 1395 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2363 1328 68 68 2430 1328 2295 1328 +4 1 0 50 -1 18 5 0.0000 4 60 60 2363 1361 1\001 +-6 +6 1845 2295 1980 2430 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 1913 2363 68 68 1980 2363 1845 2363 +4 1 0 50 -1 18 5 0.0000 4 60 60 1913 2396 2\001 +-6 +6 2475 2340 2610 2475 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2543 2408 68 68 2610 2408 2475 2408 +4 1 0 50 -1 18 5 0.0000 4 60 60 2543 2441 9\001 +-6 +6 2835 2610 2970 2745 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2903 2678 68 68 2970 2678 2835 2678 +4 1 0 50 -1 18 5 0.0000 4 60 60 2903 2711 7\001 +-6 +6 3195 2025 3330 2160 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 3263 2093 68 68 3330 2093 3195 2093 +4 1 0 50 -1 18 5 0.0000 4 60 60 3263 2126 8\001 +-6 +6 2745 2160 2880 2295 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2813 2228 68 68 2880 2228 2745 2228 +4 1 0 50 -1 18 5 0.0000 4 60 60 2813 2261 6\001 +-6 +6 990 2700 1125 2835 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 1058 2768 68 68 1125 2768 990 2768 +4 1 0 50 -1 18 5 0.0000 4 60 120 1058 2801 13\001 +-6 +6 1305 2970 1440 3105 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 1373 3038 68 68 1440 3038 1305 3038 +4 1 0 50 -1 18 5 0.0000 4 60 120 1373 3071 12\001 +-6 +6 3105 1710 3240 1845 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 3173 1778 68 68 3240 1778 3105 1778 +4 1 0 50 -1 18 5 0.0000 4 60 120 3173 1811 15\001 +-6 +6 4275 1260 4410 1395 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 1328 68 68 4410 1328 4275 1328 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 1361 1\001 +-6 +6 4275 1440 4410 1575 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 1508 68 68 4410 1508 4275 1508 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 1541 2\001 +-6 +6 4275 1620 4410 1755 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 1688 68 68 4410 1688 4275 1688 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 1721 3\001 +-6 +6 4275 1800 4410 1935 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 1868 68 68 4410 1868 4275 1868 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 1901 4\001 +-6 +6 3240 2835 3375 2970 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 3308 2903 68 68 3375 2903 3240 2903 +4 1 0 50 -1 18 5 0.0000 4 60 120 3308 2936 16\001 +-6 +6 2835 3015 2970 3150 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2903 3083 68 68 2970 3083 2835 3083 +4 1 0 50 -1 18 5 0.0000 4 60 120 2903 3116 17\001 +-6 +6 2295 3195 2430 3330 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2363 3263 68 68 2430 3263 2295 3263 +4 1 0 50 -1 18 5 0.0000 4 60 60 2363 3296 3\001 +-6 +6 1440 4815 1620 4995 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 1508 4883 68 68 1575 4883 1440 4883 +4 1 0 50 -1 18 5 0.0000 4 60 120 1508 4916 19\001 +-6 +6 1800 3960 1980 4140 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 1868 4028 68 68 1935 4028 1800 4028 +4 1 0 50 -1 18 5 0.0000 4 60 120 1868 4061 18\001 +-6 +6 4275 1980 4410 2115 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 2048 68 68 4410 2048 4275 2048 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 2081 5\001 +-6 +6 4275 2340 4410 2475 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 2408 68 68 4410 2408 4275 2408 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 2441 6\001 +-6 +6 4275 2520 4410 2655 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 2588 68 68 4410 2588 4275 2588 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 2621 7\001 +-6 +6 4275 2700 4410 2835 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 2768 68 68 4410 2768 4275 2768 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 2801 8\001 +-6 +6 4275 2880 4410 3015 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 2948 68 68 4410 2948 4275 2948 +4 1 0 50 -1 18 5 0.0000 4 60 60 4343 2981 9\001 +-6 +6 4275 3060 4410 3195 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 3128 68 68 4410 3128 4275 3128 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 3161 10\001 +-6 +6 4275 3240 4410 3375 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 3308 68 68 4410 3308 4275 3308 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 3341 11\001 +-6 +6 4275 3420 4410 3555 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 3488 68 68 4410 3488 4275 3488 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 3521 12\001 +-6 +6 4275 3600 4410 3735 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 3668 68 68 4410 3668 4275 3668 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 3701 13\001 +-6 +6 4275 3960 4410 4095 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 4028 68 68 4410 4028 4275 4028 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 4061 15\001 +-6 +6 4275 4140 4410 4275 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 4208 68 68 4410 4208 4275 4208 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 4241 16\001 +-6 +6 4275 4320 4410 4455 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 4388 68 68 4410 4388 4275 4388 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 4421 17\001 +-6 +6 4275 3780 4455 3960 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 3848 68 68 4410 3848 4275 3848 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 3881 14\001 +-6 +6 4275 4590 4455 4770 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 4658 68 68 4410 4658 4275 4658 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 4691 18\001 +-6 +6 4275 4770 4455 4950 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 4838 68 68 4410 4838 4275 4838 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 4871 19\001 +-6 +6 4275 4950 4455 5130 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 5018 68 68 4410 5018 4275 5018 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 5051 20\001 +-6 +6 1170 3690 1350 3870 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 1238 3758 68 68 1305 3758 1170 3758 +4 1 0 50 -1 18 5 0.0000 4 60 120 1238 3791 11\001 +-6 +6 1530 3555 1710 3735 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 1598 3623 68 68 1665 3623 1530 3623 +4 1 0 50 -1 18 5 0.0000 4 60 120 1598 3656 10\001 +-6 +6 720 4095 900 4275 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 788 4163 68 68 855 4163 720 4163 +4 1 0 50 -1 18 5 0.0000 4 60 120 788 4196 14\001 +-6 +6 855 3645 1035 3825 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 923 3713 68 68 990 3713 855 3713 +4 1 0 50 -1 18 5 0.0000 4 60 120 923 3746 21\001 +-6 +6 4275 5130 4455 5310 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 5198 68 68 4410 5198 4275 5198 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 5231 21\001 +-6 +6 2295 4140 2430 4275 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2363 4208 68 68 2430 4208 2295 4208 +4 1 0 50 -1 18 5 0.0000 4 60 60 2363 4241 4\001 +-6 +6 2475 3870 2655 4050 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2543 3938 68 68 2610 3938 2475 3938 +4 1 0 50 -1 18 5 0.0000 4 60 120 2543 3971 22\001 +-6 +6 4275 5310 4455 5490 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 4343 5378 68 68 4410 5378 4275 5378 +4 1 0 50 -1 18 5 0.0000 4 60 120 4343 5411 22\001 +-6 +6 2295 5625 2430 5760 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2363 5693 68 68 2430 5693 2295 5693 +4 1 0 50 -1 18 5 0.0000 4 60 60 2363 5726 5\001 +-6 +6 2295 6480 2475 6660 +1 4 0 1 0 7 50 -1 -1 0.000 1 0.0000 2363 6548 68 68 2430 6548 2295 6548 +4 1 0 50 -1 18 5 0.0000 4 60 120 2363 6581 20\001 +-6 +1 2 0 1 0 6 50 -1 20 0.000 1 0.0000 1350 4612 225 112 1125 4612 1575 4612 +1 2 0 1 0 6 50 -1 20 0.000 1 0.0000 2250 1912 225 112 2025 1912 2475 1912 +1 2 0 1 0 7 50 -1 20 0.000 1 0.0000 1125 3487 225 112 900 3487 1350 3487 +1 2 0 1 0 7 50 -1 20 0.000 1 0.0000 2250 3712 225 112 2025 3712 2475 3712 +1 2 0 1 0 6 50 -1 20 0.000 1 0.0000 2250 2812 225 112 2025 2812 2475 2812 +1 2 0 1 0 7 50 -1 20 0.000 1 0.0000 3375 2362 225 112 3150 2362 3600 2362 +1 2 0 1 0 7 50 -1 20 0.000 1 0.0000 2250 1012 225 112 2025 1012 2475 1012 +1 2 0 1 0 6 50 -1 20 0.000 1 0.0000 2250 6232 225 112 2025 6232 2475 6232 +1 2 0 1 0 7 50 -1 20 0.000 1 0.0000 2250 5422 225 112 2025 5422 2475 5422 +1 2 0 1 0 7 50 -1 20 0.000 1 0.0000 2250 6997 225 112 2025 6997 2475 6997 +1 2 0 1 0 6 50 -1 20 0.000 1 0.0000 2250 4587 225 112 2025 4587 2475 4587 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 2250 1125 2250 1800 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8910 5805 4500 5805 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 3 + 6885 5900 6930 5990 6975 5810 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 3 + 6885 6570 6930 6660 6975 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5310 5589 5310 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5670 5589 5670 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6030 5589 6030 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6390 5589 6390 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6750 5589 6750 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 7110 5589 7110 6921 +2 1 0 2 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 4950 5589 4950 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8910 6705 4500 6705 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 2250 5535 2250 6120 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 2250 6345 2250 6885 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 4500 5580 8910 5580 8910 6930 4500 6930 4500 5580 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 4500 5580 8910 5580 8910 6930 4500 6930 4500 5580 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8910 6030 4500 6030 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8910 6255 4500 6255 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8910 6480 4500 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5310 5589 5310 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5670 5589 5670 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6030 5589 6030 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6390 5589 6390 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6750 5589 6750 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 7110 5589 7110 6921 +2 1 0 2 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 4950 5589 4950 6921 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 4500 5580 8910 5580 8910 6930 4500 6930 4500 5580 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 5805 4500 5805 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6030 4500 6030 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6255 4500 6255 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6480 4500 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5310 5589 5310 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5670 5589 5670 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6030 5589 6030 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6390 5589 6390 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6750 5589 6750 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 7110 5589 7110 6921 +2 1 0 2 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 4950 5589 4950 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6705 4500 6705 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 5805 4500 5805 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6030 4500 6030 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6255 4500 6255 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6480 4500 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5310 5589 5310 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 5670 5589 5670 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6030 5589 6030 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6390 5589 6390 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 6750 5589 6750 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 7110 5589 7110 6921 +2 1 0 2 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 4950 5589 4950 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8865 6705 4500 6705 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 3 + 7605 5890 7650 5980 7695 5800 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 3 + 7605 6570 7650 6660 7695 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 7470 5589 7470 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 3 + 7965 5890 8010 5980 8055 5800 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 3 + 7965 6570 8010 6660 8055 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 7830 5589 7830 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 3 + 8325 6570 8370 6660 8415 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8190 5589 8190 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8550 5589 8550 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8190 5589 8190 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8550 5589 8550 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8190 5589 8190 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8550 5589 8550 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8190 5589 8190 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2 + 8550 5589 8550 6921 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 1 + 4500 5805 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 1 + 4500 6030 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 1 + 4500 6255 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 1 + 4500 6480 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 1 + 4500 6705 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 2250 2700 2475 2475 2475 2250 2250 2025 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 3375 2250 2925 2025 2475 1935 + 0.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 3375 2475 3375 2700 2475 2835 + 0.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 3420 2475 3420 4320 3150 5850 2475 6165 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 1125 3375 1125 2925 2025 2790 + 0.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 1125 3375 1125 2250 2025 1935 + 0.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 6 + 1 1 1.00 60.00 120.00 + 2475 1890 3825 1800 3825 2520 3825 4500 3150 6075 2475 6210 + 0.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 2250 2025 2025 2250 2025 2475 2250 2700 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 2250 3825 2250 4500 + 0.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 2475 1980 2880 2115 3150 2340 + 0.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 2250 2925 2250 3600 + 0.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 2205 3825 2070 4140 1622 4221 1440 4500 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 1350 4725 1350 4950 1485 5760 2025 6165 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 7 + 1 1 1.00 60.00 120.00 + 1125 4590 720 4455 675 4050 675 3600 675 2250 1350 1800 + 2025 1935 + 0.000 1.000 1.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 3 + 1 1 1.00 60.00 120.00 + 1260 4500 1125 4320 1125 3600 + 0.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 1350 4500 1440 3645 1575 3330 2070 2880 + 0.000 1.000 1.000 0.000 +3 0 0 1 32 7 51 -1 -1 0.000 0 1 0 5 + 1 1 1.00 60.00 120.00 + 1035 3600 990 4365 990 5040 1395 5895 2025 6210 + 0.000 1.000 1.000 1.000 0.000 +3 0 0 1 32 7 51 -1 -1 0.000 0 1 0 5 + 1 1 1.00 60.00 120.00 + 2340 3825 2385 4005 2925 4275 2655 4815 2295 5310 + 0.000 1.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 4 + 1 1 1.00 60.00 120.00 + 2475 2835 3150 3375 3150 5625 2475 6120 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 7 50 -1 -1 0.000 0 1 0 2 + 1 1 1.00 60.00 120.00 + 2250 4725 2250 5310 + 0.000 0.000 +4 0 0 50 -1 14 6 0.0000 4 105 2880 4500 1710 ASS-CON: ssui(): connect_server() == SN_ERR_NONE\001 +4 0 0 50 -1 14 6 0.0000 4 90 540 4500 1350 INI-REQ: \001 +4 0 0 50 -1 14 6 0.0000 4 120 3720 4500 1530 REQ-ASS: prepare_conn_request(): srv_redispatch_connect() == 0\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 2475 2700 4\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 1620 4500 6\001 +4 0 0 50 -1 14 6 0.0000 4 120 3360 4500 1890 CON-EST: sess_update_st_con_tcp(): !timeout && !conn_err\001 +4 0 0 50 -1 14 6 0.0000 4 105 2460 4500 3510 TAR-ASS: ssui(): SI_FL_EXP && SN_ASSIGNED\001 +4 0 0 50 -1 14 6 0.0000 4 105 3420 4500 2970 ASS-REQ: connect_server: conn_retries == 0 && PR_O_REDISP\001 +4 0 0 50 -1 14 6 0.0000 4 120 2460 4500 2610 QUE-REQ: ssui(): !pend_pos && SN_ASSIGNED\001 +4 0 0 50 -1 14 6 0.0000 4 120 2520 4500 2790 QUE-REQ: ssui(): !pend_pos && !SN_ASSIGNED\001 +4 0 0 50 -1 14 6 0.0000 4 120 3300 4500 4230 QUE-CLO: ssui(): pend_pos && (SI_FL_EXP || req_aborted)\001 +4 0 0 50 -1 14 6 0.0000 4 105 2520 4500 3690 TAR-REQ: ssui(): SI_FL_EXP && !SN_ASSIGNED\001 +4 0 0 50 -1 14 6 0.0000 4 120 3960 4500 4545 ASS-CLO: PR_O_REDISP && SN_REDIRECTABLE && perform_http_redirect()\001 +4 0 0 50 -1 14 6 0.0000 4 120 4440 4500 2430 REQ-QUE: prepare_conn_request(): srv_redispatch_connect() != 0 (SI_ST_QUE)\001 +4 0 0 50 -1 14 6 0.0000 4 120 4200 4500 4050 REQ-CLO: prepare_conn_request(): srv_redispatch_connect() != 0 (error)\001 +4 0 0 50 -1 14 6 0.0000 4 105 4320 4500 4410 ASS-CLO: ssui(): connect_server() == SN_ERR_INTERNAL || conn_retries < 0\001 +4 0 0 50 -1 14 6 0.0000 4 120 3120 4500 4680 CON-CER: sess_update_st_con_tcp(): timeout/SI_FL_ERR\001 +4 0 0 50 -1 14 6 0.0000 4 120 3600 4500 4860 CER-CLO: sess_update_st_cer(): (ERR/EXP) && conn_retries < 0\001 +4 0 0 50 -1 14 6 0.0000 4 120 4200 4500 3870 CER-REQ: sess_update_st_cer(): timeout && !conn_retries && PR_O_REDISP\001 +4 0 0 50 -1 14 6 0.0000 4 120 3600 4500 3330 CER-TAR: sess_update_st_cer(): conn_err && conn_retries >= 0\001 +4 0 0 50 -1 14 6 0.0000 4 120 4620 4500 3150 CER-ASS: sess_update_st_cer(): timeout && (conn_retries >= 0 || !PR_O_REDISP)\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 1305 3375 3\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 2430 3600 5\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 3555 2250 2\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 2430 1800 1\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 2430 900 0\001 +4 0 0 50 -1 14 6 0.0000 4 105 3000 4500 2070 EST-DIS: stream_sock_read/write/shutr/shutw: close\001 +4 0 0 50 -1 14 6 0.0000 4 120 1980 4500 2250 EST-DIS: process_session(): error\001 +4 0 0 50 -1 14 6 0.0000 4 120 2100 4500 5040 DIS-CLO: process_session(): cleanup\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 1350 4680 CER\001 +4 1 0 50 -1 14 10 0.0000 4 135 315 2250 1980 REQ\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 1125 3555 TAR\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 2250 2880 ASS\001 +4 1 0 50 -1 14 10 0.0000 4 135 315 3375 2430 QUE\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 2250 3780 CON\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 2250 1080 INI\001 +4 0 0 50 -1 14 6 0.0000 4 120 2820 4500 5220 TAR-CLO: sess_update_stream_int(): client abort\001 +4 0 0 50 -1 14 6 0.0000 4 120 2820 4500 5400 CON-DIS: sess_update_st_con_tcp(): client abort\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 5130 5985 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 5490 5985 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 5850 5985 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 6210 5985 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 6570 5985 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 7290 5985 -\001 +4 1 0 50 -1 16 7 0.0000 4 105 120 4725 5985 fd\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 5130 5760 INI\001 +4 1 0 50 -1 16 7 0.0000 4 105 270 4725 5760 state\001 +4 1 0 50 -1 14 8 0.0000 4 120 225 5490 5760 REQ\001 +4 1 0 50 -1 14 8 0.0000 4 120 225 5850 5760 QUE\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 6210 5760 TAR\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 6570 5760 ASS\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 6930 5760 CON\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 7290 5760 CER\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5850 6210 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5130 6210 0\001 +4 1 0 50 -1 16 7 0.0000 4 90 270 4725 6210 ERR\001 +4 1 0 50 -1 16 7 0.0000 4 90 270 4725 6435 EXP\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5490 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6210 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6570 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6570 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5490 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5130 6435 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5850 6435 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6210 6435 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 7290 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6930 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 7290 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6930 6210 X\001 +4 1 0 50 -1 16 7 0.0000 4 75 240 4725 6660 sess\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 5130 6660 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 5490 6660 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 5850 6660 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 6210 6660 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 6570 6660 -\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 7290 6660 -\001 +4 0 0 50 -1 16 6 0.0000 4 120 5970 675 7335 Note: states painted yellow above are transient ; process_session() will never leave a stream interface in any of those upon return.\001 +4 1 0 50 -1 16 7 0.0000 4 90 330 4725 6840 SHUT\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 7290 6840 -\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6930 6840 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6570 6840 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 6210 6840 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5850 6840 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5490 6840 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 5130 6840 0\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 2250 6300 DIS\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 2250 5490 EST\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 2250 7065 CLO\001 +4 1 0 50 -1 14 10 0.0000 4 105 315 2250 4635 RDY\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 2430 4455 7\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 2430 5310 8\001 +4 0 4 50 -1 14 10 0.0000 4 105 105 2385 6120 9\001 +4 0 4 50 -1 14 10 0.0000 4 105 210 2385 6840 10\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 7650 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 7650 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 7650 6840 0\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 8010 5760 EST\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8010 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8010 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8010 6840 0\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 8370 5985 -\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8370 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8370 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 8730 5760 CLO\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 8370 5760 DIS\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 8730 5985 -\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8730 6210 X\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8730 6435 X\001 +4 1 0 50 -1 14 8 0.0000 4 15 75 8730 6660 -\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8370 6840 1\001 +4 1 0 50 -1 14 8 0.0000 4 90 75 8730 6840 1\001 +4 1 0 50 -1 14 8 0.0000 4 90 225 7650 5760 RDY\001 |
