diff options
Diffstat (limited to 'doc/internals/api/initcalls.txt')
| -rw-r--r-- | doc/internals/api/initcalls.txt | 360 |
1 files changed, 360 insertions, 0 deletions
diff --git a/doc/internals/api/initcalls.txt b/doc/internals/api/initcalls.txt new file mode 100644 index 0000000..30d8737 --- /dev/null +++ b/doc/internals/api/initcalls.txt @@ -0,0 +1,360 @@ +Initialization stages aka how to get your code initialized at the right moment + + +1. Background + +Originally all subsystems were initialized via a dedicated function call +from the huge main() function. Then some code started to become conditional +or a bit more modular and the #ifdef placed there became a mess, resulting +in init code being moved to function constructors in each subsystem's own +file. Then pools of various things were introduced, starting to make the +whole init sequence more complicated due to some forms of internal +dependencies. Later epoll was introduced, requiring a post-fork callback, +and finally threads arrived also requiring some post-thread init/deinit +and allocation, marking the old architecture's last breath. Finally the +whole thing resulted in lots of init code duplication and was simplified +in 1.9 with the introduction of initcalls and initialization stages. + + +2. New architecture + +The new architecture relies on two layers : + - the registration functions + - the INITCALL macros and initialization stages + +The first ones are mostly used to add a callback to a list. The second ones +are used to specify when to call a function. Both are totally independent, +however they are generally combined via another set consisting in the REGISTER +macros which make some registration functions be called at some specific points +during the init sequence. + + +3. Registration functions + +Registration functions never fail. Or more precisely, if they fail it will only +be on out-of-memory condition, and they will cause the process to immediately +exit. As such they do not return any status and the caller doesn't have to care +about their success. + +All available functions are described below in alphanumeric ordering. Please +make sure to respect this ordering when adding new ones. + +- void hap_register_build_opts(const char *str, int must_free) + + This appends the zero-terminated constant string <str> to the list of known + build options that will be reported on the output of "haproxy -vv". A line + feed character ('\n') will automatically be appended after the string when it + is displayed. The <must_free> argument must be zero, unless the string was + allocated by any malloc-compatible function such as malloc()/calloc()/ + realloc()/strdup() or memprintf(), in which case it's better to pass a + non-null value so that the string is freed upon exit. Note that despite the + function's prototype taking a "const char *", the pointer will actually be + cast and freed. The const char* is here to leave more freedom to use consts + when making such options lists. + +- void hap_register_per_thread_alloc(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called when + threads are started, at the beginning of the polling loop. This is also valid + for the main thread and will be called even if threads are disabled, so that + it is guaranteed that this function will be called in any circumstance. Each + thread will first call all these functions exactly once when it starts. Calls + are serialized by the init_mutex, so that locking is not necessary in these + functions. There is no relation between the thread numbers and the callback + ordering. The function is expected to return non-zero on success, or zero on + failure. A failure will make the process emit a succinct error message and + immediately exit. See also hap_register_per_thread_free() for functions + called after these ones. + +- void hap_register_per_thread_deinit(void (*fct)()); + + This adds a call to function <fct> to the list of functions to be called when + threads are gracefully stopped, at the end of the polling loop. This is also + valid for the main thread and will be called even if threads are disabled, so + that it is guaranteed that this function will be called in any circumstance + if the process experiences a soft stop. Each thread will call this function + exactly once when it stops. However contrary to _alloc() and _init(), the + calls are made without any protection, thus if any shared resource if touched + by the function, the function is responsible for protecting it. The reason + behind this is that such resources are very likely to be still in use in one + other thread and that most of the time the functions will in fact only touch + a refcount or deinitialize their private resources. See also + hap_register_per_thread_free() for functions called after these ones. + +- void hap_register_per_thread_free(void (*fct)()); + + This adds a call to function <fct> to the list of functions to be called when + threads are gracefully stopped, at the end of the polling loop, after all calls + to _deinit() callbacks are done for this thread. This is also valid for the + main thread and will be called even if threads are disabled, so that it is + guaranteed that this function will be called in any circumstance if the + process experiences a soft stop. Each thread will call this function exactly + once when it stops. However contrary to _alloc() and _init(), the calls are + made without any protection, thus if any shared resource if touched by the + function, the function is responsible for protecting it. The reason behind + this is that such resources are very likely to be still in use in one other + thread and that most of the time the functions will in fact only touch a + refcount or deinitialize their private resources. See also + hap_register_per_thread_deinit() for functions called before these ones. + +- void hap_register_per_thread_init(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called when + threads are started, at the beginning of the polling loop, right after the + list of _alloc() functions. This is also valid for the main thread and will + be called even if threads are disabled, so that it is guaranteed that this + function will be called in any circumstance. Each thread will call this + function exactly once when it starts, and calls are serialized by the + init_mutex which is held over all _alloc() and _init() calls, so that locking + is not necessary in these functions. In other words for all threads but the + current one, the sequence of _alloc() and _init() calls will be atomic. There + is no relation between the thread numbers and the callback ordering. The + function is expected to return non-zero on success, or zero on failure. A + failure will make the process emit a succinct error message and immediately + exit. See also hap_register_per_thread_alloc() for functions called before + these ones. + +- void hap_register_pre_check(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called at + the step just before the configuration validity checks. This is useful when you + need to create things like it would have been done during the configuration + parsing and where the initialization should continue in the configuration + check. + It could be used for example to generate a proxy with multiple servers using + the configuration parser itself. At this step the trash buffers are allocated. + Threads are not yet started so no protection is required. The function is + expected to return non-zero on success, or zero on failure. A failure will make + the process emit a succinct error message and immediately exit. + +- void hap_register_post_check(int (*fct)()) + + This adds a call to function <fct> to the list of functions to be called at + the end of the configuration validity checks, just at the point where the + program either forks or exits depending whether it's called with "-c" or not. + Such calls are suited for memory allocation or internal table pre-computation + that would preferably not be done on the fly to avoid inducing extra time to + a pure configuration check. Threads are not yet started so no protection is + required. The function is expected to return non-zero on success, or zero on + failure. A failure will make the process emit a succinct error message and + immediately exit. + +- void hap_register_post_deinit(void (*fct)()) + + This adds a call to function <fct> to the list of functions to be called when + freeing the global sections at the end of deinit(), after everything is + stopped. The process is single-threaded at this point, thus these functions + are suitable for releasing configuration elements provided that no other + _deinit() function uses them, i.e. only close/release what is strictly + private to the subsystem. Since such functions are mostly only called during + soft stops (reloads) or failed startups, they tend to experience much less + test coverage than others despite being more exposed, and as such a lot of + care must be taken to test them especially when facing partial subsystem + initializations followed by errors. + +- void hap_register_post_proxy_check(int (*fct)(struct proxy *)) + + This adds a call to function <fct> to the list of functions to be called for + each proxy, after the calls to _post_server_check(). This can allow, for + example, to pre-configure default values for an option in a frontend based on + the "bind" lines or something in a backend based on the "server" lines. It's + worth being aware that such a function must be careful not to waste too much + time in order not to significantly slow down configurations with tens of + thousands of backends. The function is expected to return non-zero on + success, or zero on failure. A failure will make the process emit a succinct + error message and immediately exit. + +- void hap_register_post_server_check(int (*fct)(struct server *)) + + This adds a call to function <fct> to the list of functions to be called for + each server, after the call to check_config_validity(). This can allow, for + example, to preset a health state on a server or to allocate a protocol- + specific memory area. It's worth being aware that such a function must be + careful not to waste too much time in order not to significantly slow down + configurations with tens of thousands of servers. The function is expected + to return non-zero on success, or zero on failure. A failure will make the + process emit a succinct error message and immediately exit. + +- void hap_register_proxy_deinit(void (*fct)(struct proxy *)) + + This adds a call to function <fct> to the list of functions to be called when + freeing the resources during deinit(). These functions will be called as part + of the proxy's resource cleanup. Note that some of the proxy's fields will + already have been freed and others not, so such a function must not use any + information from the proxy that is subject to being released. In particular, + all servers have already been deleted. Since such functions are mostly only + called during soft stops (reloads) or failed startups, they tend to + experience much less test coverage than others despite being more exposed, + and as such a lot of care must be taken to test them especially when facing + partial subsystem initializations followed by errors. It's worth mentioning + that too slow functions could have a significant impact on the configuration + check or exit time especially on large configurations. + +- void hap_register_server_deinit(void (*fct)(struct server *)) + + This adds a call to function <fct> to the list of functions to be called when + freeing the resources during deinit(). These functions will be called as part + of the server's resource cleanup. Note that some of the server's fields will + already have been freed and others not, so such a function must not use any + information from the server that is subject to being released. Since such + functions are mostly only called during soft stops (reloads) or failed + startups, they tend to experience much less test coverage than others despite + being more exposed, and as such a lot of care must be taken to test them + especially when facing partial subsystem initializations followed by errors. + It's worth mentioning that too slow functions could have a significant impact + on the configuration check or exit time especially on large configurations. + + +4. Initialization stages + +In order to offer some guarantees, the startup of the program is split into +several stages. Some callbacks can be placed into each of these stages using +an INITCALL macro, with 0 to 3 arguments, respectively called INITCALL0 to +INITCALL3. These macros must be placed anywhere at the top level of a C file, +preferably at the end so that the referenced symbols have already been met, +but it may also be fine to place them right after the callbacks themselves. + +Such callbacks are referenced into small structures containing a pointer to the +function and 3 arguments. NULL replaces unused arguments. The callbacks are +cast to (void (*)(void *, void *, void *)) and the arguments to (void *). + +The first argument to the INITCALL macro is the initialization stage. The +second one is the callback function, and others if any are the arguments. +The init stage must be among the values of the "init_stage" enum, currently, +and in this execution order: + + - STG_PREPARE : used to preset variables, pre-initialize lookup tables and + pre-initialize list heads + - STG_LOCK : used to pre-initialize locks + - STG_REGISTER : used to register static lists such as keywords + - STG_ALLOC : used to allocate the required structures + - STG_POOL : used to create pools + - STG_INIT : used to initialize subsystems + +Each stage is guaranteed that previous stages have successfully completed. This +means that an INITCALL placed at stage STG_INIT is guaranteed that all pools +were already created and will be usable. Conversely, an INITCALL placed at +stage STG_REGISTER must not rely on any field that requires preliminary +allocation nor initialization. A callback cannot rely on other callbacks of the +same stage, as the execution order within a stage is undefined and essentially +depends on the linking order. + +The STG_REGISTER level is made for run-time linking of the various modules that +compose the executable. Keywords, protocols and various other elements that are +local known to each compilation unit can will be appended into common lists at +boot time. This is why this call is placed just before STG_ALLOC. + +Example: register a very early call to init_log() with no argument, and another + call to cli_register_kw(&cli_kws) much later: + + INITCALL0(STG_PREPARE, init_log); + INITCALL1(STG_REGISTER, cli_register_kw, &cli_kws); + +Technically speaking, each call to such a macro adds a distinct local symbol +whose dynamic name involves the line number. These symbols are placed into a +separate section and the beginning and end section pointers are provided by the +linker. When too old a linker is used, a fallback is applied consisting in +placing them into a linked list which is built by a constructor function for +each initcall (this takes more room). + +Due to the symbols internally using the line number, it is very important not +to place more than one INITCALL per line in the source file. + +It is also strongly recommended that functions and referenced arguments are +static symbols local to the source file, unless they are global registration +functions like in the example above with cli_register_kw(), where only the +argument is a local keywords table. + +INITCALLs do not expect the callback function to return anything and as such +do not perform any error check. As such, they are very similar to constructors +offered by the compiler except that they are segmented in stages. It is thus +the responsibility of the called functions to perform their own error checking +and to exit in case of error. This may change in the future. + + +5. REGISTER family of macros + +The association of INITCALLs and registration functions allows to perform some +early dynamic registration of functions to be used anywhere, as well as values +to be added to existing lists without having to manipulate list elements. For +the sake of simplification, these combinations are available as a set of +REGISTER macros which register calls to certain functions at the appropriate +init stage. Such macros must be used at the top level in a file, just like +INITCALL macros. The following macros are currently supported. Please keep them +alphanumerically ordered: + +- REGISTER_BUILD_OPTS(str) + + Adds the constant string <str> to the list of build options. This is done by + registering a call to hap_register_build_opts(str, 0) at stage STG_REGISTER. + The string will not be freed. + +- REGISTER_CONFIG_POSTPARSER(name, parser) + + Adds a call to function <parser> at the end of the config parsing. The + function is called at the very end of check_config_validity() and may be used + to initialize a subsystem based on global settings for example. This is done + by registering a call to cfg_register_postparser(name, parser) at stage + STG_REGISTER. + +- REGISTER_CONFIG_SECTION(name, parse, post) + + Registers a new config section name <name> which will be parsed by function + <parse> (if not null), and with an optional call to function <post> at the + end of the section. Function <parse> must be of type (int (*parse)(const char + *file, int linenum, char **args, int inv)), and returns 0 on success or an + error code among the ERR_* set on failure. The <post> callback takes no + argument and returns a similar error code. This is achieved by registering a + call to cfg_register_section() with the three arguments at stage + STG_REGISTER. + +- REGISTER_PER_THREAD_ALLOC(fct) + + Registers a call to register_per_thread_alloc(fct) at stage STG_REGISTER. + +- REGISTER_PER_THREAD_DEINIT(fct) + + Registers a call to register_per_thread_deinit(fct) at stage STG_REGISTER. + +- REGISTER_PER_THREAD_FREE(fct) + + Registers a call to register_per_thread_free(fct) at stage STG_REGISTER. + +- REGISTER_PER_THREAD_INIT(fct) + + Registers a call to register_per_thread_init(fct) at stage STG_REGISTER. + +- REGISTER_POOL(ptr, name, size) + + Used internally to declare a new pool. This is made by calling function + create_pool_callback() with these arguments at stage STG_POOL. Do not use it + directly, use either DECLARE_POOL() or DECLARE_STATIC_POOL() instead. + +- REGISTER_PRE_CHECK(fct) + + Registers a call to register_pre_check(fct) at stage STG_REGISTER. + +- REGISTER_POST_CHECK(fct) + + Registers a call to register_post_check(fct) at stage STG_REGISTER. + +- REGISTER_POST_DEINIT(fct) + + Registers a call to register_post_deinit(fct) at stage STG_REGISTER. + +- REGISTER_POST_PROXY_CHECK(fct) + + Registers a call to register_post_proxy_check(fct) at stage STG_REGISTER. + +- REGISTER_POST_SERVER_CHECK(fct) + + Registers a call to register_post_server_check(fct) at stage STG_REGISTER. + +- REGISTER_PROXY_DEINIT(fct) + + Registers a call to register_proxy_deinit(fct) at stage STG_REGISTER. + +- REGISTER_SERVER_DEINIT(fct) + + Registers a call to register_server_deinit(fct) at stage STG_REGISTER. + |