Initialization stages aka how to get your code initialized at the right moment 1. Background Originally all subsystems were initialized via a dedicated function call from the huge main() function. Then some code started to become conditional or a bit more modular and the #ifdef placed there became a mess, resulting in init code being moved to function constructors in each subsystem's own file. Then pools of various things were introduced, starting to make the whole init sequence more complicated due to some forms of internal dependencies. Later epoll was introduced, requiring a post-fork callback, and finally threads arrived also requiring some post-thread init/deinit and allocation, marking the old architecture's last breath. Finally the whole thing resulted in lots of init code duplication and was simplified in 1.9 with the introduction of initcalls and initialization stages. 2. New architecture The new architecture relies on two layers : - the registration functions - the INITCALL macros and initialization stages The first ones are mostly used to add a callback to a list. The second ones are used to specify when to call a function. Both are totally independent, however they are generally combined via another set consisting in the REGISTER macros which make some registration functions be called at some specific points during the init sequence. 3. Registration functions Registration functions never fail. Or more precisely, if they fail it will only be on out-of-memory condition, and they will cause the process to immediately exit. As such they do not return any status and the caller doesn't have to care about their success. All available functions are described below in alphanumeric ordering. Please make sure to respect this ordering when adding new ones. - void hap_register_build_opts(const char *str, int must_free) This appends the zero-terminated constant string to the list of known build options that will be reported on the output of "haproxy -vv". A line feed character ('\n') will automatically be appended after the string when it is displayed. The argument must be zero, unless the string was allocated by any malloc-compatible function such as malloc()/calloc()/ realloc()/strdup() or memprintf(), in which case it's better to pass a non-null value so that the string is freed upon exit. Note that despite the function's prototype taking a "const char *", the pointer will actually be cast and freed. The const char* is here to leave more freedom to use consts when making such options lists. - void hap_register_per_thread_alloc(int (*fct)()) This adds a call to function to the list of functions to be called when threads are started, at the beginning of the polling loop. This is also valid for the main thread and will be called even if threads are disabled, so that it is guaranteed that this function will be called in any circumstance. Each thread will first call all these functions exactly once when it starts. Calls are serialized by the init_mutex, so that locking is not necessary in these functions. There is no relation between the thread numbers and the callback ordering. The function is expected to return non-zero on success, or zero on failure. A failure will make the process emit a succinct error message and immediately exit. See also hap_register_per_thread_free() for functions called after these ones. - void hap_register_per_thread_deinit(void (*fct)()); This adds a call to function to the list of functions to be called when threads are gracefully stopped, at the end of the polling loop. This is also valid for the main thread and will be called even if threads are disabled, so that it is guaranteed that this function will be called in any circumstance if the process experiences a soft stop. Each thread will call this function exactly once when it stops. However contrary to _alloc() and _init(), the calls are made without any protection, thus if any shared resource if touched by the function, the function is responsible for protecting it. The reason behind this is that such resources are very likely to be still in use in one other thread and that most of the time the functions will in fact only touch a refcount or deinitialize their private resources. See also hap_register_per_thread_free() for functions called after these ones. - void hap_register_per_thread_free(void (*fct)()); This adds a call to function to the list of functions to be called when threads are gracefully stopped, at the end of the polling loop, after all calls to _deinit() callbacks are done for this thread. This is also valid for the main thread and will be called even if threads are disabled, so that it is guaranteed that this function will be called in any circumstance if the process experiences a soft stop. Each thread will call this function exactly once when it stops. However contrary to _alloc() and _init(), the calls are made without any protection, thus if any shared resource if touched by the function, the function is responsible for protecting it. The reason behind this is that such resources are very likely to be still in use in one other thread and that most of the time the functions will in fact only touch a refcount or deinitialize their private resources. See also hap_register_per_thread_deinit() for functions called before these ones. - void hap_register_per_thread_init(int (*fct)()) This adds a call to function to the list of functions to be called when threads are started, at the beginning of the polling loop, right after the list of _alloc() functions. This is also valid for the main thread and will be called even if threads are disabled, so that it is guaranteed that this function will be called in any circumstance. Each thread will call this function exactly once when it starts, and calls are serialized by the init_mutex which is held over all _alloc() and _init() calls, so that locking is not necessary in these functions. In other words for all threads but the current one, the sequence of _alloc() and _init() calls will be atomic. There is no relation between the thread numbers and the callback ordering. The function is expected to return non-zero on success, or zero on failure. A failure will make the process emit a succinct error message and immediately exit. See also hap_register_per_thread_alloc() for functions called before these ones. - void hap_register_pre_check(int (*fct)()) This adds a call to function to the list of functions to be called at the step just before the configuration validity checks. This is useful when you need to create things like it would have been done during the configuration parsing and where the initialization should continue in the configuration check. It could be used for example to generate a proxy with multiple servers using the configuration parser itself. At this step the final trash buffers are allocated. Threads are not yet started so no protection is required. The function is expected to return non-zero on success, or zero on failure. A failure will make the process emit a succinct error message and immediately exit. - void hap_register_post_check(int (*fct)()) This adds a call to function to the list of functions to be called at the end of the configuration validity checks, just at the point where the program either forks or exits depending whether it's called with "-c" or not. Such calls are suited for memory allocation or internal table pre-computation that would preferably not be done on the fly to avoid inducing extra time to a pure configuration check. Threads are not yet started so no protection is required. The function is expected to return non-zero on success, or zero on failure. A failure will make the process emit a succinct error message and immediately exit. - void hap_register_post_deinit(void (*fct)()) This adds a call to function to the list of functions to be called when freeing the global sections at the end of deinit(), after everything is stopped. The process is single-threaded at this point, thus these functions are suitable for releasing configuration elements provided that no other _deinit() function uses them, i.e. only close/release what is strictly private to the subsystem. Since such functions are mostly only called during soft stops (reloads) or failed startups, they tend to experience much less test coverage than others despite being more exposed, and as such a lot of care must be taken to test them especially when facing partial subsystem initializations followed by errors. - void hap_register_post_proxy_check(int (*fct)(struct proxy *)) This adds a call to function to the list of functions to be called for each proxy, after the calls to _post_server_check(). This can allow, for example, to pre-configure default values for an option in a frontend based on the "bind" lines or something in a backend based on the "server" lines. It's worth being aware that such a function must be careful not to waste too much time in order not to significantly slow down configurations with tens of thousands of backends. The function is expected to return non-zero on success, or zero on failure. A failure will make the process emit a succinct error message and immediately exit. - void hap_register_post_server_check(int (*fct)(struct server *)) This adds a call to function to the list of functions to be called for each server, after the call to check_config_validity(). This can allow, for example, to preset a health state on a server or to allocate a protocol- specific memory area. It's worth being aware that such a function must be careful not to waste too much time in order not to significantly slow down configurations with tens of thousands of servers. The function is expected to return non-zero on success, or zero on failure. A failure will make the process emit a succinct error message and immediately exit. - void hap_register_proxy_deinit(void (*fct)(struct proxy *)) This adds a call to function to the list of functions to be called when freeing the resources during deinit(). These functions will be called as part of the proxy's resource cleanup. Note that some of the proxy's fields will already have been freed and others not, so such a function must not use any information from the proxy that is subject to being released. In particular, all servers have already been deleted. Since such functions are mostly only called during soft stops (reloads) or failed startups, they tend to experience much less test coverage than others despite being more exposed, and as such a lot of care must be taken to test them especially when facing partial subsystem initializations followed by errors. It's worth mentioning that too slow functions could have a significant impact on the configuration check or exit time especially on large configurations. - void hap_register_server_deinit(void (*fct)(struct server *)) This adds a call to function to the list of functions to be called when freeing the resources during deinit(). These functions will be called as part of the server's resource cleanup. Note that some of the server's fields will already have been freed and others not, so such a function must not use any information from the server that is subject to being released. Since such functions are mostly only called during soft stops (reloads) or failed startups, they tend to experience much less test coverage than others despite being more exposed, and as such a lot of care must be taken to test them especially when facing partial subsystem initializations followed by errors. It's worth mentioning that too slow functions could have a significant impact on the configuration check or exit time especially on large configurations. 4. Initialization stages In order to offer some guarantees, the startup of the program is split into several stages. Some callbacks can be placed into each of these stages using an INITCALL macro, with 0 to 3 arguments, respectively called INITCALL0 to INITCALL3. These macros must be placed anywhere at the top level of a C file, preferably at the end so that the referenced symbols have already been met, but it may also be fine to place them right after the callbacks themselves. Such callbacks are referenced into small structures containing a pointer to the function and 3 arguments. NULL replaces unused arguments. The callbacks are cast to (void (*)(void *, void *, void *)) and the arguments to (void *). The first argument to the INITCALL macro is the initialization stage. The second one is the callback function, and others if any are the arguments. The init stage must be among the values of the "init_stage" enum, currently, and in this execution order: - STG_PREPARE : used to preset variables, pre-initialize lookup tables and pre-initialize list heads - STG_LOCK : used to pre-initialize locks - STG_REGISTER : used to register static lists such as keywords - STG_ALLOC : used to allocate the required structures - STG_POOL : used to create pools - STG_INIT : used to initialize subsystems Each stage is guaranteed that previous stages have successfully completed. This means that an INITCALL placed at stage STG_INIT is guaranteed that all pools were already created and will be usable. Conversely, an INITCALL placed at stage STG_REGISTER must not rely on any field that requires preliminary allocation nor initialization. A callback cannot rely on other callbacks of the same stage, as the execution order within a stage is undefined and essentially depends on the linking order. The STG_REGISTER level is made for run-time linking of the various modules that compose the executable. Keywords, protocols and various other elements that are local known to each compilation unit can will be appended into common lists at boot time. This is why this call is placed just before STG_ALLOC. Note that trash is needed in various functions. Trash is a pool and is allocated during STG_POOL, so it's not permitted to use it before STG_INIT, where it will only use the default size, and may be reallocated later with a different size. Example: register a very early call to init_log() with no argument, and another call to cli_register_kw(&cli_kws) much later: INITCALL0(STG_PREPARE, init_log); INITCALL1(STG_REGISTER, cli_register_kw, &cli_kws); Technically speaking, each call to such a macro adds a distinct local symbol whose dynamic name involves the line number. These symbols are placed into a separate section and the beginning and end section pointers are provided by the linker. When too old a linker is used, a fallback is applied consisting in placing them into a linked list which is built by a constructor function for each initcall (this takes more room). Due to the symbols internally using the line number, it is very important not to place more than one INITCALL per line in the source file. It is also strongly recommended that functions and referenced arguments are static symbols local to the source file, unless they are global registration functions like in the example above with cli_register_kw(), where only the argument is a local keywords table. INITCALLs do not expect the callback function to return anything and as such do not perform any error check. As such, they are very similar to constructors offered by the compiler except that they are segmented in stages. It is thus the responsibility of the called functions to perform their own error checking and to exit in case of error. This may change in the future. 5. REGISTER family of macros The association of INITCALLs and registration functions allows to perform some early dynamic registration of functions to be used anywhere, as well as values to be added to existing lists without having to manipulate list elements. For the sake of simplification, these combinations are available as a set of REGISTER macros which register calls to certain functions at the appropriate init stage. Such macros must be used at the top level in a file, just like INITCALL macros. The following macros are currently supported. Please keep them alphanumerically ordered: - REGISTER_BUILD_OPTS(str) Adds the constant string to the list of build options. This is done by registering a call to hap_register_build_opts(str, 0) at stage STG_REGISTER. The string will not be freed. - REGISTER_CONFIG_POSTPARSER(name, parser) Adds a call to function at the end of the config parsing. The function is called at the very end of check_config_validity() and may be used to initialize a subsystem based on global settings for example. This is done by registering a call to cfg_register_postparser(name, parser) at stage STG_REGISTER. - REGISTER_CONFIG_SECTION(name, parse, post) Registers a new config section name which will be parsed by function (if not null), and with an optional call to function at the end of the section. Function must be of type (int (*parse)(const char *file, int linenum, char **args, int inv)), and returns 0 on success or an error code among the ERR_* set on failure. The callback takes no argument and returns a similar error code. This is achieved by registering a call to cfg_register_section() with the three arguments at stage STG_REGISTER. - REGISTER_PER_THREAD_ALLOC(fct) Registers a call to register_per_thread_alloc(fct) at stage STG_REGISTER. - REGISTER_PER_THREAD_DEINIT(fct) Registers a call to register_per_thread_deinit(fct) at stage STG_REGISTER. - REGISTER_PER_THREAD_FREE(fct) Registers a call to register_per_thread_free(fct) at stage STG_REGISTER. - REGISTER_PER_THREAD_INIT(fct) Registers a call to register_per_thread_init(fct) at stage STG_REGISTER. - REGISTER_POOL(ptr, name, size) Used internally to declare a new pool. This is made by calling function create_pool_callback() with these arguments at stage STG_POOL. Do not use it directly, use either DECLARE_POOL() or DECLARE_STATIC_POOL() instead. - REGISTER_PRE_CHECK(fct) Registers a call to register_pre_check(fct) at stage STG_REGISTER. - REGISTER_POST_CHECK(fct) Registers a call to register_post_check(fct) at stage STG_REGISTER. - REGISTER_POST_DEINIT(fct) Registers a call to register_post_deinit(fct) at stage STG_REGISTER. - REGISTER_POST_PROXY_CHECK(fct) Registers a call to register_post_proxy_check(fct) at stage STG_REGISTER. - REGISTER_POST_SERVER_CHECK(fct) Registers a call to register_post_server_check(fct) at stage STG_REGISTER. - REGISTER_PROXY_DEINIT(fct) Registers a call to register_proxy_deinit(fct) at stage STG_REGISTER. - REGISTER_SERVER_DEINIT(fct) Registers a call to register_server_deinit(fct) at stage STG_REGISTER.