path: root/src/lib/hooks/hooks_maintenance.dox

// Copyright (C) 2013-2020 Internet Systems Consortium, Inc. ("ISC")
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.

// Note: the prefix "hooksmg" to all labels is an abbreviation for "Hooks
// Maintenance Guide" and is used to prevent a clash with symbols in any
// other Doxygen file.

/**
 @page hooksmgMaintenanceGuide Hooks Maintenance Guide

 @section hooksmgIntroduction Introduction

 This document is aimed at Kea maintainers responsible for the hooks
 system.  It provides an overview of the classes that make up the hooks
 framework and notes important aspects of processing.  More detailed
 information can be found in the source code.

 It is assumed that the reader is familiar with the contents of the @ref
 hooksdgDevelopersGuide and the @ref hooksComponentDeveloperGuide.

 @section hooksmgObjects Hooks Framework Objects

 The relationships between the various objects in the hooks framework
 is shown below:

 @image html HooksUml.png "High-Level Class Diagram of the Hooks Framework"

 (To avoid clutter, the @ref hooksmgServerHooks object, used to pass
 information about registered hooks to the components, is not shown on
 the diagram.)

 The hooks framework objects can be split into user-side objects and
 server-side objects.  The former are those objects used or referenced
 by user-written hooks libraries.  The latter are those objects used in
 the hooks framework.

 @subsection hooksmgUserObjects User-Side Objects

 The user-side code is able to access two objects in the framework,
 the @ref hooksmgCalloutHandle and the @ref hooksmgLibraryHandle.
 The @ref hooksmgCalloutHandle is used to pass data between the Kea
 component and the loaded library; the @ref hooksmgLibraryHandle is used
 for registering callouts.

 @subsubsection hooksmgCalloutHandle Callout Handle

 The @ref isc::hooks::CalloutHandle has two functions: passing arguments
 between the Kea component and the user-written library, and storing
 per-request context between library calls.  In both cases the data is
 stored in a @c std::map structure, keyed by argument (or context item) name.
 The actual data is stored in a @c boost::any object, which allows any
 data type to be stored, although a penalty for this flexibility is
 the restriction (mentioned in the @ref hooksdgDevelopersGuide) that
 the type of data retrieved must be identical (and not just compatible)
 with that stored.

 The storage of context data is slightly complex because there is
 separate context for each user library.  For this reason, the @ref
 hooksmgCalloutHandle has multiple maps, one for each library loaded.
 The maps are stored in another map, the appropriate map being identified
 by the "current library index" (this index is explained further below).
 The reason for the second map (rather than a structure such as a vector)
 is to avoid creating individual context maps unless needed; given the
 key to the map (in this case the current library index) accessing an
 element in a map using the operator[] method returns the element in
 question if it exists, or creates a new one (and stores it in the map)
 if its doesn't.

 @subsubsection hooksmgLibraryHandle Library Handle

 Little more than a restricted interface to the @ref
 hooksmgCalloutManager, the @ref isc::hooks::LibraryHandle allows a
 callout to register and deregister callouts.  However, there are some
 quirks to callout registration which, although the processing involved
 is in the @ref hooksmgCalloutManager, are best described here.

 Firstly, a callout can be deregistered by a function within a user
 library only if it was registered by a function within that library. That
 is to say, if library A registers the callout A_func() on hook "alpha"
 and library B registers B_func(), functions within library A are only
 able to remove A_func() (and functions in library B remove B_func()).
 The restriction - here to prevent one library interfering with the
 callouts of another - is enforced by means of the current library index.
 As described below, each entry in the vector of callouts associated with
 a hook is a pair object, comprising a pointer to the callout and
 the index of the library with which it is associated. A callout
 can only modify entries in that vector where the current library index
 matches the index element of the pair.

 A second quirk is that when dynamically modifying the list of callouts,
 the change only takes effect when the current call out from the server
 completes. To clarify this, suppose that functions A_func(), B_func()
 and C_func() are registered on a hook, and the server executes a callout
 on the hook. Suppose also during this call, A_func() removes the callout
 C_func() and that B_func() adds D_func(). As changes only take effect
 when the current call out completes, the user callouts executed will be
 A_func(), B_func() then C_func(). When the server calls the hook callouts
 again, the functions executed will be A_func(), B_func() and D_func().

 This restriction is down to implementation.  When a set of callouts on a hook
 is being called, the @ref hooksmgCalloutManager iterates through a
 vector (the "callout vector") of (index, callout pointer) pairs.  Since
 registration or deregistration of a callout on that hook would change the
 vector (and so potentially invalidate the iterators used to access the it),
 a copy of the vector is taken before the iteration starts.  The @ref
 hooksmgCalloutManager iterates over this copy while any changes made
 by the callout registration functions affect the relevant callout vector.
 Such approach was chosen because of performance considerations.

 @subsection hooksmgServerObjects Server-Side Objects

 Those objects are not accessible by user libraries. Please do not
 attempt to use them if you are developing user callouts.

 @subsubsection hooksmgServerHooks Server Hooks

 The singleton @ref isc::hooks::ServerHooks object is used to register
 hooks. It is little more than a wrapper around a map of (hook index,
 hook name), generating a unique number (the hook index) for each
 hook registered.  It also handles the registration of the pre-defined
 context_create and context_destroy hooks.

 In operation, the @ref hooksmgHooksManager provides a thin wrapper
 around it, so that the Kea component developer does not have to
 worry about another object.

 @subsubsection hooksmgLibraryManager Library Manager

 An @ref isc::hooks::LibraryManager is created by the @ref
 hooksmgHooksManager object for each shared library loaded. It
 controls the loading and unloading of the library and in essence
 represents the library in the hooks framework. It also handles the
 registration of the standard callouts (functions in the library with
 the same name as the hook name).

 Of particular importance is the "library's index", a number associated
 with the library.  This is passed to the LibraryManager at creation
 time and is used to tag the callout pointers.  It is discussed
 further below.

 As the @c LibraryManager provides all the methods needed to manage the
 shared library, it is the natural home for the static @c validateLibrary()
 method. The function called the parsing of the Kea configuration, when
 the "hooks-libraries" element is processed. It checks that shared library
 exists, that it can be opened, that it contains the @c version() function
 and that that function returns a valid value. It then closes the shared
 library and returns an appropriate indication as to the library status.

 @subsubsection hooksmgLibraryManagerCollection Library Manager Collection

 The hooks framework can handle multiple libraries and as
 a result will create a @ref hooksmgLibraryManager for each
 of them. The collection of LibraryManagers is managed by the
 @ref isc::hooks::LibraryManagerCollection object which, in most
 cases has a method corresponding to a @ref hooksmgLibraryManager
 method, e.g. it has a @c loadLibraries() that corresponds to the @ref
 hooksmgLibraryManager's loadLibrary() call. As would be expected, methods
 on the @c LibraryManagerCollection iterate through all specified libraries,
 calling the corresponding LibraryManager method for each library.

 One point of note is that @c LibraryManagerCollection operates on an "all
 or none" principle. When @c loadLibraries() is called, on exit either all
 libraries have been successfully opened or none of them have. There
 is no use-case in Kea where, after a user has specified the shared
 libraries they want to load, the system will operate with only some of
 them loaded.

 The @c LibraryManagerCollection is the place where each library's index is set.
 Each library is assigned a number ranging from 1 through to the number
 of libraries being loaded.  As mentioned in the previous section, this
 index is used to tag callout pointers, something that is discussed
 in the next section.

 (Whilst on the subject of library index numbers, two additional
 numbers - 0 and @c INT_MAX - are also valid as "current library index".
 For flexibility, the Kea component is able to register its own
 functions as hook callouts.  It does this by obtaining a suitable @ref
 hooksmgLibraryHandle from the @ref hooksmgHooksManager.  A choice
 of two is available: one @ref hooksmgLibraryHandle (with an index
 of 0) can be used to register a callout on a hook to execute before
 any user-supplied callouts.  The second (with an index of @c INT_MAX)
 is used to register a callout to run after user-specified callouts.
 Apart from the index number, the hooks framework does not treat these
 callouts any differently from user-supplied ones.)

 @subsubsection hooksmgCalloutManager Callout Manager

 The @ref isc::hooks::CalloutManager is the core of the framework insofar
 as the registration and calling of callouts is concerned.

 It maintains a "hook vector" - a vector with one element for
 each registered hook. Each element in this vector is itself a
 vector (the callout vector), each element of which is a pair of
 (library index, callback pointer). When a callout is registered, the
 CalloutManager's current library index is used to supply the "library
 index" part of the pair. The library index is set explicitly by the
 @ref hooksmgLibraryManager prior to calling the user library's load()
 function (and prior to registering the standard callbacks).

 The situation is slightly more complex when a callout is executing. In
 order to execute a callout, the CalloutManager's @c callCallouts()
 method must be called. This iterates through the callout vector for
 a hook and for each element in the vector, uses the "library index"
 part of the pair to set the "current library index" before calling the
 callout function recorded in the second part of the pair. In most cases,
 the setting of the library index has no effect on the callout. However,
 if the callout wishes to dynamically register or deregister a callout,
 the @ref hooksmgLibraryHandle (see above) calls a method on the
 @ref hooksmgCalloutManager which in turn uses that information.

 @subsubsection hooksmgHooksManager Hooks Manager

 The @ref isc::hooks::HooksManager is the main object insofar as the
 server is concerned. It controls the creation of the library-related
 objects and provides the framework in which they interact. It also
 provides a shell around objects such as @ref hooksmgServerHooks so that all
 interaction with the hooks framework by the server is through the
 HooksManager object.  Apart from this, it supplies no functionality to
 the hooks framework.

 @section hooksmgOtherIssues Other Issues

 @subsection hooksmgMemoryAllocation Memory Allocation

 Unloading a shared library works by unmapping the part of the process's
 virtual address space in which the library lies. This may lead to
 problems if there are still references to that address space elsewhere
 in the process.

 In many operating systems, heap storage allowed by a shared library
 will lie in the virtual address allocated to the library. This has
 implications in the hooks framework because:

 - Argument information stored in a @ref hooksmgCalloutHandle by a
 callout in a library may lie in the library's address space.

 - Data modified in objects passed as arguments may lie in the address
 space. For example, it is common for a DHCP callout to add "options"
 to a packet: the memory allocated for those options will most likely
 lie in library address space.

 The problem really arises because of the extensive use by Kea of
 boost smart pointers. When the pointer is destroyed, the pointed-to
 memory is deallocated. If the pointer points to address space that is
 unmapped because a library has been unloaded, the deletion causes a
 segmentation fault.

 The hooks framework addresses the issue for the @ref hooksmgCalloutHandle
 by keeping in that object a shared pointer to the object controlling
 library unloading (the @ref hooksmgLibraryManagerCollection). Although
 the libraries can be unloaded at any time, it is only when every
 @ref hooksmgCalloutHandle that could possibly reference address space in the
 library have been deleted that the library will actually be unloaded
 and the address space unmapped.

 The hooks framework cannot solve the second issue as the objects in
 question are under control of the Kea server incorporating the
 hooks. It is up to the server developer to ensure that all such objects
 have been destroyed before libraries are reloaded. In extreme cases
 this may mean the server suspending all processing of incoming requests
 until all currently executing requests have completed and data object
 destroyed, reloading the libraries, then resuming processing.

 Since Kea 1.7.10 the unload() entry point is called as the first phase
 of unloading. This gives more chance to hooks writer to perform
 necessary cleanup actions so the second phase, memory unmapping
 can safely happen. The @c isc::hooks::unloadLibraries() function
 was updated too to return false when at least one active callout
 handle remained.

 @subsection hooksmgStaticLinking Hooks and Statically-Linked Kea

 Kea has the configuration option to allow static linking.  What this
 means is that it links against the static Kea libraries and not
 the sharable ones - although it links against the sharable system
 libraries like "libc" and "libstdc++" and well as the sharable libraries
 for third-party packages such as log4cplus and MySql.

 Static linking poses a problem for dynamically-loaded hooks libraries
 as some of the code in them - in particular the hooks framework and
 the logging code - depend on global objects created within the Kea
 libraries.  In the normal course of events (Kea linked against
 shared libraries), when Kea is run and the operating system loads
 a Kea shared library containing a global object, address space
 is assigned for it.  When the hooks framework loads a user-library
 linked against the same Kea shared library, the operating system
 recognizes that the library is already loaded (and initialized) and
 uses its definition of the global object.  Thus both the code in the
 Kea image and the code in the user-written shared library
 reference the same object.

 If Kea is statically linked, the linker allocates address space
 in the Kea image for the global object and does not include any
 reference to the shared library containing it.  When Kea now loads
 the user-written shared library - and so loads the Kea library code
 containing the global object - the operating system does not know that
 the object already exists.  Instead, it allocates new address space.
 The version of Kea in memory therefore has two copies of the object:
 one referenced by code in the Kea image, and one referenced by code
 in the user-written hooks library.  This causes problems - information
 put in one copy is not available to the other.

 Particular problems were encountered with global objects the hooks library
 and in the logging library, so some code to alleviate the problem has been
 included.

 The issue in the hooks library is the singleton @ref
 isc::hooks::ServerHooks object, used by the user-written hooks library
 if it attempts to register or deregister callouts.  The contents of the
 singleton - the names of the hook points and their index - are set by
 the relevant Kea server; this information is not available in the
 singleton created in the user's hooks library.

 Within the code users by the user's hooks library, the @c ServerHooks
 object is used by @ref isc::hooks::CalloutHandle and @ref
 isc::hooks::CalloutManager objects.  Both these objects are passed to the
 hooks library code when a callout is called: the former directly through
 the callout argument list, the latter indirectly as a pointer to it is
 stored in the CalloutHandle.  This allows a solution to the problem:
 instead of accessing the singleton via @c ServerHooks::getServerHooks(),
 the constructors of these objects store a reference to the singleton
 @c ServerHooks when they are created and use that reference to access
 @c ServerHooks data.  Since both @c CalloutHandle and @c CalloutManager are
 created in the statically-linked Kea server, use of the reference
 means that it is the singleton within the server - and not the one
 within the user's hooks library - that is referenced.

 The solution of the logging problem is not so straightforward.  Within
 Kea, there are two logging components, the Kea logging framework
 and the log4cplus libraries.  Owing to static linking, there are two
 instances of the former; but as static linking uses shared libraries of
 third-party products, there is one instance of the latter.  What further
 complicates matters is that initialization of the logging framework is
 in two parts: static initialization and run-time initialization.

 The logging initialization comprises the following:

 -# Static initialization of the log4cplus global variables.
 -# Static initialization of messages in the various Kea libraries.
 -# Static initialization of logging framework.
 -# Run-time initialization of the logging framework.
 -# Run-time initialization of log4cplus

 As both the Kea server and the user-written hooks libraries use the
 log4cplus shared library, item 1 - the static initialization of the log4cplus
 global variables is performed once.

 The next two tasks - static initialization of the messages in the Kea
 libraries and the static initialization of the logging framework -
 are performed twice, once in the context of the Kea server and
 once in the context of the hooks library.  For this reason, run-time
 initialization of the logging framework needs to be performed twice,
 once in the context of the Kea server and once in the context of the
 user-written hooks library.  However, the standard logging framework
 initialization code also performs the last task, initialization of
 log4cplus, something that causes problems if executed more than once.

 To get round this, the function @ref isc::hooks::hooksStaticLinkInit()
 has been written.  It executes the only part of the logging framework
 run-time initialization that actually pertains to the logging framework
 and not log4cplus, namely loading the message dictionary with the
 statically-initialized messages in the Kea libraries.
 This should be executed by any hooks library linking against a statically
 initialized Kea.  (In fact, running it against a dynamically-linked
 Kea should have no effect, as the load operation discards any duplicate
 message entries.)  The hooks library tests do this, the code being
 conditionally compiled within a test of the @c USE_STATIC_LINK macro, set
 by the configure script.

 @note Not everything is completely rosy with logging and static linking.
 In particular, there appears to be an issue with the scenario where a
 user-written hooks library is run by a statically-linked Kea and then
 unloaded.  As far as can be determined, on unload the system attempts to
 delete the same logger twice.  This is alleviated by explicitly clearing
 the loggerptr_ variable in the @ref isc::log::Logger destructor, but there
 is a suspicion that some memory might be lost in these circumstances.
 This is still under investigation.
*/