diff options
Diffstat (limited to '')
| -rw-r--r-- | doc/19-technical-concepts.md | 2217 |
1 files changed, 2217 insertions, 0 deletions
diff --git a/doc/19-technical-concepts.md b/doc/19-technical-concepts.md new file mode 100644 index 0000000..0fb5895 --- /dev/null +++ b/doc/19-technical-concepts.md @@ -0,0 +1,2217 @@ +# Technical Concepts <a id="technical-concepts"></a> + +This chapter provides technical concepts and design insights +into specific Icinga 2 components such as: + +* [Application](19-technical-concepts.md#technical-concepts-application) +* [Configuration](19-technical-concepts.md#technical-concepts-configuration) +* [Features](19-technical-concepts.md#technical-concepts-features) +* [Check Scheduler](19-technical-concepts.md#technical-concepts-check-scheduler) +* [Checks](19-technical-concepts.md#technical-concepts-checks) +* [Cluster](19-technical-concepts.md#technical-concepts-cluster) +* [TLS Network IO](19-technical-concepts.md#technical-concepts-tls-network-io) + +## Application <a id="technical-concepts-application"></a> + +### CLI Commands <a id="technical-concepts-application-cli-commands"></a> + +The Icinga 2 application is managed with different CLI sub commands. +`daemon` takes care about loading the configuration files, running the +application as daemon, etc. +Other sub commands allow to enable features, generate and request +TLS certificates or enter the debug console. + +The main entry point for each CLI command parses the command line +parameters and then triggers the required actions. + +### daemon CLI command <a id="technical-concepts-application-cli-commands-daemon"></a> + +This CLI command loads the configuration files, starting with `icinga2.conf`. +The [configuration compiler](19-technical-concepts.md#technical-concepts-configuration) parses the +file and detects additional file includes, constants, and any other DSL +specific declaration. + +At this stage, the configuration will already be checked against the +defined grammar in the scanner, and custom object validators will also be +checked. + +If the user provided `-C/--validate`, the CLI command returns with the +validation exit code. + +When running as daemon, additional parameters are checked, e.g. whether +this application was triggered by a reload, needs to daemonize with fork() +involved and update the object's authority. The latter is important for +HA-enabled cluster zones. + +## Configuration <a id="technical-concepts-configuration"></a> + +### Lexer <a id="technical-concepts-configuration-lexer"></a> + +The lexer stage does not understand the DSL itself, it only +maps specific character sequences into identifiers. + +This allows Icinga to detect the beginning of a string with `"`, +reading the following characters and determining the end of the +string with again `"`. + +Other parts covered by the lexer a escape sequences insides a string, +e.g. `"\"abc"`. + +The lexer also identifiers logical operators, e.g. `&` or `in`, +specific keywords like `object`, `import`, etc. and comment blocks. + +Please check `lib/config/config_lexer.ll` for details. + +Icinga uses [Flex](https://github.com/westes/flex) in the first stage. + +> Flex (The Fast Lexical Analyzer) +> +> Flex is a fast lexical analyser generator. It is a tool for generating programs +> that perform pattern-matching on text. Flex is a free (but non-GNU) implementation +> of the original Unix lex program. + +### Parser <a id="technical-concepts-configuration-parser"></a> + +The parser stage puts the identifiers from the lexer into more +context with flow control and sequences. + +The following comparison is parsed into a left term, an operator +and a right term. + +``` +x > 5 +``` + +The DSL contains many elements which require a specific order, +and sometimes only a left term for example. + +The parser also takes care of parsing an object declaration for +example. It already knows from the lexer that `object` marks the +beginning of an object. It then expects a type string afterwards, +and the object name - which can be either a string with double quotes +or a previously defined constant. + +An opening bracket `{` in this specific context starts the object +scope, which also is stored for later scope specific variable access. + +If there's an apply rule defined, this follows the same principle. +The config parser detects the scope of an apply rule and generates +Icinga 2 C++ code for the parsed string tokens. + +``` +assign where host.vars.sla == "24x7" +``` + +is parsed into an assign token identifier, and the string expression +is compiled into a new `ApplyExpression` object. + +The flow control inside the parser ensures that for example `ignore where` +can only be defined when a previous `assign where` was given - or when +inside an apply for rule. + +Another example are specific object types which allow assign expression, +specifically group objects. Others objects must throw a configuration error. + +Please check `lib/config/config_parser.yy` for more details, +and the [language reference](17-language-reference.md#language-reference) chapter for +documented DSL keywords and sequences. + +> Icinga uses [Bison](https://en.wikipedia.org/wiki/GNU_bison) as parser generator +> which reads a specification of a context-free language, warns about any parsing +> ambiguities, and generates a parser in C++ which reads sequences of tokens and +> decides whether the sequence conforms to the syntax specified by the grammar. + + +### Compiler <a id="technical-concepts-configuration-compiler"></a> + +The config compiler initializes the scanner inside the [lexer](19-technical-concepts.md#technical-concepts-configuration-lexer) +stage. + +The configuration files are parsed into memory from inside the [daemon CLI command](19-technical-concepts.md#technical-concepts-application-cli-commands-daemon) +which invokes the config validation in `ValidateConfigFiles()`. This compiles the +files into an AST expression which is executed. + +At this stage, the expressions generate so-called "config items" which +are a pre-stage of the later compiled object. + +`ConfigItem::CommitItems` takes care of committing the items, and doing a +rollback on failure. It also checks against matching apply rules from the previous run +and generates statistics about the objects which can be seen by the config validation. + +`ConfigItem::CommitNewItems` collects the registered types and items, +and checks for a specific required order, e.g. a service object needs +a host object first. + +The following stages happen then: + +- **Commit**: A workqueue then commits the items in a parallel fashion for this specific type. The object gets its name, and the AST expression is executed. It is then registered into the item into `m_Object` as reference. +- **OnAllConfigLoaded**: Special signal for each object to pre-load required object attributes, resolve group membership, initialize functions and timers. +- **CreateChildObjects**: Run apply rules for this specific type. +- **CommitNewItems**: Apply rules may generate new config items, this is to ensure that they again run through the stages. + +Note that the items are now committed and the configuration is validated and loaded +into memory. The final config objects are not yet activated though. + +This only happens after the validation, when the application is about to be run +with `ConfigItem::ActivateItems`. + +Each item has an object created in `m_Object` which is checked in a loop. +Again, the dependency order of activated objects is important here, e.g. logger features come first, then +config objects and last the checker, api, etc. features. This is done by sorting the objects +based on their type specific activation priority. + +The following signals are triggered in the stages: + +- **PreActivate**: Setting the `active` flag for the config object. +- **Activate**: Calls `Start()` on the object, sets the local HA authority and notifies subscribers that this object is now activated (e.g. for config updates in the DB backend). + + +### References <a id="technical-concepts-configuration-references"></a> + +* [The Icinga Config Compiler: An Overview](https://www.netways.de/blog/2018/07/12/the-icinga-config-compiler-an-overview/) +* [A parser/lexer/compiler for the Leonardo language](https://github.com/EmilGedda/Leonardo) +* [I wrote a programming language. Here’s how you can, too.](https://medium.freecodecamp.org/the-programming-language-pipeline-91d3f449c919) +* [http://onoffswitch.net/building-a-custom-lexer/](http://onoffswitch.net/building-a-custom-lexer/) +* [Writing an Interpreter with Lex, Yacc, and Memphis](http://memphis.compilertools.net/interpreter.html) +* [Flex](https://github.com/westes/flex) +* [GNU Bison](https://www.gnu.org/software/bison/) + +## Core <a id="technical-concepts-core"></a> + +### Core: Reload Handling <a id="technical-concepts-core-reload"></a> + +The initial design of the reload state machine looks like this: + +* receive reload signal SIGHUP +* fork a child process, start configuration validation in parallel work queues +* parent process continues with old configuration objects and the event scheduling +(doing checks, replicating cluster events, triggering alert notifications, etc.) +* validation NOT ok: child process terminates, parent process continues with old configuration state +* validation ok: child process signals parent process to terminate and save its current state (all events until now) into the icinga2 state file +* parent process shuts down writing icinga2.state file +* child process waits for parent process gone, reads the icinga2 state file and synchronizes all historical and status data +* child becomes the new session leader + +Since Icinga 2.6, there are two processes when checked with `ps aux | grep icinga2` or `pidof icinga2`. +This was to ensure that feature file descriptors don't leak into the plugin process (e.g. DB IDO MySQL sockets). + +Icinga 2.9 changed the reload handling a bit with SIGUSR2 signals +and systemd notifies. + +With systemd, it could occur that the tree was broken thus resulting +in killing all remaining processes on stop, instead of a clean exit. +You can read the full story [here](https://github.com/Icinga/icinga2/issues/7309). + +With 2.11 you'll now see 3 processes: + +- The umbrella process which takes care about signal handling and process spawning/stopping +- The main process with the check scheduler, notifications, etc. +- The execution helper process + +During reload, the umbrella process spawns a new reload process which validates the configuration. +Once successful, the new reload process signals the umbrella process that it is finished. +The umbrella process forwards the signal and tells the old main process to shutdown. +The old main process writes the icinga2.state file. The umbrella process signals +the reload process that the main process terminated. + +The reload process was in idle wait before, and now continues to read the written +state file and run the event loop (checks, notifications, "events", ...). The reload +process itself also spawns the execution helper process again. + + +## Features <a id="technical-concepts-features"></a> + +Features are implemented in specific libraries and can be enabled +using CLI commands. + +Features either write specific data or receive data. + +Examples for writing data: [DB IDO](14-features.md#db-ido), [Graphite](14-features.md#graphite-carbon-cache-writer), [InfluxDB](14-features.md#influxdb-writer). [GELF](14-features.md#gelfwriter), etc. +Examples for receiving data: [REST API](12-icinga2-api.md#icinga2-api), etc. + +The implementation of features makes use of existing libraries +and functionality. This makes the code more abstract, but shorter +and easier to read. + +Features register callback functions on specific events they want +to handle. For example the `GraphiteWriter` feature subscribes to +new CheckResult events. + +Each time Icinga 2 receives and processes a new check result, this +event is triggered and forwarded to all subscribers. + +The GraphiteWriter feature calls the registered function and processes +the received data. Features which connect Icinga 2 to external interfaces +normally parse and reformat the received data into an applicable format. + +Since this check result signal is blocking, many of the features include a work queue +with asynchronous task handling. + +The GraphiteWriter uses a TCP socket to communicate with the carbon cache +daemon of Graphite. The InfluxDBWriter is instead writing bulk metric messages +to InfluxDB's HTTP API, similar to Elasticsearch. + + +## Check Scheduler <a id="technical-concepts-check-scheduler"></a> + +The check scheduler starts a thread which loops forever. It waits for +check events being inserted into `m_IdleCheckables`. + +If the current pending check event number is larger than the configured +max concurrent checks, the thread waits up until it there's slots again. + +In addition, further checks on enabled checks, check periods, etc. are +performed. Once all conditions have passed, the next check timestamp is +calculated and updated. This also is the timestamp where Icinga expects +a new check result ("freshness check"). + +The object is removed from idle checkables, and inserted into the +pending checkables list. This can be seen via REST API metrics for the +checker component feature as well. + +The actual check execution happens asynchronously using the application's +thread pool. + +Once the check returns, it is removed from pending checkables and again +inserted into idle checkables. This ensures that the scheduler takes this +checkable event into account in the next iteration. + +### Start <a id="technical-concepts-check-scheduler-start"></a> + +When checkable objects get activated during the startup phase, +the checker feature registers a handler for this event. This is due +to the fact that the `checker` feature is fully optional, and e.g. not +used on command endpoint clients. + +Whenever such an object activation signal is triggered, Icinga 2 checks +whether it is [authoritative for this object](19-technical-concepts.md#technical-concepts-cluster-ha-object-authority). +This means that inside an HA enabled zone with two endpoints, only non-paused checkable objects are +actively inserted into the idle checkable list for the check scheduler. + +### Initial Check <a id="technical-concepts-check-scheduler-initial"></a> + +When a new checkable object (host or service) is initially added to the +configuration, Icinga 2 performs the following during startup: + +* `Checkable::Start()` is called and calculates the first check time +* With a spread delta, the next check time is actually set. + +If the next check should happen within a time frame of 60 seconds, +Icinga 2 calculates a delta from a random value. The minimum of `check_interval` +and 60 seconds is used as basis, multiplied with a random value between 0 and 1. + +In the best case, this check gets immediately executed after application start. +The worst case scenario is that the check is scheduled 60 seconds after start +the latest. + +The reasons for delaying and spreading checks during startup is that +the application typically needs more resources at this time (cluster connections, +feature warmup, initial syncs, etc.). Immediate check execution with +thousands of checks could lead into performance problems, and additional +events for each received check results. + +Therefore the initial check window is 60 seconds on application startup, +random seed for all checkables. This is not predictable over multiple restarts +for specific checkable objects, the delta changes every time. + +### Scheduling Offset <a id="technical-concepts-check-scheduler-offset"></a> + +There's a high chance that many checkable objects get executed at the same time +and interval after startup. The initial scheduling spreads that a little, but +Icinga 2 also attempts to ensure to keep fixed intervals, even with high check latency. + +During startup, Icinga 2 calculates the scheduling offset from a random number: + +* `Checkable::Checkable()` calls `SetSchedulingOffset()` with `Utility::Random()` +* The offset is a pseudo-random integral value between `0` and `RAND_MAX`. + +Whenever the next check time is updated with `Checkable::UpdateNextCheck()`, +the scheduling offset is taken into account. + +Depending on the state type (SOFT or HARD), either the `retry_interval` or `check_interval` +is used. If the interval is greater than 1 second, the time adjustment is calculated in the +following way: + +`now * 100 + offset` divided by `interval * 100`, using the remainder (that's what `fmod()` is for) +and dividing this again onto base 100. + +Example: offset is 6500, interval 300, now is 1542190472. + +``` +1542190472 * 100 + 6500 = 154219053714 +300 * 100 = 30000 +154219053714 / 30000 = 5140635.1238 + +(5140635.1238 - 5140635.0) * 30000 = 3714 +3714 / 100 = 37.14 +``` + +37.15 seconds as an offset would be far too much, so this is again used as a calculation divider for the +real offset with the base of 5 times the actual interval. + +Again, the remainder is calculated from the offset and `interval * 5`. This is divided onto base 100 again, +with an additional 0.5 seconds delay. + +Example: offset is 6500, interval 300. + +``` +6500 / 300 = 21.666666666666667 +(21.666666666666667 - 21.0) * 300 = 200 +200 / 100 = 2 +2 + 0.5 = 2.5 +``` + +The minimum value between the first adjustment and the second offset calculation based on the interval is +taken, in the above example `2.5` wins. + +The actual next check time substracts the adjusted time from the future interval addition to provide +a more widespread scheduling time among all checkable objects. + +`nextCheck = now - adj + interval` + +You may ask, what other values can happen with this offset calculation. Consider calculating more examples +with different interval settings. + +Example: offset is 34567, interval 60, now is 1542190472. + +``` +1542190472 * 100 + 34567 = 154219081767 +60 * 100 = 6000 +154219081767 / 6000 = 25703180.2945 +(25703180.2945 - 25703180.0) * 6000 / 100 = 17.67 + +34567 / 60 = 576.116666666666667 +(576.116666666666667 - 576.0) * 60 / 100 + 0.5 = 1.2 +``` + +`1m` interval starts at `now + 1.2s`. + +Example: offset is 12345, interval 86400, now is 1542190472. + +``` +1542190472 * 100 + 12345 = 154219059545 +86400 * 100 = 8640000 +154219059545 / 8640000 = 17849.428188078703704 +(17849.428188078703704 - 17849) * 8640000 = 3699545 +3699545 / 100 = 36995.45 + +12345 / 86400 = 0.142881944444444 +0.142881944444444 * 86400 / 100 + 0.5 = 123.95 +``` + +`1d` interval starts at `now + 2m4s`. + +> **Note** +> +> In case you have a better algorithm at hand, feel free to discuss this in a PR on GitHub. +> It needs to fulfill two things: 1) spread and shuffle execution times on each `next_check` update +> 2) not too narrowed window for both long and short intervals +> Application startup and initial checks need to be handled with care in a slightly different +> fashion. + +When `SetNextCheck()` is called, there are signals registered. One of them sits +inside the `CheckerComponent` class whose handler `CheckerComponent::NextCheckChangedHandler()` +deletes/inserts the next check event from the scheduling queue. This basically +is a list with multiple indexes with the keys for scheduling info and the object. + + +## Checks<a id="technical-concepts-checks"></a> + +### Check Latency and Execution Time <a id="technical-concepts-checks-latency"></a> + +Each check command execution logs the start and end time where +Icinga 2 (and the end user) is able to calculate the plugin execution time from it. + +```cpp +GetExecutionEnd() - GetExecutionStart() +``` + +The higher the execution time, the higher the command timeout must be set. Furthermore +users and developers are encouraged to look into plugin optimizations to minimize the +execution time. Sometimes it is better to let an external daemon/script do the checks +and feed them back via REST API. + +Icinga 2 stores the scheduled start and end time for a check. If the actual +check execution time differs from the scheduled time, e.g. due to performance +problems or limited execution slots (concurrent checks), this value is stored +and computed from inside the check result. + +The difference between the two deltas is called `check latency`. + +```cpp +(GetScheduleEnd() - GetScheduleStart()) - CalculateExecutionTime() +``` + +### Severity <a id="technical-concepts-checks-severity"></a> + +The severity attribute is introduced with Icinga v2.11 and provides +a bit mask calculated value from specific checkable object states. + +The severity value is pre-calculated for visualization interfaces +such as Icinga Web which sorts the problem dashboard by severity by default. + +The higher the severity number is, the more important the problem is. +However, the formula can change across Icinga 2 releases. + + +## Cluster <a id="technical-concepts-cluster"></a> + +This documentation refers to technical roles between cluster +endpoints. + +- The `server` or `parent` role accepts incoming connection attempts and handles requests +- The `client` role actively connects to remote endpoints receiving config/commands, requesting certificates, etc. + +A client role is not necessarily bound to the Icinga agent. +It may also be a satellite which actively connects to the +master. + +### Communication <a id="technical-concepts-cluster-communication"></a> + +Icinga 2 uses its own certificate authority (CA) by default. The +public and private CA keys can be generated on the signing master. + +Each node certificate must be signed by the private CA key. + +Note: The following description uses `parent node` and `child node`. +This also applies to nodes in the same cluster zone. + +During the connection attempt, a TLS handshake is performed. +If the public certificate of a child node is not signed by the same +CA, the child node is not trusted and the connection will be closed. + +If the TLS handshake succeeds, the parent node reads the +certificate's common name (CN) of the child node and looks for +a local Endpoint object name configuration. + +If there is no Endpoint object found, further communication +(runtime and config sync, etc.) is terminated. + +The child node also checks the CN from the parent node's public +certificate. If the child node does not find any local Endpoint +object name configuration, it will not trust the parent node. + +Both checks prevent accepting cluster messages from an untrusted +source endpoint. + +If an Endpoint match was found, there is one additional security +mechanism in place: Endpoints belong to a Zone hierarchy. + +Several cluster messages can only be sent "top down", others like +check results are allowed being sent from the child to the parent node. + +Once this check succeeds the cluster messages are exchanged and processed. + + +### CSR Signing <a id="technical-concepts-cluster-csr-signing"></a> + +In order to make things easier, Icinga 2 provides built-in methods +to allow child nodes to request a signed certificate from the +signing master. + +Icinga 2 v2.8 introduces the possibility to request certificates +from indirectly connected nodes. This is required for multi level +cluster environments with masters, satellites and agents. + +CSR Signing in general starts with the master setup. This step +ensures that the master is in a working CSR signing state with: + +* public and private CA key in `/var/lib/icinga2/ca` +* private `TicketSalt` constant defined inside the `api` feature +* Cluster communication is ready and Icinga 2 listens on port 5665 + +The child node setup which is run with CLI commands will now +attempt to connect to the parent node. This is not necessarily +the signing master instance, but could also be a parent satellite node. + +During this process the child node asks the user to verify the +parent node's public certificate to prevent MITM attacks. + +There are two methods to request signed certificates: + +* Add the ticket into the request. This ticket was generated on the master +beforehand and contains hashed details for which client it has been created. +The signing master uses this information to automatically sign the certificate +request. + +* Do not add a ticket into the request. It will be sent to the signing master +which stores the pending request. Manual user interaction with CLI commands +is necessary to sign the request. + +The certificate request is sent as `pki::RequestCertificate` cluster +message to the parent node. + +If the parent node is not the signing master, it stores the request +in `/var/lib/icinga2/certificate-requests` and forwards the +cluster message to its parent node. + +Once the message arrives on the signing master, it first verifies that +the sent certificate request is valid. This is to prevent unwanted errors +or modified requests from the "proxy" node. + +After verification, the signing master checks if the request contains +a valid signing ticket. It hashes the certificate's common name and +compares the value to the received ticket number. + +If the ticket is valid, the certificate request is immediately signed +with CA key. The request is sent back to the client inside a `pki::UpdateCertificate` +cluster message. + +If the child node was not the certificate request origin, it only updates +the cached request for the child node and send another cluster message +down to its child node (e.g. from a satellite to an agent). + + +If no ticket was specified, the signing master waits until the +`ca sign` CLI command manually signed the certificate. + +> **Note** +> +> Push notifications for manual request signing is not yet implemented (TODO). + +Once the child node reconnects it synchronizes all signed certificate requests. +This takes some minutes and requires all nodes to reconnect to each other. + + +#### CSR Signing: Clients without parent connection <a id="technical-concepts-cluster-csr-signing-clients-no-connection"></a> + +There is an additional scenario: The setup on a child node does +not necessarily need a connection to the parent node. + +This mode leaves the node in a semi-configured state. You need +to manually copy the master's public CA key into `/var/lib/icinga2/certs/ca.crt` +on the client before starting Icinga 2. + +> **Note** +> +> The `client` in this case can be either a satellite or an agent. + +The parent node needs to actively connect to the child node. +Once this connections succeeds, the child node will actively +request a signed certificate. + +The update procedure works the same way as above. + +### High Availability <a id="technical-concepts-cluster-ha"></a> + +General high availability is automatically enabled between two endpoints in the same +cluster zone. + +**This requires the same configuration and enabled features on both nodes.** + +HA zone members trust each other and share event updates as cluster messages. +This includes for example check results, next check timestamp updates, acknowledgements +or notifications. + +This ensures that both nodes are synchronized. If one node goes away, the +remaining node takes over and continues as normal. + +#### High Availability: Object Authority <a id="technical-concepts-cluster-ha-object-authority"></a> + +Cluster nodes automatically determine the authority for configuration +objects. By default, all config objects are set to `HARunEverywhere` and +as such the object authority is true for any config object on any instance. + +Specific objects can override and influence this setting, e.g. with `HARunOnce` +instead prior to config object activation. + +This is done when the daemon starts and in a regular interval inside +the ApiListener class, specifically calling `ApiListener::UpdateObjectAuthority()`. + +The algorithm works like this: + +* Determine whether this instance is assigned to a local zone and endpoint. +* Collects all endpoints in this zone if they are connected. +* If there's two endpoints, but only us seeing ourselves and the application start is less than 60 seconds in the past, do nothing (wait for cluster reconnect to take place, grace period). +* Sort the collected endpoints by name. +* Iterate over all config types and their respective objects + * Ignore !active objects + * Ignore objects which are !HARunOnce. This means, they can run multiple times in a zone and don't need an authority update. + * If this instance doesn't have a local zone, set authority to true. This is for non-clustered standalone environments where everything belongs to this instance. + * Calculate the object authority based on the connected endpoint names. + * Set the authority (true or false) + +The object authority calculation works "offline" without any message exchange. +Each instance alculates the SDBM hash of the config object name, puts that in contrast +modulo the connected endpoints size. +This index is used to lookup the corresponding endpoint in the connected endpoints array, +including the local endpoint. Whether the local endpoint is equal to the selected endpoint, +or not, this sets the authority to `true` or `false`. + +```cpp +authority = endpoints[Utility::SDBM(object->GetName()) % endpoints.size()] == my_endpoint; +``` + +`ConfigObject::SetAuthority(bool authority)` triggers the following events: + +* Authority is true and object now paused: Resume the object and set `paused` to `false`. +* Authority is false, object not paused: Pause the object and set `paused` to true. + +**This results in activated but paused objects on one endpoint.** You can verify +that by querying the `paused` attribute for all objects via REST API +or debug console on both endpoints. + +Endpoints inside a HA zone calculate the object authority independent from each other. +This object authority is important for selected features explained below. + +Since features are configuration objects too, you must ensure that all nodes +inside the HA zone share the same enabled features. If configured otherwise, +one might have a checker feature on the left node, nothing on the right node. +This leads to late check results because one half is not executed by the right +node which holds half of the object authorities. + +By default, features are enabled to "Run-Everywhere". Specific features which +support HA awareness, provide the `enable_ha` configuration attribute. When `enable_ha` +is set to `true` (usually the default), "Run-Once" is set and the feature pauses on one side. + +``` +vim /etc/icinga2/features-enabled/graphite.conf + +object GraphiteWriter "graphite" { + ... + enable_ha = true +} +``` + +Once such a feature is paused, there won't be any more event handling, e.g. the Elasticsearch +feature won't process any checkresults nor write to the Elasticsearch REST API. + +When the cluster connection drops, the feature configuration object is updated with +the new object authority by the ApiListener timer and resumes its operation. You can see +that by grepping the log file for `resumed` and `paused`. + +``` +[2018-10-24 13:28:28 +0200] information/GraphiteWriter: 'g-ha' paused. +``` + +``` +[2018-10-24 13:28:28 +0200] information/GraphiteWriter: 'g-ha' resumed. +``` + +Specific features with HA capabilities are explained below. + +#### High Availability: Checker <a id="technical-concepts-cluster-ha-checker"></a> + +The `checker` feature only executes checks for `Checkable` objects (Host, Service) +where it is authoritative. + +That way each node only executes checks for a segment of the overall configuration objects. + +The cluster message routing ensures that all check results are synchronized +to nodes which are not authoritative for this configuration object. + + +#### High Availability: Notifications <a id="technical-concepts-cluster-notifications"></a> + +The `notification` feature only sends notifications for `Notification` objects +where it is authoritative. + +That way each node only executes notifications for a segment of all notification objects. + +Notified users and other event details are synchronized throughout the cluster. +This is required if for example the DB IDO feature is active on the other node. + +#### High Availability: DB IDO <a id="technical-concepts-cluster-ha-ido"></a> + +If you don't have HA enabled for the IDO feature, both nodes will +write their status and historical data to their own separate database +backends. + +In order to avoid data separation and a split view (each node would require its +own Icinga Web 2 installation on top), the high availability option was added +to the DB IDO feature. This is enabled by default with the `enable_ha` setting. + +This requires a central database backend. Best practice is to use a MySQL cluster +with a virtual IP. + +Both Icinga 2 nodes require the connection and credential details configured in +their DB IDO feature. + +During startup Icinga 2 calculates whether the feature configuration object +is authoritative on this node or not. The order is an alpha-numeric +comparison, e.g. if you have `master1` and `master2`, Icinga 2 will enable +the DB IDO feature on `master2` by default. + +If the connection between endpoints drops, the object authority is re-calculated. + +In order to prevent data duplication in a split-brain scenario where both +nodes would write into the same database, there is another safety mechanism +in place. + +The split-brain decision which node will write to the database is calculated +from a quorum inside the `programstatus` table. Each node +verifies whether the `endpoint_name` column is not itself on database connect. +In addition to that the DB IDO feature compares the `last_update_time` column +against the current timestamp plus the configured `failover_timeout` offset. + +That way only one active DB IDO feature writes to the database, even if they +are not currently connected in a cluster zone. This prevents data duplication +in historical tables. + +### Health Checks <a id="technical-concepts-cluster-health-checks"></a> + +#### cluster-zone <a id="technical-concepts-cluster-health-checks-cluster-zone"></a> + +This built-in check provides the possibility to check for connectivity between +zones. + +If you for example need to know whether the `master` zone is connected and processing +messages with the child zone called `satellite` in this example, you can configure +the [cluster-zone](10-icinga-template-library.md#itl-icinga-cluster-zone) check as new service on all `master` zone hosts. + +``` +vim /etc/zones.d/master/host1.conf + +object Service "cluster-zone-satellite" { + check_command = "cluster-zone" + host_name = "host1" + + vars.cluster_zone = "satellite" +} +``` + +The check itself changes to NOT-OK if one or more child endpoints in the child zone +are not connected to parent zone endpoints. + +In addition to the overall connectivity check, the log lag is calculated based +on the to-be-sent replay log. Each instance stores that for its configured endpoint +objects. + +This health check iterates over the target zone (`cluster_zone`) and their endpoints. + +The log lag is greater than zero if + +* the replay log synchronization is in progress and not yet finished or +* the endpoint is not connected, and no replay log sync happened (obviously). + +The final log lag value is the worst value detected. If satellite1 has a log lag of +`1.5` and satellite2 only has `0.5`, the computed value will be `1.5.`. + +You can control the check state by using optional warning and critical thresholds +for the log lag value. + +If this service exists multiple times, e.g. for each master host object, the log lag +may differ based on the execution time. This happens for example on restart of +an instance when the log replay is in progress and a health check is executed at different +times. +If the endpoint is not connected, both master instances may have saved a different log replay +position from the last synchronisation. + +The lag value is returned as performance metric key `slave_lag`. + +Icinga 2 v2.9+ adds more performance metrics for these values: + +* `last_messages_sent` and `last_messages_received` as UNIX timestamp +* `sum_messages_sent_per_second` and `sum_messages_received_per_second` +* `sum_bytes_sent_per_second` and `sum_bytes_received_per_second` + + +### Config Sync <a id="technical-concepts-cluster-config-sync"></a> + +The visible feature for the user is to put configuration files in `/etc/icinga2/zones.d/<zonename>` +and have them synced automatically to all involved zones and endpoints. + +This not only includes host and service objects being checked +in a satellite zone, but also additional config objects such as +commands, groups, timeperiods and also templates. + +Additional thoughts and complexity added: + +- Putting files into zone directory names removes the burden to set the `zone` attribute on each object in this directory. This is done automatically by the config compiler. +- Inclusion of `zones.d` happens automatically, the user shouldn't be bothered about this. +- Before the REST API was created, only static configuration files in `/etc/icinga2/zones.d` existed. With the addition of config packages, additional `zones.d` targets must be registered (e.g. used by the Director) +- Only one config master is allowed. This one identifies itself with configuration files in `/etc/icinga2/zones.d`. This is not necessarily the zone master seen in the debug logs, that one is important for message routing internally. +- Objects and templates which cannot be bound into a specific zone (e.g. hosts in the satellite zone) must be made available "globally". +- Users must be able to deny the synchronisation of specific zones, e.g. for security reasons. + +#### Config Sync: Config Master <a id="technical-concepts-cluster-config-sync-config-master"></a> + +All zones must be configured and included in the `zones.conf` config file beforehand. +The zone names are the identifier for the directories underneath the `/etc/icinga2/zones.d` +directory. If a zone is not configured, it will not be included in the config sync - keep this +in mind for troubleshooting. + +When the config master starts, the content of `/etc/icinga2/zones.d` is automatically +included. There's no need for an additional entry in `icinga2.conf` like `conf.d`. +You can verify this by running the config validation on debug level: + +``` +icinga2 daemon -C -x debug | grep 'zones.d' + +[2019-06-19 15:16:19 +0200] notice/ConfigCompiler: Compiling config file: /etc/icinga2/zones.d/global-templates/commands.conf +``` + +Once the config validation succeeds, the startup routine for the daemon +copies the files into the "production" directory in `/var/lib/icinga2/api/zones`. +This directory is used for all endpoints where Icinga stores the received configuration. +With the exception of the config master retrieving this from `/etc/icinga2/zones.d` instead. + +These operations are logged for better visibility. + +``` +[2019-06-19 15:26:38 +0200] information/ApiListener: Copying 1 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'. +[2019-06-19 15:26:38 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf +``` + +The master is finished at this point. Depending on the cluster configuration, +the next iteration is a connected endpoint after successful TLS handshake and certificate +authentication. + +It calls `SendConfigUpdate(client)` which sends the [config::Update](19-technical-concepts.md#technical-concepts-json-rpc-messages-config-update) +JSON-RPC message including all required zones and their configuration file content. + + +#### Config Sync: Receive Config <a id="technical-concepts-cluster-config-sync-receive-config"></a> + +The secondary master endpoint and endpoints in a child zone will be connected to the config +master. The endpoint receives the [config::Update](19-technical-concepts.md#technical-concepts-json-rpc-messages-config-update) +JSON-RPC message and processes the content in `ConfigUpdateHandler()`. This method checks +whether config should be accepted. In addition to that, it locks a local mutex to avoid race conditions +with multiple syncs in parallel. + +After that, the received configuration content is analysed. + +> **Note** +> +> The cluster design allows that satellite endpoints may connect to the secondary master first. +> There is no immediate need to always connect to the config master first, especially since +> the satellite endpoints don't know that. +> +> The secondary master not only stores the master zone config files, but also all child zones. +> This is also the case for any HA enabled zone with more than one endpoint. + + +2.11 puts the received configuration files into a staging directory in +`/var/lib/icinga2/api/zones-stage`. Previous versions directly wrote the +files into production which could have led to broken configuration on the +next manual restart. + +``` +[2019-06-19 16:08:29 +0200] information/ApiListener: New client connection for identity 'master1' to [127.0.0.1]:5665 +[2019-06-19 16:08:30 +0200] information/ApiListener: Applying config update from endpoint 'master1' of zone 'master'. +[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums. +[2019-06-19 16:08:30 +0200] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/agent//_etc/host.conf' for zone 'agent'. +[2019-06-19 16:08:30 +0200] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/agent' (176 Bytes). +[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the checksums. +[2019-06-19 16:08:30 +0200] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/master' (17 Bytes). +[2019-06-19 16:08:30 +0200] information/ApiListener: Received configuration from endpoint 'master1' is different to production, triggering validation and reload. +``` + +It then validates the received configuration in its own config stage. There is +an parameter override in place which disables the automatic inclusion of the production +config in `/var/lib/icinga2/api/zones`. + +Once completed, the reload is triggered. This follows the same configurable timeout +as with the global reload. + +``` +[2019-06-19 16:52:26 +0200] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload. +[2019-06-19 16:52:27 +0200] information/Application: Got reload command: Started new instance with PID '19945' (timeout is 300s). +[2019-06-19 16:52:28 +0200] information/Application: Reload requested, letting new process take over. +``` + +Whenever the staged configuration validation fails, Icinga logs this including a reference +to the startup log file which includes additional errors. + +``` +[2019-06-19 15:45:27 +0200] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Aborting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log' +``` + + +#### Config Sync: Changes and Reload <a id="technical-concepts-cluster-config-sync-changes-reload"></a> + +Whenever a new configuration is received, it is validated and upon success, the +daemon automatically reloads. While the daemon continues with checks, the reload +cannot hand over open TCP connections. That being said, reloading the daemon everytime +a configuration is synchronized would lead into many not connected endpoints. + +Therefore the cluster config sync checks whether the configuration files actually +changed, and will only trigger a reload when such a change happened. + +2.11 calculates a checksum from each file content and compares this to the +production configuration. Previous versions used additional metadata with timestamps from +files which sometimes led to problems with asynchronous dates. + +> **Note** +> +> For compatibility reasons, the timestamp metadata algorithm is still intact, e.g. +> when the client is 2.11 already, but the parent endpoint is still on 2.10. + +Icinga logs a warning when this happens. + +``` +Received configuration update without checksums from parent endpoint satellite1. This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+ +``` + + +The debug log provides more details on the actual checksums and checks. Future output +may change, use this solely for troubleshooting and debugging whenever the cluster +config sync fails. + +``` +[2019-06-19 16:13:16 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums. +[2019-06-19 16:13:16 +0200] debug/ApiListener: Checking for config change between stage and production. Old (3): '{"/.checksums":"7ede1276a9a32019c1412a52779804a976e163943e268ec4066e6b6ec4d15d73","/.timestamp":"ec4354b0eca455f7c2ca386fddf5b9ea810d826d402b3b6ac56ba63b55c2892c","/_etc/host.conf":"35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44"}' vs. new (3): '{"/.checksums":"84a586435d732327e2152e7c9b6d85a340cc917b89ae30972042f3dc344ea7cf","/.timestamp":"0fd6facf35e49ab1b2a161872fa7ad794564eba08624373d99d31c32a7a4c7d3","/_etc/host.conf":"0d62075e89be14088de1979644b40f33a8f185fcb4bb6ff1f7da2f63c7723fcb"}'. +[2019-06-19 16:13:16 +0200] debug/ApiListener: Checking /_etc/host.conf for checksum: 35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44 +[2019-06-19 16:13:16 +0200] debug/ApiListener: Path '/_etc/host.conf' doesn't match old checksum '0d62075e89be14088de1979644b40f33a8f185fcb4bb6ff1f7da2f63c7723fcb' with new checksum '35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44'. +``` + + +#### Config Sync: Trust <a id="technical-concepts-cluster-config-sync-trust"></a> + +The config sync follows the "top down" approach, where the master endpoint in the master +zone is allowed to synchronize configuration to the child zone, e.g. the satellite zone. + +Endpoints in the same zone, e.g. a secondary master, receive configuration for the same +zone and all child zones. + +Endpoints in the satellite zone trust the parent zone, and will accept the pushed +configuration via JSON-RPC cluster messages. By default, this is disabled and must +be enabled with the `accept_config` attribute in the ApiListener feature (manually or with CLI +helpers). + +The satellite zone will not only accept zone configuration for its own zone, but also +all configured child zones. That is why it is important to configure the zone hierarchy +on the satellite as well. + +Child zones are not allowed to sync configuration up to the parent zone. Each Icinga instance +evaluates this in startup and knows on endpoint connect which config zones need to be synced. + + +Global zones have a special trust relationship: They are synced to all child zones, be it +a satellite zone or agent zone. Since checkable objects such as a Host or a Service object +must have only one endpoint as authority, they cannot be put into a global zone (denied by +the config compiler). + +Apply rules and templates are allowed, since they are evaluated in the endpoint which received +the synced configuration. Keep in mind that there may be differences on the master and the satellite +when e.g. hostgroup membership is used for assign where expressions, but the groups are only +available on the master. + + +### Cluster: Message Routing <a id="technical-concepts-cluster-message-routing"></a> + +One fundamental part of the cluster message routing is the MessageOrigin object. +This is created when a new JSON-RPC message is received in `JsonRpcConnection::MessageHandler()`. + +It contains + +- FromZone being extracted from the endpoint object which owns the JsonRpcConnection +- FromClient being the JsonRpcConnection bound to the endpoint object + +These attributes are checked in message receive api handlers for security access. E.g. whether a +message origin is from a child zone which is not allowed, etc. +This is explained in the [JSON-RPC messages](19-technical-concepts.md#technical-concepts-json-rpc-messages) chapter. + +Whenever such a message is processed on the client, it may trigger additional cluster events +which are sent back to other endpoints. Therefore it is key to always pass the MessageOrigin +`origin` when processing these messages locally. + +Example: + +- Client receives a CheckResult from another endpoint in the same zone, call it `sender` for now +- Calls ProcessCheckResult() to store the CR and calculcate states, notifications, etc. +- Calls the OnNewCheckResult() signal to trigger IDO updates + +OnNewCheckResult() also calls a registered cluster handler which forwards the CheckResult to other cluster members. + +Without any origin details, this CheckResult would be relayed to the `sender` endpoint again. +Which processes the message, ProcessCheckResult(), OnNewCheckResult(), sends back and so on. + +That creates a loop which our cluster protocol needs to prevent at all cost. + +RelayMessageOne() takes care of the routing. This involves fetching the targetZone for this message and its endpoints. + +- Don't relay messages to ourselves. +- Don't relay messages to disconnected endpoints. +- Don't relay the message to the zone through more than one endpoint unless this is our own zone. +- Don't relay messages back to the endpoint which we got the message from. **THIS** +- Don't relay messages back to the zone which we got the message from. +- Only relay message to the zone master if we're not currently the zone master. + +``` + e1 is zone master, e2 and e3 are zone members. + + Message is sent from e2 or e3: + !isMaster == true + targetEndpoint e1 is zone master -> send the message + targetEndpoint e3 is not zone master -> skip it, avoid routing loops + + Message is sent from e1: + !isMaster == false -> send the messages to e2 and e3 being the zone routing master. +``` + +With passing the `origin` the following condition prevents sending a message back to sender: + +```cpp +if (origin && origin->FromClient && targetEndpoint == origin->FromClient->GetEndpoint()) { +``` + +This message then simply gets skipped for this specific Endpoint and is never sent. + +This analysis originates from a long-lasting [downtime loop bug](https://github.com/Icinga/icinga2/issues/7198). + +## TLS Network IO <a id="technical-concepts-tls-network-io"></a> + +### TLS Connection Handling <a id="technical-concepts-tls-network-io-connection-handling"></a> + +Icinga supports two connection directions, controlled via the `host` attribute +inside the Endpoint objects: + +* Outgoing connection attempts +* Incoming connection handling + +Once the connection is established, higher layers can exchange JSON-RPC and +HTTP messages. It doesn't matter which direction these message go. + +This offers a big advantage over single direction connections, just like +polling via HTTP only. Also, connections are kept alive as long as data +is transmitted. + +When the master connects to the child zone member(s), this requires more +resources there. Keep this in mind when endpoints are not reachable, the +TCP timeout blocks other resources. Moving a satellite zone in the middle +between masters and agents helps to split the tasks - the master +processes and stores data, deploys configuration and serves the API. The +satellites schedule the checks, connect to the agents and receive +check results. + +Agents/Clients can also connect to the parent endpoints - be it a master or +a satellite. This is the preferred way out of a DMZ, and also reduces the +overhead with connecting to e.g. 2000 agents on the master. You can +benchmark this when TCP connections are broken and timeouts are encountered. + +#### Master Processes Incoming Connection <a id="technical-concepts-tls-network-io-connection-handling-incoming"></a> + +* The node starts a new ApiListener, this invokes `AddListener()` + * Setup TLS Context (SslContext) + * Initialize global I/O engine and create a TCP acceptor + * Resolve bind host/port (optional) + * Listen on IPv4 and IPv6 + * Re-use socket address and port + * Listen on port 5665 with `INT_MAX` possible sockets +* Spawn a new Coroutine which listens for new incoming connections as 'TCP server' pattern + * Accept new connections asynchronously + * Spawn a new Coroutine which handles the new client connection in a different context, Role: Server + +#### Master Connects Outgoing <a id="technical-concepts-tls-network-io-connection-handling-outgoing"></a> + +* The node starts a timer in a 10 seconds interval with `ApiReconnectTimerHandler()` as callback + * Loop over all configured zones, exclude global zones and not direct parent/child zones + * Get the endpoints configured in the zones, exclude: local endpoint, no 'host' attribute, already connected or in progress + * Call `AddConnection()` +* Spawn a new Coroutine after making the TLS context + * Use the global I/O engine for socket I/O + * Create TLS stream + * Connect to endpoint host/port details + * Handle the client connection, Role: Client + +#### TLS Handshake <a id="technical-concepts-tls-network-io-connection-handling-handshake"></a> + +* Create a TLS connection in sslConn and perform an asynchronous TLS handshake +* Get the peer certificate +* Verify the presented certificate: `ssl::verify_peer` and `ssl::verify_client_once` +* Get the certificate CN and compare it against the endpoint name - if not matching, return and close the connection + +#### Data Exchange <a id="technical-concepts-tls-network-io-connection-data-exchange"></a> + +Everything runs through TLS, we don't use any "raw" connections nor plain message handling. + +HTTP and JSON-RPC messages share the same port and API, so additional handling is required. + +On a new connection and successful TLS handshake, the first byte is read. This either +is a JSON-RPC message in Netstring format starting with a number, or plain HTTP. + +``` +HTTP/1.1 + +2:{} +``` + +Depending on this, `ClientJsonRpc` or `ClientHttp` are assigned. + +JSON-RPC: + +* Create a new JsonRpcConnection object + * When the endpoint object is configured, spawn a Coroutine which takes care of syncing the client (file and runtime config, replay log, etc.) + * No endpoint treats this connection as anonymous client, with a configurable limit. This client may send a CSR signing request for example. + * Start the JsonRpcConnection - this spawns Coroutines to HandleIncomingMessages, WriteOutgoingMessages, HandleAndWriteHeartbeats and CheckLiveness + +HTTP: + +* Create a new HttpServerConnection + * Start the HttpServerConnection - this spawns Coroutines to ProcessMessages and CheckLiveness + + +All the mentioned Coroutines run asynchronously using the global I/O engine's context. +More details on this topic can be found in [this blogpost](https://www.netways.de/blog/2019/04/04/modern-c-programming-coroutines-with-boost/). + +The lower levels of context switching and sharing or event polling are +hidden in Boost ASIO, Beast, Coroutine and Context libraries. + +#### Data Exchange: Coroutines and I/O Engine <a id="technical-concepts-tls-network-io-connection-data-exchange-coroutines"></a> + +Light-weight and fast operations such as connection handling or TLS handshakes +are performed in the default `IoBoundWorkSlot` pool inside the I/O engine. + +The I/O engine has another pool available: `CpuBoundWork`. + +This is used for processing CPU intensive tasks, such as handling a HTTP request. +Depending on the available CPU cores, this is limited to `std::thread::hardware_concurrency() * 3u / 2u`. + +``` +1 core * 3 / 2 = 1 +2 cores * 3 / 2 = 3 +8 cores * 3 / 2 = 12 +16 cores * 3 / 2 = 24 +``` + +The I/O engine itself is used with all network I/O in Icinga, not only the cluster +and the REST API. Features such as Graphite, InfluxDB, etc. also consume its functionality. + +There are 2 * CPU cores threads available which run the event loop +in the I/O engine. This polls the I/O service with `m_IoService.run();` +and triggers an asynchronous event progress for waiting coroutines. + +<!-- +## REST API <a id="technical-concepts-rest-api"></a> + +Icinga 2 provides its own HTTP server which shares the port 5665 with +the JSON-RPC cluster protocol. +--> + +## JSON-RPC Message API <a id="technical-concepts-json-rpc-messages"></a> + +**The JSON-RPC message API is not a public API for end users.** In case you want +to interact with Icinga, use the [REST API](12-icinga2-api.md#icinga2-api). + +This section describes the internal cluster messages exchanged between endpoints. + +> **Tip** +> +> Debug builds with `icinga2 daemon -DInternal.DebugJsonRpc=1` unveils the JSON-RPC messages. + +### Registered Handler Functions + +Functions by example: + +Event Sender: `Checkable::OnNewCheckResult` + +``` +On<xyz>.connect(&xyzHandler) +``` + +Event Receiver (Client): `CheckResultAPIHandler` in `REGISTER_APIFUNCTION` + +``` +<xyz>APIHandler() +``` + +### Messages + +#### icinga::Hello <a id="technical-concepts-json-rpc-messages-icinga-hello"></a> + +> Location: `apilistener.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | icinga::Hello +params | Dictionary + +##### Params + +Key | Type | Description +---------------------|-------------|------------------ +capabilities | Number | Bitmask, see `lib/remote/apilistener.hpp`. +version | Number | Icinga 2 version, e.g. 21300 for v2.13.0. + +##### Functions + +Event Sender: When a new client connects in `NewClientHandlerInternal()`. +Event Receiver: `HelloAPIHandler` + +##### Permissions + +None, this is a required message. + +#### event::Heartbeat <a id="technical-concepts-json-rpc-messages-event-heartbeat"></a> + +> Location: `jsonrpcconnection-heartbeat.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::Heartbeat +params | Dictionary + +##### Params + +Key | Type | Description +----------|---------------|------------------ +timeout | Number | Heartbeat timeout, sender sets 120s. + + +##### Functions + +Event Sender: `JsonRpcConnection::HeartbeatTimerHandler` +Event Receiver: `HeartbeatAPIHandler` + +Both sender and receiver exchange this heartbeat message. If the sender detects +that a client endpoint hasn't sent anything in the updated timeout span, it disconnects +the client. This is to avoid stale connections with no message processing. + +##### Permissions + +None, this is a required message. + +#### event::CheckResult <a id="technical-concepts-json-rpc-messages-event-checkresult"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::CheckResult +params | Dictionary + +##### Params + +Key | Type | Description +----------|---------------|------------------ +host | String | Host name +service | String | Service name +cr | Serialized CR | Check result + +##### Functions + +Event Sender: `Checkable::OnNewCheckResult` +Event Receiver: `CheckResultAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Hosts/services do not exist +* Origin is a remote command endpoint different to the configured, and whose zone is not allowed to access this checkable. + +#### event::SetNextCheck <a id="technical-concepts-json-rpc-messages-event-setnextcheck"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetNextCheck +params | Dictionary + +##### Params + +Key | Type | Description +------------|---------------|------------------ +host | String | Host name +service | String | Service name +next\_check | Timestamp | Next scheduled time as UNIX timestamp. + +##### Functions + +Event Sender: `Checkable::OnNextCheckChanged` +Event Receiver: `NextCheckChangedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::SetLastCheckStarted <a id="technical-concepts-json-rpc-messages-event-setlastcheckstarted"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetLastCheckStarted +params | Dictionary + +##### Params + +Key | Type | Description +---------------------|-----------|------------------ +host | String | Host name +service | String | Service name +last\_check\_started | Timestamp | Last check's start time as UNIX timestamp. + +##### Functions + +Event Sender: `Checkable::OnLastCheckStartedChanged` +Event Receiver: `LastCheckStartedChangedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::SetStateBeforeSuppression <a id="technical-concepts-json-rpc-messages-event-setstatebeforesuppression"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------------------------------- +jsonrpc | 2.0 +method | event::SetStateBeforeSuppression +params | Dictionary + +##### Params + +Key | Type | Description +---------------------------|--------|----------------------------------------------- +host | String | Host name +service | String | Service name +state\_before\_suppression | Number | Checkable state before the current suppression + +##### Functions + +Event Sender: `Checkable::OnStateBeforeSuppressionChanged` +Event Receiver: `StateBeforeSuppressionChangedAPIHandler` + +Used to sync the checkable state from before a notification suppression (for example +because the checkable is in a downtime) started within the same HA zone. + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint is not within the local zone. + +#### event::SetSuppressedNotifications <a id="technical-concepts-json-rpc-messages-event-setsupressednotifications"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetSuppressedNotifications +params | Dictionary + +##### Params + +Key | Type | Description +-------------------------|---------------|------------------ +host | String | Host name +service | String | Service name +supressed\_notifications | Number | Bitmask for suppressed notifications. + +##### Functions + +Event Sender: `Checkable::OnSuppressedNotificationsChanged` +Event Receiver: `SuppressedNotificationsChangedAPIHandler` + +Used to sync the notification state of a host or service object within the same HA zone. + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint is not within the local zone. + +#### event::SetSuppressedNotificationTypes <a id="technical-concepts-json-rpc-messages-event-setsuppressednotificationtypes"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetSuppressedNotificationTypes +params | Dictionary + +##### Params + +Key | Type | Description +-------------------------|--------|------------------ +notification | String | Notification name +supressed\_notifications | Number | Bitmask for suppressed notifications. + +Used to sync the state of a notification object within the same HA zone. + +##### Functions + +Event Sender: `Notification::OnSuppressedNotificationsChanged` +Event Receiver: `SuppressedNotificationTypesChangedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Notification does not exist. +* Origin endpoint is not within the local zone. + + +#### event::SetNextNotification <a id="technical-concepts-json-rpc-messages-event-setnextnotification"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetNextNotification +params | Dictionary + +##### Params + +Key | Type | Description +-------------------|---------------|------------------ +host | String | Host name +service | String | Service name +notification | String | Notification name +next\_notification | Timestamp | Next scheduled notification time as UNIX timestamp. + +##### Functions + +Event Sender: `Notification::OnNextNotificationChanged` +Event Receiver: `NextNotificationChangedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Notification does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::SetForceNextCheck <a id="technical-concepts-json-rpc-messages-event-setforcenextcheck"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetForceNextCheck +params | Dictionary + +##### Params + +Key | Type | Description +----------|---------------|------------------ +host | String | Host name +service | String | Service name +forced | Boolean | Forced next check (execute now) + +##### Functions + +Event Sender: `Checkable::OnForceNextCheckChanged` +Event Receiver: `ForceNextCheckChangedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::SetForceNextNotification <a id="technical-concepts-json-rpc-messages-event-setforcenextnotification"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetForceNextNotification +params | Dictionary + +##### Params + +Key | Type | Description +----------|---------------|------------------ +host | String | Host name +service | String | Service name +forced | Boolean | Forced next check (execute now) + +##### Functions + +Event Sender: `Checkable::SetForceNextNotification` +Event Receiver: `ForceNextNotificationChangedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::SetAcknowledgement <a id="technical-concepts-json-rpc-messages-event-setacknowledgement"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetAcknowledgement +params | Dictionary + +##### Params + +Key | Type | Description +-----------|---------------|------------------ +host | String | Host name +service | String | Service name +author | String | Acknowledgement author name. +comment | String | Acknowledgement comment content. +acktype | Number | Acknowledgement type (0=None, 1=Normal, 2=Sticky) +notify | Boolean | Notification should be sent. +persistent | Boolean | Whether the comment is persistent. +expiry | Timestamp | Optional expire time as UNIX timestamp. + +##### Functions + +Event Sender: `Checkable::OnForceNextCheckChanged` +Event Receiver: `ForceNextCheckChangedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::ClearAcknowledgement <a id="technical-concepts-json-rpc-messages-event-clearacknowledgement"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::ClearAcknowledgement +params | Dictionary + +##### Params + +Key | Type | Description +----------|---------------|------------------ +host | String | Host name +service | String | Service name + +##### Functions + +Event Sender: `Checkable::OnAcknowledgementCleared` +Event Receiver: `AcknowledgementClearedAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::SendNotifications <a id="technical-concepts-json-rpc-messages-event-sendnotifications"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SendNotifications +params | Dictionary + +##### Params + +Key | Type | Description +----------|---------------|------------------ +host | String | Host name +service | String | Service name +cr | Serialized CR | Check result +type | Number | enum NotificationType, same as `types` for notification objects. +author | String | Author name +text | String | Notification text + +##### Functions + +Event Sender: `Checkable::OnNotificationsRequested` +Event Receiver: `SendNotificationsAPIHandler` + +Signals that notifications have to be sent within the same HA zone. This is relevant if the checkable and its +notifications are active on different endpoints. + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint is not within the local zone. + +#### event::NotificationSentUser <a id="technical-concepts-json-rpc-messages-event-notificationsentuser"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::NotificationSentUser +params | Dictionary + +##### Params + +Key | Type | Description +--------------|-----------------|------------------ +host | String | Host name +service | String | Service name +notification | String | Notification name. +user | String | Notified user name. +type | Number | enum NotificationType, same as `types` in Notification objects. +cr | Serialized CR | Check result. +author | String | Notification author (for specific types) +text | String | Notification text (for specific types) +command | String | Notification command name. + +##### Functions + +Event Sender: `Checkable::OnNotificationSentToUser` +Event Receiver: `NotificationSentUserAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone the same as the receiver. This binds notification messages to the HA zone. + +#### event::NotificationSentToAllUsers <a id="technical-concepts-json-rpc-messages-event-notificationsenttoallusers"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::NotificationSentToAllUsers +params | Dictionary + +##### Params + +Key | Type | Description +----------------------------|-----------------|------------------ +host | String | Host name +service | String | Service name +notification | String | Notification name. +users | Array of String | Notified user names. +type | Number | enum NotificationType, same as `types` in Notification objects. +cr | Serialized CR | Check result. +author | String | Notification author (for specific types) +text | String | Notification text (for specific types) +last\_notification | Timestamp | Last notification time as UNIX timestamp. +next\_notification | Timestamp | Next scheduled notification time as UNIX timestamp. +notification\_number | Number | Current notification number in problem state. +last\_problem\_notification | Timestamp | Last problem notification time as UNIX timestamp. +no\_more\_notifications | Boolean | Whether to send future notifications when this notification becomes active on this HA node. + +##### Functions + +Event Sender: `Checkable::OnNotificationSentToAllUsers` +Event Receiver: `NotificationSentToAllUsersAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone the same as the receiver. This binds notification messages to the HA zone. + +#### event::ExecuteCommand <a id="technical-concepts-json-rpc-messages-event-executecommand"></a> + +> Location: `clusterevents-check.cpp` and `checkable-check.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::ExecuteCommand +params | Dictionary + +##### Params + +Key | Type | Description +---------------|---------------|------------------ +host | String | Host name. +service | String | Service name. +command\_type | String | `check_command` or `event_command`. +command | String | CheckCommand or EventCommand name. +check\_timeout | Number | Check timeout of the checkable object, if specified as `check_timeout` attribute. +macros | Dictionary | Command arguments as key/value pairs for remote execution. +endpoint | String | The endpoint to execute the command on. +deadline | Number | A Unix timestamp indicating the execution deadline +source | String | The execution UUID + + +##### Functions + +**Event Sender:** This gets constructed directly in `Checkable::ExecuteCheck()`, `Checkable::ExecuteEventHandler()` or `ApiActions::ExecuteCommand()` when a remote command endpoint is configured. + +* `Get{CheckCommand,EventCommand}()->Execute()` simulates an execution and extracts all command arguments into the `macro` dictionary (inside lib/methods tasks). +* When the endpoint is connected, the message is constructed and sent directly. +* When the endpoint is not connected and not syncing replay logs and 5m after application start, generate an UNKNOWN check result for the user ("not connected"). + +**Event Receiver:** `ExecuteCommandAPIHandler` + +Special handling, calls `ClusterEvents::EnqueueCheck()` for command endpoint checks. +This function enqueues check tasks into a queue which is controlled in `RemoteCheckThreadProc()`. +If the `endpoint` parameter is specified and is not equal to the local endpoint then the message is forwarded to the correct endpoint zone. + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Origin endpoint's zone is not a parent zone of the receiver endpoint. +* `accept_commands = false` in the `api` feature configuration sends back an UNKNOWN check result to the sender. + +The receiver constructs a virtual host object and looks for the local CheckCommand object. + +Returns UNKNOWN as check result to the sender + +* when the CheckCommand object does not exist. +* when there was an exception triggered from check execution, e.g. the plugin binary could not be executed or similar. + +The returned messages are synced directly to the sender's endpoint, no cluster broadcast. + +> **Note**: EventCommand errors are just logged on the remote endpoint. + +### event::UpdateExecutions <a id="technical-concepts-json-rpc-messages-event-updateexecutions"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::UpdateExecutions +params | Dictionary + +##### Params + +Key | Type | Description +---------------|---------------|------------------ +host | String | Host name. +service | String | Service name. +executions | Dictionary | Executions to be updated + +##### Functions + +**Event Sender:** `ClusterEvents::ExecutedCommandAPIHandler`, `ClusterEvents::UpdateExecutionsAPIHandler`, `ApiActions::ExecuteCommand` +**Event Receiver:** `ClusterEvents::UpdateExecutionsAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +### event::ExecutedCommand <a id="technical-concepts-json-rpc-messages-event-executedcommand"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::ExecutedCommand +params | Dictionary + +##### Params + +Key | Type | Description +---------------|---------------|------------------ +host | String | Host name. +service | String | Service name. +execution | String | The execution ID executed. +exitStatus | Number | The command exit status. +output | String | The command output. +start | Number | The unix timestamp at the start of the command execution +end | Number | The unix timestamp at the end of the command execution + +##### Functions + +**Event Sender:** `ClusterEvents::ExecuteCheckFromQueue`, `ClusterEvents::ExecuteCommandAPIHandler` +**Event Receiver:** `ClusterEvents::ExecutedCommandAPIHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Checkable does not exist. +* Origin endpoint's zone is not allowed to access this checkable. + +#### event::SetRemovalInfo <a id="technical-concepts-json-rpc-messages-event-setremovalinfo"></a> + +> Location: `clusterevents.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | event::SetRemovalInfo +params | Dictionary + +##### Params + +Key | Type | Description +---------------|-------------|--------------------------------- +object\_type | String | Object type (`"Comment"` or `"Downtime"`) +object\_name | String | Object name +removed\_by | String | Name of the removal requestor +remove\_time | Timestamp | Time of the remove operation + +##### Functions + +**Event Sender**: `Comment::OnRemovalInfoChanged` and `Downtime::OnRemovalInfoChanged` +**Event Receiver**: `SetRemovalInfoAPIHandler` + +This message is used to synchronize information about manual comment and downtime removals before deleting the +corresponding object. + +##### Permissions + +This message is only accepted from the local zone and from parent zones. + +#### config::Update <a id="technical-concepts-json-rpc-messages-config-update"></a> + +> Location: `apilistener-filesync.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | config::Update +params | Dictionary + +##### Params + +Key | Type | Description +-----------|---------------|------------------ +update | Dictionary | Config file paths and their content. +update\_v2 | Dictionary | Additional meta config files introduced in 2.4+ for compatibility reasons. + +##### Functions + +**Event Sender:** `SendConfigUpdate()` called in `ApiListener::SyncClient()` when a new client endpoint connects. +**Event Receiver:** `ConfigUpdateHandler` reads the config update content and stores them in `/var/lib/icinga2/api`. +When it detects a configuration change, the function requests and application restart. + +##### Permissions + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* The origin sender is not in a parent zone of the receiver. +* `api` feature does not accept config. + +Config updates will be ignored when: + +* The zone is not configured on the receiver endpoint. +* The zone is authoritative on this instance (this only happens on a master which has `/etc/icinga2/zones.d` populated, and prevents sync loops) + +#### config::UpdateObject <a id="technical-concepts-json-rpc-messages-config-updateobject"></a> + +> Location: `apilistener-configsync.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | config::UpdateObject +params | Dictionary + +##### Params + +Key | Type | Description +---------------------|-------------|------------------ +name | String | Object name. +type | String | Object type name. +version | Number | Object version. +config | String | Config file content for `_api` packages. +modified\_attributes | Dictionary | Modified attributes at runtime as key value pairs. +original\_attributes | Array | Original attributes as array of keys. + + +##### Functions + +**Event Sender:** Either on client connect (full sync), or runtime created/updated object + +`ApiListener::SendRuntimeConfigObjects()` gets called when a new endpoint is connected +and runtime created config objects need to be synced. This invokes a call to `UpdateConfigObject()` +to only sync this JsonRpcConnection client. + +`ConfigObject::OnActiveChanged` (created or deleted) or `ConfigObject::OnVersionChanged` (updated) +also call `UpdateConfigObject()`. + +**Event Receiver:** `ConfigUpdateObjectAPIHandler` calls `ConfigObjectUtility::CreateObject()` in order +to create the object if it is not already existing. Afterwards, all modified attributes are applied +and in case, original attributes are restored. The object version is set as well, keeping it in sync +with the sender. + +##### Permissions + +###### Sender + +Client receiver connects: + +The sender only syncs config object updates to a client which can access +the config object, in `ApiListener::SendRuntimeConfigObjects()`. + +In addition to that, the client endpoint's zone is checked whether this zone may access +the config object. + +Runtime updated object: + +Only if the config object belongs to the `_api` package. + + +###### Receiver + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Origin sender endpoint's zone is in a child zone. +* `api` feature does not accept config +* The received config object type does not exist (this is to prevent failures with older nodes and new object types). + +Error handling: + +* Log an error if `CreateObject` fails (only if the object does not already exist) +* Local object version is newer than the received version, object will not be updated. +* Compare modified and original attributes and restore any type of change here. + + +#### config::DeleteObject <a id="technical-concepts-json-rpc-messages-config-deleteobject"></a> + +> Location: `apilistener-configsync.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | config::DeleteObject +params | Dictionary + +##### Params + +Key | Type | Description +--------------------|-------------|------------------ +name | String | Object name. +type | String | Object type name. +version | Number | Object version. + +##### Functions + +**Event Sender:** + +`ConfigObject::OnActiveChanged` (created or deleted) or `ConfigObject::OnVersionChanged` (updated) +call `DeleteConfigObject()`. + +**Event Receiver:** `ConfigDeleteObjectAPIHandler` + +##### Permissions + +###### Sender + +Runtime deleted object: + +Only if the config object belongs to the `_api` package. + +###### Receiver + +The receiver will not process messages from not configured endpoints. + +Message updates will be dropped when: + +* Origin sender endpoint's zone is in a child zone. +* `api` feature does not accept config +* The received config object type does not exist (this is to prevent failures with older nodes and new object types). +* The object in question was not created at runtime, it does not belong to the `_api` package. + +Error handling: + +* Log an error if `DeleteObject` fails (only if the object does not already exist) + +#### pki::RequestCertificate <a id="technical-concepts-json-rpc-messages-pki-requestcertificate"></a> + +> Location: `jsonrpcconnection-pki.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | pki::RequestCertificate +params | Dictionary + +##### Params + +Key | Type | Description +--------------|---------------|------------------ +ticket | String | Own ticket, or as satellite in CA proxy from local store. +cert\_request | String | Certificate request content from local store, optional. + +##### Functions + +Event Sender: `RequestCertificateHandler` +Event Receiver: `RequestCertificateHandler` + +##### Permissions + +This is an anonymous request, and the number of anonymous clients can be configured +in the `api` feature. + +Only valid certificate request messages are processed, and valid signed certificates +won't be signed again. + +#### pki::UpdateCertificate <a id="technical-concepts-json-rpc-messages-pki-updatecertificate"></a> + +> Location: `jsonrpcconnection-pki.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | pki::UpdateCertificate +params | Dictionary + +##### Params + +Key | Type | Description +---------------------|---------------|------------------ +status\_code | Number | Status code, 0=ok. +cert | String | Signed certificate content. +ca | String | Public CA certificate content. +fingerprint\_request | String | Certificate fingerprint from the CSR. + + +##### Functions + +**Event Sender:** + +* When a client requests a certificate in `RequestCertificateHandler` and the satellite +already has a signed certificate, the `pki::UpdateCertificate` message is constructed and sent back. +* When the endpoint holding the master's CA private key (and TicketSalt private key) is able to sign +the request, the `pki::UpdateCertificate` message is constructed and sent back. + +**Event Receiver:** `UpdateCertificateHandler` + +##### Permissions + +Message updates are dropped when + +* The origin sender is not in a parent zone of the receiver. +* The certificate fingerprint is in an invalid format. + +#### log::SetLogPosition <a id="technical-concepts-json-rpc-messages-log-setlogposition"></a> + +> Location: `apilistener.cpp` and `jsonrpcconnection.cpp` + +##### Message Body + +Key | Value +----------|--------- +jsonrpc | 2.0 +method | log::SetLogPosition +params | Dictionary + +##### Params + +Key | Type | Description +--------------------|---------------|------------------ +log\_position | Timestamp | The endpoint's log position as UNIX timestamp. + + +##### Functions + +**Event Sender:** + +During log replay to a client endpoint in `ApiListener::ReplayLog()`, each processed +file generates a message which updates the log position timestamp. + +`ApiListener::ApiTimerHandler()` invokes a check to keep all connected endpoints and +their log position in sync during replay log. + +**Event Receiver:** `SetLogPositionHandler` + +##### Permissions + +The receiver will not process messages from not configured endpoints. |