diff options
Diffstat (limited to 'health/README.md')
| -rw-r--r-- | health/README.md | 234 |
1 files changed, 118 insertions, 116 deletions
diff --git a/health/README.md b/health/README.md index 345f7fc70..848c1bc3b 100644 --- a/health/README.md +++ b/health/README.md @@ -1,9 +1,9 @@ # Health monitoring -Each netdata node runs an independent thread evaluating health monitoring checks. +Each Netdata node runs an independent thread evaluating health monitoring checks. This thread has lock free access to the database, so that it can operate as a watchdog. -Health checks (alarms) are attached to netdata charts, allowing netdata to automatically +Health checks (alarms) are attached to Netdata charts, allowing Netdata to automatically activate an alarm as soon as a chart is created. This is very important for netdata, since many charts are dynamically created during runtime (for example, the chart tracking network interface packet drops, is automatically created on the first @@ -20,15 +20,15 @@ use expressions combining the latest value of any number of metrics. ## Health configuration reference -Stock netdata health configuration is in `/usr/lib/netdata/conf.d/health.d`. +Stock Netdata health configuration is in `/usr/lib/netdata/conf.d/health.d`. These files can be overwritten by copying them and editing them in `/etc/netdata/health.d` (run `/etc/netdata/edit-config` to edit them). In `/etc/netdata/health.d` you can also put any number of files (in any number of sub-directories) -with a suffix `.conf` to have them processed by netdata. +with a suffix `.conf` to have them processed by Netdata. -Health configuration can be reloaded at any time, without restarting netdata. -Just send netdata the SIGUSR2 signal, like this: +Health configuration can be reloaded at any time, without restarting Netdata. +Just send Netdata the SIGUSR2 signal, like this: ```sh killall -USR2 netdata @@ -38,11 +38,11 @@ killall -USR2 netdata There are 2 entities: -1. **alarms**, which are attached to specific charts, and +1. **alarms**, which are attached to specific charts, and -1. **templates**, which define rules that should be applied to all charts having a - specific `context`. You can use this feature to apply **alarms** to all disks, - all network interfaces, all mysql databases, all nginx web servers, etc. +2. **templates**, which define rules that should be applied to all charts having a + specific `context`. You can use this feature to apply **alarms** to all disks, + all network interfaces, all mysql databases, all nginx web servers, etc. Both of these entities have exactly the same format and feature set. The only difference is the label `alarm` or `template`. @@ -50,7 +50,7 @@ The only difference is the label `alarm` or `template`. Netdata supports overriding **templates** with **alarms**. For example, when a template is defined for a set of charts, an alarm with exactly the same name attached to the same chart the template matches, will have higher precedence -(i.e. netdata will use the alarm on this chart and prevent the template from being applied +(i.e. Netdata will use the alarm on this chart and prevent the template from being applied to it). ### The format @@ -135,7 +135,7 @@ hosts: server1 server2 database* !redis3 redis* The above says: use this alarm on all hosts named `server1`, `server2`, `database*`, and all `redis*` except `redis3`. -This is useful when you centralize metrics from multiple hosts, to one netdata. +This is useful when you centralize metrics from multiple hosts, to one Netdata. --- @@ -168,27 +168,27 @@ lookup: METHOD AFTER [at BEFORE] [every DURATION] [OPTIONS] [of DIMENSIONS] Everything is the same with [badges](../web/api/badges/). In short: -- `METHOD` is one of `average`, `min`, `max`, `sum`, `incremental-sum`. - This is required. +- `METHOD` is one of `average`, `min`, `max`, `sum`, `incremental-sum`. + This is required. -- `AFTER` is a relative number of seconds, but it also accepts a single letter for changing - the units, like `-1s` = 1 second in the past, `-1m` = 1 minute in the past, `-1h` = 1 hour - in the past, `-1d` = 1 day in the past. You need a negative number (i.e. how far in the past - to look for the value). **This is required**. +- `AFTER` is a relative number of seconds, but it also accepts a single letter for changing + the units, like `-1s` = 1 second in the past, `-1m` = 1 minute in the past, `-1h` = 1 hour + in the past, `-1d` = 1 day in the past. You need a negative number (i.e. how far in the past + to look for the value). **This is required**. -- `at BEFORE` is by default 0 and is not required. Using this you can define the end of the - lookup. So data will be evaluated between `AFTER` and `BEFORE`. +- `at BEFORE` is by default 0 and is not required. Using this you can define the end of the + lookup. So data will be evaluated between `AFTER` and `BEFORE`. -- `every DURATION` sets the updated frequency of the lookup (supports single letter units as - above too). +- `every DURATION` sets the updated frequency of the lookup (supports single letter units as + above too). -- `OPTIONS` is a space separated list of `percentage`, `absolute`, `min2max`, `unaligned`, - `match-ids`, `match-names`. Check the badges documentation for more info. +- `OPTIONS` is a space separated list of `percentage`, `absolute`, `min2max`, `unaligned`, + `match-ids`, `match-names`. Check the badges documentation for more info. -- `of DIMENSIONS` is optional and has to be the last parameter. Dimensions have to be separated - by `,` or `|`. The space characters found in dimensions will be kept as-is (a few dimensions - have spaces in their names). This accepts netdata simple patterns and the `match-ids` and - `match-names` options affect the searches for dimensions. +- `of DIMENSIONS` is optional and has to be the last parameter. Dimensions have to be separated + by `,` or `|`. The space characters found in dimensions will be kept as-is (a few dimensions + have spaces in their names). This accepts Netdata simple patterns and the `match-ids` and + `match-names` options affect the searches for dimensions. The result of the lookup will be available as `$this` and `$NAME` in expressions. The timestamps of the timeframe evaluated by the database lookup is available as variables @@ -259,6 +259,7 @@ Format: warn: EXPRESSION crit: EXPRESSION ``` + Check [Expressions](#expressions) for more information. --- @@ -289,8 +290,8 @@ Format: exec: SCRIPT ``` -The default `SCRIPT` is netdata's `alarm-notify.sh`, which supports all the notifications -methods netdata supports, including custom hooks. +The default `SCRIPT` is Netdata's `alarm-notify.sh`, which supports all the notifications +methods Netdata supports, including custom hooks. --- @@ -306,40 +307,41 @@ Format: delay: [[[up U] [down D] multiplier M] max X] ``` -- `up U` defines the delay to be applied to a notification for an alarm that raised its status - (i.e. CLEAR to WARNING, CLEAR to CRITICAL, WARNING to CRITICAL). For example, `up 10s`, the - notification for this event will be sent 10 seconds after the actual event. This is used in - hope the alarm will get back to its previous state within the duration given. The default `U` - is zero. +- `up U` defines the delay to be applied to a notification for an alarm that raised its status + (i.e. CLEAR to WARNING, CLEAR to CRITICAL, WARNING to CRITICAL). For example, `up 10s`, the + notification for this event will be sent 10 seconds after the actual event. This is used in + hope the alarm will get back to its previous state within the duration given. The default `U` + is zero. + +- `down D` defines the delay to be applied to a notification for an alarm that moves to lower + state (i.e. CRITICAL to WARNING, CRITICAL to CLEAR, WARNING to CLEAR). For example, `down 1m` + will delay the notification by 1 minute. This is used to prevent notifications for flapping + alarms. The default `D` is zero. -- `down D` defines the delay to be applied to a notification for an alarm that moves to lower - state (i.e. CRITICAL to WARNING, CRITICAL to CLEAR, WARNING to CLEAR). For example, `down 1m` - will delay the notification by 1 minute. This is used to prevent notifications for flapping - alarms. The default `D` is zero. +- `mutliplier M` multiplies `U` and `D` when an alarm changes state, while a notification is + delayed. The default multiplier is `1.0`. -- `mutliplier M` multiplies `U` and `D` when an alarm changes state, while a notification is - delayed. The default multiplier is `1.0`. +- `max X` defines the maximum absolute notification delay an alarm may get. The default `X` + is `max(U * M, D * M)` (i.e. the max duration of `U` or `D` multiplied once with `M`). -- `max X` defines the maximum absolute notification delay an alarm may get. The default `X` - is `max(U * M, D * M)` (i.e. the max duration of `U` or `D` multiplied once with `M`). + Example: - Example: + `delay: up 10s down 15m multiplier 2 max 1h` - `delay: up 10s down 15m multiplier 2 max 1h` + The time is `00:00:00` and the status of the alarm is CLEAR. - The time is `00:00:00` and the status of the alarm is CLEAR. + | time of event | new status | delay | notification will be sent|why| + |-------------|----------|:---:|-------------------------|---| + | 00:00:01 | WARNING | `up 10s` | 00:00:11|first state switch| + | 00:00:05 | CLEAR | `down 15m x2` | 00:30:05|the alarm changes state while a notification is delayed, so it was multiplied| + | 00:00:06 | WARNING | `up 10s x2 x2` | 00:00:26|multiplied twice| + | 00:00:07 | CLEAR | `down 15m x2 x2 x2` | 00:45:07|multiplied 3 times.| - time of event|new status|delay|notification will be sent|why - -------------|----------|:---:|-------------------------|--- - 00:00:01 | WARNING | `up 10s` | 00:00:11 |first state switch - 00:00:05 | CLEAR | `down 15m x2`| 00:30:05 |the alarm changes state while a notification is delayed, so it was multiplied - 00:00:06 | WARNING | `up 10s x2 x2` | 00:00:26 |multiplied twice - 00:00:07|CLEAR|`down 15m x2 x2 x2`|00:45:07|multiplied 3 times. + So: - So: - - `U` and `D` are multiplied by `M` every time the alarm changes state (any state, not just - their matching one) and a delay is in place. - - All are reset to their defaults when the alarm switches state without a delay in place. + - `U` and `D` are multiplied by `M` every time the alarm changes state (any state, not just + their matching one) and a delay is in place. + - All are reset to their defaults when the alarm switches state without a delay in place. --- @@ -353,9 +355,9 @@ Format: repeat: [off] [warning DURATION] [critical DURATION] ``` -* `off`: Turns off the repeating feature for the current alarm. This is effective when the default repeat settings has been enabled in health configuration. -* `warning DURATION`: Defines the interval when the alarm is in WARNING state. Use `0s` to turn off the repeating notification for WARNING mode. -* `critical DURATION`: Defines the interval when the alarm is in CRITICAL state. Use `0s` to turn off the repeating notification for CRITICAL mode. +- `off`: Turns off the repeating feature for the current alarm. This is effective when the default repeat settings has been enabled in health configuration. +- `warning DURATION`: Defines the interval when the alarm is in WARNING state. Use `0s` to turn off the repeating notification for WARNING mode. +- `critical DURATION`: Defines the interval when the alarm is in CRITICAL state. Use `0s` to turn off the repeating notification for CRITICAL mode. --- @@ -373,19 +375,17 @@ For some alarms we need compare two time-frames, to detect anomalies. For exampl ### Expressions -netdata has an internal [infix expression parser](../libnetdata/eval). +Netdata has an internal [infix expression parser](../libnetdata/eval). This parses expressions and creates an internal structure that allows fast execution of them. These operators are supported `+`, `-`, `*`, `/`, `<`, `<=`, `<>`, `!=`, `>`, `>=`, `&&`, `||`, `!`, `AND`, `OR`, `NOT`. Boolean operators result in either `1` (true) or `0` (false). -The conditional evaluation operator `?` is supported too. Using this operator IF-THEN-ELSE -conditional statements can be specified. The format is: `(condition) ? (true expression) : -(false expression)`. So, netdata will first evaluate the `condition` and based on the result -will either evaluate `true expression` or `false expression`. +The conditional evaluation operator `?` is supported too. Using this operator IF-THEN-ELSE conditional statements can be specified. The format is: `(condition) ? (true expression) :(false expression)`. So, Netdata will first evaluate the `condition` and based on the result will either evaluate `true expression` or `false expression`. + Example: `($this > 0) ? ($avail * 2) : ($used / 2)`. -Nested such expressions are also supported (i.e. `true expression` and `false expression` can -contain conditional evaluations). + +Nested such expressions are also supported (i.e. `true expression` and `false expression` can contain conditional evaluations). Expressions also support the `abs()` function. @@ -393,9 +393,9 @@ Expressions can have variables. Variables start with `$`. Check below for more i There are two special values you can use: -- `nan`, for example `$this != nan` will check if the variable `this` is available. A variable can be `nan` if the database lookup failed. All calculations (i.e. addition, multiplication, etc) with a `nan` result in a `nan`. +- `nan`, for example `$this != nan` will check if the variable `this` is available. A variable can be `nan` if the database lookup failed. All calculations (i.e. addition, multiplication, etc) with a `nan` result in a `nan`. -- `inf`, for example `$this != inf` will check if `this` is not infinite. A value or variable can be infinite if divided by zero. All calculations (i.e. addition, multiplication, etc) with a `inf` result in a `inf`. +- `inf`, for example `$this != inf` will check if `this` is not infinite. A value or variable can be infinite if divided by zero. All calculations (i.e. addition, multiplication, etc) with a `inf` result in a `inf`. --- @@ -407,7 +407,7 @@ or warning thresholds. This usage helps to avoid bogus messages resulting from variations in the value when it is varying regularly but staying close to the threshold value, without needing to delay sending messages at all. -An example of such usage from the default CPU usage alarms bundled with netdata is: +An example of such usage from the default CPU usage alarms bundled with Netdata is: ``` warn: $this > (($status >= $WARNING) ? (75) : (85)) @@ -415,25 +415,27 @@ crit: $this > (($status == $CRITICAL) ? (85) : (95)) ``` The above say: -* If the alarm is currently a warning, then the threshold for being considered a warning - is 75, otherwise it's 85. -* If the alarm is currently critical, then the threshold for being considered critical - is 85, otherwise it's 95. +- If the alarm is currently a warning, then the threshold for being considered a warning + is 75, otherwise it's 85. + +- If the alarm is currently critical, then the threshold for being considered critical + is 85, otherwise it's 95. Which in turn, results in the following behavior: -* While the value is rising, it will trigger a warning when it exceeds 85, and a critical - alert when it exceeds 95. -* While the value is falling, it will return to a warning state when it goes below 85, - and a normal state when it goes below 75. +- While the value is rising, it will trigger a warning when it exceeds 85, and a critical + alert when it exceeds 95. + +- While the value is falling, it will return to a warning state when it goes below 85, + and a normal state when it goes below 75. -* If the value is constantly varying between 80 and 90, then it will trigger a warning the - first time it goes above 85, but will remain a warning until it goes below 75 (or goes above 85). +- If the value is constantly varying between 80 and 90, then it will trigger a warning the + first time it goes above 85, but will remain a warning until it goes below 75 (or goes above 85). -* If the value is constantly varying between 90 and 100, then it will trigger a critical alert - the first time it goes above 95, but will remain a critical alert goes below 85 (at which - point it will return to being a warning). +- If the value is constantly varying between 90 and 100, then it will trigger a critical alert + the first time it goes above 95, but will remain a critical alert goes below 85 (at which + point it will return to being a warning). --- @@ -445,21 +447,21 @@ Example: [variables for the `system.cpu` chart of the registry](https://registry _Hint: If you don't know how to find the CHART_NAME, you can read about it [here](../docs/Charts.md#charts)._ - Netdata supports 3 internal indexes for variables that will be used in health monitoring. + <details markdown="1"><summary>The variables below can be used in both chart alarms and context templates.</summary> Although the `alarm_variables` link shows you variables for a particular chart, the same variables can also be used in templates for charts belonging to the same [context](../docs/Charts.md#contexts). The reason is that all charts of a given contexts are essentially identical, with the only difference being the [family](../docs/Charts.md#families) that identifies a particular hardware or software instance. Charts and templates do not apply to specific families anyway, unless if you explicitly limit an alarm with the [alarm line `families`](#alarm-line-families). </details> - - **chart local variables**. All the dimensions of the chart are exposed as local variables. The value of $this for the other configured alarms of the chart also appears, under the name of each configured alarm. +- **chart local variables**. All the dimensions of the chart are exposed as local variables. The value of $this for the other configured alarms of the chart also appears, under the name of each configured alarm. Charts also define a few special variables: - - `$last_collected_t` is the unix timestamp of the last data collection - - `$collected_total_raw` is the sum of all the dimensions (their last collected values) - - `$update_every` is the update frequency of the chart - - `$green` and `$red` the threshold defined in alarms (these are per chart - the charts - inherits them from the the first alarm that defined them) + - `$last_collected_t` is the unix timestamp of the last data collection + - `$collected_total_raw` is the sum of all the dimensions (their last collected values) + - `$update_every` is the update frequency of the chart + - `$green` and `$red` the threshold defined in alarms (these are per chart - the charts + inherits them from the the first alarm that defined them) Chart dimensions define their last calculated (i.e. interpolated) value, exactly as shown on the charts, but also a variable with their name and suffix `_raw` that resolves @@ -467,49 +469,49 @@ Although the `alarm_variables` link shows you variables for a particular chart, that resolves to unix timestamp the dimension was last collected (there may be dimensions that fail to be collected while others continue normally). - - **family variables**. Families are used to group charts together. For example all `eth0` +- **family variables**. Families are used to group charts together. For example all `eth0` charts, have `family = eth0`. This index includes all local variables, but if there are overlapping variables, only the first are exposed. - - **host variables**. All the dimensions of all charts, including all alarms, in fullname. +- **host variables**. All the dimensions of all charts, including all alarms, in fullname. Fullname is `CHART.VARIABLE`, where `CHART` is either the chart id or the chart name (both are supported). - - **special variables*** are: +- **special variables\*** are: - - `$this`, which is resolved to the value of the current alarm. + - `$this`, which is resolved to the value of the current alarm. - - `$status`, which is resolved to the current status of the alarm (the current = the last - status, i.e. before the current database lookup and the evaluation of the `calc` line). - This values can be compared with `$REMOVED`, `$UNINITIALIZED`, `$UNDEFINED`, `$CLEAR`, - `$WARNING`, `$CRITICAL`. These values are incremental, ie. `$status > $CLEAR` works as - expected. + - `$status`, which is resolved to the current status of the alarm (the current = the last + status, i.e. before the current database lookup and the evaluation of the `calc` line). + This values can be compared with `$REMOVED`, `$UNINITIALIZED`, `$UNDEFINED`, `$CLEAR`, + `$WARNING`, `$CRITICAL`. These values are incremental, ie. `$status > $CLEAR` works as + expected. - - `$now`, which is resolved to current unix timestamp. + - `$now`, which is resolved to current unix timestamp. ## Alarm Statuses Alarms can have the following statuses: - - `REMOVED` - the alarm has been deleted (this happens when a SIGUSR2 is sent to netdata +- `REMOVED` - the alarm has been deleted (this happens when a SIGUSR2 is sent to Netdata to reload health configuration) - - `UNINITIALIZED` - the alarm is not initialized yet +- `UNINITIALIZED` - the alarm is not initialized yet - - `UNDEFINED` - the alarm failed to be calculated (i.e. the database lookup failed, +- `UNDEFINED` - the alarm failed to be calculated (i.e. the database lookup failed, a division by zero occurred, etc) - - `CLEAR` - the alarm is not armed / raised (i.e. is OK) +- `CLEAR` - the alarm is not armed / raised (i.e. is OK) - - `WARNING` - the warning expression resulted in true or non-zero +- `WARNING` - the warning expression resulted in true or non-zero - - `CRITICAL` - the critical expression resulted in true or non-zero +- `CRITICAL` - the critical expression resulted in true or non-zero The external script will be called for all status changes. ## Examples -Check the `health/health.d/` directory for all alarms shipped with netdata. +Check the `health/health.d/` directory for all alarms shipped with Netdata. Here are a few examples: @@ -526,7 +528,7 @@ template: apache_last_collected_secs crit: $this > (10 * $update_every) ``` -The above checks that netdata is able to collect data from apache. In detail: +The above checks that Netdata is able to collect data from apache. In detail: ``` template: apache_last_collected_secs @@ -547,10 +549,10 @@ The above applies the **template** to all charts that have `context = apache.req calc: $now - $last_collected_t ``` -- `$now` is a standard variable that resolves to the current timestamp. +- `$now` is a standard variable that resolves to the current timestamp. -- `$last_collected_t` is the last data collection timestamp of the chart. - So this calculation gives the number of seconds passed since the last data collection. +- `$last_collected_t` is the last data collection timestamp of the chart. + So this calculation gives the number of seconds passed since the last data collection. ``` every: 10s @@ -565,8 +567,8 @@ The alarm will be evaluated every 10 seconds. If these result in non-zero or true, they trigger the alarm. -- `$this` refers to the value of this alarm (i.e. the result of the `calc` line. - We could also use `$apache_last_collected_secs`. +- `$this` refers to the value of this alarm (i.e. the result of the `calc` line. + We could also use `$apache_last_collected_secs`. `$update_every` is the update frequency of the chart, in seconds. @@ -653,12 +655,12 @@ The `lookup` line will calculate the sum of the all dropped packets in the last The `crit` line will issue a critical alarm if even a single packet has been dropped. Note that the drops chart does not exist if a network interface has never dropped a single packet. -When netdata detects a dropped packet, it will add the chart and it will automatically attach this +When Netdata detects a dropped packet, it will add the chart and it will automatically attach this alarm to it. ## Troubleshooting -You can compile netdata with [debugging](../daemon#debugging) and then set in `netdata.conf`: +You can compile Netdata with [debugging](../daemon#debugging) and then set in `netdata.conf`: ``` [global] @@ -671,10 +673,10 @@ Important: this will generate a lot of output in debug.log. You can find the context of charts by looking up the chart in either `http://your.netdata:19999/netdata.conf` or `http://your.netdata:19999/api/v1/charts`. -You can find how netdata interpreted the expressions by examining the alarm at `http://your.netdata:19999/api/v1/alarms?all`. For each expression, netdata will return the expression as given in its config file, and the same expression with additional parentheses added to indicate the evaluation flow of the expression. +You can find how Netdata interpreted the expressions by examining the alarm at `http://your.netdata:19999/api/v1/alarms?all`. For each expression, Netdata will return the expression as given in its config file, and the same expression with additional parentheses added to indicate the evaluation flow of the expression. ## Disabling health checks or silencing notifications at runtime It's currently not possible to schedule notifications from within the alarm template. For those scenarios where you need to temporary disable notifications (for instance when running backups triggers a disk alert) you can disable or silence notifications are runtime. The health checks can be controlled at runtime via the [health management api](../web/api/health/#health-management-api). -[]() +[](<>) |