1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
|
# Agent alert notifications
This is a reference documentation for Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
The `script to execute on alarm` line in `netdata.conf` defines the external script that will be called once the alert is triggered.
The default script is `alarm-notify.sh`.
> ### Info
>
> This file mentions editing configuration files.
>
> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.
> Note that to run the script you need to be inside your Netdata config directory.
>
> - Please also note that after most configuration changes you will need to [restart the Agent](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for the changes to take effect.
>
> It is recommended to use this way for configuring Netdata.
You can change the default script globally by editing `netdata.conf` and changing the `script to execute on alarm` in the `[health]` section.
`alarm-notify.sh` is capable of sending notifications:
- to multiple recipients
- using multiple notification methods
- filtering severity per recipient
It uses **roles**. For example `sysadmin`, `webmaster`, `dba`, etc.
Each alert is assigned to one or more roles, using the `to` line of the alert configuration. For example, here is the alert configuration for `ram.conf` that defaults to the role `sysadmin`:
```conf
alarm: ram_in_use
on: system.ram
class: Utilization
type: System
component: Memory
os: linux
hosts: *
calc: $used * 100 / ($used + $cached + $free + $buffers)
units: %
every: 10s
warn: $this > (($status >= $WARNING) ? (80) : (90))
crit: $this > (($status == $CRITICAL) ? (90) : (98))
delay: down 15m multiplier 1.5 max 1h
info: system memory utilization
to: sysadmin
```
Then `alarm-notify.sh` uses its own configuration file `health_alarm_notify.conf`, which at the bottom of the file stores the recipients per role, for all notification methods.
Here is an example, of the `sysadmin`'s role recipients for the email notification.
You can send the notification to multiple recipients by separating the emails with a space.
```conf
###############################################################################
# RECIPIENTS PER ROLE
# -----------------------------------------------------------------------------
# generic system alarms
# CPU, disks, network interfaces, entropy, etc
role_recipients_email[sysadmin]="someone@exaple.com someoneelse@example.com"
```
Each role may have one or more destinations and one or more notification methods.
So, for example the `sysadmin` role may send:
1. emails to admin1@example.com and admin2@example.com
2. pushover.net notifications to USERTOKENS `A`, `B` and `C`.
3. pushbullet.com push notifications to admin1@example.com and admin2@example.com
4. messages to the `#alerts` and `#systems` channels of a Slack workspace.
5. messages to Discord channels `#alerts` and `#systems`.
## Configuration
You can edit `health_alarm_notify.conf` using the `edit-config` script to configure:
- **Settings** per notification method:
All notification methods except email, require some configuration (i.e. API keys, tokens, destination rooms, channels, etc). Please check this section's content to find the configuration guides for your notification option of choice
- **Recipients** per role per notification method
```conf
role_recipients_email[sysadmin]="${DEFAULT_RECIPIENT_EMAIL}"
role_recipients_pushover[sysadmin]="${DEFAULT_RECIPIENT_PUSHOVER}"
role_recipients_pushbullet[sysadmin]="${DEFAULT_RECIPIENT_PUSHBULLET}"
role_recipients_telegram[sysadmin]="${DEFAULT_RECIPIENT_TELEGRAM}"
role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"
...
```
Here you can change the `${DEFAULT_...}` values to the values of the recipients you want, separated by a space if you have multiple recipients.
## Testing Alert Notifications
You can run the following command by hand, to test alerts configuration:
```sh
# become user netdata
sudo su -s /bin/bash netdata
# enable debugging info on the console
export NETDATA_ALARM_NOTIFY_DEBUG=1
# send test alarms to sysadmin
/usr/libexec/netdata/plugins.d/alarm-notify.sh test
# send test alarms to any role
/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
```
If you are [running your own registry](https://github.com/netdata/netdata/blob/master/registry/README.md#run-your-own-registry), add `export NETDATA_REGISTRY_URL=[YOUR_URL]` before calling `alarm-notify.sh`.
> If you need to dig even deeper, you can trace the execution with `bash -x`. Note that in test mode, `alarm-notify.sh` calls itself with many more arguments. So first do:
>
>```sh
>bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test
>```
>
> And then look in the output for the alarm-notify.sh calls and run the one you want to trace with `bash -x`.
## Global configuration options
### Notification Filtering
When you define recipients per role for notification methods, you can append `|critical` to limit the notifications that are sent.
In the following examples, the first recipient receives all the alarms, while the second one receives only notifications for alarms that have at some point become critical.
The second user may still receive warning and clear notifications, but only for the event that previously caused a critical alarm.
```conf
email : "user1@example.com user2@example.com|critical"
pushover : "2987343...9437837 8756278...2362736|critical"
telegram : "111827421 112746832|critical"
slack : "alarms disasters|critical"
alerta : "alarms disasters|critical"
flock : "alarms disasters|critical"
discord : "alarms disasters|critical"
twilio : "+15555555555 +17777777777|critical"
messagebird: "+15555555555 +17777777777|critical"
kavenegar : "09155555555 09177777777|critical"
pd : "<pd_service_key_1> <pd_service_key_2>|critical"
irc : "<irc_channel_1> <irc_channel_2>|critical"
```
If a per role recipient is set to an empty string, the default recipient of the given
notification method (email, pushover, telegram, slack, alerta, etc) will be used.
To disable a notification, use the recipient called: disabled
This works for all notification methods (including the default recipients).
### Proxy configuration
If you need to send curl based notifications (pushover, pushbullet, slack, alerta,
flock, discord, telegram) via a proxy, you should set these variables to your proxy address:
```conf
export http_proxy="http://10.0.0.1:3128/"
export https_proxy="http://10.0.0.1:3128/"
```
### Notification images
Images in notifications need to be downloaded from an Internet facing site.
To allow notification providers to fetch the icons/images, by default we set the URL of the global public netdata registry.
If you have an Internet facing netdata (or you have copied the images/ folder
of netdata to your web server), set its URL here, to fetch the notification
images from it.
```conf
images_base_url="http://my.public.netdata.server:19999"
```
### Date handling
You can configure netdata alerts to send dates in any format you want via editing the `date_format` variable.
This uses standard `date` command format strings. See `man date` for
more info on what formats are supported.
Note that this has to start with a '+', otherwise it won't work.
- For ISO 8601 dates, use `+%FT%T%z`
- For RFC 5322 dates, use `+%a, %d %b %Y %H:%M:%S %z`
- For RFC 3339 dates, use `+%F %T%:z`
- For RFC 1123 dates, use `+%a, %d %b %Y %H:%M:%S %Z`
- For RFC 1036 dates, use `+%A, %d-%b-%y %H:%M:%S %Z`
- For a reasonably local date and time (in that order), use `+%x %X`
- For the old default behavior (compatible with ANSI C's `asctime()` function), leave the `date_format` field empty.
### Hostname handling
By default, Netdata will use the simple hostname for the system (the hostname with everything after the first `.` removed) when displaying the hostname in alert notifications.
If you instead prefer to have Netdata use the host's fully qualified domain name, you can set `use_fdqn` to `YES`.
This setting does not account for child systems for which the system you are configuring is a parent.
> ### Note
>
> If the system's host name is overridden in `/etc/netdata.conf` with the `hostname` option, that name will be used unconditionally.
|
