1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
<!--
title: "Enable alarm notifications"
description: "Send Netdata alarms from a centralized place with Netdata Cloud, or configure nodes individually, to enable incident response and faster resolution."
custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/monitor/enable-notifications.md"
sidebar_label: "Enable alarm notifications"
learn_status: "Published"
learn_topic_type: "Tasks"
learn_rel_path: "Setup"
-->
# Enable alarm notifications
Netdata offers two ways to receive alarm notifications on external platforms. These methods work independently _or_ in
parallel, which means you can enable both at the same time to send alarm notifications to any number of endpoints.
Both methods use a node's health alarms to generate the content of alarm notifications. Read the doc on [configuring
alarms](https://github.com/netdata/netdata/blob/master/docs/monitor/configure-alarms.md) to change the preconfigured thresholds or to create tailored alarms for your
infrastructure.
Netdata Cloud offers [centralized alarm notifications](#netdata-cloud) via email, which leverages the health status
information already streamed to Netdata Cloud from connected nodes to send notifications to those who have enabled them.
The Netdata Agent has a [notification system](#netdata-agent) that supports more than a dozen services, such as email,
Slack, PagerDuty, Twilio, Amazon SNS, Discord, and much more.
For example, use centralized alarm notifications in Netdata Cloud for immediate, zero-configuration alarm notifications
for your team, then configure individual nodes send notifications to a PagerDuty endpoint for an automated incident
response process.
## Netdata Cloud
Netdata Cloud's [centralized alarm
notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.mdx) is a zero-configuration way to
get notified when an anomaly or incident strikes any node or application in your infrastructure. The advantage of using
centralized alarm notifications from Netdata Cloud is that you don't have to worry about configuring each node in your
infrastructure.
To enable centralized alarm notifications for a Space, click on **Manage Space** in the left-hand menu, then click on
the **Notifications** tab. Click the toggle switch next to **E-mail** to enable this notification method.
Next, enable notifications on a user level by clicking on your profile icon, then **Profile** in the dropdown. The
**Notifications** tab reveals rich management settings, including the ability to enable/disable methods entirely or
choose what types of notifications to receive from each War Room.
See the [centralized alarm notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.mdx)
reference doc for further details about what information is conveyed in an email notification, flood protection, and
more.
## Netdata Agent
The Netdata Agent's [notification system](https://github.com/netdata/netdata/blob/master/health/notifications/README.md) runs on every node and dispatches
notifications based on configured endpoints and roles. You can enable multiple endpoints on any one node _and_ use Agent
notifications in parallel with centralized alarm notifications in Netdata Cloud.
> ❗ If you want to enable notifications from multiple nodes in your infrastructure, each running the Netdata Agent, you
> must configure each node individually.
Below, we'll use [Slack notifications](#enable-slack-notifications) as an example of the process of enabling any
notification platform.
### Supported notification endpoints
- [**alerta.io**](https://github.com/netdata/netdata/blob/master/health/notifications/alerta/README.md)
- [**Amazon SNS**](https://github.com/netdata/netdata/blob/master/health/notifications/awssns/README.md)
- [**Custom endpoint**](https://github.com/netdata/netdata/blob/master/health/notifications/custom/README.md)
- [**Discord**](https://github.com/netdata/netdata/blob/master/health/notifications/discord/README.md)
- [**Dynatrace**](https://github.com/netdata/netdata/blob/master/health/notifications/dynatrace/README.md)
- [**Email**](https://github.com/netdata/netdata/blob/master/health/notifications/email/README.md)
- [**Flock**](https://github.com/netdata/netdata/blob/master/health/notifications/flock/README.md)
- [**Google Hangouts**](https://github.com/netdata/netdata/blob/master/health/notifications/hangouts/README.md)
- [**Gotify**](https://github.com/netdata/netdata/blob/master/health/notifications/gotify/README.md)
- [**IRC**](https://github.com/netdata/netdata/blob/master/health/notifications/irc/README.md)
- [**Kavenegar**](https://github.com/netdata/netdata/blob/master/health/notifications/kavenegar/README.md)
- [**Matrix**](https://github.com/netdata/netdata/blob/master/health/notifications/matrix/README.md)
- [**Messagebird**](https://github.com/netdata/netdata/blob/master/health/notifications/messagebird/README.md)
- [**Microsoft Teams**](https://github.com/netdata/netdata/blob/master/health/notifications/msteams/README.md)
- [**Netdata Agent dashboard**](https://github.com/netdata/netdata/blob/master/health/notifications/web/README.md)
- [**Opsgenie**](https://github.com/netdata/netdata/blob/master/health/notifications/opsgenie/README.md)
- [**PagerDuty**](https://github.com/netdata/netdata/blob/master/health/notifications/pagerduty/README.md)
- [**Prowl**](https://github.com/netdata/netdata/blob/master/health/notifications/prowl/README.md)
- [**PushBullet**](https://github.com/netdata/netdata/blob/master/health/notifications/pushbullet/README.md)
- [**PushOver**](https://github.com/netdata/netdata/blob/master/health/notifications/pushover/README.md)
- [**Rocket.Chat**](https://github.com/netdata/netdata/blob/master/health/notifications/rocketchat/README.md)
- [**Slack**](https://github.com/netdata/netdata/blob/master/health/notifications/slack/README.md)
- [**SMS Server Tools 3**](https://github.com/netdata/netdata/blob/master/health/notifications/smstools3/README.md)
- [**StackPulse**](https://github.com/netdata/netdata/blob/master/health/notifications/stackpulse/README.md)
- [**Syslog**](https://github.com/netdata/netdata/blob/master/health/notifications/syslog/README.md)
- [**Telegram**](https://github.com/netdata/netdata/blob/master/health/notifications/telegram/README.md)
- [**Twilio**](https://github.com/netdata/netdata/blob/master/health/notifications/twilio/README.md)
### Enable Slack notifications
First, [Add an incoming webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) in Slack for the channel where you
want to see alarm notifications from Netdata. Click the green **Add to Slack** button, choose the channel, and click the
**Add Incoming WebHooks Integration** button.
On the following page, you'll receive a **Webhook URL**. That's what you'll need to configure Netdata, so keep it handy.
Navigate to your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) and use `edit-config` to
open the `health_alarm_notify.conf` file:
```bash
sudo ./edit-config health_alarm_notify.conf
```
Look for the `SLACK_WEBHOOK_URL=" "` line and add the incoming webhook URL you got from Slack:
```conf
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/XXXXXXXXXXXX"
```
A few lines down, edit the `DEFAULT_RECIPIENT_SLACK` line to contain a single hash `#` character. This instructs Netdata
to send a notification to the channel you configured with the incoming webhook.
```conf
DEFAULT_RECIPIENT_SLACK="#"
```
To test Slack notifications, switch to the Netdata user.
```bash
sudo su -s /bin/bash netdata
```
Next, run the `alarm-notify` script using the `test` option.
```bash
/usr/libexec/netdata/plugins.d/alarm-notify.sh test
```
You should receive three notifications in your Slack channel for each health status change: `WARNING`, `CRITICAL`, and
`CLEAR`.
See the [Agent Slack notifications](https://github.com/netdata/netdata/blob/master/health/notifications/slack/README.md) doc for more options and information.
## What's next?
Now that you have health entities configured to your infrastructure's needs and notifications to inform you of anomalies
or incidents, your health monitoring setup is complete.
To make your dashboards most useful during root cause analysis, use Netdata's [distributed data
architecture](https://github.com/netdata/netdata/blob/master/docs/store/distributed-data-architecture.md) for the best-in-class performance and scalability.
### Related reference documentation
- [Netdata Cloud · Alarm notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.mdx)
- [Netdata Agent · Notifications](https://github.com/netdata/netdata/blob/master/health/notifications/README.md)
|
