Merging upstream version 1.45.3+dfsg.

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

58 files changed, 3439 insertions, 18 deletions

diff --git a/src/health/notifications/README.md b/src/health/notifications/README.md
new file mode 100644
index 000000000..5a0b74045
--- /dev/null
+++ b/src/health/notifications/README.md
@@ -0,0 +1,207 @@
+# Agent alert notifications
+
+This is a reference documentation for Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+The `script to execute on alarm` line in `netdata.conf` defines the external script that will be called once the alert is triggered.
+
+The default script is `alarm-notify.sh`.
+
+> ### Info
+>
+> This file mentions editing configuration files.  
+>
+> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.  
+> Note that to run the script you need to be inside your Netdata config directory.
+>
+> - Please also note that after most configuration changes you will need to [restart the Agent](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for the changes to take effect.
+>
+> It is recommended to use this way for configuring Netdata.
+
+You can change the default script globally by editing `netdata.conf` and changing the `script to execute on alarm` in the `[health]` section.
+
+`alarm-notify.sh` is capable of sending notifications:
+
+- to multiple recipients
+- using multiple notification methods
+- filtering severity per recipient
+
+It uses **roles**. For example `sysadmin`, `webmaster`, `dba`, etc.
+
+Each alert is assigned to one or more roles, using the `to` line of the alert  configuration. For example, here is the alert configuration for `ram.conf` that defaults to the role `sysadmin`:
+
+```conf
+    alarm: ram_in_use
+       on: system.ram
+    class: Utilization
+     type: System
+component: Memory
+       os: linux
+    hosts: *
+     calc: $used * 100 / ($used + $cached + $free + $buffers)
+    units: %
+    every: 10s
+     warn: $this > (($status >= $WARNING)  ? (80) : (90))
+     crit: $this > (($status == $CRITICAL) ? (90) : (98))
+    delay: down 15m multiplier 1.5 max 1h
+     info: system memory utilization
+       to: sysadmin
+```
+
+Then `alarm-notify.sh` uses its own configuration file `health_alarm_notify.conf`, which at the bottom of the file stores the recipients per role, for all notification methods.
+
+Here is an example, of the `sysadmin`'s role recipients for the email notification.  
+You can send the notification to multiple recipients by separating the emails with a space.
+
+```conf
+
+###############################################################################
+# RECIPIENTS PER ROLE
+
+# -----------------------------------------------------------------------------
+# generic system alerts
+# CPU, disks, network interfaces, entropy, etc
+
+role_recipients_email[sysadmin]="someone@exaple.com someoneelse@example.com"
+```
+
+Each role may have one or more destinations and one or more notification methods.
+
+So, for example the `sysadmin` role may send:
+
+1. emails to admin1@example.com and admin2@example.com
+2. pushover.net notifications to USERTOKENS `A`, `B` and `C`.
+3. pushbullet.com push notifications to admin1@example.com and admin2@example.com
+4. messages to the `#alerts` and `#systems` channels of a Slack workspace.
+5. messages to Discord channels `#alerts` and `#systems`.
+
+## Configuration
+
+You can edit `health_alarm_notify.conf` using the `edit-config` script to configure:
+
+- **Settings** per notification method:
+
+     All notification methods except email, require some configuration (i.e. API keys, tokens, destination rooms, channels, etc). Please check this section's content to find the configuration guides for your notification option of choice
+
+- **Recipients** per role per notification method
+
+     ```conf
+     role_recipients_email[sysadmin]="${DEFAULT_RECIPIENT_EMAIL}"
+     role_recipients_pushover[sysadmin]="${DEFAULT_RECIPIENT_PUSHOVER}"
+     role_recipients_pushbullet[sysadmin]="${DEFAULT_RECIPIENT_PUSHBULLET}"
+     role_recipients_telegram[sysadmin]="${DEFAULT_RECIPIENT_TELEGRAM}"
+     role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"
+     ...
+     ```
+
+     Here you can change the `${DEFAULT_...}` values to the values of the recipients you want, separated by a space if you have multiple recipients.
+
+## Testing Alert Notifications
+
+You can run the following command by hand, to test alerts configuration:
+
+```sh
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alerts to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alerts to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+If you are [running your own registry](https://github.com/netdata/netdata/blob/master/src/registry/README.md#run-your-own-registry), add `export NETDATA_REGISTRY_URL=[YOUR_URL]` before calling `alarm-notify.sh`.
+
+> If you need to dig even deeper, you can trace the execution with `bash -x`. Note that in test mode, `alarm-notify.sh` calls itself with many more arguments. So first do:
+>
+>```sh
+>bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test
+>```
+>
+> And then look in the output for the alarm-notify.sh calls and run the one you want to trace with `bash -x`.
+
+## Global configuration options
+
+### Notification Filtering
+
+When you define recipients per role for notification methods, you can append `|critical` to limit the notifications that are sent.
+
+In the following examples, the first recipient receives all the alerts, while the second one receives only notifications for alerts that have at some point become critical.
+The second user may still receive warning and clear notifications, but only for the event that previously caused a critical alert.
+
+```conf
+ email      : "user1@example.com user2@example.com|critical"
+ pushover   : "2987343...9437837 8756278...2362736|critical"
+ telegram   : "111827421 112746832|critical"
+ slack      : "alerts disasters|critical"
+ alerta     : "alerts disasters|critical"
+ flock      : "alerts disasters|critical"
+ discord    : "alerts disasters|critical"
+ twilio     : "+15555555555 +17777777777|critical"
+ messagebird: "+15555555555 +17777777777|critical"
+ kavenegar  : "09155555555 09177777777|critical"
+ pd         : "<pd_service_key_1> <pd_service_key_2>|critical"
+ irc        : "<irc_channel_1> <irc_channel_2>|critical"
+```
+
+If a per role recipient is set to an empty string, the default recipient of the given
+notification method (email, pushover, telegram, slack, alerta, etc.) will be used.
+
+To disable a notification, use the recipient called: disabled
+This works for all notification methods (including the default recipients).
+
+### Proxy configuration
+
+If you need to send curl based notifications (pushover, pushbullet, slack, alerta,
+flock, discord, telegram) via a proxy, you should set these variables to your proxy address:
+
+```conf
+export http_proxy="http://10.0.0.1:3128/"
+export https_proxy="http://10.0.0.1:3128/"
+```
+
+### Notification images
+
+Images in notifications need to be downloaded from an Internet facing site.
+
+To allow notification providers to fetch the icons/images, by default we set the URL of the global public netdata registry.
+
+If you have an Internet facing netdata (or you have copied the images/ folder
+of netdata to your web server), set its URL here, to fetch the notification
+images from it.
+
+```conf
+images_base_url="http://my.public.netdata.server:19999"
+```
+
+### Date handling
+
+You can configure netdata alerts to send dates in any format you want via editing the `date_format` variable.
+
+This uses standard `date` command format strings. See `man date` for
+more info on what formats are supported.
+
+Note that this has to start with a '+', otherwise it won't work.
+
+- For ISO 8601 dates, use `+%FT%T%z`
+- For RFC 5322 dates, use `+%a, %d %b %Y %H:%M:%S %z`
+- For RFC 3339 dates, use `+%F %T%:z`
+- For RFC 1123 dates, use `+%a, %d %b %Y %H:%M:%S %Z`
+- For RFC 1036 dates, use `+%A, %d-%b-%y %H:%M:%S %Z`
+- For a reasonably local date and time (in that order), use `+%x %X`
+- For the old default behavior (compatible with ANSI C's `asctime()` function), leave the `date_format` field empty.
+
+### Hostname handling
+
+By default, Netdata will use the simple hostname for the system (the hostname with everything after the first `.` removed) when displaying the hostname in alert notifications.
+
+If you instead prefer to have Netdata use the host's fully qualified domain name, you can set `use_fdqn` to `YES`.
+
+This setting does not account for child systems for which the system you are configuring is a parent.
+
+> ### Note
+>
+> If the system's host name is overridden in `/etc/netdata.conf` with the `hostname` option, that name will be used unconditionally.
diff --git a/health/notifications/alarm-email.sh b/src/health/notifications/alarm-email.sh
index 69c4c3f8d..69c4c3f8d 100755
--- a/health/notifications/alarm-email.sh
+++ b/src/health/notifications/alarm-email.sh
diff --git a/health/notifications/alarm-notify.sh.in b/src/health/notifications/alarm-notify.sh.in
index 9d95c21dc..9a5780de1 100755
--- a/health/notifications/alarm-notify.sh.in
+++ b/src/health/notifications/alarm-notify.sh.in
@@ -248,31 +248,32 @@ docurl() {
 # Used in a couple of places to write more compact code.
 
 method_names="
-email
-pushover
-pushbullet
-telegram
-slack
 alerta
-flock
+awssns
+custom
 discord
+dynatrace
+email
+fleep
+flock
+gotify
 hipchat
-twilio
+irc
+kavenegar
+matrix
 messagebird
-pd
-fleep
-syslog
-custom
 msteams
-kavenegar
+ntfy
+pd
 prowl
-irc
-awssns
+pushbullet
+pushover
 rocketchat
+slack
 sms
-dynatrace
-matrix
-ntfy
+syslog
+telegram
+twilio
 "
 
 # -----------------------------------------------------------------------------
@@ -2399,7 +2400,7 @@ send_ntfy() {
     msg="${host} ${status_message}: ${alarm} - ${info}"
     httpcode=$(docurl -X POST \
                       "${ntfy_auth_header[@]}" \
-                      -H "Icon: https://raw.githubusercontent.com/netdata/netdata/master/web/gui/dashboard/images/favicon-196x196.png" \
+                      -H "Icon: https://raw.githubusercontent.com/netdata/netdata/master/src/web/gui/dashboard/images/favicon-196x196.png" \
                       -H "Title: ${host}: ${name//_/ }" \
                       -H "Tags: ${emoji}" \
                       -H "Priority: ${priority}" \
diff --git a/health/notifications/alarm-test.sh b/src/health/notifications/alarm-test.sh
index 828aa756b..828aa756b 100755
--- a/health/notifications/alarm-test.sh
+++ b/src/health/notifications/alarm-test.sh
diff --git a/src/health/notifications/alerta/README.md b/src/health/notifications/alerta/README.md
new file mode 100644
index 000000000..e0e2d34da
--- /dev/null
+++ b/src/health/notifications/alerta/README.md
@@ -0,0 +1,128 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/alerta/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/alerta/metadata.yaml"
+sidebar_label: "Alerta"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Alerta
+
+
+<img src="https://netdata.cloud/img/alerta.png" width="150"/>
+
+
+The [Alerta](https://alerta.io/) monitoring system is a tool used to consolidate and de-duplicate alerts from multiple sources for quick ‘at-a-glance’ visualization. With just one system you can monitor alerts from many other monitoring tools on a single screen.
+You can send Netdata alerts to Alerta to see alerts coming from many Netdata hosts or also from a multi-host Netdata configuration.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- A working Alerta instance
+- An Alerta API key (if authentication in Alerta is enabled)
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_ALERTA | Set `SEND_ALERTA` to YES |  | yes |
+| ALERTA_WEBHOOK_URL | set `ALERTA_WEBHOOK_URL` to the API url you defined when you installed the Alerta server. |  | yes |
+| ALERTA_API_KEY | Set `ALERTA_API_KEY` to your API key. |  | yes |
+| DEFAULT_RECIPIENT_ALERTA | Set `DEFAULT_RECIPIENT_ALERTA` to the default recipient environment you want the alert notifications to be sent to. All roles will default to this variable if left unconfigured. |  | yes |
+| DEFAULT_RECIPIENT_CUSTOM | Set different recipient environments per role, by editing `DEFAULT_RECIPIENT_CUSTOM` with the environment name of your choice |  | no |
+
+##### ALERTA_API_KEY
+
+You will need an API key to send messages from any source, if Alerta is configured to use authentication (recommended). To create a new API key:
+1. Go to Configuration > API Keys.
+2. Create a new API key called "netdata" with `write:alerts` permission.
+
+
+##### DEFAULT_RECIPIENT_CUSTOM
+
+The `DEFAULT_RECIPIENT_CUSTOM` can be edited in the following entries at the bottom of the same file:
+
+```conf
+role_recipients_alerta[sysadmin]="Systems"
+role_recipients_alerta[domainadmin]="Domains"
+role_recipients_alerta[dba]="Databases Systems"
+role_recipients_alerta[webmaster]="Marketing Development"
+role_recipients_alerta[proxyadmin]="Proxy"
+role_recipients_alerta[sitemgr]="Sites"
+```
+
+The values you provide should be defined as environments in `/etc/alertad.conf` with `ALLOWED_ENVIRONMENTS` option.
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# alerta (alerta.io) global notification options
+
+SEND_ALERTA="YES"
+ALERTA_WEBHOOK_URL="http://yourserver/alerta/api"
+ALERTA_API_KEY="INSERT_YOUR_API_KEY_HERE"
+DEFAULT_RECIPIENT_ALERTA="Production"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/alerta/metadata.yaml b/src/health/notifications/alerta/metadata.yaml
index f815032b9..f815032b9 100644
--- a/health/notifications/alerta/metadata.yaml
+++ b/src/health/notifications/alerta/metadata.yaml
diff --git a/src/health/notifications/awssns/README.md b/src/health/notifications/awssns/README.md
new file mode 100644
index 000000000..350ef6b8d
--- /dev/null
+++ b/src/health/notifications/awssns/README.md
@@ -0,0 +1,180 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/awssns/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/awssns/metadata.yaml"
+sidebar_label: "AWS SNS"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# AWS SNS
+
+
+<img src="https://netdata.cloud/img/aws.svg" width="150"/>
+
+
+As part of its AWS suite, Amazon provides a notification broker service called 'Simple Notification Service' (SNS). Amazon SNS works similarly to Netdata's own notification system, allowing to dispatch a single notification to multiple subscribers of different types. Among other things, SNS supports sending notifications to:
+- Email addresses
+- Mobile Phones via SMS
+- HTTP or HTTPS web hooks
+- AWS Lambda functions
+- AWS SQS queues
+- Mobile applications via push notifications
+You can send notifications through Amazon SNS using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Limitations
+
+- While Amazon SNS supports sending differently formatted messages for different delivery methods, Netdata does not currently support this functionality.
+- For email notification support, we recommend using Netdata's email notifications, as it is has the following benefits:
+  - In most cases, it requires less configuration.
+  - Netdata's emails are nicely pre-formatted and support features like threading, which requires a lot of manual effort in SNS.
+  - It is less resource intensive and more cost-efficient than SNS.
+
+
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The [Amazon Web Services CLI tools](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) (awscli).
+- An actual home directory for the user you run Netdata as, instead of just using `/` as a home directory. The setup depends on the distribution, but `/var/lib/netdata` is the recommended directory. If you are using Netdata as a dedicated user, the permissions will already be correct.
+- An Amazon SNS topic to send notifications to with one or more subscribers. The Getting Started section of the Amazon SNS documentation covers the basics of how to set this up. Make note of the Topic ARN when you create the topic.
+- While not mandatory, it is highly recommended to create a dedicated IAM user on your account for Netdata to send notifications. This user needs to have programmatic access, and should only allow access to SNS. For an additional layer of security, you can create one for each system or group of systems.
+- Terminal access to the Agent you wish to configure.
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| aws path | The full path of the aws command. If empty, the system `$PATH` will be searched for it. If not found, Amazon SNS notifications will be silently disabled. |  | yes |
+| SEND_AWSNS | Set `SEND_AWSNS` to YES | YES | yes |
+| AWSSNS_MESSAGE_FORMAT | Set `AWSSNS_MESSAGE_FORMAT` to to the string that you want the alert to be sent into. | ${status} on ${host} at ${date}: ${chart} ${value_string} | yes |
+| DEFAULT_RECIPIENT_AWSSNS | Set `DEFAULT_RECIPIENT_AWSSNS` to the Topic ARN you noted down upon creating the Topic. |  | yes |
+
+##### AWSSNS_MESSAGE_FORMAT
+
+The supported variables are:
+
+| Variable name               | Description                                                                      |
+|:---------------------------:|:---------------------------------------------------------------------------------|
+| `${alarm}`                  | Like "name = value units"                                                        |
+| `${status_message}`         | Like "needs attention", "recovered", "is critical"                               |
+| `${severity}`               | Like "Escalated to CRITICAL", "Recovered from WARNING"                           |
+| `${raised_for}`             | Like "(alarm was raised for 10 minutes)"                                         |
+| `${host}`                   | The host generated this event                                                    |
+| `${url_host}`               | Same as ${host} but URL encoded                                                  |
+| `${unique_id}`              | The unique id of this event                                                      |
+| `${alarm_id}`               | The unique id of the alarm that generated this event                             |
+| `${event_id}`               | The incremental id of the event, for this alarm id                               |
+| `${when}`                   | The timestamp this event occurred                                                |
+| `${name}`                   | The name of the alarm, as given in netdata health.d entries                      |
+| `${url_name}`               | Same as ${name} but URL encoded                                                  |
+| `${chart}`                  | The name of the chart (type.id)                                                  |
+| `${url_chart}`              | Same as ${chart} but URL encoded                                                 |
+| `${status}`                 | The current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
+| `${old_status}`             | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
+| `${value}`                  | The current value of the alarm                                                   |
+| `${old_value}`              | The previous value of the alarm                                                  |
+| `${src}`                    | The line number and file the alarm has been configured                           |
+| `${duration}`               | The duration in seconds of the previous alarm state                              |
+| `${duration_txt}`           | Same as ${duration} for humans                                                   |
+| `${non_clear_duration}`     | The total duration in seconds this is/was non-clear                              |
+| `${non_clear_duration_txt}` | Same as ${non_clear_duration} for humans                                         |
+| `${units}`                  | The units of the value                                                           |
+| `${info}`                   | A short description of the alarm                                                 |
+| `${value_string}`           | Friendly value (with units)                                                      |
+| `${old_value_string}`       | Friendly old value (with units)                                                  |
+| `${image}`                  | The URL of an image to represent the status of the alarm                         |
+| `${color}`                  | A color in  AABBCC format for the alarm                                          |
+| `${goto_url}`               | The URL the user can click to see the netdata dashboard                          |
+| `${calc_expression}`        | The expression evaluated to provide the value for the alarm                      |
+| `${calc_param_values}`      | The value of the variables in the evaluated expression                           |
+| `${total_warnings}`         | The total number of alarms in WARNING state on the host                          |
+| `${total_critical}`         | The total number of alarms in CRITICAL state on the host                         |
+
+
+##### DEFAULT_RECIPIENT_AWSSNS
+
+All roles will default to this variable if left unconfigured.
+
+You can have different recipient Topics per **role**, by editing `DEFAULT_RECIPIENT_AWSSNS` with the Topic ARN you want, in the following entries at the bottom of the same file:
+
+```conf
+role_recipients_awssns[sysadmin]="arn:aws:sns:us-east-2:123456789012:Systems"
+role_recipients_awssns[domainadmin]="arn:aws:sns:us-east-2:123456789012:Domains"
+role_recipients_awssns[dba]="arn:aws:sns:us-east-2:123456789012:Databases"
+role_recipients_awssns[webmaster]="arn:aws:sns:us-east-2:123456789012:Development"
+role_recipients_awssns[proxyadmin]="arn:aws:sns:us-east-2:123456789012:Proxy"
+role_recipients_awssns[sitemgr]="arn:aws:sns:us-east-2:123456789012:Sites"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+An example working configuration would be:
+
+```yaml
+```conf
+#------------------------------------------------------------------------------
+# Amazon SNS notifications
+
+SEND_AWSSNS="YES"
+AWSSNS_MESSAGE_FORMAT="${status} on ${host} at ${date}: ${chart} ${value_string}"
+DEFAULT_RECIPIENT_AWSSNS="arn:aws:sns:us-east-2:123456789012:MyTopic"
+```
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/awssns/metadata.yaml b/src/health/notifications/awssns/metadata.yaml
index 93389bad0..93389bad0 100644
--- a/health/notifications/awssns/metadata.yaml
+++ b/src/health/notifications/awssns/metadata.yaml
diff --git a/src/health/notifications/custom/README.md b/src/health/notifications/custom/README.md
new file mode 100644
index 000000000..47af067ec
--- /dev/null
+++ b/src/health/notifications/custom/README.md
@@ -0,0 +1,211 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/custom/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/custom/metadata.yaml"
+sidebar_label: "Custom"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Custom
+
+
+<img src="https://netdata.cloud/img/custom.png" width="150"/>
+
+
+Netdata Agent's alert notification feature allows you to send custom notifications to any endpoint you choose.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_CUSTOM | Set `SEND_CUSTOM` to YES | YES | yes |
+| DEFAULT_RECIPIENT_CUSTOM | This value is dependent on how you handle the `${to}` variable inside the `custom_sender()` function. |  | yes |
+| custom_sender() | You can look at the other senders in `/usr/libexec/netdata/plugins.d/alarm-notify.sh` for examples of how to modify the function in this configuration file. |  | no |
+
+##### DEFAULT_RECIPIENT_CUSTOM
+
+All roles will default to this variable if left unconfigured. You can edit `DEFAULT_RECIPIENT_CUSTOM` with the variable you want, in the following entries at the bottom of the same file:
+```
+role_recipients_custom[sysadmin]="systems"
+role_recipients_custom[domainadmin]="domains"
+role_recipients_custom[dba]="databases systems"
+role_recipients_custom[webmaster]="marketing development"
+role_recipients_custom[proxyadmin]="proxy-admin"
+role_recipients_custom[sitemgr]="sites"
+```
+
+
+##### custom_sender()
+
+The following is a sample custom_sender() function in health_alarm_notify.conf, to send an SMS via an imaginary HTTPS endpoint to the SMS gateway:
+```
+custom_sender() {
+    # example human readable SMS
+    local msg="${host} ${status_message}: ${alarm} ${raised_for}"
+
+    # limit it to 160 characters and encode it for use in a URL
+    urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}"
+
+    # a space separated list of the recipients to send alarms to
+    to="${1}"
+
+    for phone in ${to}; do
+      httpcode=$(docurl -X POST \
+            --data-urlencode "From=XXX" \
+            --data-urlencode "To=${phone}" \
+            --data-urlencode "Body=${msg}" \
+            -u "${accountsid}:${accounttoken}" \
+        https://domain.website.com/)
+
+      if [ "${httpcode}" = "200" ]; then
+        info "sent custom notification ${msg} to ${phone}"
+        sent=$((sent + 1))
+      else
+        error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."
+      fi
+    done
+}
+```
+
+The supported variables that you can use for the function's `msg` variable are:
+
+| Variable name               | Description                                                                      |
+|:---------------------------:|:---------------------------------------------------------------------------------|
+| `${alarm}`                  | Like "name = value units"                                                        |
+| `${status_message}`         | Like "needs attention", "recovered", "is critical"                               |
+| `${severity}`               | Like "Escalated to CRITICAL", "Recovered from WARNING"                           |
+| `${raised_for}`             | Like "(alarm was raised for 10 minutes)"                                         |
+| `${host}`                   | The host generated this event                                                    |
+| `${url_host}`               | Same as ${host} but URL encoded                                                  |
+| `${unique_id}`              | The unique id of this event                                                      |
+| `${alarm_id}`               | The unique id of the alarm that generated this event                             |
+| `${event_id}`               | The incremental id of the event, for this alarm id                               |
+| `${when}`                   | The timestamp this event occurred                                                |
+| `${name}`                   | The name of the alarm, as given in netdata health.d entries                      |
+| `${url_name}`               | Same as ${name} but URL encoded                                                  |
+| `${chart}`                  | The name of the chart (type.id)                                                  |
+| `${url_chart}`              | Same as ${chart} but URL encoded                                                 |
+| `${status}`                 | The current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
+| `${old_status}`             | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
+| `${value}`                  | The current value of the alarm                                                   |
+| `${old_value}`              | The previous value of the alarm                                                  |
+| `${src}`                    | The line number and file the alarm has been configured                           |
+| `${duration}`               | The duration in seconds of the previous alarm state                              |
+| `${duration_txt}`           | Same as ${duration} for humans                                                   |
+| `${non_clear_duration}`     | The total duration in seconds this is/was non-clear                              |
+| `${non_clear_duration_txt}` | Same as ${non_clear_duration} for humans                                         |
+| `${units}`                  | The units of the value                                                           |
+| `${info}`                   | A short description of the alarm                                                 |
+| `${value_string}`           | Friendly value (with units)                                                      |
+| `${old_value_string}`       | Friendly old value (with units)                                                  |
+| `${image}`                  | The URL of an image to represent the status of the alarm                         |
+| `${color}`                  | A color in  AABBCC format for the alarm                                          |
+| `${goto_url}`               | The URL the user can click to see the netdata dashboard                          |
+| `${calc_expression}`        | The expression evaluated to provide the value for the alarm                      |
+| `${calc_param_values}`      | The value of the variables in the evaluated expression                           |
+| `${total_warnings}`         | The total number of alarms in WARNING state on the host                          |
+| `${total_critical}`         | The total number of alarms in CRITICAL state on the host                         |
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# custom notifications
+
+SEND_CUSTOM="YES"
+DEFAULT_RECIPIENT_CUSTOM=""
+
+# The custom_sender() is a custom function to do whatever you need to do
+custom_sender() {
+    # example human readable SMS
+    local msg="${host} ${status_message}: ${alarm} ${raised_for}"
+
+    # limit it to 160 characters and encode it for use in a URL
+    urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}"
+
+    # a space separated list of the recipients to send alarms to
+    to="${1}"
+
+    for phone in ${to}; do
+      httpcode=$(docurl -X POST \
+            --data-urlencode "From=XXX" \
+            --data-urlencode "To=${phone}" \
+            --data-urlencode "Body=${msg}" \
+            -u "${accountsid}:${accounttoken}" \
+        https://domain.website.com/)
+
+      if [ "${httpcode}" = "200" ]; then
+        info "sent custom notification ${msg} to ${phone}"
+        sent=$((sent + 1))
+      else
+        error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."
+      fi
+    done
+}
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/custom/metadata.yaml b/src/health/notifications/custom/metadata.yaml
index 557539cfb..557539cfb 100644
--- a/health/notifications/custom/metadata.yaml
+++ b/src/health/notifications/custom/metadata.yaml
diff --git a/src/health/notifications/discord/README.md b/src/health/notifications/discord/README.md
new file mode 100644
index 000000000..c6200e45c
--- /dev/null
+++ b/src/health/notifications/discord/README.md
@@ -0,0 +1,117 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/discord/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/discord/metadata.yaml"
+sidebar_label: "Discord"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Discord
+
+
+<img src="https://netdata.cloud/img/discord.png" width="150"/>
+
+
+Send notifications to Discord using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The incoming webhook URL as given by Discord. Create a webhook by following the official [Discord documentation](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks). You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).
+- One or more Discord channels to post the messages to
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_DISCORD | Set `SEND_DISCORD` to YES | YES | yes |
+| DISCORD_WEBHOOK_URL | set `DISCORD_WEBHOOK_URL` to your webhook URL. |  | yes |
+| DEFAULT_RECIPIENT_DISCORD | Set `DEFAULT_RECIPIENT_DISCORD` to the channel you want the alert notifications to be sent to. You can define multiple channels like this: `alerts` `systems`. |  | yes |
+
+##### DEFAULT_RECIPIENT_DISCORD
+
+All roles will default to this variable if left unconfigured.
+You can then have different channels per role, by editing `DEFAULT_RECIPIENT_DISCORD` with the channel you want, in the following entries at the bottom of the same file:
+```conf
+role_recipients_discord[sysadmin]="systems"
+role_recipients_discord[domainadmin]="domains"
+role_recipients_discord[dba]="databases systems"
+role_recipients_discord[webmaster]="marketing development"
+role_recipients_discord[proxyadmin]="proxy-admin"
+role_recipients_discord[sitemgr]="sites"
+```
+
+The values you provide should already exist as Discord channels in your server.
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# discord (discordapp.com) global notification options
+
+SEND_DISCORD="YES"
+DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/XXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+DEFAULT_RECIPIENT_DISCORD="alerts"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/discord/metadata.yaml b/src/health/notifications/discord/metadata.yaml
index a46a8ec98..a46a8ec98 100644
--- a/health/notifications/discord/metadata.yaml
+++ b/src/health/notifications/discord/metadata.yaml
diff --git a/src/health/notifications/dynatrace/README.md b/src/health/notifications/dynatrace/README.md
new file mode 100644
index 000000000..4e70f9016
--- /dev/null
+++ b/src/health/notifications/dynatrace/README.md
@@ -0,0 +1,124 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/dynatrace/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/dynatrace/metadata.yaml"
+sidebar_label: "Dynatrace"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Dynatrace
+
+
+<img src="https://netdata.cloud/img/dynatrace.svg" width="150"/>
+
+
+Dynatrace allows you to receive notifications using their Events REST API. See the [Dynatrace documentation](https://www.dynatrace.com/support/help/dynatrace-api/environment-api/events-v2/post-event) about POSTing an event in the Events API for more details.
+You can send notifications to Dynatrace using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- A Dynatrace Server. You can use the same on all your Netdata servers but make sure the server is network visible from your Netdata hosts. The Dynatrace server should be with protocol prefixed (http:// or https://), for example: https://monitor.example.com.
+- An API Token. Generate a secure access API token that enables access to your Dynatrace monitoring data via the REST-based API. See [Dynatrace API - Authentication](https://www.dynatrace.com/support/help/extend-dynatrace/dynatrace-api/basics/dynatrace-api-authentication/) for more details.
+- An API Space. This is the URL part of the page you have access in order to generate the API Token. For example, the URL for a generated API token might look like: https://monitor.illumineit.com/e/2a93fe0e-4cd5-469a-9d0d-1a064235cfce/#settings/integration/apikeys;gf=all In that case, the Space is 2a93fe0e-4cd5-469a-9d0d-1a064235cfce.
+- A Server Tag. To generate one on your Dynatrace Server, go to Settings --> Tags --> Manually applied tags and create the Tag. The Netdata alarm is sent as a Dynatrace Event to be correlated with all those hosts tagged with this Tag you have created.
+- Terminal access to the Agent you wish to configure
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_DYNATRACE | Set `SEND_DYNATRACE` to YES | YES | yes |
+| DYNATRACE_SERVER | Set `DYNATRACE_SERVER` to the Dynatrace server with the protocol prefix, for example `https://monitor.example.com`. |  | yes |
+| DYNATRACE_TOKEN | Set `DYNATRACE_TOKEN` to your Dynatrace API authentication token |  | yes |
+| DYNATRACE_SPACE | Set `DYNATRACE_SPACE` to the API Space, it is the URL part of the page you have access in order to generate the API Token. |  | yes |
+| DYNATRACE_TAG_VALUE | Set `DYNATRACE_TAG_VALUE` to your Dynatrace Server Tag. |  | yes |
+| DYNATRACE_ANNOTATION_TYPE | `DYNATRACE_ANNOTATION_TYPE` can be left to its default value Netdata Alarm, but you can change it to better fit your needs. | Netdata Alarm | no |
+| DYNATRACE_EVENT | Set `DYNATRACE_EVENT` to the Dynatrace eventType you want. | Netdata Alarm | no |
+
+##### DYNATRACE_SPACE
+
+For example, the URL for a generated API token might look like: https://monitor.illumineit.com/e/2a93fe0e-4cd5-469a-9d0d-1a064235cfce/#settings/integration/apikeys;gf=all In that case, the Space is 2a93fe0e-4cd5-469a-9d0d-1a064235cfce.
+
+
+##### DYNATRACE_EVENT
+
+`AVAILABILITY_EVENT`, `CUSTOM_ALERT`, `CUSTOM_ANNOTATION`, `CUSTOM_CONFIGURATION`, `CUSTOM_DEPLOYMENT`, `CUSTOM_INFO`, `ERROR_EVENT`,
+`MARKED_FOR_TERMINATION`, `PERFORMANCE_EVENT`, `RESOURCE_CONTENTION_EVENT`.
+You can read more [here](https://www.dynatrace.com/support/help/dynatrace-api/environment-api/events-v2/post-event#request-body-objects).
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# Dynatrace global notification options
+
+SEND_DYNATRACE="YES"
+DYNATRACE_SERVER="https://monitor.example.com"
+DYNATRACE_TOKEN="XXXXXXX"
+DYNATRACE_SPACE="2a93fe0e-4cd5-469a-9d0d-1a064235cfce"
+DYNATRACE_TAG_VALUE="SERVERTAG"
+DYNATRACE_ANNOTATION_TYPE="Netdata Alert"
+DYNATRACE_EVENT="AVAILABILITY_EVENT"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/dynatrace/metadata.yaml b/src/health/notifications/dynatrace/metadata.yaml
index a88c766fd..a88c766fd 100644
--- a/health/notifications/dynatrace/metadata.yaml
+++ b/src/health/notifications/dynatrace/metadata.yaml
diff --git a/src/health/notifications/email/README.md b/src/health/notifications/email/README.md
new file mode 100644
index 000000000..bc469c7af
--- /dev/null
+++ b/src/health/notifications/email/README.md
@@ -0,0 +1,114 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/email/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/email/metadata.yaml"
+sidebar_label: "Email"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Email
+
+
+<img src="https://netdata.cloud/img/email.png" width="150"/>
+
+
+Send notifications via Email using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- A working sendmail command is required for email alerts to work. Almost all MTAs provide a sendmail interface. Netdata sends all emails as user netdata, so make sure your sendmail works for local users.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| EMAIL_SENDER | You can change `EMAIL_SENDER` to the email address sending the notifications. | netdata | no |
+| SEND_EMAIL | Set `SEND_EMAIL` to YES | YES | yes |
+| DEFAULT_RECIPIENT_EMAIL | Set `DEFAULT_RECIPIENT_EMAIL` to the email address you want the email to be sent by default. You can define multiple email addresses like this: `alarms@example.com` `systems@example.com`. | root | yes |
+
+##### DEFAULT_RECIPIENT_EMAIL
+
+All roles will default to this variable if left unconfigured.
+The `DEFAULT_RECIPIENT_CUSTOM` can be edited in the following entries at the bottom of the same file:
+```conf
+role_recipients_email[sysadmin]="systems@example.com"
+role_recipients_email[domainadmin]="domains@example.com"
+role_recipients_email[dba]="databases@example.com systems@example.com"
+role_recipients_email[webmaster]="marketing@example.com development@example.com"
+role_recipients_email[proxyadmin]="proxy-admin@example.com"
+role_recipients_email[sitemgr]="sites@example.com"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# email global notification options
+
+EMAIL_SENDER="example@domain.com"
+SEND_EMAIL="YES"
+DEFAULT_RECIPIENT_EMAIL="recipient@example.com"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/email/metadata.yaml b/src/health/notifications/email/metadata.yaml
index f0d4a62a9..f0d4a62a9 100644
--- a/health/notifications/email/metadata.yaml
+++ b/src/health/notifications/email/metadata.yaml
diff --git a/src/health/notifications/flock/README.md b/src/health/notifications/flock/README.md
new file mode 100644
index 000000000..8bd9d5cf9
--- /dev/null
+++ b/src/health/notifications/flock/README.md
@@ -0,0 +1,113 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/flock/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/flock/metadata.yaml"
+sidebar_label: "Flock"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Flock
+
+
+<img src="https://netdata.cloud/img/flock.png" width="150"/>
+
+
+Send notifications to Flock using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The incoming webhook URL as given by flock.com. You can use the same on all your Netdata servers (or you can have multiple if you like). Read more about flock webhooks and how to get one [here](https://admin.flock.com/webhooks).
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_FLOCK | Set `SEND_FLOCK` to YES | YES | yes |
+| FLOCK_WEBHOOK_URL | set `FLOCK_WEBHOOK_URL` to your webhook URL. |  | yes |
+| DEFAULT_RECIPIENT_FLOCK | Set `DEFAULT_RECIPIENT_FLOCK` to  the Flock channel you want the alert notifications to be sent to. All roles will default to this variable if left unconfigured. |  | yes |
+
+##### DEFAULT_RECIPIENT_FLOCK
+
+You can have different channels per role, by editing DEFAULT_RECIPIENT_FLOCK with the channel you want, in the following entries at the bottom of the same file:
+```conf
+role_recipients_flock[sysadmin]="systems"
+role_recipients_flock[domainadmin]="domains"
+role_recipients_flock[dba]="databases systems"
+role_recipients_flock[webmaster]="marketing development"
+role_recipients_flock[proxyadmin]="proxy-admin"
+role_recipients_flock[sitemgr]="sites"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# flock (flock.com) global notification options
+
+SEND_FLOCK="YES"
+FLOCK_WEBHOOK_URL="https://api.flock.com/hooks/sendMessage/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+DEFAULT_RECIPIENT_FLOCK="alarms"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/flock/metadata.yaml b/src/health/notifications/flock/metadata.yaml
index 62e7f4995..62e7f4995 100644
--- a/health/notifications/flock/metadata.yaml
+++ b/src/health/notifications/flock/metadata.yaml
diff --git a/src/health/notifications/gotify/README.md b/src/health/notifications/gotify/README.md
new file mode 100644
index 000000000..ec3b13df1
--- /dev/null
+++ b/src/health/notifications/gotify/README.md
@@ -0,0 +1,98 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/gotify/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/gotify/metadata.yaml"
+sidebar_label: "Gotify"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Gotify
+
+
+<img src="https://netdata.cloud/img/gotify.png" width="150"/>
+
+
+[Gotify](https://gotify.net/) is a self-hosted push notification service created for sending and receiving messages in real time.
+You can send alerts to your Gotify instance using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- An application token. You can generate a new token in the Gotify Web UI.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_GOTIFY | Set `SEND_GOTIFY` to YES | YES | yes |
+| GOTIFY_APP_TOKEN | set `GOTIFY_APP_TOKEN` to the app token you generated. |  | yes |
+| GOTIFY_APP_URL | Set `GOTIFY_APP_URL` to point to your Gotify instance, for example `https://push.example.domain/` |  | yes |
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+SEND_GOTIFY="YES"
+GOTIFY_APP_TOKEN="XXXXXXXXXXXXXXX"
+GOTIFY_APP_URL="https://push.example.domain/"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/gotify/metadata.yaml b/src/health/notifications/gotify/metadata.yaml
index 4552de1c4..4552de1c4 100644
--- a/health/notifications/gotify/metadata.yaml
+++ b/src/health/notifications/gotify/metadata.yaml
diff --git a/health/notifications/health_alarm_notify.conf b/src/health/notifications/health_alarm_notify.conf
index f3b67c9de..f3b67c9de 100755
--- a/health/notifications/health_alarm_notify.conf
+++ b/src/health/notifications/health_alarm_notify.conf
diff --git a/health/notifications/health_email_recipients.conf b/src/health/notifications/health_email_recipients.conf
index f56c6c64a..f56c6c64a 100644
--- a/health/notifications/health_email_recipients.conf
+++ b/src/health/notifications/health_email_recipients.conf
diff --git a/src/health/notifications/irc/README.md b/src/health/notifications/irc/README.md
new file mode 100644
index 000000000..d1d062a0e
--- /dev/null
+++ b/src/health/notifications/irc/README.md
@@ -0,0 +1,132 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/irc/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/irc/metadata.yaml"
+sidebar_label: "IRC"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# IRC
+
+
+<img src="https://netdata.cloud/img/irc.png" width="150"/>
+
+
+Send notifications to IRC using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The `nc` utility. You can set the path to it, or Netdata will search for it in your system `$PATH`.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| nc path | Set the path for nc, otherwise Netdata will search for it in your system $PATH |  | yes |
+| SEND_IRC | Set `SEND_IRC` YES. | YES | yes |
+| IRC_NETWORK | Set `IRC_NETWORK` to the IRC network which your preferred channels belong to. |  | yes |
+| IRC_PORT | Set `IRC_PORT` to the IRC port to which a connection will occur. |  | no |
+| IRC_NICKNAME | Set `IRC_NICKNAME` to the IRC nickname which is required to send the notification. It must not be an already registered name as the connection's MODE is defined as a guest. |  | yes |
+| IRC_REALNAME | Set `IRC_REALNAME` to the IRC realname which is required in order to make the connection. |  | yes |
+| DEFAULT_RECIPIENT_IRC | You can have different channels per role, by editing `DEFAULT_RECIPIENT_IRC` with the channel you want |  | yes |
+
+##### nc path
+
+```sh
+#------------------------------------------------------------------------------
+# external commands
+#
+# The full path of the nc command.
+# If empty, the system $PATH will be searched for it.
+# If not found, irc notifications will be silently disabled.
+nc="/usr/bin/nc"
+```
+
+
+##### DEFAULT_RECIPIENT_IRC
+
+The `DEFAULT_RECIPIENT_IRC` can be edited in the following entries at the bottom of the same file:
+```conf
+role_recipients_irc[sysadmin]="#systems"
+role_recipients_irc[domainadmin]="#domains"
+role_recipients_irc[dba]="#databases #systems"
+role_recipients_irc[webmaster]="#marketing #development"
+role_recipients_irc[proxyadmin]="#proxy-admin"
+role_recipients_irc[sitemgr]="#sites"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# irc notification options
+#
+SEND_IRC="YES"
+DEFAULT_RECIPIENT_IRC="#system-alarms"
+IRC_NETWORK="irc.freenode.net"
+IRC_NICKNAME="netdata-alarm-user"
+IRC_REALNAME="netdata-user"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/irc/metadata.yaml b/src/health/notifications/irc/metadata.yaml
index aa2593f91..aa2593f91 100644
--- a/health/notifications/irc/metadata.yaml
+++ b/src/health/notifications/irc/metadata.yaml
diff --git a/src/health/notifications/kavenegar/README.md b/src/health/notifications/kavenegar/README.md
new file mode 100644
index 000000000..408e2e3a9
--- /dev/null
+++ b/src/health/notifications/kavenegar/README.md
@@ -0,0 +1,120 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/kavenegar/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/kavenegar/metadata.yaml"
+sidebar_label: "Kavenegar"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Kavenegar
+
+
+<img src="https://netdata.cloud/img/kavenegar.png" width="150"/>
+
+
+[Kavenegar](https://kavenegar.com/) as service for software developers, based in Iran, provides send and receive SMS, calling voice by using its APIs.
+You can send notifications to Kavenegar using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The APIKEY and Sender from http://panel.kavenegar.com/client/setting/account
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_KAVENEGAR | Set `SEND_KAVENEGAR` to YES | YES | yes |
+| KAVENEGAR_API_KEY | Set `KAVENEGAR_API_KEY` to your API key. |  | yes |
+| KAVENEGAR_SENDER | Set `KAVENEGAR_SENDER` to the value of your Sender. |  | yes |
+| DEFAULT_RECIPIENT_KAVENEGAR | Set `DEFAULT_RECIPIENT_KAVENEGAR` to the SMS recipient you want the alert notifications to be sent to. You can define multiple recipients like this: 09155555555 09177777777. |  | yes |
+
+##### DEFAULT_RECIPIENT_KAVENEGAR
+
+All roles will default to this variable if lest unconfigured.
+
+You can then have different SMS recipients per role, by editing `DEFAULT_RECIPIENT_KAVENEGAR` with the SMS recipients you want, in the following entries at the bottom of the same file:
+```conf
+role_recipients_kavenegar[sysadmin]="09100000000"
+role_recipients_kavenegar[domainadmin]="09111111111"
+role_recipients_kavenegar[dba]="0922222222"
+role_recipients_kavenegar[webmaster]="0933333333"
+role_recipients_kavenegar[proxyadmin]="0944444444"
+role_recipients_kavenegar[sitemgr]="0955555555"
+```
+
+The values you provide should be defined as environments in `/etc/alertad.conf` with `ALLOWED_ENVIRONMENTS` option.
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# Kavenegar (Kavenegar.com) SMS options
+
+SEND_KAVENEGAR="YES"
+KAVENEGAR_API_KEY="XXXXXXXXXXXX"
+KAVENEGAR_SENDER="YYYYYYYY"
+DEFAULT_RECIPIENT_KAVENEGAR="0912345678"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/kavenegar/metadata.yaml b/src/health/notifications/kavenegar/metadata.yaml
index 559dbac09..559dbac09 100644
--- a/health/notifications/kavenegar/metadata.yaml
+++ b/src/health/notifications/kavenegar/metadata.yaml
diff --git a/src/health/notifications/matrix/README.md b/src/health/notifications/matrix/README.md
new file mode 100644
index 000000000..491806d6b
--- /dev/null
+++ b/src/health/notifications/matrix/README.md
@@ -0,0 +1,132 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/matrix/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/matrix/metadata.yaml"
+sidebar_label: "Matrix"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Matrix
+
+
+<img src="https://netdata.cloud/img/matrix.svg" width="150"/>
+
+
+Send notifications to Matrix network rooms using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The url of the homeserver (`https://homeserver:port`).
+- Credentials for connecting to the homeserver, in the form of a valid access token for your account (or for a dedicated notification account). These tokens usually don't expire.
+- The room ids that you want to sent the notification to.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_MATRIX | Set `SEND_MATRIX` to YES | YES | yes |
+| MATRIX_HOMESERVER | set `MATRIX_HOMESERVER` to the URL of the Matrix homeserver. |  | yes |
+| MATRIX_ACCESSTOKEN | Set `MATRIX_ACCESSTOKEN` to the access token from your Matrix account. |  | yes |
+| DEFAULT_RECIPIENT_MATRIX | Set `DEFAULT_RECIPIENT_MATRIX`  to the rooms you want the alert notifications to be sent to. The format is `!roomid:homeservername`. |  | yes |
+
+##### MATRIX_ACCESSTOKEN
+
+To obtain the access token, you can use the following curl command:
+```
+curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "https://homeserver:8448/_matrix/client/r0/login"
+```
+
+
+##### DEFAULT_RECIPIENT_MATRIX
+
+The room ids are unique identifiers and can be obtained from the room settings in a Matrix client (e.g. Riot).
+
+You can define multiple rooms like this: `!roomid1:homeservername` `!roomid2:homeservername`.
+
+All roles will default to this variable if left unconfigured.
+
+You can have different rooms per role, by editing `DEFAULT_RECIPIENT_MATRIX` with the `!roomid:homeservername` you want, in the following entries at the bottom of the same file:
+
+```conf
+role_recipients_matrix[sysadmin]="!roomid1:homeservername"
+role_recipients_matrix[domainadmin]="!roomid2:homeservername"
+role_recipients_matrix[dba]="!roomid3:homeservername"
+role_recipients_matrix[webmaster]="!roomid4:homeservername"
+role_recipients_matrix[proxyadmin]="!roomid5:homeservername"
+role_recipients_matrix[sitemgr]="!roomid6:homeservername"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# Matrix notifications
+
+SEND_MATRIX="YES"
+MATRIX_HOMESERVER="https://matrix.org:8448"
+MATRIX_ACCESSTOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+DEFAULT_RECIPIENT_MATRIX="!XXXXXXXXXXXX:matrix.org"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/matrix/metadata.yaml b/src/health/notifications/matrix/metadata.yaml
index 17135aa3a..17135aa3a 100644
--- a/health/notifications/matrix/metadata.yaml
+++ b/src/health/notifications/matrix/metadata.yaml
diff --git a/src/health/notifications/messagebird/README.md b/src/health/notifications/messagebird/README.md
new file mode 100644
index 000000000..a20654147
--- /dev/null
+++ b/src/health/notifications/messagebird/README.md
@@ -0,0 +1,117 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/messagebird/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/messagebird/metadata.yaml"
+sidebar_label: "MessageBird"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# MessageBird
+
+
+<img src="https://netdata.cloud/img/messagebird.svg" width="150"/>
+
+
+Send notifications to MessageBird using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- An access key under 'API ACCESS (REST)' (you will want a live key), you can read more [here](https://developers.messagebird.com/quickstarts/sms/test-credits-api-keys/).
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_MESSAGEBIRD | Set `SEND_MESSAGEBIRD` to YES | YES | yes |
+| MESSAGEBIRD_ACCESS_KEY | Set `MESSAGEBIRD_ACCESS_KEY` to your API key. |  | yes |
+| MESSAGEBIRD_NUMBER | Set `MESSAGEBIRD_NUMBER` to the MessageBird number you want to use for the alert. |  | yes |
+| DEFAULT_RECIPIENT_MESSAGEBIRD | Set `DEFAULT_RECIPIENT_MESSAGEBIRD` to the number you want the alert notification to be sent as an SMS. You can define multiple recipients like this: +15555555555 +17777777777. |  | yes |
+
+##### DEFAULT_RECIPIENT_MESSAGEBIRD
+
+All roles will default to this variable if left unconfigured.
+
+You can then have different recipients per role, by editing `DEFAULT_RECIPIENT_MESSAGEBIRD` with the number you want, in the following entries at the bottom of the same file:
+```conf
+role_recipients_messagebird[sysadmin]="+15555555555"
+role_recipients_messagebird[domainadmin]="+15555555556"
+role_recipients_messagebird[dba]="+15555555557"
+role_recipients_messagebird[webmaster]="+15555555558"
+role_recipients_messagebird[proxyadmin]="+15555555559"
+role_recipients_messagebird[sitemgr]="+15555555550"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# Messagebird (messagebird.com) SMS options
+
+SEND_MESSAGEBIRD="YES"
+MESSAGEBIRD_ACCESS_KEY="XXXXXXXX"
+MESSAGEBIRD_NUMBER="XXXXXXX"
+DEFAULT_RECIPIENT_MESSAGEBIRD="+15555555555"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/messagebird/metadata.yaml b/src/health/notifications/messagebird/metadata.yaml
index a97cdc712..a97cdc712 100644
--- a/health/notifications/messagebird/metadata.yaml
+++ b/src/health/notifications/messagebird/metadata.yaml
diff --git a/src/health/notifications/msteams/README.md b/src/health/notifications/msteams/README.md
new file mode 100644
index 000000000..4d6466afc
--- /dev/null
+++ b/src/health/notifications/msteams/README.md
@@ -0,0 +1,118 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/msteams/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/msteams/metadata.yaml"
+sidebar_label: "Microsoft Teams"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Microsoft Teams
+
+
+<img src="https://netdata.cloud/img/msteams.svg" width="150"/>
+
+
+You can send Netdata alerts to Microsoft Teams using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The incoming webhook URL as given by Microsoft Teams. You can use the same on all your Netdata servers (or you can have multiple if you like).
+- One or more channels to post the messages to
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_MSTEAMS | Set `SEND_MSTEAMS` to YES | YES | yes |
+| MSTEAMS_WEBHOOK_URL | set `MSTEAMS_WEBHOOK_URL` to the incoming webhook URL as given by Microsoft Teams. |  | yes |
+| DEFAULT_RECIPIENT_MSTEAMS | Set `DEFAULT_RECIPIENT_MSTEAMS` to the encoded Microsoft Teams channel name you want the alert notifications to be sent to. |  | yes |
+
+##### DEFAULT_RECIPIENT_MSTEAMS
+
+In Microsoft Teams the channel name is encoded in the URI after `/IncomingWebhook/`. You can define multiple channels like this: `CHANNEL1` `CHANNEL2`.
+
+All roles will default to this variable if left unconfigured.
+
+You can have different channels per role, by editing `DEFAULT_RECIPIENT_MSTEAMS` with the channel you want, in the following entries at the bottom of the same file:
+```conf
+role_recipients_msteams[sysadmin]="CHANNEL1"
+role_recipients_msteams[domainadmin]="CHANNEL2"
+role_recipients_msteams[dba]="databases CHANNEL3"
+role_recipients_msteams[webmaster]="CHANNEL4"
+role_recipients_msteams[proxyadmin]="CHANNEL5"
+role_recipients_msteams[sitemgr]="CHANNEL6"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# Microsoft Teams (office.com) global notification options
+
+SEND_MSTEAMS="YES"
+MSTEAMS_WEBHOOK_URL="https://outlook.office.com/webhook/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX@XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/IncomingWebhook/CHANNEL/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
+DEFAULT_RECIPIENT_MSTEAMS="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/msteams/metadata.yaml b/src/health/notifications/msteams/metadata.yaml
index 72de507a4..72de507a4 100644
--- a/health/notifications/msteams/metadata.yaml
+++ b/src/health/notifications/msteams/metadata.yaml
diff --git a/src/health/notifications/ntfy/README.md b/src/health/notifications/ntfy/README.md
new file mode 100644
index 000000000..6cfdee4c9
--- /dev/null
+++ b/src/health/notifications/ntfy/README.md
@@ -0,0 +1,135 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/ntfy/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/ntfy/metadata.yaml"
+sidebar_label: "ntfy"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# ntfy
+
+
+<img src="https://netdata.cloud/img/ntfy.svg" width="150"/>
+
+
+[ntfy](https://ntfy.sh/) (pronounce: notify) is a simple HTTP-based [pub-sub](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) notification service. It allows you to send notifications to your phone or desktop via scripts from any computer, entirely without signup, cost or setup. It's also [open source](https://github.com/binwiederhier/ntfy) if you want to run your own server.
+You can send alerts to an ntfy server using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- (Optional) A [self-hosted ntfy server](https://docs.ntfy.sh/faq/#can-i-self-host-it), in case you don't want to use https://ntfy.sh
+- A new [topic](https://ntfy.sh/#subscribe) for the notifications to be published to
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_NTFY | Set `SEND_NTFY` to YES | YES | yes |
+| DEFAULT_RECIPIENT_NTFY | URL formed by the server-topic combination you want the alert notifications to be sent to. Unless hosting your own server, the server should always be set to https://ntfy.sh. |  | yes |
+| NTFY_USERNAME | The username for netdata to use to authenticate with an ntfy server. |  | no |
+| NTFY_PASSWORD | The password for netdata to use to authenticate with an ntfy server. |  | no |
+| NTFY_ACCESS_TOKEN | The access token for netdata to use to authenticate with an ntfy server. |  | no |
+
+##### DEFAULT_RECIPIENT_NTFY
+
+You can define multiple recipient URLs like this: `https://SERVER1/TOPIC1` `https://SERVER2/TOPIC2`
+
+All roles will default to this variable if left unconfigured.
+
+You can then have different servers and/or topics per role, by editing DEFAULT_RECIPIENT_NTFY with the server-topic combination you want, in the following entries at the bottom of the same file:
+```conf
+role_recipients_ntfy[sysadmin]="https://SERVER1/TOPIC1"
+role_recipients_ntfy[domainadmin]="https://SERVER2/TOPIC2"
+role_recipients_ntfy[dba]="https://SERVER3/TOPIC3"
+role_recipients_ntfy[webmaster]="https://SERVER4/TOPIC4"
+role_recipients_ntfy[proxyadmin]="https://SERVER5/TOPIC5"
+role_recipients_ntfy[sitemgr]="https://SERVER6/TOPIC6"
+```
+
+
+##### NTFY_USERNAME
+
+Only useful on self-hosted ntfy instances. See [users and roles](https://docs.ntfy.sh/config/#users-and-roles) for details.
+Ensure that your user has proper read/write access to the provided topic in `DEFAULT_RECIPIENT_NTFY`
+
+
+##### NTFY_PASSWORD
+
+Only useful on self-hosted ntfy instances. See [users and roles](https://docs.ntfy.sh/config/#users-and-roles) for details.
+Ensure that your user has proper read/write access to the provided topic in `DEFAULT_RECIPIENT_NTFY`
+
+
+##### NTFY_ACCESS_TOKEN
+
+This can be used in place of `NTFY_USERNAME` and `NTFY_PASSWORD` to authenticate with a self-hosted ntfy instance. See [access tokens](https://docs.ntfy.sh/config/?h=access+to#access-tokens) for details.
+Ensure that the token user has proper read/write access to the provided topic in `DEFAULT_RECIPIENT_NTFY`
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+SEND_NTFY="YES"
+DEFAULT_RECIPIENT_NTFY="https://ntfy.sh/netdata-X7seHg7d3Tw9zGOk https://ntfy.sh/netdata-oIPm4IK1IlUtlA30"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/ntfy/metadata.yaml b/src/health/notifications/ntfy/metadata.yaml
index 0d6c0beac..0d6c0beac 100644
--- a/health/notifications/ntfy/metadata.yaml
+++ b/src/health/notifications/ntfy/metadata.yaml
diff --git a/src/health/notifications/opsgenie/README.md b/src/health/notifications/opsgenie/README.md
new file mode 100644
index 000000000..b82f36a4f
--- /dev/null
+++ b/src/health/notifications/opsgenie/README.md
@@ -0,0 +1,98 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/opsgenie/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/opsgenie/metadata.yaml"
+sidebar_label: "OpsGenie"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# OpsGenie
+
+
+<img src="https://netdata.cloud/img/opsgenie.png" width="150"/>
+
+
+Opsgenie is an alerting and incident response tool. It is designed to group and filter alarms, build custom routing rules for on-call teams, and correlate deployments and commits to incidents.
+You can send notifications to Opsgenie using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- An Opsgenie integration. You can create an [integration](https://docs.opsgenie.com/docs/api-integration) in the [Opsgenie](https://www.atlassian.com/software/opsgenie) dashboard.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_OPSGENIE | Set `SEND_OPSGENIE` to YES | YES | yes |
+| OPSGENIE_API_KEY | Set `OPSGENIE_API_KEY` to your API key. |  | yes |
+| OPSGENIE_API_URL | Set `OPSGENIE_API_URL` to the corresponding URL if required, for example there are region-specific API URLs such as `https://eu.api.opsgenie.com`. | https://api.opsgenie.com | no |
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+SEND_OPSGENIE="YES"
+OPSGENIE_API_KEY="11111111-2222-3333-4444-555555555555"
+OPSGENIE_API_URL=""
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/opsgenie/metadata.yaml b/src/health/notifications/opsgenie/metadata.yaml
index 78bd8c2bd..78bd8c2bd 100644
--- a/health/notifications/opsgenie/metadata.yaml
+++ b/src/health/notifications/opsgenie/metadata.yaml
diff --git a/src/health/notifications/pagerduty/README.md b/src/health/notifications/pagerduty/README.md
new file mode 100644
index 000000000..730f73168
--- /dev/null
+++ b/src/health/notifications/pagerduty/README.md
@@ -0,0 +1,117 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/pagerduty/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/pagerduty/metadata.yaml"
+sidebar_label: "PagerDuty"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# PagerDuty
+
+
+<img src="https://netdata.cloud/img/pagerduty.png" width="150"/>
+
+
+PagerDuty is an enterprise incident resolution service that integrates with ITOps and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into incidents, PagerDuty streamlines the incident management process by reducing alert noise and resolution times.
+You can send notifications to PagerDuty using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- An installation of the [PagerDuty](https://www.pagerduty.com/docs/guides/agent-install-guide/) agent on the node running the Netdata Agent
+- A PagerDuty Generic API service using either the `Events API v2` or `Events API v1`
+- [Add a new service](https://support.pagerduty.com/docs/services-and-integrations#section-configuring-services-and-integrations) to PagerDuty. Click Use our API directly and select either `Events API v2` or `Events API v1`. Once you finish creating the service, click on the Integrations tab to find your Integration Key.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_PD | Set `SEND_PD` to YES | YES | yes |
+| DEFAULT_RECIPIENT_PD | Set `DEFAULT_RECIPIENT_PD` to the PagerDuty service key you want the alert notifications to be sent to. You can define multiple service keys like this: `pd_service_key_1` `pd_service_key_2`. |  | yes |
+
+##### DEFAULT_RECIPIENT_PD
+
+All roles will default to this variable if left unconfigured.
+
+The `DEFAULT_RECIPIENT_PD` can be edited in the following entries at the bottom of the same file:
+```conf
+role_recipients_pd[sysadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa"
+role_recipients_pd[domainadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxb"
+role_recipients_pd[dba]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc"
+role_recipients_pd[webmaster]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxd"
+role_recipients_pd[proxyadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxe"
+role_recipients_pd[sitemgr]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxf"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# pagerduty.com notification options
+
+SEND_PD="YES"
+DEFAULT_RECIPIENT_PD="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+USE_PD_VERSION="2"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/pagerduty/metadata.yaml b/src/health/notifications/pagerduty/metadata.yaml
index 6fc1d640e..6fc1d640e 100644
--- a/health/notifications/pagerduty/metadata.yaml
+++ b/src/health/notifications/pagerduty/metadata.yaml
diff --git a/src/health/notifications/prowl/README.md b/src/health/notifications/prowl/README.md
new file mode 100644
index 000000000..bc9b2b350
--- /dev/null
+++ b/src/health/notifications/prowl/README.md
@@ -0,0 +1,119 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/prowl/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/prowl/metadata.yaml"
+sidebar_label: "Prowl"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Prowl
+
+
+<img src="https://netdata.cloud/img/prowl.png" width="150"/>
+
+
+Send notifications to Prowl using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Limitations
+
+- Because of how Netdata integrates with Prowl, there is a hard limit of at most 1000 notifications per hour (starting from the first notification sent). Any alerts beyond the first thousand in an hour will be dropped.
+- Warning messages will be sent with the 'High' priority, critical messages will be sent with the 'Emergency' priority, and all other messages will be sent with the normal priority. Opening the notification's associated URL will take you to the Netdata dashboard of the system that issued the alert, directly to the chart that it triggered on.
+
+
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- A Prowl API key, which can be requested through the Prowl website after registering
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_PROWL | Set `SEND_PROWL` to YES | YES | yes |
+| DEFAULT_RECIPIENT_PROWL | Set `DEFAULT_RECIPIENT_PROWL` to  the Prowl API key you want the alert notifications to be sent to. You can define multiple API keys like this: `APIKEY1`, `APIKEY2`. |  | yes |
+
+##### DEFAULT_RECIPIENT_PROWL
+
+All roles will default to this variable if left unconfigured.
+
+The `DEFAULT_RECIPIENT_PROWL` can be edited in the following entries at the bottom of the same file:
+```conf
+role_recipients_prowl[sysadmin]="AAAAAAAA"
+role_recipients_prowl[domainadmin]="BBBBBBBBB"
+role_recipients_prowl[dba]="CCCCCCCCC"
+role_recipients_prowl[webmaster]="DDDDDDDDDD"
+role_recipients_prowl[proxyadmin]="EEEEEEEEEE"
+role_recipients_prowl[sitemgr]="FFFFFFFFFF"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# iOS Push Notifications
+
+SEND_PROWL="YES"
+DEFAULT_RECIPIENT_PROWL="XXXXXXXXXX"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/prowl/metadata.yaml b/src/health/notifications/prowl/metadata.yaml
index b3f0e0a1e..b3f0e0a1e 100644
--- a/health/notifications/prowl/metadata.yaml
+++ b/src/health/notifications/prowl/metadata.yaml
diff --git a/src/health/notifications/pushbullet/README.md b/src/health/notifications/pushbullet/README.md
new file mode 100644
index 000000000..70e30ea3c
--- /dev/null
+++ b/src/health/notifications/pushbullet/README.md
@@ -0,0 +1,117 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/pushbullet/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/pushbullet/metadata.yaml"
+sidebar_label: "Pushbullet"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Pushbullet
+
+
+<img src="https://netdata.cloud/img/pushbullet.png" width="150"/>
+
+
+Send notifications to Pushbullet using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- A Pushbullet access token that can be created in your [account settings](https://www.pushbullet.com/#settings/account).
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| Send_PUSHBULLET | Set `Send_PUSHBULLET` to YES | YES | yes |
+| PUSHBULLET_ACCESS_TOKEN | set `PUSHBULLET_ACCESS_TOKEN` to the access token you generated. |  | yes |
+| DEFAULT_RECIPIENT_PUSHBULLET | Set `DEFAULT_RECIPIENT_PUSHBULLET` to the email (e.g. `example@domain.com`) or the channel tag (e.g. `#channel`) you want the alert notifications to be sent to. |  | yes |
+
+##### DEFAULT_RECIPIENT_PUSHBULLET
+
+You can define multiple entries like this: user1@email.com user2@email.com.
+
+All roles will default to this variable if left unconfigured.
+
+The `DEFAULT_RECIPIENT_PUSHBULLET` can be edited in the following entries at the bottom of the same file:
+```conf
+role_recipients_pushbullet[sysadmin]="user1@email.com"
+role_recipients_pushbullet[domainadmin]="user2@mail.com"
+role_recipients_pushbullet[dba]="#channel1"
+role_recipients_pushbullet[webmaster]="#channel2"
+role_recipients_pushbullet[proxyadmin]="user3@mail.com"
+role_recipients_pushbullet[sitemgr]="user4@mail.com"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# pushbullet (pushbullet.com) push notification options
+
+SEND_PUSHBULLET="YES"
+PUSHBULLET_ACCESS_TOKEN="XXXXXXXXX"
+DEFAULT_RECIPIENT_PUSHBULLET="admin1@example.com admin3@somemail.com #examplechanneltag #anotherchanneltag"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/pushbullet/metadata.yaml b/src/health/notifications/pushbullet/metadata.yaml
index 430033cca..430033cca 100644
--- a/health/notifications/pushbullet/metadata.yaml
+++ b/src/health/notifications/pushbullet/metadata.yaml
diff --git a/src/health/notifications/pushover/README.md b/src/health/notifications/pushover/README.md
new file mode 100644
index 000000000..6fe59c9ac
--- /dev/null
+++ b/src/health/notifications/pushover/README.md
@@ -0,0 +1,119 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/pushover/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/pushover/metadata.yaml"
+sidebar_label: "PushOver"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# PushOver
+
+
+<img src="https://netdata.cloud/img/pushover.png" width="150"/>
+
+
+Send notification to Pushover using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+- Netdata will send warning messages with priority 0 and critical messages with priority 1.
+- Pushover allows you to select do-not-disturb hours. The way this is configured, critical notifications will ring and vibrate your phone, even during the do-not-disturb-hours.
+- All other notifications will be delivered silently.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- An Application token. You can use the same on all your Netdata servers.
+- A User token for each user you are going to send notifications to. This is the actual recipient of the notification.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_PUSHOVER | Set `SEND_PUSHOVER` to YES | YES | yes |
+| PUSHOVER_WEBHOOK_URL | set `PUSHOVER_WEBHOOK_URL` to your Pushover Application token. |  | yes |
+| DEFAULT_RECIPIENT_PUSHOVER | Set `DEFAULT_RECIPIENT_PUSHOVER` the Pushover User token you want the alert notifications to be sent to. You can define multiple User tokens like this: `USERTOKEN1` `USERTOKEN2`. |  | yes |
+
+##### DEFAULT_RECIPIENT_PUSHOVER
+
+All roles will default to this variable if left unconfigured.
+
+The `DEFAULT_RECIPIENT_PUSHOVER` can be edited in the following entries at the bottom of the same file:
+```conf
+role_recipients_pushover[sysadmin]="USERTOKEN1"
+role_recipients_pushover[domainadmin]="USERTOKEN2"
+role_recipients_pushover[dba]="USERTOKEN3 USERTOKEN4"
+role_recipients_pushover[webmaster]="USERTOKEN5"
+role_recipients_pushover[proxyadmin]="USERTOKEN6"
+role_recipients_pushover[sitemgr]="USERTOKEN7"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# pushover (pushover.net) global notification options
+
+SEND_PUSHOVER="YES"
+PUSHOVER_APP_TOKEN="XXXXXXXXX"
+DEFAULT_RECIPIENT_PUSHOVER="USERTOKEN"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/pushover/metadata.yaml b/src/health/notifications/pushover/metadata.yaml
index 9af729ea8..9af729ea8 100644
--- a/health/notifications/pushover/metadata.yaml
+++ b/src/health/notifications/pushover/metadata.yaml
diff --git a/src/health/notifications/rocketchat/README.md b/src/health/notifications/rocketchat/README.md
new file mode 100644
index 000000000..4ff51a917
--- /dev/null
+++ b/src/health/notifications/rocketchat/README.md
@@ -0,0 +1,116 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/rocketchat/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/rocketchat/metadata.yaml"
+sidebar_label: "RocketChat"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# RocketChat
+
+
+<img src="https://netdata.cloud/img/rocketchat.png" width="150"/>
+
+
+Send notifications to Rocket.Chat using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- The incoming webhook URL as given by RocketChat. You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).
+- One or more channels to post the messages to
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_ROCKETCHAT | Set `SEND_ROCKETCHAT` to `YES` | YES | yes |
+| ROCKETCHAT_WEBHOOK_URL | set `ROCKETCHAT_WEBHOOK_URL` to your webhook URL. |  | yes |
+| DEFAULT_RECIPIENT_ROCKETCHAT | Set `DEFAULT_RECIPIENT_ROCKETCHAT` to the channel you want the alert notifications to be sent to. You can define multiple channels like this: `alerts` `systems`. |  | yes |
+
+##### DEFAULT_RECIPIENT_ROCKETCHAT
+
+All roles will default to this variable if left unconfigured.
+
+The `DEFAULT_RECIPIENT_ROCKETCHAT` can be edited in the following entries at the bottom of the same file:
+```conf
+role_recipients_rocketchat[sysadmin]="systems"
+role_recipients_rocketchat[domainadmin]="domains"
+role_recipients_rocketchat[dba]="databases systems"
+role_recipients_rocketchat[webmaster]="marketing development"
+role_recipients_rocketchat[proxyadmin]="proxy_admin"
+role_recipients_rocketchat[sitemgr]="sites"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# rocketchat (rocket.chat) global notification options
+
+SEND_ROCKETCHAT="YES"
+ROCKETCHAT_WEBHOOK_URL="<your_incoming_webhook_url>"
+DEFAULT_RECIPIENT_ROCKETCHAT="monitoring_alarms"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/rocketchat/metadata.yaml b/src/health/notifications/rocketchat/metadata.yaml
index f644b93e1..f644b93e1 100644
--- a/health/notifications/rocketchat/metadata.yaml
+++ b/src/health/notifications/rocketchat/metadata.yaml
diff --git a/health/notifications/sample-metadata.yaml b/src/health/notifications/sample-metadata.yaml
index 41a287aeb..41a287aeb 100644
--- a/health/notifications/sample-metadata.yaml
+++ b/src/health/notifications/sample-metadata.yaml
diff --git a/src/health/notifications/slack/README.md b/src/health/notifications/slack/README.md
new file mode 100644
index 000000000..3b26f01b4
--- /dev/null
+++ b/src/health/notifications/slack/README.md
@@ -0,0 +1,101 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/slack/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/slack/metadata.yaml"
+sidebar_label: "Slack"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Slack
+
+
+<img src="https://netdata.cloud/img/slack.png" width="150"/>
+
+
+Send notifications to a Slack workspace using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- Slack app along with an incoming webhook, read Slack's guide on the topic [here](https://api.slack.com/messaging/webhooks).
+- One or more channels to post the messages to
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_SLACK | Set `SEND_SLACK` to YES | YES | yes |
+| SLACK_WEBHOOK_URL | set `SLACK_WEBHOOK_URL` to your Slack app's webhook URL. |  | yes |
+| DEFAULT_RECIPIENT_SLACK | Set `DEFAULT_RECIPIENT_SLACK` to the Slack channel your Slack app is set to send messages to. The syntax for channels is `#channel` or `channel`. |  | yes |
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# slack (slack.com) global notification options
+
+SEND_SLACK="YES"
+SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXX/XXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" 
+DEFAULT_RECIPIENT_SLACK="#alarms"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/slack/metadata.yaml b/src/health/notifications/slack/metadata.yaml
index 226c7ca37..226c7ca37 100644
--- a/health/notifications/slack/metadata.yaml
+++ b/src/health/notifications/slack/metadata.yaml
diff --git a/src/health/notifications/smstools3/README.md b/src/health/notifications/smstools3/README.md
new file mode 100644
index 000000000..0931a338c
--- /dev/null
+++ b/src/health/notifications/smstools3/README.md
@@ -0,0 +1,126 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/smstools3/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/smstools3/metadata.yaml"
+sidebar_label: "SMS"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# SMS
+
+
+<img src="https://netdata.cloud/img/sms.svg" width="150"/>
+
+
+Send notifications to `smstools3` using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+The SMS Server Tools 3 is a SMS Gateway software which can send and receive short messages through GSM modems and mobile phones.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- [Install](http://smstools3.kekekasvi.com/index.php?p=compiling) and [configure](http://smstools3.kekekasvi.com/index.php?p=configure) `smsd`
+- To ensure that the user `netdata` can execute `sendsms`. Any user executing `sendsms` needs to:
+  - Have write permissions to /tmp and /var/spool/sms/outgoing
+  - Be a member of group smsd
+  - To ensure that the steps above are successful, just su netdata and execute sendsms phone message.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| sendsms | Set the path for `sendsms`, otherwise Netdata will search for it in your system `$PATH:` | YES | yes |
+| SEND_SMS | Set `SEND_SMS` to `YES`. |  | yes |
+| DEFAULT_RECIPIENT_SMS | Set DEFAULT_RECIPIENT_SMS to the phone number you want the alert notifications to be sent to. You can define multiple phone numbers like this: PHONE1 PHONE2. |  | yes |
+
+##### sendsms
+
+# The full path of the sendsms command (smstools3).
+# If empty, the system $PATH will be searched for it.
+# If not found, SMS notifications will be silently disabled.
+sendsms="/usr/bin/sendsms"
+
+
+##### DEFAULT_RECIPIENT_SMS
+
+All roles will default to this variable if left unconfigured.
+
+You can then have different phone numbers per role, by editing `DEFAULT_RECIPIENT_SMS` with the phone number you want, in the following entries at the bottom of the same file:
+```conf
+role_recipients_sms[sysadmin]="PHONE1"
+role_recipients_sms[domainadmin]="PHONE2"
+role_recipients_sms[dba]="PHONE3"
+role_recipients_sms[webmaster]="PHONE4"
+role_recipients_sms[proxyadmin]="PHONE5"
+role_recipients_sms[sitemgr]="PHONE6"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# SMS Server Tools 3 (smstools3) global notification options
+SEND_SMS="YES"
+DEFAULT_RECIPIENT_SMS="1234567890"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/smstools3/metadata.yaml b/src/health/notifications/smstools3/metadata.yaml
index 3a29183a5..3a29183a5 100644
--- a/health/notifications/smstools3/metadata.yaml
+++ b/src/health/notifications/smstools3/metadata.yaml
diff --git a/src/health/notifications/syslog/README.md b/src/health/notifications/syslog/README.md
new file mode 100644
index 000000000..7aac4dc83
--- /dev/null
+++ b/src/health/notifications/syslog/README.md
@@ -0,0 +1,132 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/syslog/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/syslog/metadata.yaml"
+sidebar_label: "syslog"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# syslog
+
+
+<img src="https://netdata.cloud/img/syslog.png" width="150"/>
+
+
+Send notifications to Syslog using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- A working `logger` command for this to work. This is the case on pretty much every Linux system in existence, and most BSD systems.
+- Access to the terminal where Netdata Agent is running
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SYSLOG_FACILITY | Set `SYSLOG_FACILITY` to the facility used for logging, by default this value is set to `local6`. |  | yes |
+| DEFAULT_RECIPIENT_SYSLOG | Set `DEFAULT_RECIPIENT_SYSLOG` to the recipient you want the alert notifications to be sent to. |  | yes |
+| SEND_SYSLOG | Set SEND_SYSLOG to YES, make sure you have everything else configured before turning this on. |  | yes |
+
+##### DEFAULT_RECIPIENT_SYSLOG
+
+Targets are defined as follows:
+
+```
+[[facility.level][@host[:port]]/]prefix
+```
+
+prefix defines what the log messages are prefixed with. By default, all lines are prefixed with 'netdata'.
+
+The facility and level are the standard syslog facility and level options, for more info on them see your local logger and syslog documentation. By default, Netdata will log to the local6 facility, with a log level dependent on the type of message (crit for CRITICAL, warning for WARNING, and info for everything else).
+
+You can configure sending directly to remote log servers by specifying a host (and optionally a port). However, this has a somewhat high overhead, so it is much preferred to use your local syslog daemon to handle the forwarding of messages to remote systems (pretty much all of them allow at least simple forwarding, and most of the really popular ones support complex queueing and routing of messages to remote log servers).
+
+You can define multiple recipients like this: daemon.notice@loghost:514/netdata daemon.notice@loghost2:514/netdata.
+All roles will default to this variable if left unconfigured.
+
+
+##### SEND_SYSLOG 
+
+You can then have different recipients per role, by editing DEFAULT_RECIPIENT_SYSLOG with the recipient you want, in the following entries at the bottom of the same file:
+
+```conf
+role_recipients_syslog[sysadmin]="daemon.notice@loghost1:514/netdata"
+role_recipients_syslog[domainadmin]="daemon.notice@loghost2:514/netdata"
+role_recipients_syslog[dba]="daemon.notice@loghost3:514/netdata"
+role_recipients_syslog[webmaster]="daemon.notice@loghost4:514/netdata"
+role_recipients_syslog[proxyadmin]="daemon.notice@loghost5:514/netdata"
+role_recipients_syslog[sitemgr]="daemon.notice@loghost6:514/netdata"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# syslog notifications
+
+SEND_SYSLOG="YES"
+SYSLOG_FACILITY='local6'
+DEFAULT_RECIPIENT_SYSLOG="daemon.notice@loghost6:514/netdata"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/syslog/metadata.yaml b/src/health/notifications/syslog/metadata.yaml
index c5f241e76..c5f241e76 100644
--- a/health/notifications/syslog/metadata.yaml
+++ b/src/health/notifications/syslog/metadata.yaml
diff --git a/src/health/notifications/telegram/README.md b/src/health/notifications/telegram/README.md
new file mode 100644
index 000000000..2ceef6963
--- /dev/null
+++ b/src/health/notifications/telegram/README.md
@@ -0,0 +1,117 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/telegram/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/telegram/metadata.yaml"
+sidebar_label: "Telegram"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Telegram
+
+
+<img src="https://netdata.cloud/img/telegram.svg" width="150"/>
+
+
+Send notifications to Telegram using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- A bot token. To get one, contact the [@BotFather](https://t.me/BotFather) bot and send the command `/newbot` and follow the instructions. Invite your bot to a group where you want it to send messages.
+- The chat ID for every chat you want to send messages to. Invite [@myidbot](https://t.me/myidbot) bot to the group that will receive notifications, and write the command `/getgroupid@myidbot` to get the group chat ID. Group IDs start with a hyphen, supergroup IDs start with `-100`.
+- Terminal access to the Agent you wish to configure.
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_TELEGRAM | Set `SEND_TELEGRAM` to YES | YES | yes |
+| TELEGRAM_BOT_TOKEN | set `TELEGRAM_BOT_TOKEN` to your bot token. |  | yes |
+| DEFAULT_RECIPIENT_TELEGRAM | Set `DEFAULT_RECIPIENT_TELEGRAM` to the chat ID you want the alert notifications to be sent to. You can define multiple chat IDs like this: -49999333322 -1009999222255. |  | yes |
+
+##### DEFAULT_RECIPIENT_TELEGRAM
+
+All roles will default to this variable if left unconfigured.
+
+The `DEFAULT_RECIPIENT_CUSTOM` can be edited in the following entries at the bottom of the same file:
+
+```conf
+role_recipients_telegram[sysadmin]="-49999333324"
+role_recipients_telegram[domainadmin]="-49999333389"
+role_recipients_telegram[dba]="-10099992222"
+role_recipients_telegram[webmaster]="-10099992222 -49999333389"
+role_recipients_telegram[proxyadmin]="-49999333344"
+role_recipients_telegram[sitemgr]="-49999333876"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# telegram (telegram.org) global notification options
+
+SEND_TELEGRAM="YES"
+TELEGRAM_BOT_TOKEN="111122223:7OpFlFFRzRBbrUUmIjj5HF9Ox2pYJZy5"
+DEFAULT_RECIPIENT_TELEGRAM="-49999333876"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/src/health/notifications/telegram/metadata.yaml b/src/health/notifications/telegram/metadata.yaml
new file mode 100644
index 000000000..cc6d8c91e
--- /dev/null
+++ b/src/health/notifications/telegram/metadata.yaml
@@ -0,0 +1,76 @@
+# yamllint disable rule:line-length
+---
+- id: 'notify-telegram'
+  meta:
+    name: 'Telegram'
+    link: 'https://telegram.org/'
+    categories:
+      - notify.agent
+    icon_filename: 'telegram.svg'
+  keywords:
+    - Telegram
+  overview:
+    notification_description: |
+      Send notifications to Telegram using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+    notification_limitations: ''
+  setup:
+    prerequisites:
+      list:
+        - title: ''
+          description: |
+            - A bot token. To get one, contact the [@BotFather](https://t.me/BotFather) bot and send the command `/newbot` and follow the instructions. Invite your bot to a group where you want it to send messages.
+            - The chat ID for every chat you want to send messages to. Invite [@myidbot](https://t.me/myidbot) bot to the group that will receive notifications, and write the command `/getgroupid@myidbot` to get the group chat ID. Group IDs start with a hyphen, supergroup IDs start with `-100`.
+            - Terminal access to the Agent you wish to configure.
+    configuration:
+      file:
+        name: 'health_alarm_notify.conf'
+      options:
+        description: 'The following options can be defined for this notification'
+        folding:
+          title: 'Config Options'
+          enabled: true
+        list:
+          - name: 'SEND_TELEGRAM'
+            default_value: 'YES'
+            description: "Set `SEND_TELEGRAM` to YES"
+            required: true
+          - name: 'TELEGRAM_BOT_TOKEN'
+            default_value: ''
+            description: "set `TELEGRAM_BOT_TOKEN` to your bot token."
+            required: true
+          - name: 'DEFAULT_RECIPIENT_TELEGRAM'
+            default_value: ''
+            description: "Set `DEFAULT_RECIPIENT_TELEGRAM` to the chat ID you want the alert notifications to be sent to. You can define multiple chat IDs like this: -49999333322 -1009999222255."
+            required: true
+            detailed_description: |
+              All roles will default to this variable if left unconfigured.
+
+              The `DEFAULT_RECIPIENT_CUSTOM` can be edited in the following entries at the bottom of the same file:
+
+              ```conf
+              role_recipients_telegram[sysadmin]="-49999333324"
+              role_recipients_telegram[domainadmin]="-49999333389"
+              role_recipients_telegram[dba]="-10099992222"
+              role_recipients_telegram[webmaster]="-10099992222 -49999333389"
+              role_recipients_telegram[proxyadmin]="-49999333344"
+              role_recipients_telegram[sitemgr]="-49999333876"
+              ```
+      examples:
+        folding:
+          enabled: true
+          title: ''
+        list:
+          - name: 'Basic Configuration'
+            folding:
+              enabled: false
+            description: ''
+            config: |
+              #------------------------------------------------------------------------------
+              # telegram (telegram.org) global notification options
+
+              SEND_TELEGRAM="YES"
+              TELEGRAM_BOT_TOKEN="111122223:7OpFlFFRzRBbrUUmIjj5HF9Ox2pYJZy5"
+              DEFAULT_RECIPIENT_TELEGRAM="-49999333876"
+  troubleshooting:
+    problems:
+      list: []
diff --git a/src/health/notifications/twilio/README.md b/src/health/notifications/twilio/README.md
new file mode 100644
index 000000000..2037c1f3b
--- /dev/null
+++ b/src/health/notifications/twilio/README.md
@@ -0,0 +1,118 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/twilio/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/health/notifications/twilio/metadata.yaml"
+sidebar_label: "Twilio"
+learn_status: "Published"
+learn_rel_path: "Alerts & Notifications/Notifications/Agent Dispatched Notifications"
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE NOTIFICATION'S metadata.yaml FILE"
+endmeta-->
+
+# Twilio
+
+
+<img src="https://netdata.cloud/img/twilio.png" width="150"/>
+
+
+Send notifications to Twilio using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
+
+
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Setup
+
+### Prerequisites
+
+#### 
+
+- Get your SID, and Token from https://www.twilio.com/console
+- Terminal access to the Agent you wish to configure
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `health_alarm_notify.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config health_alarm_notify.conf
+```
+#### Options
+
+The following options can be defined for this notification
+
+<details><summary>Config Options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| SEND_TWILIO | Set `SEND_TWILIO` to YES | YES | yes |
+| TWILIO_ACCOUNT_SID | set `TWILIO_ACCOUNT_SID`  to your account SID. |  | yes |
+| TWILIO_ACCOUNT_TOKEN | Set `TWILIO_ACCOUNT_TOKEN` to your account token. |  | yes |
+| TWILIO_NUMBER | Set `TWILIO_NUMBER` to your account's number. |  | yes |
+| DEFAULT_RECIPIENT_TWILIO | Set DEFAULT_RECIPIENT_TWILIO to the number you want the alert notifications to be sent to. You can define multiple numbers like this: +15555555555 +17777777777. |  | yes |
+
+##### DEFAULT_RECIPIENT_TWILIO
+
+You can then have different recipients per role, by editing DEFAULT_RECIPIENT_TWILIO with the recipient's number you want, in the following entries at the bottom of the same file:
+
+```conf
+role_recipients_twilio[sysadmin]="+15555555555"
+role_recipients_twilio[domainadmin]="+15555555556"
+role_recipients_twilio[dba]="+15555555557"
+role_recipients_twilio[webmaster]="+15555555558"
+role_recipients_twilio[proxyadmin]="+15555555559"
+role_recipients_twilio[sitemgr]="+15555555550"
+```
+
+
+</details>
+
+#### Examples
+
+##### Basic Configuration
+
+
+
+```yaml
+#------------------------------------------------------------------------------
+# Twilio (twilio.com) SMS options
+
+SEND_TWILIO="YES"
+TWILIO_ACCOUNT_SID="xxxxxxxxx"
+TWILIO_ACCOUNT_TOKEN="xxxxxxxxxx"
+TWILIO_NUMBER="xxxxxxxxxxx"
+DEFAULT_RECIPIENT_TWILIO="+15555555555"
+
+```
+
+
+## Troubleshooting
+
+### Test Notification
+
+You can run the following command by hand, to test alerts configuration:
+
+```bash
+# become user netdata
+sudo su -s /bin/bash netdata
+
+# enable debugging info on the console
+export NETDATA_ALARM_NOTIFY_DEBUG=1
+
+# send test alarms to sysadmin
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
+
+# send test alarms to any role
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
+```
+
+Note that this will test _all_ alert mechanisms for the selected role.
+
+
diff --git a/health/notifications/twilio/metadata.yaml b/src/health/notifications/twilio/metadata.yaml
index 35fc3f042..35fc3f042 100644
--- a/health/notifications/twilio/metadata.yaml
+++ b/src/health/notifications/twilio/metadata.yaml
diff --git a/src/health/notifications/web/README.md b/src/health/notifications/web/README.md
new file mode 100644
index 000000000..d7115be3d
--- /dev/null
+++ b/src/health/notifications/web/README.md
@@ -0,0 +1,18 @@
+<!--
+title: "Browser pop up agent alert notifications"
+sidebar_label: "Browser pop ups"
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/health/notifications/web/README.md"
+learn_status: "Published"
+learn_topic_type: "Tasks"
+learn_rel_path: "Integrations/Notify/Agent alert notifications"
+learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"
+-->
+
+# Browser pop up agent alert notifications
+
+The Netdata dashboard shows HTML notifications, when it is open.
+
+Such web notifications look like this:
+![image](https://cloud.githubusercontent.com/assets/2662304/18407279/82bac6a6-7714-11e6-847e-c2e84eeacbfb.png)
+
+