# Custom Netdata Agent's alert notification feature allows you to send custom notifications to any endpoint you choose. ## Setup ### Prerequisites #### - Access to the terminal where Netdata Agent is running ### Configuration #### File The configuration file name for this integration is `health_alarm_notify.conf`. You can edit the configuration file using the `edit-config` script from the Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). ```bash cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata sudo ./edit-config health_alarm_notify.conf ``` #### Options The following options can be defined for this notification
Config Options | Name | Description | Default | Required | |:----|:-----------|:-------|:--------:| | SEND_CUSTOM | Set `SEND_CUSTOM` to YES | YES | yes | | DEFAULT_RECIPIENT_CUSTOM | This value is dependent on how you handle the `${to}` variable inside the `custom_sender()` function. | | yes | | custom_sender() | You can look at the other senders in `/usr/libexec/netdata/plugins.d/alarm-notify.sh` for examples of how to modify the function in this configuration file. | | no | ##### DEFAULT_RECIPIENT_CUSTOM All roles will default to this variable if left unconfigured. You can edit `DEFAULT_RECIPIENT_CUSTOM` with the variable you want, in the following entries at the bottom of the same file: ``` role_recipients_custom[sysadmin]="systems" role_recipients_custom[domainadmin]="domains" role_recipients_custom[dba]="databases systems" role_recipients_custom[webmaster]="marketing development" role_recipients_custom[proxyadmin]="proxy-admin" role_recipients_custom[sitemgr]="sites" ``` ##### custom_sender() The following is a sample custom_sender() function in health_alarm_notify.conf, to send an SMS via an imaginary HTTPS endpoint to the SMS gateway: ``` custom_sender() { # example human readable SMS local msg="${host} ${status_message}: ${alarm} ${raised_for}" # limit it to 160 characters and encode it for use in a URL urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}" # a space separated list of the recipients to send alarms to to="${1}" for phone in ${to}; do httpcode=$(docurl -X POST \ --data-urlencode "From=XXX" \ --data-urlencode "To=${phone}" \ --data-urlencode "Body=${msg}" \ -u "${accountsid}:${accounttoken}" \ https://domain.website.com/) if [ "${httpcode}" = "200" ]; then info "sent custom notification ${msg} to ${phone}" sent=$((sent + 1)) else error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}." fi done } ``` The supported variables that you can use for the function's `msg` variable are: | Variable name | Description | |:---------------------------:|:---------------------------------------------------------------------------------| | `${alarm}` | Like "name = value units" | | `${status_message}` | Like "needs attention", "recovered", "is critical" | | `${severity}` | Like "Escalated to CRITICAL", "Recovered from WARNING" | | `${raised_for}` | Like "(alarm was raised for 10 minutes)" | | `${host}` | The host generated this event | | `${url_host}` | Same as ${host} but URL encoded | | `${unique_id}` | The unique id of this event | | `${alarm_id}` | The unique id of the alarm that generated this event | | `${event_id}` | The incremental id of the event, for this alarm id | | `${when}` | The timestamp this event occurred | | `${name}` | The name of the alarm, as given in netdata health.d entries | | `${url_name}` | Same as ${name} but URL encoded | | `${chart}` | The name of the chart (type.id) | | `${url_chart}` | Same as ${chart} but URL encoded | | `${status}` | The current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL | | `${old_status}` | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL | | `${value}` | The current value of the alarm | | `${old_value}` | The previous value of the alarm | | `${src}` | The line number and file the alarm has been configured | | `${duration}` | The duration in seconds of the previous alarm state | | `${duration_txt}` | Same as ${duration} for humans | | `${non_clear_duration}` | The total duration in seconds this is/was non-clear | | `${non_clear_duration_txt}` | Same as ${non_clear_duration} for humans | | `${units}` | The units of the value | | `${info}` | A short description of the alarm | | `${value_string}` | Friendly value (with units) | | `${old_value_string}` | Friendly old value (with units) | | `${image}` | The URL of an image to represent the status of the alarm | | `${color}` | A color in AABBCC format for the alarm | | `${goto_url}` | The URL the user can click to see the netdata dashboard | | `${calc_expression}` | The expression evaluated to provide the value for the alarm | | `${calc_param_values}` | The value of the variables in the evaluated expression | | `${total_warnings}` | The total number of alarms in WARNING state on the host | | `${total_critical}` | The total number of alarms in CRITICAL state on the host |
#### Examples ##### Basic Configuration ```yaml #------------------------------------------------------------------------------ # custom notifications SEND_CUSTOM="YES" DEFAULT_RECIPIENT_CUSTOM="" # The custom_sender() is a custom function to do whatever you need to do custom_sender() { # example human readable SMS local msg="${host} ${status_message}: ${alarm} ${raised_for}" # limit it to 160 characters and encode it for use in a URL urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}" # a space separated list of the recipients to send alarms to to="${1}" for phone in ${to}; do httpcode=$(docurl -X POST \ --data-urlencode "From=XXX" \ --data-urlencode "To=${phone}" \ --data-urlencode "Body=${msg}" \ -u "${accountsid}:${accounttoken}" \ https://domain.website.com/) if [ "${httpcode}" = "200" ]; then info "sent custom notification ${msg} to ${phone}" sent=$((sent + 1)) else error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}." fi done } ``` ## Troubleshooting ### Test Notification You can run the following command by hand, to test alerts configuration: ```bash # become user netdata sudo su -s /bin/bash netdata # enable debugging info on the console export NETDATA_ALARM_NOTIFY_DEBUG=1 # send test alarms to sysadmin /usr/libexec/netdata/plugins.d/alarm-notify.sh test # send test alarms to any role /usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE" ``` Note that this will test _all_ alert mechanisms for the selected role.