Adding upstream version 1.33.0.upstream/1.33.0

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

9 files changed, 127 insertions, 108 deletions

diff --git a/health/REFERENCE.md b/health/REFERENCE.md
index 66ccf88d..4feb782d 100644
--- a/health/REFERENCE.md
+++ b/health/REFERENCE.md
@@ -52,7 +52,7 @@ Netdata parses the following lines. Beneath the table is an in-depth explanation
 -   The `every` line is **required** if not using `lookup`.
 -   Each entity **must** have at least one of the following lines: `lookup`, `calc`, `warn`, or `crit`.
 -   A few lines use space-separated lists to define how the entity behaves. You can use `*` as a wildcard or prefix with
-    `!` for a negative match. Order is important, too! See our [simple patterns docs](../libnetdata/simple_pattern/) for
+    `!` for a negative match. Order is important, too! See our [simple patterns docs](/libnetdata/simple_pattern/README.md) for
     more examples.
 -   Lines terminated by a `\` are spliced together with the next line. The backslash is removed and the following line is
     joined with the current one. No space is inserted, so you may split a line anywhere, even in the middle of a word.
@@ -275,7 +275,7 @@ template: disk_svctm_alarm
 The `families` line, used only alongside templates, filters which families within the context this alarm should apply
 to. The value is a space-separated list.
 
-The value is a space-separate list of simple patterns. See our [simple patterns docs](../libnetdata/simple_pattern/) for
+The value is a space-separate list of simple patterns. See our [simple patterns docs](/libnetdata/simple_pattern/README.md) for
 some examples.
 
 For example, you can create a template on the `disk.io` context, but filter it to only the `sda` and `sdb` families:
@@ -294,7 +294,7 @@ The format is:
 lookup: METHOD AFTER [at BEFORE] [every DURATION] [OPTIONS] [of DIMENSIONS] [foreach DIMENSIONS]
 ```
 
-Everything is the same with [badges](../web/api/badges/). In short:
+Everything is the same with [badges](/web/api/badges/README.md). In short:
 
 -   `METHOD` is one of `average`, `min`, `max`, `sum`, `incremental-sum`.
      This is required.
@@ -531,14 +531,14 @@ that will be applied to all hosts installed in the last decade with the followin
 host labels: installed = 201*
 ```
 
-See our [simple patterns docs](../libnetdata/simple_pattern/) for more examples.
+See our [simple patterns docs](/libnetdata/simple_pattern/README.md) for more examples.
 
 ## Expressions
 
-Netdata has an internal [infix expression parser](../libnetdata/eval). This parses expressions and creates an internal
+Netdata has an internal [infix expression parser](/libnetdata/eval). This parses expressions and creates an internal
 structure that allows fast execution of them.
 
-These operators are supported `+`, `-`, `*`, `/`, `<`, `<=`, `<>`, `!=`, `>`, `>=`, `&&`, `||`, `!`, `AND`, `OR`, `NOT`.
+These operators are supported `+`, `-`, `*`, `/`, `<`, `==`, `<=`, `<>`, `!=`, `>`, `>=`, `&&`, `||`, `!`, `AND`, `OR`, `NOT`.
 Boolean operators result in either `1` (true) or `0` (false).
 
 The conditional evaluation operator `?` is supported too. Using this operator IF-THEN-ELSE conditional statements can be
@@ -602,15 +602,15 @@ You can find all the variables that can be used for a given chart, using
 Agent dashboard. For example, [variables for the `system.cpu` chart of the
 registry](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu).
 
-> If you don't know how to find the CHART_NAME, you can read about it [here](../web/README.md#charts).
+> If you don't know how to find the CHART_NAME, you can read about it [here](/web/README.md#charts).
 
 Netdata supports 3 internal indexes for variables that will be used in health monitoring.
 
 <details markdown="1"><summary>The variables below can be used in both chart alarms and context templates.</summary>
 
 Although the `alarm_variables` link shows you variables for a particular chart, the same variables can also be used in
-templates for charts belonging to a given [context](../web/README.md#contexts). The reason is that all charts of a given
-context are essentially identical, with the only difference being the [family](../web/README.md#families) that
+templates for charts belonging to a given [context](/web/README.md#contexts). The reason is that all charts of a given
+context are essentially identical, with the only difference being the [family](/web/README.md#families) that
 identifies a particular hardware or software instance. Charts and templates do not apply to specific families anyway,
 unless if you explicitly limit an alarm with the [alarm line `families`](#alarm-line-families).
 
diff --git a/health/health.c b/health/health.c
index d8e1d4b7..e94339fa 100644
--- a/health/health.c
+++ b/health/health.c
@@ -101,7 +101,11 @@ static void health_silencers_init(void) {
                 freez(str);
             }
         } else {
-            error("Health silencers file %s has the size %ld that is out of range[ 1 , %d ]. Aborting read.", silencers_filename, length, HEALTH_SILENCERS_MAX_FILE_LEN);
+            error(
+                "Health silencers file %s has the size %" PRId64 " that is out of range[ 1 , %d ]. Aborting read.",
+                silencers_filename,
+                (int64_t)length,
+                HEALTH_SILENCERS_MAX_FILE_LEN);
         }
         fclose(fd);
     } else {
@@ -326,7 +330,7 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
                     buffer_strcat(warn_alarms, ",");
                 buffer_strcat(warn_alarms, rc->name);
                 buffer_strcat(warn_alarms, "=");
-                buffer_snprintf(warn_alarms, 11, "%ld", rc->last_status_change);
+                buffer_snprintf(warn_alarms, 11, "%"PRId64"", (int64_t)rc->last_status_change);
                 n_warn++;
             } else if (ae->alarm_id == rc->id)
                 expr = rc->warning;
@@ -336,7 +340,7 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
                     buffer_strcat(crit_alarms, ",");
                 buffer_strcat(crit_alarms, rc->name);
                 buffer_strcat(crit_alarms, "=");
-                buffer_snprintf(crit_alarms, 11, "%ld", rc->last_status_change);
+                buffer_snprintf(crit_alarms, 11, "%"PRId64"", (int64_t)rc->last_status_change);
                 n_crit++;
             } else if (ae->alarm_id == rc->id)
                 expr = rc->critical;
@@ -346,9 +350,9 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
         }
     }
 
-    char *edit_command = ae->source ? health_edit_command_from_source(ae->source) : strdupz("UNKNOWN=0");
+    char *edit_command = ae->source ? health_edit_command_from_source(ae->source) : strdupz("UNKNOWN=0=UNKNOWN");
 
-    snprintfz(command_to_run, ALARM_EXEC_COMMAND_LENGTH, "exec %s '%s' '%s' '%u' '%u' '%u' '%lu' '%s' '%s' '%s' '%s' '%s' '" CALCULATED_NUMBER_FORMAT_ZERO "' '" CALCULATED_NUMBER_FORMAT_ZERO "' '%s' '%u' '%u' '%s' '%s' '%s' '%s' '%s' '%s' '%d' '%d' '%s' '%s' '%s' '%s' '%s'",
+    snprintfz(command_to_run, ALARM_EXEC_COMMAND_LENGTH, "exec %s '%s' '%s' '%u' '%u' '%u' '%lu' '%s' '%s' '%s' '%s' '%s' '" CALCULATED_NUMBER_FORMAT_ZERO "' '" CALCULATED_NUMBER_FORMAT_ZERO "' '%s' '%u' '%u' '%s' '%s' '%s' '%s' '%s' '%s' '%d' '%d' '%s' '%s' '%s' '%s'",
               exec,
               recipient,
               host->registry_hostname,
@@ -377,8 +381,7 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
               buffer_tostring(warn_alarms),
               buffer_tostring(crit_alarms),
               ae->classification?ae->classification:"Unknown",
-              edit_command,
-              localhost->registry_hostname
+              edit_command
     );
 
     ae->flags |= HEALTH_ENTRY_FLAG_EXEC_RUN;
@@ -673,6 +676,9 @@ void *health_main(void *ptr) {
     rrdcalc_labels_unlink();
 
     unsigned int loop = 0;
+#if defined(ENABLE_ACLK) && defined(ENABLE_NEW_CLOUD_PROTOCOL)
+    unsigned int marked_aclk_reload_loop = 0;
+#endif
     while(!netdata_exit) {
         loop++;
         debug(D_HEALTH, "Health monitoring iteration no %u started", loop);
@@ -684,9 +690,10 @@ void *health_main(void *ptr) {
         if (unlikely(check_if_resumed_from_suspension())) {
             apply_hibernation_delay = 1;
 
-            info("Postponing alarm checks for %ld seconds, because it seems that the system was just resumed from suspension.",
-                 hibernation_delay
-            );
+            info(
+                "Postponing alarm checks for %"PRId64" seconds, "
+                "because it seems that the system was just resumed from suspension.",
+                (int64_t)hibernation_delay);
         }
 
         if (unlikely(silencers->all_alarms && silencers->stype == STYPE_DISABLE_ALARMS)) {
@@ -698,6 +705,11 @@ void *health_main(void *ptr) {
             }
         }
 
+#if defined(ENABLE_ACLK) && defined(ENABLE_NEW_CLOUD_PROTOCOL)
+        if (aclk_alert_reloaded && !marked_aclk_reload_loop)
+            marked_aclk_reload_loop = loop;
+#endif
+
         rrd_rdlock();
 
         RRDHOST *host;
@@ -706,9 +718,10 @@ void *health_main(void *ptr) {
                 continue;
 
             if (unlikely(apply_hibernation_delay)) {
-
-                info("Postponing health checks for %ld seconds, on host '%s'.", hibernation_delay, host->hostname
-                );
+                info(
+                    "Postponing health checks for %"PRId64" seconds, on host '%s'.",
+                    (int64_t)hibernation_delay,
+                    host->hostname);
 
                 host->health_delay_up_to = now + hibernation_delay;
             }
@@ -1038,14 +1051,6 @@ void *health_main(void *ptr) {
                 rrdhost_unlock(host);
             }
 
-#ifdef ENABLE_ACLK
-#ifdef ENABLE_NEW_CLOUD_PROTOCOL
-            if (netdata_cloud_setting && unlikely(aclk_alert_reloaded) && loop > 2) {
-                sql_queue_removed_alerts_to_aclk(host);
-            }
-#endif
-#endif
-
             if (unlikely(netdata_exit))
                 break;
 
@@ -1070,9 +1075,16 @@ void *health_main(void *ptr) {
             health_alarm_wait_for_execution(ae);
         }
 
-#ifdef ENABLE_NEW_CLOUD_PROTOCOL
-        if (netdata_cloud_setting && unlikely(aclk_alert_reloaded))
-            aclk_alert_reloaded = 0;
+#if defined(ENABLE_ACLK) && defined(ENABLE_NEW_CLOUD_PROTOCOL)
+        if (netdata_cloud_setting && unlikely(aclk_alert_reloaded) && loop > (marked_aclk_reload_loop + 2)) {
+                rrdhost_foreach_read(host) {
+                    if (unlikely(!host->health_enabled))
+                        continue;
+                    sql_queue_removed_alerts_to_aclk(host);
+                }
+                aclk_alert_reloaded = 0;
+                marked_aclk_reload_loop = 0;
+            }
 #endif
 
         rrd_unlock();
diff --git a/health/health.h b/health/health.h
index 09040b3a..323279ba 100644
--- a/health/health.h
+++ b/health/health.h
@@ -5,17 +5,6 @@
 
 #include "daemon/common.h"
 
-#define NETDATA_PLUGIN_HOOK_HEALTH \
-    { \
-        .name = "HEALTH", \
-        .config_section = NULL, \
-        .config_name = NULL, \
-        .enabled = 1, \
-        .thread = NULL, \
-        .init_routine = NULL, \
-        .start_routine = health_main \
-    },
-
 extern unsigned int default_health_enabled;
 
 #define HEALTH_ENTRY_FLAG_PROCESSED             0x00000001
@@ -43,7 +32,6 @@ extern unsigned int default_health_enabled;
 extern char *silencers_filename;
 
 extern void health_init(void);
-extern void *health_main(void *ptr);
 
 extern void health_reload(void);
 
diff --git a/health/health_config.c b/health/health_config.c
index 35234df1..da71f13d 100644
--- a/health/health_config.c
+++ b/health/health_config.c
@@ -485,10 +485,11 @@ char *health_edit_command_from_source(const char *source)
         snprintfz(
             buffer,
             FILENAME_MAX,
-            "sudo %s/edit-config health.d/%s=%s",
+            "sudo %s/edit-config health.d/%s=%s=%s",
             netdata_configured_user_config_dir,
             file_no_path + 1,
-            temp);
+            temp,
+            localhost->registry_hostname);
     } else
         buffer[0] = '\0';
 
diff --git a/health/health_json.c b/health/health_json.c
index a21d5a4f..be95100b 100644
--- a/health/health_json.c
+++ b/health/health_json.c
@@ -14,7 +14,7 @@ void health_string2json(BUFFER *wb, const char *prefix, const char *label, const
 }
 
 void health_alarm_entry2json_nolock(BUFFER *wb, ALARM_ENTRY *ae, RRDHOST *host) {
-    char *edit_command = ae->source ? health_edit_command_from_source(ae->source) : strdupz("UNKNOWN=0");
+    char *edit_command = ae->source ? health_edit_command_from_source(ae->source) : strdupz("UNKNOWN=0=UNKNOWN");
     char config_hash_id[GUID_LEN + 1];
     uuid_unparse_lower(ae->config_hash_id, config_hash_id);
 
diff --git a/health/health_log.c b/health/health_log.c
index d20085d9..6d63966c 100644
--- a/health/health_log.c
+++ b/health/health_log.c
@@ -112,7 +112,7 @@ inline void health_alarm_log_save(RRDHOST *host, ALARM_ENTRY *ae) {
                         "\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s"
                         "\t%d\t%d\t%d\t%d"
                         "\t" CALCULATED_NUMBER_FORMAT_AUTO "\t" CALCULATED_NUMBER_FORMAT_AUTO
-                        "\t%016lx"
+                        "\t%016"PRIx64""
                         "\t%s\t%s\t%s"
                         "\n"
                             , (ae->flags & HEALTH_ENTRY_FLAG_SAVED)?'U':'A'
diff --git a/health/notifications/alarm-notify.sh.in b/health/notifications/alarm-notify.sh.in
index 08a32ff1..287cabfe 100755
--- a/health/notifications/alarm-notify.sh.in
+++ b/health/notifications/alarm-notify.sh.in
@@ -243,7 +243,6 @@ else
   total_crit_alarms="${26}"  # List of alarms in critical state
   classification="${27}"     # The class field from .conf files
   edit_command_line="${28}"  # The command to edit the alarm, with the line number
-  sender_host="${29}"        # The host sending this notification
 fi
 
 # -----------------------------------------------------------------------------
@@ -258,17 +257,6 @@ else
 fi
 
 # -----------------------------------------------------------------------------
-# Do the same for sender_host (find a suitable hostname to use, if netdata did not supply a hostname)
-
-if [ -z ${sender_host} ]; then
-  this_host=$(hostname -s 2>/dev/null)
-  s_host="${this_host}"
-  sender_host="${this_host}"
-else
-  s_host="${sender_host}"
-fi
-
-# -----------------------------------------------------------------------------
 # screen statuses we don't need to send a notification
 
 # don't do anything if this is not WARNING, CRITICAL or CLEAR
@@ -1012,10 +1000,21 @@ send_pushover() {
 # pushbullet sender
 
 send_pushbullet() {
-  local userapikey="${1}" source_device="${2}" recipients="${3}" url="${4}" title="${5}" message="${6}" httpcode sent=0 user
+  local userapikey="${1}" source_device="${2}" recipients="${3}" url="${4}" title="${5}" message="${6}" httpcode sent=0 userOrChannelTag
   if [ "${SEND_PUSHBULLET}" = "YES" ] && [ -n "${userapikey}" ] && [ -n "${recipients}" ] && [ -n "${message}" ] && [ -n "${title}" ]; then
-    #https://docs.pushbullet.com/#create-push
-    for user in ${recipients}; do
+
+    # https://docs.pushbullet.com/#create-push
+    # Accept specification of user(s) (PushBullet account email address) and/or channel tag(s), separated by spaces.
+    # If recipient begins with a "#" then send to channel tag, otherwise send to email recipient.
+
+    for userOrChannelTag in ${recipients}; do
+      if [ "${userOrChannelTag::1}" = "#" ]; then
+        userOrChannelTag_type="channel_tag"
+        userOrChannelTag="${userOrChannelTag:1}" # Remove hash from start of channel tag (required by pushbullet API)
+      else
+        userOrChannelTag_type="email"
+      fi
+
       httpcode=$(docurl \
         --header 'Access-Token: '${userapikey}'' \
         --header 'Content-Type: application/json' \
@@ -1023,7 +1022,7 @@ send_pushbullet() {
           cat <<EOF
                               {"title": "${title}",
                               "type": "link",
-                              "email": "${user}",
+                              "${userOrChannelTag_type}": "${userOrChannelTag}",
                               "body": "$(echo -n ${message})",
                               "url": "${url}",
                               "source_device_iden": "${source_device}"}
@@ -1031,10 +1030,10 @@ EOF
         ) "https://api.pushbullet.com/v2/pushes" -X POST)
 
       if [ "${httpcode}" = "200" ]; then
-        info "sent pushbullet notification for: ${host} ${chart}.${name} is ${status} to '${user}'"
+        info "sent pushbullet notification for: ${host} ${chart}.${name} is ${status} to '${userOrChannelTag}'"
         sent=$((sent + 1))
       else
-        error "failed to send pushbullet notification for: ${host} ${chart}.${name} is ${status} to '${user}' with HTTP response status code ${httpcode}."
+        error "failed to send pushbullet notification for: ${host} ${chart}.${name} is ${status} to '${userOrChannelTag}' with HTTP response status code ${httpcode}."
       fi
     done
 
@@ -2845,7 +2844,7 @@ if [ -n "$total_crit_alarms" ]; then
 fi
 
 if [ -n "$edit_command_line" ]; then
-    IFS='=' read -r edit_command line <<<"$edit_command_line"
+    IFS='=' read -r edit_command line s_host <<<"$edit_command_line"
 fi
 
 IFS='' read -r -d '' email_html_part <<EOF
@@ -3258,7 +3257,7 @@ Content-Transfer-Encoding: 8bit
                     </tr>
                     <tr>
                       <td align="left" style="font-size:0px;padding:10px 25px;padding-left:0;word-break:break-word;">
-                        <div style="font-family:Open Sans, sans-serif;font-size:14px;line-height:1.3;text-align:left;color:#35414A;">Discuss and troubleshoot with others on the Netdata <a href="https://community.netdata.cloud/" class="link" style="color: #00AB44; text-decoration: none;">community forums</a></div>
+                        <div style="font-family:Open Sans, sans-serif;font-size:14px;line-height:1.3;text-align:left;color:#35414A;">Join the troubleshooting discussion for this alert on our <a href="https://community.netdata.cloud/t/${name//[._]/-}" class="link" style="color: #00AB44; text-decoration: none;">community forums</a>.</div>
                       </td>
                     </tr>
                     </tbody>
@@ -3329,7 +3328,7 @@ Content-Transfer-Encoding: 8bit
                     <tr>
                       <td align="left" style="font-size:0px;padding:10px 25px;padding-top:8px;padding-left:0;word-break:break-word;">
                         <div style="font-family:Open Sans, sans-serif;font-size:12px;line-height:1.3;text-align:left;color:#35414A;">${edit_command} <br>
-                          The alarm to edit is at line {${line}}</div>
+                          <br>The alarm to edit is at line ${line}</div>
                       </td>
                     </tr>
                     </tbody>
diff --git a/health/notifications/pagerduty/README.md b/health/notifications/pagerduty/README.md
index b1f60d49..30db6379 100644
--- a/health/notifications/pagerduty/README.md
+++ b/health/notifications/pagerduty/README.md
@@ -1,46 +1,63 @@
 <!--
-title: "PagerDuty"
+title: "Send alert notifications to PagerDuty"
+description: "Send alerts to your PagerDuty dashboard any time an anomaly or performance issue strikes a node in your infrastructure."
+sidebar_label: "PagerDuty"
 custom_edit_url: https://github.com/netdata/netdata/edit/master/health/notifications/pagerduty/README.md
 -->
 
-# PagerDuty
+# Send alert notifications to PagerDuty
 
-[PagerDuty](https://www.pagerduty.com/company/) is the enterprise incident resolution service that integrates with ITOps and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into incidents, PagerDuty streamlines the incident management process by reducing alert noise and resolution times.
+[PagerDuty](https://www.pagerduty.com/company/) is an enterprise incident resolution service that integrates with ITOps
+and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to
+correlating them into incidents, PagerDuty streamlines the incident management process by reducing alert noise and
+resolution times.
 
-Here is an example of a PagerDuty dashboard with Netdata notifications:
+## What you need to get started
 
-![PagerDuty dashboard with Netdata notifications](https://cloud.githubusercontent.com/assets/19278582/21233877/b466a08a-c2a5-11e6-8d66-ee6eed43818f.png)
+- An installation of the open-source [Netdata](/docs/get-started.mdx) monitoring agent.
+- An installation of the [PagerDuty agent](https://www.pagerduty.com/docs/guides/agent-install-guide/) on the node
+  running Netdata.
+- A PagerDuty `Generic API` service using either the `Events API v2` or `Events API v1`.
 
-To have Netdata send notifications to PagerDuty, you'll first need to set up a PagerDuty `Generic API` service and install the PagerDuty agent on the host running Netdata.  See the following guide for details:
+## Setup
 
-<https://www.pagerduty.com/docs/guides/agent-install-guide/>
+[Add a new service](https://support.pagerduty.com/docs/services-and-integrations#section-configuring-services-and-integrations)
+to PagerDuty. Click **Use our API directly** and select either `Events API v2` or `Events API v1`. Once you finish
+creating the service, click on the **Integrations** tab to find your **Integration Key**.
 
-During the setup of the `Generic API` PagerDuty service, you'll obtain a `pagerduty service key`.  Keep this **service key** handy.
-
-Once the PagerDuty agent is installed on your host and can send notifications from your host to your `Generic API` service on PagerDuty, add the **service key** to `DEFAULT_RECIPIENT_PD` in `health_alarm_notify.conf`:
+Navigate to the [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory) and use
+[`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) to open
+`health_alarm_notify.conf`.
 
+```bash
+cd /etc/netdata
+sudo ./edit-config health_alarm_notify.conf
 ```
-#------------------------------------------------------------------------------
-# pagerduty.com notification options
-#
-# pagerduty.com notifications require the pagerduty agent to be installed and 
-# a "Generic API" pagerduty service.
-# https://www.pagerduty.com/docs/guides/agent-install-guide/
 
-# multiple recipients can be given like this:
-#              "<pd_service_key_1> <pd_service_key_2> ..."
+Scroll down to the `# pagerduty.com notification options` section.
+
+Ensure `SEND_PD` is set to `YES`, then copy your Integration Key into `DEFAULT_RECIPIENT_ID`. Change `USE_PD_VERSION` to
+`2` if you chose `Events API v2` during service setup on PagerDuty. Minus comments, the section should look like this:
 
-# enable/disable sending pagerduty notifications
+```conf
 SEND_PD="YES"
+DEFAULT_RECIPIENT_PD="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+USE_PD_VERSION="2"
+```
 
-# if a role's recipients are not configured, a notification will be sent to
-# the "General API" pagerduty.com service that uses this service key.
-# (empty = do not send a notification for unconfigured roles):
-DEFAULT_RECIPIENT_PD="<service key>"
+## Testing
 
-# Which PD API are we going to use? For version 2 or newer, it is necessary to do a request for Pagerduty
-# before to set the version(https://developer.pagerduty.com/docs/events-api-v2/overview/).
-USE_PD_VERSION="1"
+To test alert notifications to PagerDuty, run the following:
+
+```bash
+sudo su -s /bin/bash netdata
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test
 ```
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fhealth%2Fnotifications%2Fpagerduty%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+## Configuration
+
+Aside from the three values set in `health_alarm_notify.conf`, there is no further configuration required to send alert
+notifications to PagerDuty.
+
+To configure individual alarms, read our [alert configuration](/docs/monitor/configure-alarms.md) doc or
+the [health entity reference](/health/REFERENCE.md) doc.
diff --git a/health/notifications/pushbullet/README.md b/health/notifications/pushbullet/README.md
index 7a098d6a..a1c88c70 100644
--- a/health/notifications/pushbullet/README.md
+++ b/health/notifications/pushbullet/README.md
@@ -14,20 +14,22 @@ And like this on your Android device:
 
 You will need:
 
-1.  Signup and Login to pushbullet.com
-2.  Get your Access Token, go to <https://www.pushbullet.com/#settings/account> and create a new one
-3.  Fill in the PUSHBULLET_ACCESS_TOKEN with that value
-4.  Add the recipient emails to DEFAULT_RECIPIENT_PUSHBULLET
-    !!PLEASE NOTE THAT IF THE RECIPIENT DOES NOT HAVE A PUSHBULLET ACCOUNT, PUSHBULLET SERVICE WILL SEND AN EMAIL!!
+1.  Sign up and log in to [pushbullet.com](pushbullet.com)
+2.  Create a new access token in your [account settings](https://www.pushbullet.com/#settings/account).
+3.  Fill in the `PUSHBULLET_ACCESS_TOKEN` with the newly generated access token.
+4.  Add the recipient emails or channel tags (each channel tag must be prefixed with #, e.g. #channeltag) to `DEFAULT_RECIPIENT_PUSHBULLET`.
+    > 🚨 The pushbullet notification service will send emails to the email recipient, regardless of if they have a pushbullet account.
 
-Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:
+To add notification channels, run `/etc/netdata/edit-config health_alarm_notify.conf` 
+
+You can change the configuration like this:
 
 ```
 ###############################################################################
 # pushbullet (pushbullet.com) push notification options
 
-# multiple recipients can be given like this:
-#                  "user1@email.com user2@mail.com"
+# multiple recipients (a combination of email addresses or channel tags) can be given like this:
+#                  "user1@email.com user2@mail.com #channel1 #channel2"
 
 # enable/disable sending pushbullet notifications
 SEND_PUSHBULLET="YES"
@@ -35,14 +37,14 @@ SEND_PUSHBULLET="YES"
 # Signup and Login to pushbullet.com
 # To get your Access Token, go to https://www.pushbullet.com/#settings/account
 # And create a new access token
-# Then just set the recipients emails
-# Please note that the if the email in the DEFAULT_RECIPIENT_PUSHBULLET does
+# Then just set the recipients emails and/or channel tags (channel tags must be prefixed with #)
+# Please note that the if an email in the DEFAULT_RECIPIENT_PUSHBULLET does
 # not have a pushbullet account, the pushbullet service will send an email
 # to that address instead
 
 # Without an access token, Netdata cannot send pushbullet notifications.
 PUSHBULLET_ACCESS_TOKEN="o.Sometokenhere"
-DEFAULT_RECIPIENT_PUSHBULLET="admin1@example.com admin3@somemail.com"
+DEFAULT_RECIPIENT_PUSHBULLET="admin1@example.com admin3@somemail.com #examplechanneltag #anotherchanneltag"
 ```
 
 [![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fhealth%2Fnotifications%2Fpushbullet%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)