diff options
Diffstat (limited to 'docs/cloud/alerts-notifications')
9 files changed, 159 insertions, 23 deletions
diff --git a/docs/cloud/alerts-notifications/add-discord-notification.md b/docs/cloud/alerts-notifications/add-discord-notification.md index d1769f0e..3edf5002 100644 --- a/docs/cloud/alerts-notifications/add-discord-notification.md +++ b/docs/cloud/alerts-notifications/add-discord-notification.md @@ -8,7 +8,7 @@ To enable Discord notifications you need: - A Netdata Cloud account - Access to the space as an **administrator** -- Have a Discord server able to receive webhook integrations. For mode details check [how to configure this on Discord](#settings-on-discord) +- Have a Discord server able to receive webhook integrations. For more details check [how to configure this on Discord](#settings-on-discord) ## Steps diff --git a/docs/cloud/alerts-notifications/add-mattermost-notification-configuration.md b/docs/cloud/alerts-notifications/add-mattermost-notification-configuration.md new file mode 100644 index 00000000..79bc9861 --- /dev/null +++ b/docs/cloud/alerts-notifications/add-mattermost-notification-configuration.md @@ -0,0 +1,51 @@ +# Add Mattermost notification configuration + +From the Cloud interface, you can manage your space's notification settings and from these you can add a specific configuration to get notifications delivered on Mattermost. + +## Prerequisites + +To add Mattermost notification configurations you need: + +- A Netdata Cloud account +- Access to the space as an **administrator** +- Space needs to be on **Business** plan or higher +- Have a Mattermost app on your workspace to receive the webhooks, for more details check [how to configure this on Mattermost](#settings-on-mattermost) + +## Steps + +1. Click on the **Space settings** cog (located above your profile icon) +1. Click on the **Notification** tab +1. Click on the **+ Add configuration** button (near the top-right corner of your screen) +1. On the **Mattermost** card click on **+ Add** +1. A modal will be presented to you to enter the required details to enable the configuration: + 1. **Notification settings** are Netdata specific settings + - Configuration name - you can optionally provide a name for your configuration you can easily refer to it + - Rooms - by specifying a list of Rooms you are select to which nodes or areas of your infrastructure you want to be notified using this configuration + - Notification - you specify which notifications you want to be notified using this configuration: All Alerts and unreachable, All Alerts, Critical only + 1. **Integration configuration** are the specific notification integration required settings, which vary by notification method. For Mattermost: + - Webhook URL - URL provided on Mattermost for the channel you want to receive your notifications. For more details check [how to configure this on Mattermost](#settings-on-mattermost) + +## Settings on Mattermost + +To enable the webhook integrations on Mattermost you need: +1. In Mattermost, go to Product menu > Integrations > Incoming Webhook. + + + + - If you don’t have the Integrations option, incoming webhooks may not be enabled on your Mattermost server or may be disabled for non-admins. They can be enabled by a System Admin from System Console > Integrations > Integration Management. Once incoming webhooks are enabled, continue with the steps below + + + +2. Select Add Incoming Webhook and add a name and description for the webhook. The description can be up to 500 characters + +3. Select the channel to receive webhook payloads, then select Add to create the webhook + + + +4. You will end up with a webhook endpoint that looks like so: +``` +https://your-mattermost-server.com/hooks/xxx-generatedkey-xxx +``` + - Treat this endpoint as a secret. Anyone who has it will be able to post messages to your Mattermost instance. + +For more details please check Mattermost's article [Incoming webhooks for Mattermost](https://developers.mattermost.com/integrate/webhooks/incoming/). diff --git a/docs/cloud/alerts-notifications/add-opsgenie-notification-configuration.md b/docs/cloud/alerts-notifications/add-opsgenie-notification-configuration.md index 28e526c9..0a80311e 100644 --- a/docs/cloud/alerts-notifications/add-opsgenie-notification-configuration.md +++ b/docs/cloud/alerts-notifications/add-opsgenie-notification-configuration.md @@ -4,7 +4,7 @@ From the Cloud interface, you can manage your space's notification settings and ## Prerequisites -To add Opsgenie notification configurations you need +To add Opsgenie notification configurations you need: - A Netdata Cloud account - Access to the space as an **administrator** @@ -34,4 +34,4 @@ To enable the Netdata integration on Opsgenie you need: 1. Pick **API** from available integrations. Copy your API Key and press **Save Integration**. -1. Paste copied API key into the corresponding field in **Integration configuration** section of Opsgenie modal window in Netdata.
\ No newline at end of file +1. Paste copied API key into the corresponding field in **Integration configuration** section of Opsgenie modal window in Netdata. diff --git a/docs/cloud/alerts-notifications/add-pagerduty-notification-configuration.md b/docs/cloud/alerts-notifications/add-pagerduty-notification-configuration.md index 64880ebe..eec4f94c 100644 --- a/docs/cloud/alerts-notifications/add-pagerduty-notification-configuration.md +++ b/docs/cloud/alerts-notifications/add-pagerduty-notification-configuration.md @@ -4,12 +4,12 @@ From the Cloud interface, you can manage your space's notification settings and ## Prerequisites -To add PagerDuty notification configurations you need +To add PagerDuty notification configurations you need: - A Cloud account - Access to the space as and **administrator** - Space needs to be on **Business** plan or higher -- Have a PagerDuty service to receive events, for mode details check [how to configure this on PagerDuty](#settings-on-pagerduty) +- Have a PagerDuty service to receive events, for more details check [how to configure this on PagerDuty](#settings-on-pagerduty) ## Steps diff --git a/docs/cloud/alerts-notifications/add-slack-notification-configuration.md b/docs/cloud/alerts-notifications/add-slack-notification-configuration.md index 99bb2d5b..ed845b4d 100644 --- a/docs/cloud/alerts-notifications/add-slack-notification-configuration.md +++ b/docs/cloud/alerts-notifications/add-slack-notification-configuration.md @@ -4,12 +4,12 @@ From the Cloud interface, you can manage your space's notification settings and ## Prerequisites -To add discord notification configurations you need +To add slack notification configurations you need: - A Netdata Cloud account - Access to the space as an **administrator** - Space needs to be on **Business** plan or higher -- Have a Slack app on your workspace to receive the webhooks, for mode details check [how to configure this on Slack](#settings-on-slack) +- Have a Slack app on your workspace to receive the webhooks, for more details check [how to configure this on Slack](#settings-on-slack) ## Steps @@ -34,7 +34,7 @@ To enable the webhook integrations on Slack you need: - On your app go to **Incoming Webhooks** and click on **activate incoming webhooks**  - + - At the bottom of **Webhook URLs for Your Workspace** section you have **Add New Webhook to Workspace** - After pressing that specify the channel where you want your notifications to be delivered diff --git a/docs/cloud/alerts-notifications/add-webhook-notification-configuration.md b/docs/cloud/alerts-notifications/add-webhook-notification-configuration.md index 0140c30f..21d1b6ed 100644 --- a/docs/cloud/alerts-notifications/add-webhook-notification-configuration.md +++ b/docs/cloud/alerts-notifications/add-webhook-notification-configuration.md @@ -4,12 +4,12 @@ From the Cloud interface, you can manage your space's notification settings and ## Prerequisites -To add discord notification configurations you need +To add webhook notification configurations you need: - A Netdata Cloud account - Access to the space as an **administrator** - Space needs to be on **Pro** plan or higher -- Have an app that allows you to receive webhooks following a predefined schema, for mode details check [how to create the webhook service](#webhook-service) +- Have an app that allows you to receive webhooks following a predefined schema, for more details check [how to create the webhook service](#webhook-service) ## Steps @@ -24,8 +24,8 @@ To add discord notification configurations you need - Notification - you specify which notifications you want to be notified using this configuration: All Alerts and unreachable, All Alerts, Critical only 1. **Integration configuration** are the specific notification integration required settings, which vary by notification method. For webhook: - Webhook URL - webhook URL is the url of the service that Netdata will send notifications to. In order to keep the communication secured, we only accept HTTPS urls. Check [how to create the webhook service](#webhook-service). - - Extra headers - these are optional key-value pairs that you can set to be included in the HTTP requests sent to the webhook URL. For mode details check [Extra headers](#extra-headers) - - Authentication Mechanism - Netdata webhook integration supports 3 different authentication mechanisms. For mode details check [Authentication mechanisms](#authentication-mechanisms): + - Extra headers - these are optional key-value pairs that you can set to be included in the HTTP requests sent to the webhook URL. For more details check [Extra headers](#extra-headers) + - Authentication Mechanism - Netdata webhook integration supports 3 different authentication mechanisms. For more details check [Authentication mechanisms](#authentication-mechanisms): - Mutual TLS (recommended) - default authentication mechanism used if no other method is selected. - Basic - the client sends a request with an Authorization header that includes a base64-encoded string in the format **username:password**. These will settings will be required inputs. - Bearer - the client sends a request with an Authorization header that includes a **bearer token**. This setting will be a required input. @@ -134,7 +134,7 @@ nsjoQAm6OwpTN5362vE9SYu1twz7KdzBlUkDhePEOgQkWfLHBJWwB+PvB1j/cUA3 ```bash server { listen 443 ssl default_server; - + # ... existing SSL configuration for server authentication ... ssl_verify_client on; ssl_client_certificate /path/to/Netdata_CA.pem; diff --git a/docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md b/docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md new file mode 100644 index 00000000..b9806c6f --- /dev/null +++ b/docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md @@ -0,0 +1,58 @@ +# Manage alert notification silencing rules + +From the Cloud interface, you can manage your space's alert notification silencing rules settings as well as allow users to define their personal ones. + +## Prerequisites + +To manage **space's alert notification silencing rule settings**, you will need the following: + +- A Netdata Cloud account +- Access to the space as an **administrator** or **manager** (**troubleshooters** can only view space rules) + + +To manage your **personal alert notification silencing rule settings**, you will need the following: + +- A Netdata Cloud account +- Access to the space with any roles except **billing** + +### Steps + +1. Click on the **Space settings** cog (located above your profile icon) +1. Click on the **Alert & Notification** tab on the left hand-side +1. Click on the **Notification Silencing Rules** tab +1. You will be presented with a table of the configured alert notification silencing rules for: + * the space (if aren't an **observer**) + * yourself + + You will be able to: + 1. **Add a new** alert notification silencing rule configuration. + - Choose if it applies to **All users** or **Myself** (All users is only available for **administrators** and **managers**) + - You need to provide a name for the configuration so you can easily refer to it + - Define criteria for Nodes: To which Rooms will this apply? What Nodes? Does it apply to host labels key-value pairs? + - Define criteria for Alerts: Which alert name is being targeted? What alert context? Will it apply to a specific alert role? + - Define when it will be applied: + - Immediately, from now till until it is turned off or until a specific duration (start and end date automatically set) + - Scheduled, you specify the start and end time for when the rule becomes active and then inactive (time is set according to your browser local timezone) + Note: You are only able to add a rule if your space is on a [paid plan](https://github.com/netdata/netdata/edit/master/docs/cloud/manage/plans.md). + 1. **Edit an existing** alert notification silencing rule configurations. You will be able to change: + - The name provided for it + - Who it applies to + - Selection criteria for Nodes and Alert + - When it will be applied + 1. **Enable/Disable** a given alert notification silencing rule configuration. + - Use the toggle to enable or disable + 1. **Delete an existing** alert notification silencing rule. + - Use the trash icon to delete your configuration + +## Silencing rules examples + +| Rule name | War Rooms | Nodes | Host Label | Alert name | Alert context | Alert role | Description | +| :-- | :-- | :-- | :-- | :-- | :-- | :-- | :--| +| Space silencing | All Rooms | * | * | * | * | * | This rule silences the entire space, targets all nodes and for all users. E.g. infrastructure wide maintenance window. | +| DB Servers Rooms | PostgreSQL Servers | * | * | * | * | * | This rules silences the nodes in the room named PostgreSQL Servers, for example it doesn't silence the `All Nodes` room. E.g. My team with membership to this room doesn't want to receive notifications for these nodes. | +| Node child1 | All Rooms | `child1` | * | * | * | * | This rule silences all alert state transitions for node `child1` on all rooms and for all users. E.g. node could be going under maintenance. | +| Production nodes | All Rooms | * | `environment:production` | * | * | * | This rule silences all alert state transitions for nodes with the host label key-value pair `environment:production`. E.g. Maintenance window on nodes with specific host labels. | +| Third party maintenance | All Rooms | * | * | `httpcheck_posthog_netdata_cloud.request_status` | * | * | This rule silences this specific alert since third party partner will be undergoing maintenance. | +| Intended stress usage on CPU | All Rooms | * | * | * | `system.cpu` | * | This rule silences specific alerts across all nodes and their CPU cores. | +| Silence role webmaster | All Rooms | * | * | * | * | `webmaster` | This rule silences all alerts configured with the role `webmaster`. | +| Silence alert on node | All Rooms | `child1` | * | `httpcheck_posthog_netdata_cloud.request_status` | * | * | This rule silences the specific alert on the `child1` node. | diff --git a/docs/cloud/alerts-notifications/manage-notification-methods.md b/docs/cloud/alerts-notifications/manage-notification-methods.md index 17c7f879..f61b6bf6 100644 --- a/docs/cloud/alerts-notifications/manage-notification-methods.md +++ b/docs/cloud/alerts-notifications/manage-notification-methods.md @@ -27,7 +27,8 @@ Notes: ### Steps 1. Click on the **Space settings** cog (located above your profile icon) -1. Click on the **Notification** tab +1. Click on the **Alerts & Notification** tab on the left hand-side +1. Click on the **Notification Methods** tab 1. You will be presented with a table of the configured notification methods for the space. You will be able to: 1. **Add a new** notification method configuration. - Choose the service from the list of the available ones, you'll may see a list of unavailable options if your plan doesn't allow some of them (you will see on the @@ -42,7 +43,7 @@ Notes: - Service specific inputs 1. **Enable/Disable** a given notification method configuration. - Use the toggle to enable or disable the notification method configuration - 1. **Delete an existing** notification method configuartion. Netdata provided ones can't be deleted, e.g. Email + 1. **Delete an existing** notification method configuration. Netdata provided ones can't be deleted, e.g. Email - Use the trash icon to delete your configuration ## Manage user notification settings diff --git a/docs/cloud/alerts-notifications/notifications.md b/docs/cloud/alerts-notifications/notifications.md index 94cd2dc3..ad115d43 100644 --- a/docs/cloud/alerts-notifications/notifications.md +++ b/docs/cloud/alerts-notifications/notifications.md @@ -31,7 +31,7 @@ or add new alert that you see in Netdata Cloud, and receive via centralized aler </Callout> -### Alert notifications +## Alert notifications Netdata Cloud can send centralized alert notifications to your team whenever a node enters a warning, critical, or unreachable state. By enabling notifications, you ensure no alert, on any node in your infrastructure, goes unnoticed by you or your team. @@ -51,9 +51,9 @@ All users in a Space can personalize their notifications settings, for Personal > ⚠️ Netdata Cloud supports different notification methods and their availability will depend on the plan you are at. > For more details check [Service classification](#service-classification) or [netdata.cloud/pricing](https://www.netdata.cloud/pricing). -#### Service level +### Service level -##### Personal +#### Personal The notifications methods classified as **Personal** are what we consider generic, meaning that these can't have specific rules for them set by the administrators. @@ -63,7 +63,7 @@ manage what specific configurations they want for the Space / Room(s) and the de One example of such a notification method is the E-mail. -##### System +#### System For **System** notification methods, the destination of the channel will be a target that usually isn't specific to a single user, e.g. slack channel. @@ -72,23 +72,49 @@ different targets depending on Rooms or Notification level settings. Some examples of such notification methods are: Webhook, PagerDuty, Slack. -#### Service classification +### Service classification -##### Community +#### Community Notification methods classified as Community can be used by everyone independent on the plan your space is at. These are: Email and discord -##### Pro +#### Pro Notification methods classified as Pro are only available for **Pro** and **Business** plans These are: webhook -##### Business +#### Business Notification methods classified as Business are only available for **Business** plans These are: PagerDuty, Slack, Opsgenie +## Silencing Alert notifications + +Netdata Cloud provides you a Silencing Rule engine which allows you to mute alert notifications. This muting action is specific to alert state transition notifications, it doesn't include node unreachable state transitions. + +The Silencing Rule engine is flexible and allows you to enter silence rules for the two main entities involved on alert notifications and can be set using different attributes. The main entities you can enter are **Nodes** and **Alerts** which can be used in combination or isolation to target specific needs - see some examples [here](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md#silencing-rules-examples). + +### Scope definition for Nodes +* **Space:** silencing the space, selecting `All Rooms`, silences all alert state transitions from any node claimed to the space. +* **War Room:** silencing a specific room will silence all alert state transitions from any node in that room. Please note if the node belongs to +another room which isn't silenced it can trigger alert notifications to the users with membership to that other room. +* **Node:** silencing a specific node can be done for the entire space, selecting `All Rooms`, or for specific war room(s). The main difference is +if the node should be silenced for the entire space or just for specific rooms (when specific rooms are selected only users with membership to that room won't receive notifications). + +### Scope definition for Alerts +* **Alert name:** silencing a specific alert name silences all alert state transitions for that specific alert. +* **Alert context:** silencing a specific alert context will silence all alert state transitions for alerts targeting that chart context, for more details check [alert configuration docs](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#alarm-line-on). +* **Alert role:** silencing a specific alert role will silence all the alert state transitions for alerts that are configured to be specific role recipients, for more details check [alert configuration docs](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#alarm-line-to). + +Beside the above two main entities there are another two important settings that you can define on a silencing rule: +* Who does the rule affect? **All user** in the space or **Myself** +* When does is to apply? **Immediately** or on a **Schedule** (when setting immediately you can set duration) + +For further help on setting alert notification silencing rules go to [Manage Alert Notification Silencing Rules](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md). + +> ⚠️ This feature is only available for [Netdata paid plans](https://github.com/netdata/netdata/edit/master/docs/cloud/manage/plans.md). + ## Flood protection If a node has too many state changes like firing too many alerts or going from reachable to unreachable, Netdata Cloud |