diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-07-24 09:53:08 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-07-24 09:53:08 +0000 |
| commit | 6a1900e8bd84c282a500ae4032645ae55c614b7b (patch) | |
| tree | d4d31289c39fc00da064a825df13a0b98ce95b10 /docs/deployment-guides | |
| parent | Adding upstream version 1.45.3+dfsg. (diff) | |
| download | netdata-6a1900e8bd84c282a500ae4032645ae55c614b7b.tar.xz netdata-6a1900e8bd84c282a500ae4032645ae55c614b7b.zip | |
Adding upstream version 1.46.3.upstream/1.46.3
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'docs/deployment-guides')
| -rw-r--r-- | docs/deployment-guides/README.md | 4 | ||||
| -rw-r--r-- | docs/deployment-guides/deployment-strategies.md | 100 | ||||
| -rw-r--r-- | docs/deployment-guides/deployment-with-centralization-points.md | 25 | ||||
| -rw-r--r-- | docs/deployment-guides/standalone-deployment.md | 42 |
4 files changed, 71 insertions, 100 deletions
diff --git a/docs/deployment-guides/README.md b/docs/deployment-guides/README.md index 18f578857..1b6571b99 100644 --- a/docs/deployment-guides/README.md +++ b/docs/deployment-guides/README.md @@ -1,6 +1,6 @@ # Deployment Guides -Netdata can be used to monitor all kinds of infrastructure, from stand-alone tiny IoT devices to complex hybrid setups combining on-premise and cloud infrastructure, mixing bare-metal servers, virtual machines and containers. +Netdata can be used to monitor all kinds of infrastructure, from tiny stand-alone IoT devices to complex hybrid setups combining on-premise and cloud infrastructure, mixing bare-metal servers, virtual machines and containers. There are 3 components to structure your Netdata ecosystem: @@ -12,7 +12,7 @@ There are 3 components to structure your Netdata ecosystem: 2. **Netdata Parents** - To create [observability centralization points](https://github.com/netdata/netdata/blob/master/docs/observability-centralization-points/README.md) within your infrastructure, to offload Netdata Agents functions from your production systems, to provide high-availability of your data, increased data retention and isolation of your nodes. + To create [observability centralization points](/docs/observability-centralization-points/README.md) within your infrastructure, to offload Netdata Agents functions from your production systems, to provide high-availability of your data, increased data retention and isolation of your nodes. Netdata Parents are implemented using the Netdata Agent software. Any Netdata Agent can be an Agent for a node and a Parent for other Agents, at the same time. diff --git a/docs/deployment-guides/deployment-strategies.md b/docs/deployment-guides/deployment-strategies.md index 3be2edbcf..abdb36cdf 100644 --- a/docs/deployment-guides/deployment-strategies.md +++ b/docs/deployment-guides/deployment-strategies.md @@ -1,64 +1,42 @@ -# Deployment strategies - +# Deployment Examples ## Deployment Options Overview -This section provides a quick overview of a few common deployment options. The next sections go into configuration examples and further reading. - -### Stand-alone Deployment - -To help our users have a complete experience of Netdata when they install it for the first time, a Netdata Agent with default configuration -is a complete monitoring solution out of the box, having all these features enabled and available. - -The Agent will act as a _stand-alone_ Agent by default, and this is great to start out with for small setups and home labs. By [connecting each Agent to Cloud](https://github.com/netdata/netdata/blob/master/src/claim/README.md), you can see an overview of all your nodes, with aggregated charts and centralized alerting, without setting up a Parent. - - - -### Parent – Child Deployment - -An Agent connected to a Parent is called a _Child_. It will _stream_ metrics to its Parent. The Parent can then take care of storing metrics on behalf of that node (with longer retention), handle metrics queries for showing dashboards, and provide alerting. - -When using Cloud, it is recommended that just the Parent is connected to Cloud. Child Agents can then be configured to have short retention, in RAM instead of on Disk, and have alerting and other features disabled. Because they don't need to connect to Cloud themselves, those children can then be further secured by not allowing outbound traffic. - - +This section provides a quick overview for a few common deployment options for Netdata. -This setup allows for leaner Child nodes and is good for setups with more than a handful of nodes. Metrics data remains accessible if the Child node is temporarily unavailable or decommissioned, although there is no failover in case the Parent becomes unavailable. +You can read about [Standalone Deployment](/docs/deployment-guides/standalone-deployment.md) and [Deployment with Centralization Points](/docs/deployment-guides/deployment-with-centralization-points.md) in the documentation inside this section. +The sections below go into configuration examples about these deployment concepts. -### Active–Active Parent Deployment +## Deployment Configuration Details -For high availability, Parents can be configured to stream data for their children between them, and keep the data sets in sync. Child Agents are configured with the addresses of both Parent Agents, but will only stream to one of them at a time. When that Parent becomes unavailable, it reconnects to another. When the first Parent becomes available again, that Parent will catch up by receiving the backlog from the second. +### Stand-alone -With both Parent Agents connected to Cloud, Cloud will route queries to either Parent transparently, depending on their availability. Alerts trigger on either Parent will stream to Cloud, and Cloud will deduplicate and debounce state changes to prevent spurious notifications. +The stand-alone setup is configured out of the box with reasonable defaults, but please consult our [configuration documentation](/docs/netdata-agent/configuration/README.md) for details, including the overview of [common configuration changes](/docs/netdata-agent/configuration/common-configuration-changes.md). - +### Parent – Child +For setups involving Parent and Child Agents, they need to be configured for [streaming](docs/observability-centralization-points/metrics-centralization-points/configuration.md), through the configuration file `stream.conf`. -## Configuration Details +This will instruct the Child to stream data to the Parent and the Parent to accept streaming connections for one or more Child Agents. To secure this connection, both need a shared API key (to replace the string `API_KEY` in the examples below). Additionally, the Child can be configured with one or more addresses of Parent Agents (`PARENT_IP_ADDRESS`). -### Stand-alone Deployment - -The stand-alone setup is configured out of the box with reasonable defaults, but please consult our [configuration documentation](https://github.com/netdata/netdata/blob/master/docs/cloud/cheatsheet.md) for details, including the overview of [common configuration changes](https://github.com/netdata/netdata/blob/master/docs/configure/common-changes.md). - -### Parent – Child Deployment - -For setups involving Child and Parent Agents, the Agents need to be configured for [_streaming_](https://github.com/netdata/netdata/blob/master/src/streaming/README.md), through the configuration file `stream.conf`. This will instruct the Child to stream data to the Parent and the Parent to accept streaming connections for one or more Child Agents. To secure this connection, both need set up a shared API key (to replace the string `API_KEY` in the examples below). Additionally, the Child is configured with one or more addresses of Parent Agents (`PARENT_IP_ADDRESS`). - -An API key is a key created with `uuidgen` and is used for authentication and/or customization in the Parent side. I.e. a Child will stream using the API key, and a Parent is configured to accept connections from Child, but can also apply different options for children by using multiple different API keys. The easiest setup uses just one API key for all Child Agents. +An API key is a key created with `uuidgen` and is used for authentication and/or customization on the Parent side. For example, a Child can stream using the API key, and a Parent can be configured to accept connections from the Child, but it can also apply different options for Children by using multiple different API keys. The easiest setup uses just one API key for all Child Agents. #### Child config -As mentioned above, the recommendation is to not claim the Child to Cloud directly during your setup, avoiding establishing an [ACLK](https://github.com/netdata/netdata/blob/master/src/aclk/README.md) connection. +As mentioned above, we do not recommend to claim the Child to Cloud directly during your setup. -To reduce the footprint of the Netdata Agent on your production system, some capabilities can be switched OFF on the Child and kept ON on the Parent. In this example, Machine Learning and Alerting are disabled in the Child, so that the Parent can take the load. We also use RAM instead of disk to store metrics with limited retention, covering temporary network issues. +This is done in order to reduce the footprint of the Netdata Agent on your production system, as some capabilities can be switched OFF for the Child and kept ON for the Parent. + +In this example, Machine Learning and Alerting are disabled for the Child, so that the Parent can take the load. We also use RAM instead of disk to store metrics with limited retention, covering temporary network issues. ##### netdata.conf -On the child node, edit `netdata.conf` by using the edit-config script: `/etc/netdata/edit-config netdata.conf` set the following parameters: +On the child node, edit `netdata.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters: ```yaml [db] - # https://learn.netdata.cloud/docs/agent/database + # https://github.com/netdata/netdata/blob/master/src/database/README.md # none = no retention, ram = some retention in ram mode = ram # The retention in seconds. @@ -85,9 +63,7 @@ On the child node, edit `netdata.conf` by using the edit-config script: `/etc/ne ##### stream.conf -To edit `stream.conf`, again use the edit-config script: `/etc/netdata/edit-config stream.conf`. - -Set the following parameters: +To edit `stream.conf`, use again the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters: ```yaml [stream] @@ -101,7 +77,7 @@ Set the following parameters: #### Parent config -For the Parent, besides setting up streaming, the example will also provide an example configuration of multiple [tiers](https://github.com/netdata/netdata/blob/master/src/database/engine/README.md#tiering) of metrics [storage](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md), for 10 children, with about 2k metrics each. +For the Parent, besides setting up streaming, this example also provides configuration for multiple [tiers of metrics storage](/docs/netdata-agent/configuration/optimizing-metrics-database/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics), for 10 Children, with about 2k metrics each. This allows for: - 1s granularity at tier 0 for 1 week - 1m granularity at tier 1 for 1 month @@ -114,7 +90,7 @@ Requiring: ##### netdata.conf -On the Parent, edit `netdata.conf` with `/etc/netdata/edit-config netdata.conf` and set the following parameters: +On the Parent, edit `netdata.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters: ```yaml [db] @@ -149,7 +125,7 @@ On the Parent, edit `netdata.conf` with `/etc/netdata/edit-config netdata.conf` ##### stream.conf -On the Parent node, edit `stream.conf` with `/etc/netdata/edit-config stream.conf`, and then set the following parameters: +On the Parent node, edit `stream.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters: ```yaml [API_KEY] @@ -157,13 +133,13 @@ On the Parent node, edit `stream.conf` with `/etc/netdata/edit-config stream.con enabled = yes ``` -### Active–Active Parent Deployment +### Active–Active Parents -In order to setup active–active streaming between Parent 1 and Parent 2, Parent 1 needs to be instructed to stream data to Parent 2 and Parent 2 to stream data to Parent 1. The Child Agents need to be configured with the addresses of both Parent Agents. The Agent will only connect to one Parent at a time, falling back to the next if the previous failed. These examples use the same API key between Parent Agents as for connections from Child Agents. +In order to setup active–active streaming between Parent 1 and Parent 2, Parent 1 needs to be instructed to stream data to Parent 2 and Parent 2 to stream data to Parent 1. The Child Agents need to be configured with the addresses of both Parent Agents. An Agent will only connect to one Parent at a time, falling back to the next upon failure. These examples use the same API key between Parent Agents and for connections for Child Agents. -On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netdata/edit-config stream.conf`: +On both Netdata Parent and all Child Agents, edit `stream.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script: -##### stream.conf on Parent 1 +#### stream.conf on Parent 1 ```yaml [stream] @@ -178,7 +154,7 @@ On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netda enabled = yes ``` -##### stream.conf on Parent 2 +#### stream.conf on Parent 2 ```yaml [stream] @@ -192,7 +168,7 @@ On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netda enabled = yes ``` -##### stream.conf on Child Agents +#### stream.conf on Child Agents ```yaml [stream] @@ -208,32 +184,24 @@ On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netda We strongly recommend the following configuration changes for production deployments: -1. Understand Netdata's [security and privacy design](https://github.com/netdata/netdata/blob/master/docs/security-and-privacy-design/README.md) and - [secure your nodes](https://github.com/netdata/netdata/blob/master/docs/category-overview-pages/secure-nodes.md) +1. Understand Netdata's [security and privacy design](/docs/security-and-privacy-design/README.md) and [secure your nodes](/docs/netdata-agent/securing-netdata-agents.md) To safeguard your infrastructure and comply with your organization's security policies. -2. Set up [streaming and replication](https://github.com/netdata/netdata/blob/master/src/streaming/README.md) to: - - - Offload Netdata Agents running on production systems and free system resources for the production applications running on them. - - Isolate production systems from the rest of the world and improve security. - - Increase data retention. - - Make your data highly available. - -3. [Optimize the Netdata Agents system utilization and performance](https://github.com/netdata/netdata/blob/master/docs/guides/configure/performance.md) +2. [Optimize the Netdata Agents system utilization and performance](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md) To save valuable system resources, especially when running on weak IoT devices. We also suggest that you: -1. [Use Netdata Cloud to access the dashboards](https://github.com/netdata/netdata/blob/master/docs/quickstart/infrastructure.md) +1. [Use Netdata Cloud to access the dashboards](/docs/netdata-cloud/README.md) - For increased security, user management and access to our latest tools for advanced dashboarding and troubleshooting. + For increased security, user management and access to our latest features, tools and troubleshooting solutions. -2. [Change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md) +2. [Change how long Netdata stores metrics](/docs/netdata-agent/configuration/optimizing-metrics-database/change-metrics-storage.md) - To control Netdata's memory use, when you have a lot of ephemeral metrics. + To control Netdata's memory use, when you have a lot of ephemeral metrics. -3. [Use host labels](https://github.com/netdata/netdata/blob/master/docs/guides/using-host-labels.md) +3. [Use host labels](/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts.md) To organize systems, metrics, and alerts. diff --git a/docs/deployment-guides/deployment-with-centralization-points.md b/docs/deployment-guides/deployment-with-centralization-points.md index b3e2b40dc..87fd4a61a 100644 --- a/docs/deployment-guides/deployment-with-centralization-points.md +++ b/docs/deployment-guides/deployment-with-centralization-points.md @@ -14,7 +14,7 @@ When metrics and logs are centralized, the Children are never queried for metric | Unified infrastructure dashboards for logs | All logs are accessible via the same dashboard at Netdata Cloud, although they are unified per Netdata Parent | | Centrally configured alerts | Yes, at Netdata Parents | | Centrally dispatched alert notifications | Yes, at Netdata Cloud | -| Data are exclusively on-prem | Yes, Netdata Cloud queries Netdata Agents to satisfy dashboard queries. | +| Data are exclusively on-prem | Yes, Netdata Cloud queries Netdata Agents to satisfy dashboard queries. | A configuration with 2 observability centralization points, looks like this: @@ -24,7 +24,7 @@ flowchart LR dashboard for all nodes"]] NC(["<b>Netdata Cloud</b> - decides which agents + decides which Agents need to be queried"]) SA1["Netdata at AWS A1"] @@ -93,16 +93,24 @@ flowchart LR SB1 & SB2 & SBN ---|stream| PB ``` -### Configuration steps for deploying Netdata with Observability Centralization Points +## Active–Active Parent Deployment + +For high availability, Parents can be configured to stream data for their Children between them, and keep their data sets in sync. Children are configured with the addresses of both Parents, but will only stream to one of them at a time. When one Parent becomes unavailable, the Child reconnects to the other. When the first Parent becomes available again, that Parent will catch up by receiving the backlog from the second. + +With both Parent Agents connected to Netdata Cloud, it will route queries to either of them transparently, depending on their availability. Alerts trigger on either Parent will stream to Cloud, and Cloud will deduplicate and debounce state changes to prevent spurious notifications. + +## Configuration steps for deploying Netdata with Observability Centralization Points For Metrics: -- Install Netdata agents on all systems and the Netdata Parents. +- Install Netdata Agents on all systems and the Netdata Parents. - Configure `stream.conf` at the Netdata Parents to enable streaming access with an API key. - Configure `stream.conf` at the Netdata Children to enable streaming to the configured Netdata Parents. +Check the [related section in our documentation](/docs/observability-centralization-points/metrics-centralization-points/README.md) for more info + For Logs: - Install `systemd-journal-remote` on all systems and the Netdata Parents. @@ -111,11 +119,4 @@ For Logs: - Configure `systemd-journal-upload` at the Netdata Children to enable transmission of their logs to the Netdata Parents. -Optionally: - -- Disable ML, health checks and dashboard access at Netdata Children to save resources and avoid duplicate notifications. - -When using Netdata Cloud: - -- Optionally: disable dashboard access on all Netdata agents (including Netdata Parents). -- Optionally: disable alert notifications on all Netdata agents (including Netdata Parents). +Check the [related section in our documentation](/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/README.md) for more info diff --git a/docs/deployment-guides/standalone-deployment.md b/docs/deployment-guides/standalone-deployment.md index 5baef805a..3138141f7 100644 --- a/docs/deployment-guides/standalone-deployment.md +++ b/docs/deployment-guides/standalone-deployment.md @@ -1,22 +1,22 @@ # Standalone Deployment -To help our users have a complete experience of Netdata when they install it for the first time, a Netdata Agent with default configuration is a complete monitoring solution out of the box, having all its features enabled and available. +To help our users have a complete experience of Netdata when they install it for the first time, the Netdata Agent with default configuration is a complete monitoring solution out of the box, with features enabled and available. -So, each Netdata agent acts as a standalone monitoring system by default. +So, each Netdata Agent acts as a standalone monitoring system by default. -## Standalone agents, without Netdata Cloud +## Standalone Agents, without Netdata Cloud | Feature | How it works | |:---------------------------------------------:|:----------------------------------------------------:| -| Unified infrastructure dashboards for metrics | No, each Netdata agent provides its own dashboard | -| Unified infrastructure dashboards for logs | No, each Netdata agent exposes its own logs | +| Unified infrastructure dashboards for metrics | No, each Netdata Agent provides its own dashboard | +| Unified infrastructure dashboards for logs | No, each Netdata Agent exposes its own logs | | Centrally configured alerts | No, each Netdata has its own alerts configuration | -| Centrally dispatched alert notifications | No, each Netdata agent sends notifications by itself | +| Centrally dispatched alert notifications | No, each Netdata Agent sends notifications by itself | | Data are exclusively on-prem | Yes | -When using Standalone Netdata agents, each of them offers an API and a dashboard, at its own unique URL, that looks like `http://agent-ip:19999`. +When using Standalone Netdata Agents, each of them offers an API and a dashboard, at its own unique URL, that looks like `http://agent-ip:19999`. -So, each of the Netdata agents has to be accessed individually and independently of the others: +So, each of the Netdata Agents has to be accessed individually and independently of the others: ```mermaid flowchart LR @@ -37,7 +37,7 @@ flowchart LR WEB -->|URL N| SN ``` -The same is true for alert notifications. Each of the Netdata agents runs its own alerts and sends notifications by itself, according to its configuration: +The same is true for alert notifications. Each of the Netdata Agents runs its own alerts and sends notifications by itself, according to its configuration: ```mermaid flowchart LR @@ -61,23 +61,23 @@ flowchart LR S1 & S2 & SN ==> OTHER ``` -### Configuration steps for standalone Netdata agents without Netdata Cloud +### Configuration steps for standalone Netdata Agents without Netdata Cloud No special configuration needed. -- Install Netdata agents on all your systems, then access each of them via its own unique URL, that looks like `http://agent-ip:19999/`. +- Install Netdata Agents on all your systems, then access each of them via its own unique URL, that looks like `http://agent-ip:19999/`. -## Standalone agents, with Netdata Cloud +## Standalone Agents, with Netdata Cloud | Feature | How it works | |:---------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | Unified infrastructure dashboards for metrics | Yes, via Netdata Cloud, all charts aggregate metrics from all servers. | | Unified infrastructure dashboards for logs | All logs are accessible via the same dashboard at Netdata Cloud, although they are not unified (ie. logs from different servers are not multiplexed into a single view) | -| Centrally configured alerts | No, each Netdata has its own alerts configuration | +| Centrally configured alerts | No, each Netdata has its own alerts configuration | | Centrally dispatched alert notifications | Yes, via Netdata Cloud | | Data are exclusively on-prem | Yes, Netdata Cloud queries Netdata Agents to satisfy dashboard queries. | -By [connecting all Netdata agents to Netdata Cloud](https://github.com/netdata/netdata/blob/master/src/claim/README.md), you can have a unified infrastructure view of all your nodes, with aggregated charts, without configuring [observability centralization points](https://github.com/netdata/netdata/blob/master/docs/observability-centralization-points/README.md). +By [connecting all Netdata Agents to Netdata Cloud](/src/claim/README.md), you can have a unified infrastructure view of all your nodes, with aggregated charts, without configuring [observability centralization points](/docs/observability-centralization-points/README.md). ```mermaid flowchart LR @@ -85,7 +85,7 @@ flowchart LR dashboard for all nodes"]] NC(["<b>Netdata Cloud</b> - decides which agents + decides which Agents need to be queried"]) S1["Standalone Netdata @@ -100,7 +100,7 @@ flowchart LR NC -->|queries| S1 & S2 & SN ``` -Similarly for alerts, Netdata Cloud receives all alert transitions from all agents, decides which notifications should be sent and how, applies silencing rules, maintenance windows and based on each Netdata Cloud space and user settings, dispatches notifications: +Similarly for alerts, Netdata Cloud receives all alert transitions from all Agents, decides which notifications should be sent and how, applies silencing rules, maintenance windows and based on each Netdata Cloud space and user settings, dispatches notifications: ```mermaid flowchart LR @@ -128,12 +128,14 @@ flowchart LR S1 & S2 & SN -->|alert transition| NC ``` -> Note that alerts are still triggered by Netdata agents. Netdata Cloud takes care of the notifications only. +> **Note** +> +> Alerts are still triggered by Netdata Agents. Netdata Cloud only takes care of the notifications. -### Configuration steps for standalone Netdata agents with Netdata Cloud +### Configuration steps for standalone Netdata Agents with Netdata Cloud -- Install Netdata agents using the commands given by Netdata Cloud, so that they will be automatically added to your Netdata Cloud space. Otherwise, install Netdata agents and then claim them via the command line or their dashboard. +- Install Netdata Agents using the commands given by Netdata Cloud, so that they will be automatically connected to your Netdata Cloud space. Otherwise, install Netdata Agents and then claim them via the command line or their dashboard. - Optionally: disable their direct dashboard access to secure them. -- Optionally: disable their alert notifications to avoid receiving email notifications directly from them (email notifications are automatically enabled when a working MTA is found on the systems Netdata agents are installed). +- Optionally: disable their alert notifications to avoid receiving email notifications directly from them (email notifications are automatically enabled when a working MTA is found on the systems Netdata Agents are installed). |