diff options
Diffstat (limited to '')
22 files changed, 19 insertions, 595 deletions
diff --git a/docs/category-overview-pages/accessing-netdata-dashboards.md b/docs/category-overview-pages/accessing-netdata-dashboards.md deleted file mode 100644 index af7b0df82..000000000 --- a/docs/category-overview-pages/accessing-netdata-dashboards.md +++ /dev/null @@ -1,38 +0,0 @@ -# Accessing Netdata Dashboards - -This section contains documentation on how you can access the Netdata dashboard, which are the same both for the Agent and Cloud. - -A user accessing the Netdata dashboard **from the Cloud** will always be presented with the latest Netdata dashboard version. - -A user accessing the Netdata dashboard **from the Agent** will, by default, be presented with the latest Netdata dashboard version (the same as Netdata Cloud) except in the following scenarios: -* Agent doesn't have Internet access, and is unable to get the latest Netdata dashboards, as a result it falls back to the Netdata dashboard version that -was shipped with the agent. -* Users have defined, e.g. through URL bookmark, that they want to see the previous version of the dashboard (accessible `http://NODE:19999/v1`, replacing `NODE` with the IP address or hostname of your Agent). - -## Main sections - -The Netdata dashboard consists of the following main sections: -* [Netdata charts](https://github.com/netdata/netdata/blob/master/docs/cloud/visualize/interact-new-charts.md) -* [Infrastructure Overview](https://github.com/netdata/netdata/blob/master/docs/visualize/overview-infrastructure.md) -* [Nodes view](https://github.com/netdata/netdata/blob/master/docs/cloud/visualize/nodes.md) -* [Custom dashboards](https://learn.netdata.cloud/docs/visualizations/custom-dashboards) -* [Alerts](https://github.com/netdata/netdata/blob/master/docs/monitor/view-active-alerts.md) -* [Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md) -* [Functions](https://github.com/netdata/netdata/blob/master/docs/cloud/netdata-functions.md) -* [Events feed](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/events-feed.md) - -> ⚠️ Some sections of the dashboard, when accessed through the agent, may require the user to be signed in to Netdata Cloud or having the Agent claimed to Netdata Cloud for their full functionality. Examples include saving visualization settings on charts or custom dashboards, claiming the node to Netdata Cloud, or executing functions on an Agent. - -## How to access the dashboards? - -### Netdata Cloud - -You can access the dashboard at https://app.netdata.cloud/ and [sign-in](https://github.com/netdata/netdata/blob/master/docs/cloud/manage/sign-in.md) with an account or [sign-up](https://github.com/netdata/netdata/blob/master/docs/cloud/manage/sign-in.md#dont-have-a-netdata-cloud-account-yet) if you don't have an account yet. - -### Netdata Agent - -Netdata starts a web server for its dashboard at port `19999`. Open up your web browser of choice and -navigate to `http://NODE:19999`, replacing `NODE` with the IP address or hostname of your Agent. If installed on localhost, you can access it through `http://localhost:19999`. - - -Documentation for previous Agent dashboard can still be found [here](https://github.com/netdata/netdata/blob/master/src/web/gui/README.md).
\ No newline at end of file diff --git a/docs/category-overview-pages/install-netdata-on-embedded-systems.md b/docs/category-overview-pages/install-netdata-on-embedded-systems.md deleted file mode 100644 index dfaa4482c..000000000 --- a/docs/category-overview-pages/install-netdata-on-embedded-systems.md +++ /dev/null @@ -1,3 +0,0 @@ -# Install Netdata on Embedded Systems Overview - -This section contains documentation for installation methods when it comes to Embedded Systems.
\ No newline at end of file diff --git a/docs/category-overview-pages/install-with-a-cicd-provisioning-system.md b/docs/category-overview-pages/install-with-a-cicd-provisioning-system.md deleted file mode 100644 index 30a5a706c..000000000 --- a/docs/category-overview-pages/install-with-a-cicd-provisioning-system.md +++ /dev/null @@ -1,3 +0,0 @@ -# Install with a CI/CD Provisioning System Overview - -This section contains documentation on all the installation methods through a CI/CD system.
\ No newline at end of file diff --git a/docs/category-overview-pages/installation-overview.md b/docs/category-overview-pages/installation-overview.md deleted file mode 100644 index 703ca26b9..000000000 --- a/docs/category-overview-pages/installation-overview.md +++ /dev/null @@ -1,10 +0,0 @@ -# Installation - -In this category you can find instructions on all the possible ways you can install Netdata on the -[supported platforms](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/versions-and-platforms.md). - -If this is your first time using Netdata, we recommend that you first start with the -[quick installation guide](https://github.com/netdata/netdata/edit/master/packaging/installer/README.md) and then -go into the more advanced options available to you. - - diff --git a/docs/category-overview-pages/integrations-overview.md b/docs/category-overview-pages/integrations-overview.md deleted file mode 100644 index afd4cf306..000000000 --- a/docs/category-overview-pages/integrations-overview.md +++ /dev/null @@ -1,31 +0,0 @@ -<!-- -title: "Integrations" -sidebar_label: "Integrations" -custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/category-overview-pages/integrations-overview.md" -description: "Available integrations in Netdata" -learn_status: "Published" -learn_rel_path: "Integrations" -sidebar_position: 60 ---> - -# Integrations - -Netdata's ability to monitor out of the box every potentially useful aspect of a node's operation is unparalleled. -But Netdata also provides out of the box, meaningful charts and alerts for hundreds of applications, with the ability -to be easily extended to monitor anything. See the full list of Netdata's capabilities and how you can extend them in the -[supported collectors list](https://github.com/netdata/netdata/blob/master/src/collectors/COLLECTORS.md). - -Our out of the box alerts were created by expert professionals and have been validated on the field, countless times. -Use them to trigger [alert notifications](https://github.com/netdata/netdata/blob/master/docs/monitor/enable-notifications.md) -either centrally, via the -[Cloud alert notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.md) -, or by configuring individual -[agent notifications](https://github.com/netdata/netdata/blob/master/src/health/notifications/README.md). - -We designed Netdata with interoperability in mind. The Agent collects thousands of metrics every second, and then what -you do with them is up to you. You can -[store metrics in the database engine](https://github.com/netdata/netdata/blob/master/src/database/README.md), -or send them to another time series database for long-term storage or further analysis using -Netdata's [exporting engine](https://github.com/netdata/netdata/edit/master/src/exporting/README.md). - - diff --git a/docs/category-overview-pages/logs.md b/docs/category-overview-pages/logs.md deleted file mode 100644 index fbaf85631..000000000 --- a/docs/category-overview-pages/logs.md +++ /dev/null @@ -1,3 +0,0 @@ -# Logs - -This section talks about ways Netdata collects and visualizes logs, while also providing useful guides on log centralization setups that can be used with Netdata. diff --git a/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md b/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md index 9a0e4b381..f41089bba 100644 --- a/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md +++ b/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md @@ -1,97 +1,7 @@ # Machine Learning and Anomaly Detection -Machine learning (ML) is a subfield of Artificial Intelligence (AI) that enables computers to learn and improve from experience without being explicitly programmed. +Netdata provides advanced Machine Learning features to help you identify and troubleshoot anomalies and unexpected behavior in your infrastructure before they become critical issues: -In observability, machine learning can be used to detect patterns and anomalies in large datasets, enabling users to identify potential issues before they become critical. - -Machine Learning for observability is usually misunderstood, and frequently leads to unrealistic expectations. Check for example the [presentation Google gave at SreCON19](https://www.usenix.org/conference/srecon19emea/presentation/underwood) explaining that all ideas that Google SREs and DevOps came up with, about the use of Machine Learning in observability were bad, and as Todd notes they should feel bad about it. - -At Netdata we are approaching machine learning in a completely different way. Instead of trying to make machine learning do something it cannot achieve, we tried to understand if and what useful insights it can provide and eventually we turned it to an assistant that can improve troubleshooting, reduce mean time to resolution and in many case prevent issues from escalating. - -## Design Principles - -The following are the high level design principles of Machine Learning in Netdata: - -1. **Unsupervised** - - In other words: whatever machine learning can do, it should do it by itself, without any help or assistance from users. - -2. **Real-time** - - We understand that Machine Learning will have some impact on resource utilization, especially in CPU utilization, but it shouldn't prevent Netdata from being real-time and high-fidelity. - -3. **Integrated** - - Everything achieved with machine learning should be tightly integrated to the infrastructure exploration and troubleshooting practices we are used to. - -4. **Assist, Advice, Consult** - - If we can't be sure that a decision made by Machine Learning is 100% accurate, we should use this to assist and consult users in their journey. - - In other words, we don't want to wake up someone at 3 AM, just because a machine learning model detected something. - -## Machine Learning per Time-Series - -Given the samples recently collected for a time-series, Machine Learning is used to detect if a sample just collected is an outlier or not. - -Since the query combinations are infinite, Netdata detects anomalies at the time-series level, and then combines the anomaly rates of all time-series involved in each query, to provide the anomaly rate for the query. - -When a collected sample is an outlier, we set the Anomaly Bit of the collected sample and we store it together with the sample value in the time-series database. - -## Multiple Machine Learning Models per Time-Series to Eliminate Noise - -Unsupervised machine learning has some noise, random false positives. - -To remove this noise, Netdata trains multiple machine learning models for each time-series, covering more than the last 2 days, in total. - -Netdata uses all of the available ML models to detect anomalies. So, all machine learning models of a time-series need to agree that a collected sample is an outlier, for it to be marked as an anomaly. - -This process removes 99% of the false positives, offering reliable unsupervised anomaly detection. - -## Node Level Anomaly - -When a metric becomes anomalous, in many cases a lot other metrics get anomalous too. - -For example, an anomaly on a web server may also introduce unusual network bandwidth, cpu usage, memory consumption, disk I/O, context switches, interrupts, etc. If the web server is serving an API that has an application server and a database server we may see anomalies being propagated to them too. - -To represent the spread of an anomaly in a node, Netdata computes a **Node Level Anomaly**. This is the percentage of the metrics of a node being concurrently anomalous, vs the total number of metrics of that node. - -## Node Anomaly Events - -Netdata produces a "node anomaly event" when a the percentage of concurrently anomalous time-series is high enough and persists over time. - -This anomaly event signals that there was sufficient evidence among all the time-series that some strange behavior might have been detected in a more global sense across the node. - -## What is the Anomaly Bit? - -Each sample collected, carries an Anomaly Bit. This bit (true/false) is set when the collected sample found to be an outlier, based on the machine learning models available for it so far. - -This bit is embedded into the custom floating point number the Netdata database uses, so it does not introduce any overheads in memory or disk footprint. - -The query engine of Netdata uses this bit to compute anomaly rates while it executes normal time-series queries. This eliminates to need for additional queries for anomaly rates, as all `/api/v2` time-series query include anomaly rate information. - -## What is the Anomaly Rate (AR)? - -The Anomaly Rate of a query, is a percentage, representing the number of samples in the query found anomalous, vs the total number of samples participating in the query. - -## How it works - a more technical presentation - -For each time-series Netdata trains every 3 hours, a `k-means clustering` model, using the last 6 hours of samples collected for it. - -Rather than using raw samples of each time-series, the model works on a preprocessed "feature vector" of recent smoothed and differenced values. - -This enables the model to detect a wider range of potentially anomalous patterns as opposed to just point anomalies like big spikes or drops. - -Some of the types of anomalies Netdata detects are: - -1. **Point Anomalies** or **Strange Points**: Single points that represent very big or very small values, not seen before (in some statistical sense). -2. **Contextual Anomalies** or **Strange Patterns**: Not strange points in their own, but unexpected sequences of points, given the history of the time-series. -3. **Collective Anomalies** or **Strange Multivariate Patterns**: Neither strange points nor strange patterns, but in global sense something looks off. -4. **Concept Drifts** or **Strange Trends**: A slow and steady drift to a new state. -5. **Change Point Detection** or **Strange Step**: A shift occurred and gradually a new normal is established. - -For a visual representation, check this infographic: - - - -A more detailed explanation can be found on [this (informal) presentation](https://docs.google.com/presentation/d/18zkCvU3nKP-Bw_nQZuXTEa4PIVM6wppH3VUnAauq-RU/edit#slide=id.p). +- K-means clustering [Machine Learning models](/src/ml/README.md) are trained to power the [Anomaly Advisor](/docs/dashboards-and-charts/anomaly-advisor-tab.md) on the dashboard, which allows you to identify Anomalies in your infrastructure. +- [Metric Correlations](/docs/metric-correlations.md) are possible through the dashboard using the [Two-sample Kolmogorov Smirnov](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test) statistical test and Volume heuristic measures. +- The [Netdata Assistant](/docs/netdata-assistant.md) is able to answer your prompts when it comes to troubleshooting Alerts and Anomalies. diff --git a/docs/category-overview-pages/maintenance-operations-on-netdata-agents.md b/docs/category-overview-pages/maintenance-operations-on-netdata-agents.md index 207a0bd32..1867d863f 100644 --- a/docs/category-overview-pages/maintenance-operations-on-netdata-agents.md +++ b/docs/category-overview-pages/maintenance-operations-on-netdata-agents.md @@ -1,3 +1,8 @@ # Maintenance operations on Netdata Agents Overview -This section provides information on various actions you can take when maintaining a Netdata Agent.
\ No newline at end of file +This section provides information on various actions you can take while maintaining a Netdata Agent. + +- [Starting and Stopping Netdata Agents](/docs/netdata-agent/start-stop-restart.md) +- [Update Netdata Agents](/packaging/installer/UPDATE.md) +- [Reinstall Netdata Agents](/packaging/installer/REINSTALL.md) +- [Uninstall Netdata Agents](/packaging/installer/UNINSTALL.md) diff --git a/docs/category-overview-pages/metrics-streaming-and-replication.md b/docs/category-overview-pages/metrics-streaming-and-replication.md deleted file mode 100644 index f473105fd..000000000 --- a/docs/category-overview-pages/metrics-streaming-and-replication.md +++ /dev/null @@ -1,175 +0,0 @@ -# Netdata Parents (Streaming and Replication) - -## What are they and why do we need them? - -A “Parent” is a Netdata Agent, like the ones we install on all our systems, but is configured as a central node that receives, stores and processes metrics data from other Netdata “Child” nodes in our infrastructure. - -Netdata Parents are flexible. You can have one big active-active cluster of Netdata Parents, or you can spread a lot of independent Parents across the infrastructure. - -This “distributed still centralized” setup provides a lot of benefits. Let’s see them: - -## Infrastructure-Level Dashboards: All Nodes in One Dashboard - -A Parent node receives and aggregates metrics data from all child nodes that push metrics to it, presenting all of them on a single, centralized dashboard. - -Metrics streaming between Netdata nodes is real-time and low-latency, so that the Parent can provide the same resolution and detail its children provide. - -Each chart on the Parent’s dashboard is automatically turned into a multi-node chart, allowing instant aggregation of the data across the entire dashboard. This is transparent and automatic for all kinds of charts, even application-specific ones. For example, when you have 2 PostgreSQL servers in your infrastructure, the parent will present one set of charts for PostgreSQL and these charts will include data from both servers. - -## Increased Data Retention: Store More, Learn More - -Netdata’s database (`dbengine`), supports multiple tiers of variable resolution for storing metrics’ samples. Tier 0 is the high-resolution one and usually stores per second data. Tier 1 is the middle resolution one, downsampling data to per minute. Tier 2 is the low-resolution one, downsampling data to per hour. With this setup, a default Netdata setup is usually able to maintain 2-3 days of high resolution and up to a year of low-resolution data, all in less than 1 GB of disk space. - -In many cases, however, organizations require a lot more retention than this. A Netdata Parent can be configured to have weeks or even months of high-resolution data and several years of low-resolution data for all its Child nodes, by allowing the Netdata database to grow to hundreds of GiBs or even several TiBs. - -## Monitoring Ephemeral Nodes: No Node Left Behind - -Production systems are often ephemeral by nature. In containerized and orchestrated environments, like Kubernetes, nodes may come and go due to scaling policies, maintenance tasks, or as part of regular operations. - -Netdata Parents come to the rescue in such scenarios. They can continuously receive metrics from ephemeral nodes during their lifecycle. As these nodes are removed or replaced, the Parent retains their performance history, essentially archiving the life of each node. - -The Netdata dashboards on the Parents automatically bring into the charts data from archived nodes when users pan the dashboard to the time-window these nodes were alive. This means that no data is lost and visibility is maintained across the entire lifespan of every node, regardless of its ephemeral nature. - -## Unified Alerts Management: Silence the Noise - -Each Netdata Agent is able to run health checks, trigger alerts and send notifications on its own. However, in a large-scale infrastructure with numerous nodes, each capable of generating alerts, managing these notifications can quickly become a challenge. Duplicate alerts and non-centralized management can lead to unnecessary noise, causing alert fatigue and possibly overlooking critical warnings. - -Netdata Parents provide a solution to this problem. By configuring a Parent node to handle all alerts and health checks, and disabling health monitoring on the Child nodes, you centralize your alerts management, meaning that all alerts are now generated from a single place, reducing noise and ensuring that each unique issue only triggers a single notification. - -In addition to making alert management more straightforward, this setup also allows for more refined control over your alert configurations. Instead of managing alert settings across multiple nodes, you can handle all configurations in one place, ensuring consistency and ease of management. - -## Offloading Production Systems: Prioritize Performance - -In a production environment, every bit of system resources is crucial. Minimizing the overhead due to monitoring and observability is vital to ensure optimal system performance. Although the Netdata Agent is designed to be lightweight and efficient, using a Netdata Parent can allow the Netdata Agents on your production systems to focus on the absolutely necessary for collecting metrics and pushing them to their Parent. - -On your production systems, by configuring the Netdata Agents to use the `alloc` database mode with 5-10 minutes of retention time and disabling health monitoring and Machine Learning (ML) processing, you significantly reduce the system resources consumed by the monitoring system. - -Netdata, with the `alloc` database mode, doesn't touch the disk at all (apart from logging - which can also be disabled). This approach eliminates any potential disk I/O impact from Netdata on your production applications, which could be particularly beneficial in I/O-sensitive environments. - -## Fault Tolerance and Redundancy: Ensure Continuous Monitoring - -Netdata Agents stream metrics to one Netdata Parent at a time. But more than one Parent can be configured on each child. The first available at any given time is used. - -Similarly, Netdata Parents can be configured to stream/proxy the data they receive to another Netdata Parent. And they can support multiple Parents too, one of which will be used at any given time. - -Configuration allows setting up a circular streaming setup. Parent A streams to Parent B and Parent B streams to Parent A. Child nodes are configured to stream to any of Parents A and B and they will automatically fall back and switch parents as necessary. - -With the replication feature (enabled by default), all nodes replicate missing data on their Parent, before streaming live metrics, filling up any gap the Parent may have. - -The same setup can work for 2 or even more parents, to form an active-active multi-node cluster. Child nodes can connect to any of the parent nodes available and the parent nodes will automatically replicate and stream metrics to each other. - -The setup is optimized even for wide-area connections between child nodes and parents, or for cases where the bandwidth between child nodes and parents has a cost associated with it. At any given time each child node sends its data only once. The parents then replicate and stream this data to each other. - -## Security and Isolation: Protect Your Production Systems - -Parent nodes can be set up in your organization's Demilitarized Zone (DMZ), acting as a protective barrier or application firewall, shielding your production Netdata agents from the outside world. - -With Netdata Parents configured, the Netdata Agents running on your production systems need only one connection to these parents. They don’t need to run data queries, they will never send alert notifications, or even connect to Netdata Cloud. - -Especially for Netdata Cloud, when the Parent node is connected to Netdata Cloud, it registers its Child nodes to it and can serve all functions required by the Cloud on behalf of the Child nodes. So, although only the parent is connected to Netdata Cloud, there is no difference in the user features you enjoy on Netdata Cloud in regard to your production systems. They will all be there. - -## FAQ about Netdata Parents - -### How much can a Parent node scale? - -For about 1 million real-time metrics, with a default configuration: - -- collected and streamed to it per second, -- stored in 3 database tiers (high, mid, low resolution), -- with ML training and anomaly detection running, -- health for alerts and notifications - -And about 2 TiB of storage for metrics, you will need about 5-8 CPU cores and 32GiB of RAM. On such a setup you can have: - -- 15 days of high resolution metrics -- 3 months of mid resolution metrics -- 1 year of low resolution metrics - -For such a setup, we recommend a 16 CPU cores system so that there is spare capacity for queries. More RAM and faster disks will give faster queries. - -So, depending on the number of metrics per node you have and the size of your Parents, you may be able to aggregate 200 to 500 nodes per Parent. - -### If I set up 2 active-active parents, will I be able to have more Child nodes stream to them? - -No. When you set up an active-active cluster, even if child nodes connect randomly to one or the other, all the parent nodes receive all the metrics of all the child nodes. So, all of them do all the work. - -There is a feature we currently work on, to allow Parent nodes to detect that they receive ML information with the streamed metric data (they receive it already but they ignore it), to prevent them from training their own ML models and running anomaly detection again for the child node. But this is not ready yet. - -### How much retention do the child nodes need? - -Child nodes need to have only the retention required in order to connect to another Parent if one fails or stops for maintenance. - -- If you have an active-active cluster of parents, 5 to 10 minutes in `alloc` mode is enough. -- If you have only 1 parent, it would be better to run the child nodes with `dbengine` so that they will have enough retention to backfill the parent nodes if it stops for a few hours for maintenance. - -### Does streaming between child nodes and parents support encryption? - -Yes. You can configure your parent nodes to enable TLS and configure the child nodes to connect with TLS to it. The streaming connection is also compressed with LZ4 and this works even on top of TLS. - -### Can I have an HTTP proxy between parent and child nodes? - -No. The streaming protocol works on the same port as the internal web server of Netdata Agents, but the protocol is not HTTP-friendly and cannot be understood by HTTP proxy servers. - -### Should I load balance the parents with a TCP load balancer? - -Although this can be done and for streaming between child and parent nodes it could work, we recommend not doing it. It can lead to several kinds of problems. - -It is better to configure all the parent nodes directly in the child nodes `stream.conf`. The child nodes will do everything in their power to find a parent node to connect and they will never give up. - -### When I have an active-active cluster of parents, will I receive alert notifications from both of them? - -If both are configured to run health checks and trigger alerts, yes. - -We recommend using Netdata Cloud to avoid receiving duplicate alert notifications. Netdata Cloud deduplicates alert notifications so that you will receive them only once. On top of that, you can control silencing and routing directly from the Netdata Cloud UI. - -### When I have only Parents connected to Netdata Cloud, will I be able to use the Functions feature on my child nodes? - -Yes. - -Functions is a feature of data collection plugins to expose functions that can be run from the dashboard to view more detailed information about a data collection. For example, apps.plugin exposes the processes function that returns a list of all the processes running, together with information about their CPU utilization, memory consumption, disk I/O operations, bandwidth, and a lot more. - -When a parent receives a Function request, it forwards it to the plugin that exposes it. If the plugin is available over a streaming connection, the parent will forward the request to the socket it receives metrics from. This process will be repeated even if many parents are chained in order to reach the child. - -### If I have a set of 2 active-active parents and get one out for maintenance for a few hours, will it have missing data when it returns back online? - -There are 2 reasons you may have gaps in your data after you bring it back online: - -1. Replication does not replicate metrics that are not actively collected. So, when the parent comes back, if there are samples that this parent does not have, for metrics that are not currently being collected, these samples will not be propagated to that parent. [We are working to fix this issue](https://github.com/netdata/netdata/issues/15198). -2. If the parent has been offline for a long time and the child nodes run in db mode `alloc`, you need to plan how you will bring this parent back online. Child nodes in this mode do not have enough retention to backfill the parent and if they connect to it before the other parent, you will end up with missing information on that parent. - -The simplest way to solve this is to block at the firewall all connections to port 19999 from child nodes, but allow connections from the other parent nodes. Once replication finishes for all nodes, you can unblock the connections from child nodes to it. - -### I got a parent out of maintenance but it replicates (backfills) missing data slowly. Can I speed it up? - -Yes, there is a setting on `netdata.conf` under section `[db]` called `replication threads`. The default value is 1. - -Usually, each thread is able to replicate about 2-5 million samples per second. We suggest setting this to 5 threads for all parents. Generally do not use too many threads because you are risking congesting the disks and/or the CPU cores available. Keep in mind that the sending parent needs this setting. - -There is no need to increase this number on child nodes. Each node has one replication sender, so when hundreds of nodes are replicating to a parent, there are already a lot of senders pushing metrics to it. - -### I have multiple active-active parents. Which one is used by Netdata Cloud for queries? - -When you have multiple parents available, the one that is further away from the child node is used by Netdata Cloud, unless it does not have the data required. - -This works like this: The child has `hops = 0`. Each parent receiving metrics for this child increases the `hops` by 1. So the first parent will have `hops = 1`, the second parent will have `hops = 2` and so on. - -Netdata Cloud knows the retention of each parent. So, when it needs data from this child, it first checks the available retention each parent has for it and then it uses the parent with the higher `hops`. If no parent is available and the child node is directly connected to Netdata Cloud, it uses the child. - -### Is there a way to balance child nodes to the parent nodes of an active-active cluster? - -If you have 2 parent nodes A and B, you can configure them on half the child nodes as A, B, and the other half as B, A. The child nodes will connect to the first available (left to right). If both A and B are online, half of the child nodes will connect to A and the other half to B. - -Keep in mind, however, that if you restart a parent, all the child nodes that were connected to it will automatically reconnect to the other parent. Once this happens, the child nodes will stay connected to it. - -### Is there a way to get notified when a child gets disconnected? - -There are 2 kinds of production nodes: -1. **Permanent nodes** - These are nodes that should be available permanently and if they disconnect an alert should be triggered to notify you. - By default, all nodes are considered permanent (not ephemeral). -2. **Ephemeral nodes** - These are nodes that are ephemeral by nature and they may shutdown at any point in time without any impact on the services you run. - -To set the ephemeral flag on a node, edit its `netdata.conf` and in the `[health]` section set is `ephemeral = yes`. This setting is propagated to parent nodes and Netdata Cloud. - -When using Netdata Cloud (via a parent or directly) and a permanent node gets disconnected, Netdata Cloud sends node disconnection notifications. diff --git a/docs/category-overview-pages/misc-overview.md b/docs/category-overview-pages/misc-overview.md deleted file mode 100644 index dbb11e9bc..000000000 --- a/docs/category-overview-pages/misc-overview.md +++ /dev/null @@ -1,3 +0,0 @@ -# Miscellaneous material - -This section contains material that will be moved to new locations as we see fit. We keep it here to make it accessible while we make these changes.
\ No newline at end of file diff --git a/docs/category-overview-pages/monitor-your-infrastructure.md b/docs/category-overview-pages/monitor-your-infrastructure.md deleted file mode 100644 index 3582e88a6..000000000 --- a/docs/category-overview-pages/monitor-your-infrastructure.md +++ /dev/null @@ -1,3 +0,0 @@ -# Monitor your Infrastructure Overview - -This section contains documentation on how you can use Netdata Cloud and it's features to monitor your entire infrastructure.
\ No newline at end of file diff --git a/docs/category-overview-pages/netdata-apis.md b/docs/category-overview-pages/netdata-apis.md deleted file mode 100644 index 82d1c1752..000000000 --- a/docs/category-overview-pages/netdata-apis.md +++ /dev/null @@ -1,5 +0,0 @@ -# Netdata APIs Overview - -This section contains information about Netdata's APIs. - -You can access the Netdata Agent's API through swagger UI [here](/api).
\ No newline at end of file diff --git a/docs/category-overview-pages/netdata-architecture.md b/docs/category-overview-pages/netdata-architecture.md deleted file mode 100644 index 70f126597..000000000 --- a/docs/category-overview-pages/netdata-architecture.md +++ /dev/null @@ -1,3 +0,0 @@ -# Netdata Architecture Overview - -This section's purpose is to explain the architecture of Netdata, the role of the Agent and the Cloud, and more.
\ No newline at end of file diff --git a/docs/category-overview-pages/netdata-dashboards-and-visualizations.md b/docs/category-overview-pages/netdata-dashboards-and-visualizations.md deleted file mode 100644 index cc9304365..000000000 --- a/docs/category-overview-pages/netdata-dashboards-and-visualizations.md +++ /dev/null @@ -1,3 +0,0 @@ -# Netdata Dashboards and Visualizations Overview - -This section provides documentation about all the visualization operations, features and insights that Netdata provides.
\ No newline at end of file diff --git a/docs/category-overview-pages/reverse-proxies.md b/docs/category-overview-pages/reverse-proxies.md deleted file mode 100644 index 1b4d935a3..000000000 --- a/docs/category-overview-pages/reverse-proxies.md +++ /dev/null @@ -1,34 +0,0 @@ -# Running Netdata behind a reverse proxy - -If you need to access a Netdata agent's user interface or API in a production environment we recommend you put Netdata behind -another web server and secure access to the dashboard via SSL, user authentication and firewall rules. - -A dedicated web server also provides more robustness and capabilities than the Agent's [internal web server](https://github.com/netdata/netdata/blob/master/src/web/README.md). - -We have documented running behind -[nginx](https://github.com/netdata/netdata/blob/master/docs/Running-behind-nginx.md), -[Apache](https://github.com/netdata/netdata/blob/master/docs/Running-behind-apache.md), -[HAProxy](https://github.com/netdata/netdata/blob/master/docs/Running-behind-haproxy.md), -[Lighttpd](https://github.com/netdata/netdata/blob/master/docs/Running-behind-lighttpd.md), -[Caddy](https://github.com/netdata/netdata/blob/master/docs/Running-behind-caddy.md), -and [H2O](https://github.com/netdata/netdata/blob/master/docs/Running-behind-h2o.md). -If you prefer a different web server, we suggest you follow the documentation for nginx and tell us how you did it - by adding your own "Running behind webserverX" document. - -When you run Netdata behind a reverse proxy, we recommend you firewall protect all your Netdata servers, so that only the web server IP will be allowed to directly access Netdata. To do this, run this on each of your servers (or use your firewall manager): - -```sh -PROXY_IP="1.2.3.4" -iptables -t filter -I INPUT -p tcp --dport 19999 \! -s ${PROXY_IP} -m conntrack --ctstate NEW -j DROP -``` - -The above will prevent anyone except your web server to access a Netdata dashboard running on the host. - -You can also use `netdata.conf`: - -``` -[web] - allow connections from = localhost 1.2.3.4 -``` - -Of course, you can add more IPs. diff --git a/docs/category-overview-pages/secure-nodes.md b/docs/category-overview-pages/secure-nodes.md deleted file mode 100644 index 99e273920..000000000 --- a/docs/category-overview-pages/secure-nodes.md +++ /dev/null @@ -1,177 +0,0 @@ -# Secure your nodes
-
-Netdata is a monitoring system. It should be protected, the same way you protect all your admin apps. We assume Netdata
-will be installed privately, for your eyes only.
-
-Upon installation, the Netdata Agent serves the **local dashboard** at port `19999`. If the node is accessible to the
-internet at large, anyone can access the dashboard and your node's metrics at `http://NODE:19999`. We made this decision
-so that the local dashboard was immediately accessible to users, and so that we don't dictate how professionals set up
-and secure their infrastructures.
-
-Viewers will be able to get some information about the system Netdata is running. This information is everything the dashboard
-provides. The dashboard includes a list of the services each system runs (the legends of the charts under the `Systemd Services`
-section), the applications running (the legends of the charts under the `Applications` section), the disks of the system and
-their names, the user accounts of the system that are running processes (the `Users` and `User Groups` section of the dashboard),
-the network interfaces and their names (not the IPs) and detailed information about the performance of the system and its applications.
-
-This information is not sensitive (meaning that it is not your business data), but **it is important for possible attackers**.
-It will give them clues on what to check, what to try and in the case of DDoS against your applications, they will know if they
-are doing it right or not.
-
-Also, viewers could use Netdata itself to stress your servers. Although the Netdata daemon runs unprivileged, with the minimum
-process priority (scheduling priority `idle` - lower than nice 19) and adjusts its OutOfMemory (OOM) score to 1000 (so that it
-will be first to be killed by the kernel if the system starves for memory), some pressure can be applied on your systems if
-someone attempts a DDoS against Netdata.
-
-Instead of dictating how to secure your infrastructure, we give you many options to establish security best practices
-that align with your goals and your organization's standards.
-
-- [Disable the local dashboard](#disable-the-local-dashboard): **Simplest and recommended method** for those who have
- added nodes to Netdata Cloud and view dashboards and metrics there.
-
-- [Expose Netdata only in a private LAN](#expose-netdata-only-in-a-private-lan). Simplest and recommended method for those who do not use Netdata Cloud.
-
-- [Fine-grained access control](#fine-grained-access-control): Allow local dashboard access from
- only certain IP addresses, such as a trusted static IP or connections from behind a management LAN. Full support for Netdata Cloud.
-
-- [Use a reverse proxy (authenticating web server in proxy mode)](#use-an-authenticating-web-server-in-proxy-mode): Password-protect
- a local dashboard and enable TLS to secure it. Full support for Netdata Cloud.
-
-- [Use Netdata parents as Web Application Firewalls](#use-netdata-parents-as-web-application-firewalls)
-
-- [Other methods](#other-methods) list some less common methods of protecting Netdata.
-
-## Disable the local dashboard
-
-This is the _recommended method for those who have connected their nodes to Netdata Cloud_ and prefer viewing real-time
-metrics using the War Room Overview, Nodes tab, and Cloud dashboards.
-
-You can disable the local dashboard (and API) but retain the encrypted Agent-Cloud link
-([ACLK](https://github.com/netdata/netdata/blob/master/src/aclk/README.md)) that
-allows you to stream metrics on demand from your nodes via the Netdata Cloud interface. This change mitigates all
-concerns about revealing metrics and system design to the internet at large, while keeping all the functionality you
-need to view metrics and troubleshoot issues with Netdata Cloud.
-
-Open `netdata.conf` with `./edit-config netdata.conf`. Scroll down to the `[web]` section, and find the `mode =
-static-threaded` setting, and change it to `none`.
-
-```conf
-[web]
- mode = none
-```
-
-Save and close the editor, then [restart your Agent](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md#maintaining-a-netdata-agent-installation)
-using `sudo systemctl
-restart netdata`. If you try to visit the local dashboard to `http://NODE:19999` again, the connection will fail because
-that node no longer serves its local dashboard.
-
-> See the [configuration basics doc](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md) for details on how to find
-`netdata.conf` and use
-> `edit-config`.
-
-## Expose Netdata only in a private LAN
-
-If your organisation has a private administration and management LAN, you can bind Netdata on this network interface on all your servers.
-This is done in `Netdata.conf` with these settings:
-
-```
-[web]
- bind to = 10.1.1.1:19999 localhost:19999
-```
-
-You can bind Netdata to multiple IPs and ports. If you use hostnames, Netdata will resolve them and use all the IPs
-(in the above example `localhost` usually resolves to both `127.0.0.1` and `::1`).
-
-**This is the best and the suggested way to protect Netdata**. Your systems **should** have a private administration and management
-LAN, so that all management tasks are performed without any possibility of them being exposed on the internet.
-
-For cloud based installations, if your cloud provider does not provide such a private LAN (or if you use multiple providers),
-you can create a virtual management and administration LAN with tools like `tincd` or `gvpe`. These tools create a mesh VPN
-allowing all servers to communicate securely and privately. Your administration stations join this mesh VPN to get access to
-management and administration tasks on all your cloud servers.
-
-For `gvpe` we have developed a [simple provisioning tool](https://github.com/netdata/netdata-demo-site/tree/master/gvpe) you
-may find handy (it includes statically compiled `gvpe` binaries for Linux and FreeBSD, and also a script to compile `gvpe`
-on your macOS system). We use this to create a management and administration LAN for all Netdata demo sites (spread all over
-the internet using multiple hosting providers).
-
-## Fine-grained access control
-
-If you want to keep using the local dashboard, but don't want it exposed to the internet, you can restrict access with
-[access lists](https://github.com/netdata/netdata/blob/master/src/web/server/README.md#access-lists). This method also fully
-retains the ability to stream metrics
-on-demand through Netdata Cloud.
-
-The `allow connections from` setting helps you allow only certain IP addresses or FQDN/hostnames, such as a trusted
-static IP, only `localhost`, or connections from behind a management LAN.
-
-By default, this setting is `localhost *`. This setting allows connections from `localhost` in addition to _all_
-connections, using the `*` wildcard. You can change this setting using Netdata's [simple
-patterns](https://github.com/netdata/netdata/blob/master/src/libnetdata/simple_pattern/README.md).
-
-```conf
-[web]
- # Allow only localhost connections
- allow connections from = localhost
-
- # Allow only from management LAN running on `10.X.X.X`
- allow connections from = 10.*
-
- # Allow connections only from a specific FQDN/hostname
- allow connections from = example*
-```
-
-The `allow connections from` setting is global and restricts access to the dashboard, badges, streaming, API, and
-`netdata.conf`, but you can also set each of those access lists more granularly if you choose:
-
-```conf
-[web]
- allow connections from = localhost *
- allow dashboard from = localhost *
- allow badges from = *
- allow streaming from = *
- allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.*
- allow management from = localhost
-```
-
-See the [web server](https://github.com/netdata/netdata/blob/master/src/web/server/README.md#access-lists) docs for additional details
-about access lists. You can take
-access lists one step further by [enabling SSL](https://github.com/netdata/netdata/blob/master/src/web/server/README.md#enabling-tls-support) to encrypt data from local
-dashboard in transit. The connection to Netdata Cloud is always secured with TLS.
-
-## Use an authenticating web server in proxy mode
-
-Use one web server to provide authentication in front of **all your Netdata servers**. So, you will be accessing all your Netdata with
-URLs like `http://{HOST}/netdata/{NETDATA_HOSTNAME}/` and authentication will be shared among all of them (you will sign-in once for all your servers).
-Instructions are provided on how to set the proxy configuration to have Netdata run behind
-[nginx](https://github.com/netdata/netdata/blob/master/docs/Running-behind-nginx.md),
-[HAproxy](https://github.com/netdata/netdata/blob/master/docs/Running-behind-haproxy.md),
-[Apache](https://github.com/netdata/netdata/blob/master/docs/Running-behind-apache.md),
-[lighthttpd](https://github.com/netdata/netdata/blob/master/docs/Running-behind-lighttpd.md),
-[caddy](https://github.com/netdata/netdata/blob/master/docs/Running-behind-caddy.md), and
-[H2O](https://github.com/netdata/netdata/blob/master/docs/Running-behind-h2o.md).
-
-## Use Netdata parents as Web Application Firewalls
-
-The Netdata Agents you install on your production systems do not need direct access to the Internet. Even when you use
-Netdata Cloud, you can appoint one or more Netdata Parents to act as border gateways or application firewalls, isolating
-your production systems from the rest of the world. Netdata
-Parents receive metric data from Netdata Agents or other Netdata Parents on one side, and serve most queries using their own
-copy of the data to satisfy dashboard requests on the other side.
-
-For more information see [Streaming and replication](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/enable-streaming.md).
-
-## Other methods
-
-Of course, there are many more methods you could use to protect Netdata:
-
-- Bind Netdata to localhost and use `ssh -L 19998:127.0.0.1:19999 remote.netdata.ip` to forward connections of local port 19998 to remote port 19999.
-This way you can ssh to a Netdata server and then use `http://127.0.0.1:19998/` on your computer to access the remote Netdata dashboard.
-
-- If you are always under a static IP, you can use the script given above to allow direct access to your Netdata servers without authentication,
-from all your static IPs.
-
-- Install all your Netdata in **headless data collector** mode, forwarding all metrics in real-time to a parent
- Netdata server, which will be protected with authentication using an nginx server running locally at the parent
- Netdata server. This requires more resources (you will need a bigger parent Netdata server), but does not require
- any firewall changes, since all the child Netdata servers will not be listening for incoming connections.
diff --git a/docs/category-overview-pages/troubleshooting-overview.md b/docs/category-overview-pages/troubleshooting-overview.md deleted file mode 100644 index 60406edd6..000000000 --- a/docs/category-overview-pages/troubleshooting-overview.md +++ /dev/null @@ -1,5 +0,0 @@ -# Troubleshooting and machine learning - -In this section you can learn about Netdata's advanced tools that can assist you in troubleshooting issues with -your infrastructure, to facilitate the identification of a root cause. - diff --git a/docs/category-overview-pages/visualizations-overview.md b/docs/category-overview-pages/visualizations-overview.md deleted file mode 100644 index d07af062c..000000000 --- a/docs/category-overview-pages/visualizations-overview.md +++ /dev/null @@ -1,4 +0,0 @@ -# Visualizations, charts and dashboards - -In this section you can learn about the various ways Netdata visualizes the collected metrics at an infrastructure level with Netdata Cloud -and at a single node level, with the Netdata Agent Dashboard. diff --git a/docs/category-overview-pages/working-with-logs.md b/docs/category-overview-pages/working-with-logs.md new file mode 100644 index 000000000..e1f027529 --- /dev/null +++ b/docs/category-overview-pages/working-with-logs.md @@ -0,0 +1,9 @@ +# Working with Logs + +This section talks about ways Netdata collects and visualizes logs. + +The [systemd journal plugin](/src/collectors/systemd-journal.plugin/) is the core Netdata component for reading systemd journal logs. + +For structured logs, Netdata provides tools like [log2journal](/src/collectors/log2journal/README.md) and [systemd-cat-native](/src/libnetdata/log/systemd-cat-native.md) to convert them into compatible systemd journal entries. + +You can also find useful guides on how to set up log centralization points in the [Observability Cetralization Points](/docs/observability-centralization-points/README.md) section of our docs. diff --git a/docs/category-overview-pages/developer-and-contributor-corner.md b/docs/developer-and-contributor-corner/README.md index d4d86382a..d4d86382a 100644 --- a/docs/category-overview-pages/developer-and-contributor-corner.md +++ b/docs/developer-and-contributor-corner/README.md diff --git a/docs/category-overview-pages/build-the-netdata-agent-yourself.md b/docs/developer-and-contributor-corner/build-the-netdata-agent-yourself.md index 99166ad95..99166ad95 100644 --- a/docs/category-overview-pages/build-the-netdata-agent-yourself.md +++ b/docs/developer-and-contributor-corner/build-the-netdata-agent-yourself.md diff --git a/docs/category-overview-pages/optimizing-metrics-database.md b/docs/netdata-agent/configuration/optimizing-metrics-database/README.md index fdbd3b690..fdbd3b690 100644 --- a/docs/category-overview-pages/optimizing-metrics-database.md +++ b/docs/netdata-agent/configuration/optimizing-metrics-database/README.md |