diff options
Diffstat (limited to 'docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md')
| -rw-r--r-- | docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md | 98 |
1 files changed, 4 insertions, 94 deletions
diff --git a/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md b/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md index 9a0e4b381..f41089bba 100644 --- a/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md +++ b/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md @@ -1,97 +1,7 @@ # Machine Learning and Anomaly Detection -Machine learning (ML) is a subfield of Artificial Intelligence (AI) that enables computers to learn and improve from experience without being explicitly programmed. +Netdata provides advanced Machine Learning features to help you identify and troubleshoot anomalies and unexpected behavior in your infrastructure before they become critical issues: -In observability, machine learning can be used to detect patterns and anomalies in large datasets, enabling users to identify potential issues before they become critical. - -Machine Learning for observability is usually misunderstood, and frequently leads to unrealistic expectations. Check for example the [presentation Google gave at SreCON19](https://www.usenix.org/conference/srecon19emea/presentation/underwood) explaining that all ideas that Google SREs and DevOps came up with, about the use of Machine Learning in observability were bad, and as Todd notes they should feel bad about it. - -At Netdata we are approaching machine learning in a completely different way. Instead of trying to make machine learning do something it cannot achieve, we tried to understand if and what useful insights it can provide and eventually we turned it to an assistant that can improve troubleshooting, reduce mean time to resolution and in many case prevent issues from escalating. - -## Design Principles - -The following are the high level design principles of Machine Learning in Netdata: - -1. **Unsupervised** - - In other words: whatever machine learning can do, it should do it by itself, without any help or assistance from users. - -2. **Real-time** - - We understand that Machine Learning will have some impact on resource utilization, especially in CPU utilization, but it shouldn't prevent Netdata from being real-time and high-fidelity. - -3. **Integrated** - - Everything achieved with machine learning should be tightly integrated to the infrastructure exploration and troubleshooting practices we are used to. - -4. **Assist, Advice, Consult** - - If we can't be sure that a decision made by Machine Learning is 100% accurate, we should use this to assist and consult users in their journey. - - In other words, we don't want to wake up someone at 3 AM, just because a machine learning model detected something. - -## Machine Learning per Time-Series - -Given the samples recently collected for a time-series, Machine Learning is used to detect if a sample just collected is an outlier or not. - -Since the query combinations are infinite, Netdata detects anomalies at the time-series level, and then combines the anomaly rates of all time-series involved in each query, to provide the anomaly rate for the query. - -When a collected sample is an outlier, we set the Anomaly Bit of the collected sample and we store it together with the sample value in the time-series database. - -## Multiple Machine Learning Models per Time-Series to Eliminate Noise - -Unsupervised machine learning has some noise, random false positives. - -To remove this noise, Netdata trains multiple machine learning models for each time-series, covering more than the last 2 days, in total. - -Netdata uses all of the available ML models to detect anomalies. So, all machine learning models of a time-series need to agree that a collected sample is an outlier, for it to be marked as an anomaly. - -This process removes 99% of the false positives, offering reliable unsupervised anomaly detection. - -## Node Level Anomaly - -When a metric becomes anomalous, in many cases a lot other metrics get anomalous too. - -For example, an anomaly on a web server may also introduce unusual network bandwidth, cpu usage, memory consumption, disk I/O, context switches, interrupts, etc. If the web server is serving an API that has an application server and a database server we may see anomalies being propagated to them too. - -To represent the spread of an anomaly in a node, Netdata computes a **Node Level Anomaly**. This is the percentage of the metrics of a node being concurrently anomalous, vs the total number of metrics of that node. - -## Node Anomaly Events - -Netdata produces a "node anomaly event" when a the percentage of concurrently anomalous time-series is high enough and persists over time. - -This anomaly event signals that there was sufficient evidence among all the time-series that some strange behavior might have been detected in a more global sense across the node. - -## What is the Anomaly Bit? - -Each sample collected, carries an Anomaly Bit. This bit (true/false) is set when the collected sample found to be an outlier, based on the machine learning models available for it so far. - -This bit is embedded into the custom floating point number the Netdata database uses, so it does not introduce any overheads in memory or disk footprint. - -The query engine of Netdata uses this bit to compute anomaly rates while it executes normal time-series queries. This eliminates to need for additional queries for anomaly rates, as all `/api/v2` time-series query include anomaly rate information. - -## What is the Anomaly Rate (AR)? - -The Anomaly Rate of a query, is a percentage, representing the number of samples in the query found anomalous, vs the total number of samples participating in the query. - -## How it works - a more technical presentation - -For each time-series Netdata trains every 3 hours, a `k-means clustering` model, using the last 6 hours of samples collected for it. - -Rather than using raw samples of each time-series, the model works on a preprocessed "feature vector" of recent smoothed and differenced values. - -This enables the model to detect a wider range of potentially anomalous patterns as opposed to just point anomalies like big spikes or drops. - -Some of the types of anomalies Netdata detects are: - -1. **Point Anomalies** or **Strange Points**: Single points that represent very big or very small values, not seen before (in some statistical sense). -2. **Contextual Anomalies** or **Strange Patterns**: Not strange points in their own, but unexpected sequences of points, given the history of the time-series. -3. **Collective Anomalies** or **Strange Multivariate Patterns**: Neither strange points nor strange patterns, but in global sense something looks off. -4. **Concept Drifts** or **Strange Trends**: A slow and steady drift to a new state. -5. **Change Point Detection** or **Strange Step**: A shift occurred and gradually a new normal is established. - -For a visual representation, check this infographic: - - - -A more detailed explanation can be found on [this (informal) presentation](https://docs.google.com/presentation/d/18zkCvU3nKP-Bw_nQZuXTEa4PIVM6wppH3VUnAauq-RU/edit#slide=id.p). +- K-means clustering [Machine Learning models](/src/ml/README.md) are trained to power the [Anomaly Advisor](/docs/dashboards-and-charts/anomaly-advisor-tab.md) on the dashboard, which allows you to identify Anomalies in your infrastructure. +- [Metric Correlations](/docs/metric-correlations.md) are possible through the dashboard using the [Two-sample Kolmogorov Smirnov](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test) statistical test and Volume heuristic measures. +- The [Netdata Assistant](/docs/netdata-assistant.md) is able to answer your prompts when it comes to troubleshooting Alerts and Anomalies. |