From 8020f71afd34d7696d7933659df2d763ab05542f Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sat, 4 May 2024 16:31:17 +0200 Subject: Adding upstream version 1.37.1. Signed-off-by: Daniel Baumann --- .../.page-level/_collector-page-template.mdx | 68 +++ .../.page-level/_concept-page-template.md | 30 ++ docs/.templates/.page-level/_task-page-template.md | 41 ++ .../.page-level/_tutorial-page-template.mdx | 54 +++ docs/Add-more-charts-to-netdata.md | 13 + docs/Demo-Sites.md | 24 + docs/Donations-netdata-has-received.md | 29 ++ docs/README.md | 17 + docs/Running-behind-apache.md | 371 ++++++++++++++++ docs/Running-behind-caddy.md | 34 ++ docs/Running-behind-h2o.md | 183 ++++++++ docs/Running-behind-haproxy.md | 293 ++++++++++++ docs/Running-behind-lighttpd.md | 71 +++ docs/Running-behind-nginx.md | 270 +++++++++++ docs/a-github-star-is-important.md | 24 + docs/agent-cloud.md | 77 ++++ docs/anonymous-statistics.md | 105 +++++ docs/collect/application-metrics.md | 81 ++++ docs/collect/container-metrics.md | 99 +++++ docs/collect/enable-configure.md | 68 +++ docs/collect/how-collectors-work.md | 78 ++++ docs/collect/system-metrics.md | 60 +++ docs/configure/common-changes.md | 214 +++++++++ docs/configure/nodes.md | 170 +++++++ docs/configure/secure-nodes.md | 123 ++++++ docs/configure/start-stop-restart.md | 98 ++++ docs/contributing/contributing-documentation.md | 109 +++++ docs/contributing/style-guide.md | 492 +++++++++++++++++++++ docs/dashboard/customize.mdx | 93 ++++ docs/dashboard/dimensions-contexts-families.mdx | 96 ++++ docs/dashboard/how-dashboard-works.mdx | 112 +++++ docs/dashboard/import-export-print-snapshot.mdx | 83 ++++ docs/dashboard/interact-charts.mdx | 197 +++++++++ docs/dashboard/reference-web-server.mdx | 278 ++++++++++++ .../visualization-date-and-time-controls.mdx | 121 +++++ docs/export/enable-connector.md | 95 ++++ docs/export/external-databases.md | 89 ++++ docs/get-started.mdx | 107 +++++ docs/guides/collect-apache-nginx-web-logs.md | 127 ++++++ docs/guides/collect-unbound-metrics.md | 140 ++++++ docs/guides/configure/performance.md | 277 ++++++++++++ docs/guides/deploy/ansible.md | 176 ++++++++ .../export/export-netdata-metrics-graphite.md | 183 ++++++++ docs/guides/longer-metrics-storage.md | 158 +++++++ docs/guides/monitor-cockroachdb.md | 136 ++++++ docs/guides/monitor-hadoop-cluster.md | 203 +++++++++ docs/guides/monitor/anomaly-detection-python.md | 189 ++++++++ docs/guides/monitor/anomaly-detection.md | 75 ++++ docs/guides/monitor/dimension-templates.md | 176 ++++++++ docs/guides/monitor/kubernetes-k8s-netdata.md | 254 +++++++++++ docs/guides/monitor/lamp-stack.md | 246 +++++++++++ docs/guides/monitor/pi-hole-raspberry-pi.md | 162 +++++++ docs/guides/monitor/process.md | 301 +++++++++++++ .../monitor/raspberry-pi-anomaly-detection.md | 125 ++++++ docs/guides/monitor/statsd.md | 298 +++++++++++++ docs/guides/monitor/stop-notifications-alarms.md | 92 ++++ docs/guides/monitor/visualize-monitor-anomalies.md | 142 ++++++ docs/guides/python-collector.md | 486 ++++++++++++++++++++ docs/guides/step-by-step/step-00.md | 120 +++++ docs/guides/step-by-step/step-01.md | 156 +++++++ docs/guides/step-by-step/step-02.md | 208 +++++++++ docs/guides/step-by-step/step-03.md | 91 ++++ docs/guides/step-by-step/step-04.md | 144 ++++++ docs/guides/step-by-step/step-05.md | 350 +++++++++++++++ docs/guides/step-by-step/step-06.md | 122 +++++ docs/guides/step-by-step/step-07.md | 114 +++++ docs/guides/step-by-step/step-08.md | 395 +++++++++++++++++ docs/guides/step-by-step/step-09.md | 162 +++++++ docs/guides/step-by-step/step-10.md | 232 ++++++++++ docs/guides/step-by-step/step-99.md | 51 +++ .../monitor-debug-applications-ebpf.md | 270 +++++++++++ .../troubleshooting-agent-with-cloud-connection.md | 117 +++++ docs/guides/using-host-labels.md | 212 +++++++++ .../enable-streaming.mdx | 151 +++++++ .../how-streaming-works.mdx | 92 ++++ .../reference-streaming.mdx | 486 ++++++++++++++++++++ docs/monitor/configure-alarms.md | 148 +++++++ docs/monitor/enable-notifications.md | 147 ++++++ docs/monitor/view-active-alarms.md | 75 ++++ docs/netdata-for-IoT.md | 74 ++++ docs/netdata-security.md | 229 ++++++++++ docs/overview/netdata-monitoring-stack.md | 62 +++ docs/overview/what-is-netdata.md | 74 ++++ docs/overview/why-netdata.md | 63 +++ docs/quickstart/infrastructure.md | 185 ++++++++ docs/quickstart/single-node.md | 92 ++++ docs/store/change-metrics-storage.md | 98 ++++ docs/store/distributed-data-architecture.md | 88 ++++ docs/visualize/create-dashboards.md | 64 +++ docs/visualize/interact-dashboards-charts.md | 131 ++++++ docs/visualize/overview-infrastructure.md | 111 +++++ docs/why-netdata/1s-granularity.md | 59 +++ docs/why-netdata/README.md | 35 ++ docs/why-netdata/immediate-results.md | 46 ++ docs/why-netdata/meaningful-presentation.md | 68 +++ docs/why-netdata/unlimited-metrics.md | 49 ++ 96 files changed, 13884 insertions(+) create mode 100644 docs/.templates/.page-level/_collector-page-template.mdx create mode 100644 docs/.templates/.page-level/_concept-page-template.md create mode 100644 docs/.templates/.page-level/_task-page-template.md create mode 100644 docs/.templates/.page-level/_tutorial-page-template.mdx create mode 100644 docs/Add-more-charts-to-netdata.md create mode 100644 docs/Demo-Sites.md create mode 100644 docs/Donations-netdata-has-received.md create mode 100644 docs/README.md create mode 100644 docs/Running-behind-apache.md create mode 100644 docs/Running-behind-caddy.md create mode 100644 docs/Running-behind-h2o.md create mode 100644 docs/Running-behind-haproxy.md create mode 100644 docs/Running-behind-lighttpd.md create mode 100644 docs/Running-behind-nginx.md create mode 100644 docs/a-github-star-is-important.md create mode 100644 docs/agent-cloud.md create mode 100644 docs/anonymous-statistics.md create mode 100644 docs/collect/application-metrics.md create mode 100644 docs/collect/container-metrics.md create mode 100644 docs/collect/enable-configure.md create mode 100644 docs/collect/how-collectors-work.md create mode 100644 docs/collect/system-metrics.md create mode 100644 docs/configure/common-changes.md create mode 100644 docs/configure/nodes.md create mode 100644 docs/configure/secure-nodes.md create mode 100644 docs/configure/start-stop-restart.md create mode 100644 docs/contributing/contributing-documentation.md create mode 100644 docs/contributing/style-guide.md create mode 100644 docs/dashboard/customize.mdx create mode 100644 docs/dashboard/dimensions-contexts-families.mdx create mode 100644 docs/dashboard/how-dashboard-works.mdx create mode 100644 docs/dashboard/import-export-print-snapshot.mdx create mode 100644 docs/dashboard/interact-charts.mdx create mode 100644 docs/dashboard/reference-web-server.mdx create mode 100644 docs/dashboard/visualization-date-and-time-controls.mdx create mode 100644 docs/export/enable-connector.md create mode 100644 docs/export/external-databases.md create mode 100644 docs/get-started.mdx create mode 100644 docs/guides/collect-apache-nginx-web-logs.md create mode 100644 docs/guides/collect-unbound-metrics.md create mode 100644 docs/guides/configure/performance.md create mode 100644 docs/guides/deploy/ansible.md create mode 100644 docs/guides/export/export-netdata-metrics-graphite.md create mode 100644 docs/guides/longer-metrics-storage.md create mode 100644 docs/guides/monitor-cockroachdb.md create mode 100644 docs/guides/monitor-hadoop-cluster.md create mode 100644 docs/guides/monitor/anomaly-detection-python.md create mode 100644 docs/guides/monitor/anomaly-detection.md create mode 100644 docs/guides/monitor/dimension-templates.md create mode 100644 docs/guides/monitor/kubernetes-k8s-netdata.md create mode 100644 docs/guides/monitor/lamp-stack.md create mode 100644 docs/guides/monitor/pi-hole-raspberry-pi.md create mode 100644 docs/guides/monitor/process.md create mode 100644 docs/guides/monitor/raspberry-pi-anomaly-detection.md create mode 100644 docs/guides/monitor/statsd.md create mode 100644 docs/guides/monitor/stop-notifications-alarms.md create mode 100644 docs/guides/monitor/visualize-monitor-anomalies.md create mode 100644 docs/guides/python-collector.md create mode 100644 docs/guides/step-by-step/step-00.md create mode 100644 docs/guides/step-by-step/step-01.md create mode 100644 docs/guides/step-by-step/step-02.md create mode 100644 docs/guides/step-by-step/step-03.md create mode 100644 docs/guides/step-by-step/step-04.md create mode 100644 docs/guides/step-by-step/step-05.md create mode 100644 docs/guides/step-by-step/step-06.md create mode 100644 docs/guides/step-by-step/step-07.md create mode 100644 docs/guides/step-by-step/step-08.md create mode 100644 docs/guides/step-by-step/step-09.md create mode 100644 docs/guides/step-by-step/step-10.md create mode 100644 docs/guides/step-by-step/step-99.md create mode 100644 docs/guides/troubleshoot/monitor-debug-applications-ebpf.md create mode 100644 docs/guides/troubleshoot/troubleshooting-agent-with-cloud-connection.md create mode 100644 docs/guides/using-host-labels.md create mode 100644 docs/metrics-storage-management/enable-streaming.mdx create mode 100644 docs/metrics-storage-management/how-streaming-works.mdx create mode 100644 docs/metrics-storage-management/reference-streaming.mdx create mode 100644 docs/monitor/configure-alarms.md create mode 100644 docs/monitor/enable-notifications.md create mode 100644 docs/monitor/view-active-alarms.md create mode 100644 docs/netdata-for-IoT.md create mode 100644 docs/netdata-security.md create mode 100644 docs/overview/netdata-monitoring-stack.md create mode 100644 docs/overview/what-is-netdata.md create mode 100644 docs/overview/why-netdata.md create mode 100644 docs/quickstart/infrastructure.md create mode 100644 docs/quickstart/single-node.md create mode 100644 docs/store/change-metrics-storage.md create mode 100644 docs/store/distributed-data-architecture.md create mode 100644 docs/visualize/create-dashboards.md create mode 100644 docs/visualize/interact-dashboards-charts.md create mode 100644 docs/visualize/overview-infrastructure.md create mode 100644 docs/why-netdata/1s-granularity.md create mode 100644 docs/why-netdata/README.md create mode 100644 docs/why-netdata/immediate-results.md create mode 100644 docs/why-netdata/meaningful-presentation.md create mode 100644 docs/why-netdata/unlimited-metrics.md (limited to 'docs') diff --git a/docs/.templates/.page-level/_collector-page-template.mdx b/docs/.templates/.page-level/_collector-page-template.mdx new file mode 100644 index 0000000..fcbe7ce --- /dev/null +++ b/docs/.templates/.page-level/_collector-page-template.mdx @@ -0,0 +1,68 @@ + + +# Title + +import { + EnableCollector, + CollectorDebug, +} from '@site/src/components/Collectors/_collector-components.jsx'; + +Short description of what the collector does on a high level. +Why should I use this collector? + +## Configuring $COLLECTOR_NAME + +#### Prerequisites + +List all needed prerequisites: + +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + + + +### Example + +TODO: Check if we can automatically fetch the [JOBS] section of netdata.conf + +## Other configuration information + +Explain other configuration options, as needed. + +#### Prerequisites + + + +Optional. List all needed prerequisites: + +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## Debugging $COLLECTOR_NAME (optional) + + + +## Metrics and Alerts produced by this collector + +| Chart | Metrics | Alert | +| ---------- | ----------- | ------------------------ | +| Chart Name | Metric name | [Alert 1](Link to alert) | +| Chart Name | Metric name | [Alert 2](Link to alert) | +| Chart Name | Metric name | [Alert 3](Link to alert) | diff --git a/docs/.templates/.page-level/_concept-page-template.md b/docs/.templates/.page-level/_concept-page-template.md new file mode 100644 index 0000000..685dd2f --- /dev/null +++ b/docs/.templates/.page-level/_concept-page-template.md @@ -0,0 +1,30 @@ + + +# Title + +Why should the reader care: “What’s in it for me?” + +## Subheading + +Ideally, try to explain one core idea per section. Questions that you could keep in mind while writing: + +- How does it work? +- What are the outcomes? +- What are the positive and negative effects of it? +- Are there alternatives that provide a similar result? + +## Subheading + +Add more subheadings and anything else that needs to be explained... +Remember, if you start to describe about another concept, stop yourself. +Each concept should be about one concept only. + + +### Related links + +- Visit the [related thing documentation](www.related-thing.com) to learn more about related thing. diff --git a/docs/.templates/.page-level/_task-page-template.md b/docs/.templates/.page-level/_task-page-template.md new file mode 100644 index 0000000..0f49201 --- /dev/null +++ b/docs/.templates/.page-level/_task-page-template.md @@ -0,0 +1,41 @@ + +# Title + +Short description of why or when the procedure makes sense. + +## Subheading that describes the task +#### Prerequisites + +Optional. List all needed prerequisites: +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## If needed, another task section + +See lines 11-24 + + +## What's next? + +Optional section that explains the next logical steps. + + +## Related links + +- Visit the [related thing documentation](www.related-thing.com) to learn more about related thing. diff --git a/docs/.templates/.page-level/_tutorial-page-template.mdx b/docs/.templates/.page-level/_tutorial-page-template.mdx new file mode 100644 index 0000000..9f64b5e --- /dev/null +++ b/docs/.templates/.page-level/_tutorial-page-template.mdx @@ -0,0 +1,54 @@ + + +A paragraph that explains what the tutorial does, why it matters, and the expected outcome. + +To achieve goal: + +1. [Do the first task](#first-task) +2. [Do the second task](#second-task) + +## Prerequisites + + + +Optional. List all needed prerequisites: + +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + +## First task + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## Second task + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## What's next? + +Optional section that explains the next logical steps. diff --git a/docs/Add-more-charts-to-netdata.md b/docs/Add-more-charts-to-netdata.md new file mode 100644 index 0000000..6090644 --- /dev/null +++ b/docs/Add-more-charts-to-netdata.md @@ -0,0 +1,13 @@ + + +# Add more charts to Netdata + +This file has been deprecated. Please see our [collectors docs](/collectors/README.md) for more information. + +## Available data collection modules + +See the [list of supported collectors](/collectors/COLLECTORS.md) to see all the sources Netdata can collect metrics +from. diff --git a/docs/Demo-Sites.md b/docs/Demo-Sites.md new file mode 100644 index 0000000..5c4d101 --- /dev/null +++ b/docs/Demo-Sites.md @@ -0,0 +1,24 @@ + + +# Demo sites + +You can also view live demos of Netdata at **https://app.netdata.cloud/spaces/netdata-demo** + +| Location | Netdata demo URL | 60 mins reqs | VM donated by | +| :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :------------------------------------------------- | +| London (UK) | **[london3.my-netdata.io](https://london3.my-netdata.io)**
(this is the global Netdata **registry** and has **named** and **mysql** charts) | [![Requests Per Second](https://london3.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://london3.my-netdata.io) | [DigitalOcean.com](https://m.do.co/c/83dc9f941745) | +| Atlanta (USA) | **[cdn77.my-netdata.io](https://cdn77.my-netdata.io)**
(with **named** and **mysql** charts) | [![Requests Per Second](https://cdn77.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://cdn77.my-netdata.io) | [CDN77.com](https://www.cdn77.com/) | +| Bangalore (India) | **[bangalore.my-netdata.io](https://bangalore.my-netdata.io)** | [![Requests Per Second](https://bangalore.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://bangalore.my-netdata.io) | [DigitalOcean.com](https://m.do.co/c/83dc9f941745) | +| Frankfurt (Germany) | **[frankfurt.my-netdata.io](https://frankfurt.my-netdata.io)** | [![Requests Per Second](https://frankfurt.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://frankfurt.my-netdata.io) | [DigitalOcean.com](https://m.do.co/c/83dc9f941745) | +| New York (USA) | **[newyork.my-netdata.io](https://newyork.my-netdata.io)** | [![Requests Per Second](https://newyork.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://newyork.my-netdata.io) | [DigitalOcean.com](https://m.do.co/c/83dc9f941745) | +| San Francisco (USA) | **[sanfrancisco.my-netdata.io](https://sanfrancisco.my-netdata.io)** | [![Requests Per Second](https://sanfrancisco.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://sanfrancisco.my-netdata.io) | [DigitalOcean.com](https://m.do.co/c/83dc9f941745) | +| Singapore | **[singapore.my-netdata.io](https://singapore.my-netdata.io)** | [![Requests Per Second](https://singapore.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://singapore.my-netdata.io) | [DigitalOcean.com](https://m.do.co/c/83dc9f941745) | +| Toronto (Canada) | **[toronto.my-netdata.io](https://toronto.my-netdata.io)** | [![Requests Per Second](https://toronto.my-netdata.io/api/v1/badge.svg?chart=netdata.requests&dimensions=requests&after=-3600&options=unaligned&group=sum&label=reqs&units=empty&value_color=blue&precision=0&v42)](https://toronto.my-netdata.io) | [DigitalOcean.com](https://m.do.co/c/83dc9f941745) | + +Netdata dashboards are mobile- and touch-friendly. + + diff --git a/docs/Donations-netdata-has-received.md b/docs/Donations-netdata-has-received.md new file mode 100644 index 0000000..a8623c5 --- /dev/null +++ b/docs/Donations-netdata-has-received.md @@ -0,0 +1,29 @@ + + +# Donations + +This is a list of the donations we have received for Netdata (sorted alphabetically on their name): + +| what donated|related links|who donated|description of the donation| +|-----------:|:-----------:|:---------:|:--------------------------| +| Packages Distribution|-|**[PackageCloud.io](https://packagecloud.io/)**|**PackageCloud.io** donated to a free open-source subscription to their awesome Package Distribution services.| +| Cross Browser Testing|-|**[BrowserStack.com](https://www.browserstack.com/)**|**BrowserStack.com** donated a free subscription to their awesome Browser Testing services (all three of them: Live, Screenshots, Responsive).| +| Cloud VM|[cdn77.my-netdata.io](http://cdn77.my-netdata.io)|**[CDN77.com](https://www.cdn77.com/)**|**CDN77.com** donated a VM with 2 CPU cores, 4GB RAM and 20GB HD, on their excellent CDN network.| +| Localization Management|[Netdata localization project](https://crowdin.com/project/netdata) (check issue [#279](https://github.com/netdata/netdata/issues/279))|**[Crowdin.com](https://crowdin.com/)**|**Crowdin.com** donated an open source license to their Localization Management Platform.| +| Cloud VMs|[london.my-netdata.io](https://london.my-netdata.io) (Several VMs)|**[DigitalOcean.com](https://www.digitalocean.com/)**|**DigitalOcean.com** donated 1000 USD to be used in their excellent Cloud Computing services. Many thanks to [Justin Paine](https://github.com/xxdesmus) for making this happen.| +| Development IDE|-|**[JetBrains.com](https://www.jetbrains.com/)**|**JetBrains.com** donated an open source license for 4 developers for 1 year, to their excellent IDEs.| +| Cloud VM|[octopuscs.my-netdata.io](https://octopuscs.my-netdata.io)|**[OctopusCS.com](https://octopuscs.com/)**|**OctopusCS.com** donated a VM with 4 CPU cores, 16GB RAM and 50GB HD in their excellent Cloud Computing services.| +| Cloud VM|[stackscale.my-netdata.io](https://stackscale.my-netdata.io)|**[stackscale.com](https://www.stackscale.com/)**|**StackScale.com** donated a VM with 4 CPU cores, 16GB RAM and 100GB HD in their excellent Cloud Computing services.| + +Thank you! + +--- + +**Do you want to donate?** We are thirsty for on-line services that can help us make Netdata better. We also try to build a network of demo sites (VMs) that can help us show the full potential of Netdata. + +Please contact me at costa@tsaousis.gr. + + diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..e3b9156 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,17 @@ + + +# Read documentation on + +Welcome to the Netdata documentation! While you can read Netdata documentation here, or throughout the Netdata +repository, our intention is that these pages are read on [learn.netdata.cloud](https://learn.netdata.cloud). + +Links between documentation pages will work fine here, but the formatting may not be perfect, as our documentation site +uses a few extra Markdown features that GitHub doesn't support natively. Other things might be missing or look less than +perfect. + +Now get out there and build an exceptional infrastructure. + + diff --git a/docs/Running-behind-apache.md b/docs/Running-behind-apache.md new file mode 100644 index 0000000..989c51f --- /dev/null +++ b/docs/Running-behind-apache.md @@ -0,0 +1,371 @@ + + +# Netdata via apache's mod_proxy + +Below you can find instructions for configuring an apache server to: + +1. Proxy a single Netdata via an HTTP and HTTPS virtual host. +2. Dynamically proxy any number of Netdata servers. +3. Add user authentication. +4. Adjust Netdata settings to get optimal results. + +## Requirements + +Make sure your apache has `mod_proxy` and `mod_proxy_http` installed and enabled. + +On Debian/Ubuntu systems, install apache, which already includes the two modules, using: + +```sh +sudo apt-get install apache2 +``` + +Enable them: + +```sh +sudo a2enmod proxy +sudo a2enmod proxy_http +``` + +Also, enable the rewrite module: + +```sh +sudo a2enmod rewrite +``` + +--- + +## Netdata on an existing virtual host + +On any **existing** and already **working** apache virtual host, you can redirect requests for URL `/netdata/` to one or more Netdata servers. + +### proxy one Netdata, running on the same server apache runs + +Add the following on top of any existing virtual host. It will allow you to access Netdata as `http://virtual.host/netdata/`. + +```conf + + + RewriteEngine On + ProxyRequests Off + ProxyPreserveHost On + + + Require all granted + + + # Local Netdata server accessed with '/netdata/', at localhost:19999 + ProxyPass "/netdata/" "http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on + ProxyPassReverse "/netdata/" "http://localhost:19999/" + + # if the user did not give the trailing /, add it + # for HTTP (if the virtualhost is HTTP, use this) + RewriteRule ^/netdata$ http://%{HTTP_HOST}/netdata/ [L,R=301] + # for HTTPS (if the virtualhost is HTTPS, use this) + #RewriteRule ^/netdata$ https://%{HTTP_HOST}/netdata/ [L,R=301] + + # rest of virtual host config here + + +``` + +### proxy multiple Netdata running on multiple servers + +Add the following on top of any existing virtual host. It will allow you to access multiple Netdata as `http://virtual.host/netdata/HOSTNAME/`, where `HOSTNAME` is the hostname of any other Netdata server you have (to access the `localhost` Netdata, use `http://virtual.host/netdata/localhost/`). + +```conf + + + RewriteEngine On + ProxyRequests Off + ProxyPreserveHost On + + + Require all granted + + + # proxy any host, on port 19999 + ProxyPassMatch "^/netdata/([A-Za-z0-9\._-]+)/(.*)" "http://$1:19999/$2" connectiontimeout=5 timeout=30 keepalive=on + + # make sure the user did not forget to add a trailing / + # for HTTP (if the virtualhost is HTTP, use this) + RewriteRule "^/netdata/([A-Za-z0-9\._-]+)$" http://%{HTTP_HOST}/netdata/$1/ [L,R=301] + # for HTTPS (if the virtualhost is HTTPS, use this) + RewriteRule "^/netdata/([A-Za-z0-9\._-]+)$" https://%{HTTP_HOST}/netdata/$1/ [L,R=301] + + # rest of virtual host config here + + +``` + +> IMPORTANT
+> The above config allows your apache users to connect to port 19999 on any server on your network. + +If you want to control the servers your users can connect to, replace the `ProxyPassMatch` line with the following. This allows only `server1`, `server2`, `server3` and `server4`. + +``` + ProxyPassMatch "^/netdata/(server1|server2|server3|server4)/(.*)" "http://$1:19999/$2" connectiontimeout=5 timeout=30 keepalive=on +``` + +## Netdata on a dedicated virtual host + +You can proxy Netdata through apache, using a dedicated apache virtual host. + +Create a new apache site: + +```sh +nano /etc/apache2/sites-available/netdata.conf +``` + +with this content: + +```conf + + ProxyRequests Off + ProxyPreserveHost On + + ServerName netdata.domain.tld + + + Require all granted + + + ProxyPass "/" "http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on + ProxyPassReverse "/" "http://localhost:19999/" + + ErrorLog ${APACHE_LOG_DIR}/netdata-error.log + CustomLog ${APACHE_LOG_DIR}/netdata-access.log combined + +``` + +Enable the VirtualHost: + +```sh +sudo a2ensite netdata.conf && service apache2 reload +``` + +## Netdata proxy in Plesk + +_Assuming the main goal is to make Netdata running in HTTPS._ + +1. Make a subdomain for Netdata on which you enable and force HTTPS - You can use a free Let's Encrypt certificate +2. Go to "Apache & nginx Settings", and in the following section, add: + +```conf +RewriteEngine on +RewriteRule (.*) http://localhost:19999/$1 [P,L] +``` + +3. Optional: If your server is remote, then just replace "localhost" with your actual hostname or IP, it just works. + +Repeat the operation for as many servers as you need. + +## Enable Basic Auth + +If you wish to add an authentication (user/password) to access your Netdata, do these: + +Install the package `apache2-utils`. On Debian/Ubuntu run `sudo apt-get install apache2-utils`. + +Then, generate password for user `netdata`, using `htpasswd -c /etc/apache2/.htpasswd netdata` + +**Apache 2.2 Example:**\ +Modify the virtual host with these: + +```conf + # replace the section + + Order deny,allow + Allow from all + + + # add a section + + AuthType Basic + AuthName "Protected site" + AuthUserFile /etc/apache2/.htpasswd + Require valid-user + Order deny,allow + Allow from all + +``` + +Specify `Location /` if Netdata is running on dedicated virtual host. + +**Apache 2.4 (dedicated virtual host) Example:** + +```conf + + RewriteEngine On + ProxyRequests Off + ProxyPreserveHost On + + ServerName netdata.domain.tld + + + AllowOverride None + AuthType Basic + AuthName "Protected site" + AuthUserFile /etc/apache2/.htpasswd + Require valid-user + + + ProxyPass "/" "http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on + ProxyPassReverse "/" "http://localhost:19999/" + + ErrorLog ${APACHE_LOG_DIR}/netdata-error.log + CustomLog ${APACHE_LOG_DIR}/netdata-access.log combined + +``` + +Note: Changes are applied by reloading or restarting Apache. + +## Configuration of Content Security Policy + +If you want to enable CSP within your Apache, you should consider some special requirements of the headers. Modify your configuration like that: + +``` + Header always set Content-Security-Policy "default-src http: 'unsafe-inline' 'self' 'unsafe-eval'; script-src http: 'unsafe-inline' 'self' 'unsafe-eval'; style-src http: 'self' 'unsafe-inline'" +``` + +Note: Changes are applied by reloading or restarting Apache. + +## Using Netdata with Apache's `mod_evasive` module + +The `mod_evasive` Apache module helps system administrators protect their web server from brute force and distributed +denial of service attack (DDoS) attacks. + +Because Netdata sends a request to the web server for every chart update, it's normal to create 20-30 requests per +second, per client. If you're using `mod_evasive` on your Apache web server, this volume of requests will trigger the +module's protection, and your dashboard will become unresponsive. You may even begin to see 403 errors. + +To mitigate this issue, you will need to change the value of the `DOSPageCount` option in your `mod_evasive.conf` file, +which can typically be found at `/etc/httpd/conf.d/mod_evasive.conf` or `/etc/apache2/mods-enabled/evasive.conf`. + +The `DOSPageCount` option sets the limit of the number of requests from a single IP address for the same page per page +interval, which is usually 1 second. The default value is `2` requests per second. Clearly, Netdata's typical usage will +exceed that threshold, and `mod_evasive` will add your IP address to a blocklist. + +Our users have found success by setting `DOSPageCount` to `30`. Try this, and raise the value if you continue to see 403 +errors while accessing the dashboard. + +```conf +DOSPageCount 30 +``` + +Restart Apache with `sudo systemctl restart apache2`, or the appropriate method to restart services on your system, to +reload its configuration with your new values. + +### Virtual host + +To adjust the `DOSPageCount` for a specific virtual host, open your virtual host config, which can be found at +`/etc/httpd/conf/sites-available/my-domain.conf` or `/etc/apache2/sites-available/my-domain.conf` and add the +following: + +```conf + + ... + # Increase the DOSPageCount to prevent 403 errors and IP addresses being blocked. + + DOSPageCount 30 + + +``` + +See issues [#2011](https://github.com/netdata/netdata/issues/2011) and +[#7658](https://github.com/netdata/netdata/issues/7568) for more information. + +# Netdata configuration + +You might edit `/etc/netdata/netdata.conf` to optimize your setup a bit. For applying these changes you need to restart Netdata. + +## Response compression + +If you plan to use Netdata exclusively via apache, you can gain some performance by preventing double compression of its output (Netdata compresses its response, apache re-compresses it) by editing `/etc/netdata/netdata.conf` and setting: + +``` +[web] + enable gzip compression = no +``` + +Once you disable compression at Netdata (and restart it), please verify you receive compressed responses from apache (it is important to receive compressed responses - the charts will be more snappy). + +## Limit direct access to Netdata + +You would also need to instruct Netdata to listen only on `localhost`, `127.0.0.1` or `::1`. + +``` +[web] + bind to = localhost +``` + +or + +``` +[web] + bind to = 127.0.0.1 +``` + +or + +``` +[web] + bind to = ::1 +``` + +--- + +You can also use a unix domain socket. This will also provide a faster route between apache and Netdata: + +``` +[web] + bind to = unix:/tmp/netdata.sock +``` + +Apache 2.4.24+ can not read from `/tmp` so create your socket in `/var/run/netdata` + +``` +[web] + bind to = unix:/var/run/netdata/netdata.sock +``` + +_note: Netdata v1.8+ support unix domain sockets_ + +At the apache side, prepend the 2nd argument to `ProxyPass` with `unix:/tmp/netdata.sock|`, like this: + +``` +ProxyPass "/netdata/" "unix:/tmp/netdata.sock|http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on +``` + +--- + +If your apache server is not on localhost, you can set: + +``` +[web] + bind to = * + allow connections from = IP_OF_APACHE_SERVER +``` + +*note: Netdata v1.9+ support `allow connections from`* + +`allow connections from` accepts [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to match against the connection IP address. + +## prevent the double access.log + +apache logs accesses and Netdata logs them too. You can prevent Netdata from generating its access log, by setting this in `/etc/netdata/netdata.conf`: + +``` +[global] + access log = none +``` + +## Troubleshooting mod_proxy + +Make sure the requests reach Netdata, by examining `/var/log/netdata/access.log`. + +1. if the requests do not reach Netdata, your apache does not forward them. +2. if the requests reach Netdata but the URLs are wrong, you have not re-written them properly. + + diff --git a/docs/Running-behind-caddy.md b/docs/Running-behind-caddy.md new file mode 100644 index 0000000..0282d07 --- /dev/null +++ b/docs/Running-behind-caddy.md @@ -0,0 +1,34 @@ + + +# Netdata via Caddy + +To run Netdata via [Caddy v2 proxying,](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy) set your Caddyfile up like this: + +```caddyfile +netdata.domain.tld { + reverse_proxy localhost:19999 +} +``` + +Other directives can be added between the curly brackets as needed. + +To run Netdata in a subfolder: + +```caddyfile +netdata.domain.tld { + handle_path /netdata/* { + reverse_proxy localhost:19999 + } +} +``` + +## limit direct access to Netdata + +You would also need to instruct Netdata to listen only to `127.0.0.1` or `::1`. + +To limit access to Netdata only from localhost, set `bind socket to IP = 127.0.0.1` or `bind socket to IP = ::1` in `/etc/netdata/netdata.conf`. + + diff --git a/docs/Running-behind-h2o.md b/docs/Running-behind-h2o.md new file mode 100644 index 0000000..c49e4e1 --- /dev/null +++ b/docs/Running-behind-h2o.md @@ -0,0 +1,183 @@ + + +# Running Netdata behind H2O + +[H2O](https://h2o.examp1e.net/) is a new generation HTTP server that provides quicker response to users with less CPU utilization when compared to older generation of web servers. + +It is notable for having much simpler configuration than many popular HTTP servers, low resource requirements, and integrated native support for many things that other HTTP servers may need special setup to use. + +## Why H2O + +- Sane configuration defaults mean that typical configurations are very minimalistic and easy to work with. + +- Native support for HTTP/2 provides improved performance when accessing the Netdata dashboard remotely. + +- Password protect access to the Netdata dashboard without requiring Netdata Cloud. + +## H2O configuration file. + +On most systems, the H2O configuration is found under `/etc/h2o`. H2O uses [YAML 1.1](https://yaml.org/spec/1.1/), with a few special extensions, for it’s configuration files, with the main configuration file being `/etc/h2o/h2o.conf`. + +You can edit the H2O configuration file with Nano, Vim or any other text editors with which you are comfortable. + +After making changes to the configuration files, perform the following: + +- Test the configuration with `h2o -m test -c /etc/h2o/h2o.conf` + +- Restart H2O to apply tha changes with `/etc/init.d/h2o restart` or `service h2o restart` + +## Ways to access Netdata via H2O + +### As a virtual host + +With this method instead of `SERVER_IP_ADDRESS:19999`, the Netdata dashboard can be accessed via a human-readable URL such as `netdata.example.com` used in the configuration below. + +```yaml +hosts: + netdata.example.com: + listen: + port: 80 + paths: + /: + proxy.preserve-host: ON + proxy.reverse.url: http://127.0.0.1:19999 +``` + +### As a subfolder of an existing virtual host + +This method is recommended when Netdata is to be served from a subfolder (or directory). +In this case, the virtual host `netdata.example.com` already exists and Netdata has to be accessed via `netdata.example.com/netdata/`. + +```yaml +hosts: + netdata.example.com: + listen: + port: 80 + paths: + /netdata: + redirect: + status: 301 + url: /netdata/ + /netdata/: + proxy.preserve-host: ON + proxy.reverse.url: http://127.0.0.1:19999 +``` + +### As a subfolder for multiple Netdata servers, via one H2O instance + +This is the recommended configuration when one H2O instance will be used to manage multiple Netdata servers via subfolders. + +```yaml +hosts: + netdata.example.com: + listen: + port: 80 + paths: + /netdata/server1: + redirect: + status: 301 + url: /netdata/server1/ + /netdata/server1/: + proxy.preserve-host: ON + proxy.reverse.url: http://198.51.100.1:19999 + /netdata/server2: + redirect: + status: 301 + url: /netdata/server2/ + /netdata/server2/: + proxy.preserve-host: ON + proxy.reverse.url: http://198.51.100.2:19999 +``` + +Of course you can add as many backend servers as you like. + +Using the above, you access Netdata on the backend servers, like this: + +- `http://netdata.example.com/netdata/server1/` to reach Netdata on `198.51.100.1:19999` +- `http://netdata.example.com/netdata/server2/` to reach Netdata on `198.51.100.2:19999` + +### Encrypt the communication between H2O and Netdata + +In case Netdata's web server has been [configured to use TLS](/web/server/README.md#enabling-tls-support), it is +necessary to specify inside the H2O configuration that the final destination is using TLS. To do this, change the +`http://` on the `proxy.reverse.url` line in your H2O configuration with `https://` + +### Enable authentication + +Create an authentication file to enable basic authentication via H2O, this secures your Netdata dashboard. + +If you don't have an authentication file, you can use the following command: + +```sh +printf "yourusername:$(openssl passwd -apr1)" > /etc/h2o/passwords +``` + +And then add a basic authentication handler to each path definition: + +```yaml +hosts: + netdata.example.com: + listen: + port: 80 + paths: + /: + mruby.handler: | + require "htpasswd.rb" + Htpasswd.new("/etc/h2o/passwords", "netdata.example.com") + proxy.preserve-host: ON + proxy.reverse.url: http://127.0.0.1:19999 +``` + +For more information on using basic authentication with H2O, see [their official documentation](https://h2o.examp1e.net/configure/basic_auth.html). + +## Limit direct access to Netdata + +If your H2O server is on `localhost`, you can use this to ensure external access is only possible through H2O: + +``` +[web] + bind to = 127.0.0.1 ::1 +``` + +--- + +You can also use a unix domain socket. This will provide faster communication between H2O and Netdata as well: + +``` +[web] + bind to = unix:/run/netdata/netdata.sock +``` + +In the H2O configuration, use a line like the following to connect to Netdata via the unix socket: + +```yaml +proxy.reverse.url http://[unix:/run/netdata/netdata.sock] +``` + +--- + +If your H2O server is not on localhost, you can set: + +``` +[web] + bind to = * + allow connections from = IP_OF_H2O_SERVER +``` + +*note: Netdata v1.9+ support `allow connections from`* + +`allow connections from` accepts [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to match against +the connection IP address. + +## Prevent the double access.log + +H2O logs accesses and Netdata logs them too. You can prevent Netdata from generating its access log, by setting +this in `/etc/netdata/netdata.conf`: + +``` +[global] + access log = none +``` diff --git a/docs/Running-behind-haproxy.md b/docs/Running-behind-haproxy.md new file mode 100644 index 0000000..ee1790c --- /dev/null +++ b/docs/Running-behind-haproxy.md @@ -0,0 +1,293 @@ + + +# Netdata via HAProxy + +> HAProxy is a free, very fast and reliable solution offering high availability, load balancing, +> and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic websites +> and powers quite a number of the world's most visited ones. + +If Netdata is running on a host running HAProxy, rather than connecting to Netdata from a port number, a domain name can +be pointed at HAProxy, and HAProxy can redirect connections to the Netdata port. This can make it possible to connect to +Netdata at `https://example.com` or `https://example.com/netdata/`, which is a much nicer experience then +`http://example.com:19999`. + +To proxy requests from [HAProxy](https://github.com/haproxy/haproxy) to Netdata, +the following configuration can be used: + +## Default Configuration + +For all examples, set the mode to `http` + +```conf +defaults + mode http +``` + +## Simple Configuration + +A simple example where the base URL, say `http://example.com`, is used with no subpath: + +### Frontend + +Create a frontend to receive the request. + +```conf +frontend http_frontend + ## HTTP ipv4 and ipv6 on all ips ## + bind :::80 v4v6 + + default_backend netdata_backend +``` + +### Backend + +Create the Netdata backend which will send requests to port `19999`. + +```conf +backend netdata_backend + option forwardfor + server netdata_local 127.0.0.1:19999 + + http-request set-header Host %[src] + http-request set-header X-Forwarded-For %[src] + http-request set-header X-Forwarded-Port %[dst_port] + http-request set-header Connection "keep-alive" +``` + +## Configuration with subpath + +An example where the base URL is used with a subpath `/netdata/`: + +### Frontend + +To use a subpath, create an ACL, which will set a variable based on the subpath. + +```conf +frontend http_frontend + ## HTTP ipv4 and ipv6 on all ips ## + bind :::80 v4v6 + + # URL begins with /netdata + acl is_netdata url_beg /netdata + + # if trailing slash is missing, redirect to /netdata/ + http-request redirect scheme https drop-query append-slash if is_netdata ! { path_beg /netdata/ } + + ## Backends ## + use_backend netdata_backend if is_netdata + + # Other requests go here (optional) + # put netdata_backend here if no others are used + default_backend www_backend +``` + +### Backend + +Same as simple example, except remove `/netdata/` with regex. + +```conf +backend netdata_backend + option forwardfor + server netdata_local 127.0.0.1:19999 + + http-request set-path %[path,regsub(^/netdata/,/)] + + http-request set-header Host %[src] + http-request set-header X-Forwarded-For %[src] + http-request set-header X-Forwarded-Port %[dst_port] + http-request set-header Connection "keep-alive" +``` + +## Using TLS communication + +TLS can be used by adding port `443` and a cert to the frontend. +This example will only use Netdata if host matches example.com (replace with your domain). + +### Frontend + +This frontend uses a certificate list. + +```conf +frontend https_frontend + ## HTTP ## + bind :::80 v4v6 + # Redirect all HTTP traffic to HTTPS with 301 redirect + redirect scheme https code 301 if !{ ssl_fc } + + ## HTTPS ## + # Bind to all v4/v6 addresses, use a list of certs in file + bind :::443 v4v6 ssl crt-list /etc/letsencrypt/certslist.txt + + ## ACL ## + # Optionally check host for Netdata + acl is_example_host hdr_sub(host) -i example.com + + ## Backends ## + use_backend netdata_backend if is_example_host + # Other requests go here (optional) + default_backend www_backend +``` + +In the cert list file place a mapping from a certificate file to the domain used: + +`/etc/letsencrypt/certslist.txt`: + +```txt +example.com /etc/letsencrypt/live/example.com/example.com.pem +``` + +The file `/etc/letsencrypt/live/example.com/example.com.pem` should contain the key and +certificate (in that order) concatenated into a `.pem` file.: + +```sh +cat /etc/letsencrypt/live/example.com/fullchain.pem \ + /etc/letsencrypt/live/example.com/privkey.pem > \ + /etc/letsencrypt/live/example.com/example.com.pem +``` + +### Backend + +Same as simple, except set protocol `https`. + +```conf +backend netdata_backend + option forwardfor + server netdata_local 127.0.0.1:19999 + + http-request add-header X-Forwarded-Proto https + http-request set-header Host %[src] + http-request set-header X-Forwarded-For %[src] + http-request set-header X-Forwarded-Port %[dst_port] + http-request set-header Connection "keep-alive" +``` + +## Enable authentication + +To use basic HTTP Authentication, create an authentication list: + +```conf +# HTTP Auth +userlist basic-auth-list + group is-admin + # Plaintext password + user admin password passwordhere groups is-admin +``` + +You can create a hashed password using the `mkpassword` utility. + +```sh + printf "passwordhere" | mkpasswd --stdin --method=sha-256 +$5$l7Gk0VPIpKO$f5iEcxvjfdF11khw.utzSKqP7W.0oq8wX9nJwPLwzy1 +``` + +Replace `passwordhere` with hash: + +```conf +user admin password $5$l7Gk0VPIpKO$f5iEcxvjfdF11khw.utzSKqP7W.0oq8wX9nJwPLwzy1 groups is-admin +``` + +Now add at the top of the backend: + +```conf +acl devops-auth http_auth_group(basic-auth-list) is-admin +http-request auth realm netdata_local unless devops-auth +``` + +## Full Example + +Full example configuration with HTTP auth over TLS with subpath: + +```conf +global + maxconn 20000 + + log /dev/log local0 + log /dev/log local1 notice + user haproxy + group haproxy + pidfile /run/haproxy.pid + + stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners + stats timeout 30s + daemon + + tune.ssl.default-dh-param 4096 # Max size of DHE key + + # Default ciphers to use on SSL-enabled listening sockets. + ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS + ssl-default-bind-options no-sslv3 + +defaults + log global + mode http + option httplog + option dontlognull + timeout connect 5000 + timeout client 50000 + timeout server 50000 + errorfile 400 /etc/haproxy/errors/400.http + errorfile 403 /etc/haproxy/errors/403.http + errorfile 408 /etc/haproxy/errors/408.http + errorfile 500 /etc/haproxy/errors/500.http + errorfile 502 /etc/haproxy/errors/502.http + errorfile 503 /etc/haproxy/errors/503.http + errorfile 504 /etc/haproxy/errors/504.http + +frontend https_frontend + ## HTTP ## + bind :::80 v4v6 + # Redirect all HTTP traffic to HTTPS with 301 redirect + redirect scheme https code 301 if !{ ssl_fc } + + ## HTTPS ## + # Bind to all v4/v6 addresses, use a list of certs in file + bind :::443 v4v6 ssl crt-list /etc/letsencrypt/certslist.txt + + ## ACL ## + # Optionally check host for Netdata + acl is_example_host hdr_sub(host) -i example.com + acl is_netdata url_beg /netdata + + http-request redirect scheme https drop-query append-slash if is_netdata ! { path_beg /netdata/ } + + ## Backends ## + use_backend netdata_backend if is_example_host is_netdata + default_backend www_backend + +# HTTP Auth +userlist basic-auth-list + group is-admin + # Hashed password + user admin password $5$l7Gk0VPIpKO$f5iEcxvjfdF11khw.utzSKqP7W.0oq8wX9nJwPLwzy1 groups is-admin + +## Default server(s) (optional)## +backend www_backend + mode http + balance roundrobin + timeout connect 5s + timeout server 30s + timeout queue 30s + + http-request add-header 'X-Forwarded-Proto: https' + server other_server 111.111.111.111:80 check + +backend netdata_backend + acl devops-auth http_auth_group(basic-auth-list) is-admin + http-request auth realm netdata_local unless devops-auth + + option forwardfor + server netdata_local 127.0.0.1:19999 + + http-request set-path %[path,regsub(^/netdata/,/)] + + http-request add-header X-Forwarded-Proto https + http-request set-header Host %[src] + http-request set-header X-Forwarded-For %[src] + http-request set-header X-Forwarded-Port %[dst_port] + http-request set-header Connection "keep-alive" +``` + + diff --git a/docs/Running-behind-lighttpd.md b/docs/Running-behind-lighttpd.md new file mode 100644 index 0000000..2623560 --- /dev/null +++ b/docs/Running-behind-lighttpd.md @@ -0,0 +1,71 @@ + + +# Netdata via lighttpd v1.4.x + +Here is a config for accessing Netdata in a suburl via lighttpd 1.4.46 and newer: + +```txt +$HTTP["url"] =~ "^/netdata/" { + proxy.server = ( "" => ("netdata" => ( "host" => "127.0.0.1", "port" => 19999 ))) + proxy.header = ( "map-urlpath" => ( "/netdata/" => "/") ) +} +``` + +If you have older lighttpd you have to use a chain (such as below), as explained [at this stackoverflow answer](http://stackoverflow.com/questions/14536554/lighttpd-configuration-to-proxy-rewrite-from-one-domain-to-another). + +```txt +$HTTP["url"] =~ "^/netdata/" { + proxy.server = ( "" => ("" => ( "host" => "127.0.0.1", "port" => 19998 ))) +} + +$SERVER["socket"] == ":19998" { + url.rewrite-once = ( "^/netdata(.*)$" => "/$1" ) + proxy.server = ( "" => ( "" => ( "host" => "127.0.0.1", "port" => 19999 ))) +} +``` + +--- + +If the only thing the server is exposing via the web is Netdata (and thus no suburl rewriting required), +then you can get away with just + +``` +proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => 19999 ))) +``` + +Though if it's public facing you might then want to put some authentication on it. htdigest support +looks like: + +``` +auth.backend = "htdigest" +auth.backend.htdigest.userfile = "/etc/lighttpd/lighttpd.htdigest" +auth.require = ( "" => ( "method" => "digest", + "realm" => "netdata", + "require" => "valid-user" + ) + ) +``` + +other auth methods, and more info on htdigest, can be found in lighttpd's [mod_auth docs](http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModAuth). + +--- + +It seems that lighttpd (or some versions of it), fail to proxy compressed web responses. +To solve this issue, disable web response compression in Netdata. + +Open `/etc/netdata/netdata.conf` and set in [global]\: + +``` +enable web responses gzip compression = no +``` + +## limit direct access to Netdata + +You would also need to instruct Netdata to listen only to `127.0.0.1` or `::1`. + +To limit access to Netdata only from localhost, set `bind socket to IP = 127.0.0.1` or `bind socket to IP = ::1` in `/etc/netdata/netdata.conf`. + + diff --git a/docs/Running-behind-nginx.md b/docs/Running-behind-nginx.md new file mode 100644 index 0000000..0cb1630 --- /dev/null +++ b/docs/Running-behind-nginx.md @@ -0,0 +1,270 @@ + + +# Running Netdata behind Nginx + +## Intro + +[Nginx](https://nginx.org/en/) is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server used to host websites and applications of all sizes. + +The software is known for its low impact on memory resources, high scalability, and its modular, event-driven architecture which can offer secure, predictable performance. + +## Why Nginx + +- By default, Nginx is fast and lightweight out of the box. + +- Nginx is used and useful in cases when you want to access different instances of Netdata from a single server. + +- Password-protect access to Netdata, until distributed authentication is implemented via the Netdata cloud Sign In mechanism. + +- A proxy was necessary to encrypt the communication to Netdata, until v1.16.0, which provided TLS (HTTPS) support. + +## Nginx configuration file + +All Nginx configurations can be found in the `/etc/nginx/` directory. The main configuration file is `/etc/nginx/nginx.conf`. Website or app-specific configurations can be found in the `/etc/nginx/site-available/` directory. + +Configuration options in Nginx are known as directives. Directives are organized into groups known as blocks or contexts. The two terms can be used interchangeably. + +Depending on your installation source, you’ll find an example configuration file at `/etc/nginx/conf.d/default.conf` or `etc/nginx/sites-enabled/default`, in some cases you may have to manually create the `sites-available` and `sites-enabled` directories. + +You can edit the Nginx configuration file with Nano, Vim or any other text editors you are comfortable with. + +After making changes to the configuration files: + +- Test Nginx configuration with `nginx -t`. + +- Restart Nginx to effect the change with `/etc/init.d/nginx restart` or `service nginx restart`. + +## Ways to access Netdata via Nginx + +### As a virtual host + +With this method instead of `SERVER_IP_ADDRESS:19999`, the Netdata dashboard can be accessed via a human-readable URL such as `netdata.example.com` used in the configuration below. + +```conf +upstream backend { + # the Netdata server + server 127.0.0.1:19999; + keepalive 64; +} + +server { + # nginx listens to this + listen 80; + # uncomment the line if you want nginx to listen on IPv6 address + #listen [::]:80; + + # the virtual host name of this + server_name netdata.example.com; + + location / { + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_pass http://backend; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + } +} +``` + +### As a subfolder to an existing virtual host + +This method is recommended when Netdata is to be served from a subfolder (or directory). +In this case, the virtual host `netdata.example.com` already exists and Netdata has to be accessed via `netdata.example.com/netdata/`. + +```conf +upstream netdata { + server 127.0.0.1:19999; + keepalive 64; +} + +server { + listen 80; + # uncomment the line if you want nginx to listen on IPv6 address + #listen [::]:80; + + # the virtual host name of this subfolder should be exposed + #server_name netdata.example.com; + + location = /netdata { + return 301 /netdata/; + } + + location ~ /netdata/(?.*) { + proxy_redirect off; + proxy_set_header Host $host; + + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + proxy_pass http://netdata/$ndpath$is_args$args; + + gzip on; + gzip_proxied any; + gzip_types *; + } +} +``` + +### As a subfolder for multiple Netdata servers, via one Nginx + +This is the recommended configuration when one Nginx will be used to manage multiple Netdata servers via subfolders. + +```conf +upstream backend-server1 { + server 10.1.1.103:19999; + keepalive 64; +} +upstream backend-server2 { + server 10.1.1.104:19999; + keepalive 64; +} + +server { + listen 80; + # uncomment the line if you want nginx to listen on IPv6 address + #listen [::]:80; + + # the virtual host name of this subfolder should be exposed + #server_name netdata.example.com; + + location ~ /netdata/(?.*?)/(?.*) { + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + proxy_pass http://backend-$behost/$ndpath$is_args$args; + + gzip on; + gzip_proxied any; + gzip_types *; + } + + # make sure there is a trailing slash at the browser + # or the URLs will be wrong + location ~ /netdata/(?.*) { + return 301 /netdata/$behost/; + } +} +``` + +Of course you can add as many backend servers as you like. + +Using the above, you access Netdata on the backend servers, like this: + +- `http://netdata.example.com/netdata/server1/` to reach `backend-server1` +- `http://netdata.example.com/netdata/server2/` to reach `backend-server2` + +### Encrypt the communication between Nginx and Netdata + +In case Netdata's web server has been [configured to use TLS](/web/server/README.md#enabling-tls-support), it is +necessary to specify inside the Nginx configuration that the final destination is using TLS. To do this, please, append +the following parameters in your `nginx.conf` + +```conf +proxy_set_header X-Forwarded-Proto https; +proxy_pass https://localhost:19999; +``` + +Optionally it is also possible to [enable TLS/SSL on Nginx](http://nginx.org/en/docs/http/configuring_https_servers.html), this way the user will encrypt not only the communication between Nginx and Netdata but also between the user and Nginx. + +If Nginx is not configured as described here, you will probably receive the error `SSL_ERROR_RX_RECORD_TOO_LONG`. + +### Enable authentication + +Create an authentication file to enable basic authentication via Nginx, this secures your Netdata dashboard. + +If you don't have an authentication file, you can use the following command: + +```sh +printf "yourusername:$(openssl passwd -apr1)" > /etc/nginx/passwords +``` + +And then enable the authentication inside your server directive: + +```conf +server { + # ... + auth_basic "Protected"; + auth_basic_user_file passwords; + # ... +} +``` + +## Limit direct access to Netdata + +If your Nginx is on `localhost`, you can use this to protect your Netdata: + +``` +[web] + bind to = 127.0.0.1 ::1 +``` + +--- + +You can also use a unix domain socket. This will also provide a faster route between Nginx and Netdata: + +``` +[web] + bind to = unix:/var/run/netdata/netdata.sock +``` + +*note: Netdata v1.8+ support unix domain sockets* + +At the Nginx side, use something like this to use the same unix domain socket: + +```conf +upstream backend { + server unix:/var/run/netdata/netdata.sock; + keepalive 64; +} +``` + +--- + +If your Nginx server is not on localhost, you can set: + +``` +[web] + bind to = * + allow connections from = IP_OF_NGINX_SERVER +``` + +*note: Netdata v1.9+ support `allow connections from`* + +`allow connections from` accepts [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to match against the +connection IP address. + +## Prevent the double access.log + +Nginx logs accesses and Netdata logs them too. You can prevent Netdata from generating its access log, by setting this in `/etc/netdata/netdata.conf`: + +``` +[global] + access log = none +``` + +## SELinux + +If you get an 502 Bad Gateway error you might check your Nginx error log: + +```sh +# cat /var/log/nginx/error.log: +2016/09/09 12:34:05 [crit] 5731#5731: *1 connect() to 127.0.0.1:19999 failed (13: Permission denied) while connecting to upstream, client: 1.2.3.4, server: netdata.example.com, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:19999/", host: "netdata.example.com" +``` + +If you see something like the above, chances are high that SELinux prevents nginx from connecting to the backend server. To fix that, just use this policy: `setsebool -P httpd_can_network_connect true`. + + diff --git a/docs/a-github-star-is-important.md b/docs/a-github-star-is-important.md new file mode 100644 index 0000000..22659ea --- /dev/null +++ b/docs/a-github-star-is-important.md @@ -0,0 +1,24 @@ + + +# A GitHub star is important + +**GitHub stars** allow Netdata to expand its reach, its community, especially attract people with skills willing to +contribute to it. + +Compared to its first release, Netdata is now **twice as fast**, has all its bugs settled and a lot more functionality. +This happened because a lot of people find it useful, use it daily at home and work, **rely on it** and **contribute to +it**. + +**GitHub stars** also **motivate** us. They state that you find our work **useful**. They give us strength to continue, +to work **harder** to make it even **better**. + +So, give Netdata a **GitHub star**, at the top right of this page. + +Thank you! + +Costa Tsaousis + + diff --git a/docs/agent-cloud.md b/docs/agent-cloud.md new file mode 100644 index 0000000..ed54325 --- /dev/null +++ b/docs/agent-cloud.md @@ -0,0 +1,77 @@ + + +# Use the Agent with Netdata Cloud + +While the Netdata Agent is an enormously powerful _distributed_ health monitoring and performance troubleshooting tool, +many of its users need to monitor dozens or hundreds of systems at the same time. That's why we built Netdata Cloud, a +hosted web interface that gives you real-time visibility into your entire infrastructure. + +There are two main ways to use your Agent(s) with Netdata Cloud. You can use both these methods simultaneously, or just +one, based on your needs: + +- Use Netdata Cloud's web interface for monitoring an entire infrastructure, with any number of Agents, in one + centralized dashboard. +- Use **Visited nodes** to quickly navigate between the dashboards of nodes you've recently visited. + +## Monitor an infrastructure with Netdata Cloud + +We designed Netdata Cloud to help you see health and performance metrics, plus active alarms, in a single interface. +Here's what a small infrastructure might look like: + +![Animated GIF of Netdata +Cloud](https://user-images.githubusercontent.com/1153921/80828986-1ebb3b00-8b9b-11ea-957f-2c8d0d009e44.gif) + +[Read more about Netdata Cloud](https://learn.netdata.cloud/docs/cloud/) to better understand how it gives you real-time +visibility into your entire infrastructure, and why you might consider using it. + +Next, [get started in 5 minutes](https://learn.netdata.cloud/docs/cloud/get-started/), or read our [connection to Cloud +reference](/claim/README.md) for a complete investigation of Cloud's security and encryption features, plus instructions +for Docker containers. + +## Navigate between dashboards with Visited nodes + +If you don't want to use Netdata Cloud's web interface, you can still connect multiple nodes through the **Visited +nodes** menu, which appears on the left-hand side of the dashboard. + +You can use the Visited nodes menu to navigate between the dashboards of many different Agent-monitored systems quickly. + +To add nodes to your Visited nodes menu, you first need to navigate to that node's dashboard, then click the **Sign in** +button at the top of the dashboard. On the screen that appears, which states your node is requesting access to your +Netdata Cloud account, sign in with your preferred method. + +Cloud redirects you back to your node's dashboard, which is now connected to your Netdata Cloud account. You can now see +the Visited nodes menu, which is populated by a single node. + +![An Agent's dashboard with the Visited nodes +menu](https://user-images.githubusercontent.com/1153921/80830383-b6ba2400-8b9d-11ea-9eb2-379c7eccd22f.png) + +If you previously went through the Cloud onboarding process to create a Space and War Room, you will also see these in +the Visited Nodes menu. You can click on your Space or any of your War Rooms to navigate to Netdata Cloud and continue +monitoring your infrastructure from there. + +![A Agent's dashboard with the Visited nodes menu, plus Spaces and War +Rooms](https://user-images.githubusercontent.com/1153921/80830382-b6218d80-8b9d-11ea-869c-1170b95eeb4a.png) + +To add more Agents to your Visited nodes menu, visit them and sign in again. This process connects that node to your +Cloud account and further populates the menu. + +Once you've added more than one node, you can use the menu to switch between various dashboards without remembering IP +addresses or hostnames or saving bookmarks for every node you want to monitor. + +![Switching between dashboards with Visited +nodes](https://user-images.githubusercontent.com/1153921/80831018-e158ac80-8b9e-11ea-882e-1d82cdc028cd.gif) + +## What's next? + +The Agent-Cloud integration is highly adaptable to the needs of any infrastructure or user. If you want to learn more +about how you might want to use or configure Cloud, we recommend the following: + +- Get an overview of Cloud's features by reading [Cloud documentation](https://learn.netdata.cloud/docs/cloud/). +- Follow the 5-minute [get started with Cloud](https://learn.netdata.cloud/docs/cloud/get-started/) guide to finish + onboarding and connect your first nodes. +- Better understand how agents connect securely to the Cloud with [connect agent to Cloud](/claim/README.md) and [Agent-Cloud + link](/aclk/README.md) documentation. diff --git a/docs/anonymous-statistics.md b/docs/anonymous-statistics.md new file mode 100644 index 0000000..99bd3dc --- /dev/null +++ b/docs/anonymous-statistics.md @@ -0,0 +1,105 @@ + + +# Anonymous statistics + +By default, Netdata collects anonymous usage information from the open-source monitoring agent using the open-source +product analytics platform [PostHog](https://github.com/PostHog/posthog). We use their [cloud enterprise platform](https://posthog.com/product). + +We are strongly committed to your [data privacy](https://netdata.cloud/privacy/). + +We use the statistics gathered from this information for two purposes: + +1. **Quality assurance**, to help us understand if Netdata behaves as expected, and to help us classify repeated + issues with certain distributions or environments. + +2. **Usage statistics**, to help us interpret how people use the Netdata agent in real-world environments, and to help + us identify how our development/design decisions influence the community. + +Netdata collects usage information via two different channels: + +- **Agent dashboard**: We use the [PostHog JavaScript integration](https://posthog.com/docs/integrations/js-integration) (with sensitive event attributes overwritten to be anonymized) to send product usage events when you access an [Agent's dashboard](/web/gui/README.md). +- **Agent backend**: The `netdata` daemon executes the [`anonymous-statistics.sh`](https://github.com/netdata/netdata/blob/6469cf92724644f5facf343e4bdd76ac0551a418/daemon/anonymous-statistics.sh.in) script when Netdata starts, stops cleanly, or fails. + +You can opt-out from sending anonymous statistics to Netdata through three different [opt-out mechanisms](#opt-out). + +## Agent Dashboard - PostHog JavaScript + +When you kick off an Agent dashboard session by visiting `http://NODE:19999`, Netdata initializes a PostHog session and masks various event attributes. + +_Note_: You can see the relevant code in the [dashboard repository](https://github.com/netdata/dashboard/blob/master/src/domains/global/sagas.ts#L107) where the `window.posthog.register()` call is made. + +```JavaScript +window.posthog.register({ + distinct_id: machineGuid, + $ip: "127.0.0.1", + $current_url: "agent dashboard", + $pathname: "netdata-dashboard", + $host: "dashboard.netdata.io", +}) +``` + +In the above snippet a Netdata PostHog session is initialized and the `ip`, `current_url`, `pathname` and `host` attributes are set to constant values for all events that may be sent during the session. This way, information like the IP or hostname of the Agent will not be sent as part of the product usage event data. + +We have configured the dashboard to trigger the PostHog JavaScript code only when the variable `anonymous_statistics` is true. The value of this +variable is controlled via the [opt-out mechanism](#opt-out). + +## Agent Backend - Anonymous Statistics Script + +Every time the daemon is started or stopped and every time a fatal condition is encountered, Netdata uses the anonymous +statistics script to collect system information and send it to the Netdata PostHog via an http call. The information collected for all +events is: + +- Netdata version +- OS name, version, id, id_like +- Kernel name, version, architecture +- Virtualization technology +- Containerization technology + +Furthermore, the FATAL event sends the Netdata process & thread name, along with the source code function, source code +filename and source code line number of the fatal error. + +Starting with v1.21, we additionally collect information about: + +- Failures to build the dependencies required to use Cloud features. +- Unavailability of Cloud features in an agent. +- Failures to connect to the Cloud in case the [connection process](/claim/README.md) has been completed. This includes error codes + to inform the Netdata team about the reason why the connection failed. + +To see exactly what and how is collected, you can review the script template `daemon/anonymous-statistics.sh.in`. The +template is converted to a bash script called `anonymous-statistics.sh`, installed under the Netdata `plugins +directory`, which is usually `/usr/libexec/netdata/plugins.d`. + +## Opt-out + +You can opt-out from sending anonymous statistics to Netdata through three different opt-out mechanisms: + +**Create a file called `.opt-out-from-anonymous-statistics`.** This empty file, stored in your Netdata configuration +directory (usually `/etc/netdata`), immediately stops the statistics script from running, and works with any type of +installation, including manual, offline, and macOS installations. Create the file by running `touch +.opt-out-from-anonymous-statistics` from your Netdata configuration directory. + +**Pass the option `--disable-telemetry` to any of the installer scripts in the [installation +docs](/packaging/installer/README.md).** You can append this option during the initial installation or a manual +update. You can also export the environment variable `DISABLE_TELEMETRY` with a non-zero or non-empty value +(e.g: `export DISABLE_TELEMETRY=1`). + +When using Docker, **set your `DISABLE_TELEMETRY` environment variable to `1`.** You can set this variable with the following +command: `export DISABLE_TELEMETRY=1`. When creating a container using Netdata's [Docker +image](/packaging/docker/README.md#create-a-new-netdata-agent-container) for the first time, this variable will disable +the anonymous statistics script inside of the container. + +Each of these opt-out processes does the following: + +- Prevents the daemon from executing the anonymous statistics script. +- Forces the anonymous statistics script to exit immediately. +- Stops the PostHog JavaScript snippet, which remains on the dashboard, from firing and sending any data to the Netdata PostHog. + +## Migration from Google Analytics and Google Tag Manager. + +Prior to v1.29.4 we used Google Analytics to capture this information. This led to discomfort with some of our users in sending any product usage data to a third party like Google. It was also not even that useful in terms of generating the insights we needed to help catch bugs early and find opportunities for product improvement as Google Analytics does not allow its users access to the raw underlying data without paying a significant amount of money which would be infeasible for a project like Netdata. + +While we migrate fully away from Google Analytics to PostHog there maybe be a small period of time where we run both in parallel before we remove all Google Analytics related code. This is to ensure we can fully test and validate the Netdata PostHog implementation before fully defaulting to it. + diff --git a/docs/collect/application-metrics.md b/docs/collect/application-metrics.md new file mode 100644 index 0000000..c9bc4e2 --- /dev/null +++ b/docs/collect/application-metrics.md @@ -0,0 +1,81 @@ + + +# Collect application metrics with Netdata + +Netdata instantly collects per-second metrics from many different types of applications running on your systems, such as +web servers, databases, message brokers, email servers, search platforms, and much more. Metrics collectors are +pre-installed with every Netdata Agent and usually require zero configuration. Netdata also collects and visualizes +resource utilization per application on Linux systems using `apps.plugin`. + +[**apps.plugin**](/collectors/apps.plugin/README.md) looks at the Linux process tree every second, much like `top` or +`ps fax`, and collects resource utilization information on every running process. By reading the process tree, Netdata +shows CPU, disk, networking, processes, and eBPF for every application or Linux user. Unlike `top` or `ps fax`, Netdata +adds a layer of meaningful visualization on top of the process tree metrics, such as grouping applications into useful +dimensions, and then creates per-application charts under the **Applications** section of a Netdata dashboard, per-user +charts under **Users**, and per-user group charts under **User Groups**. + +Our most popular application collectors: + +- [Prometheus endpoints](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/prometheus): Gathers + metrics from one or more Prometheus endpoints that use the OpenMetrics exposition format. Auto-detects more than 600 + endpoints. +- [Web server logs (Apache, NGINX)](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog/): + Tail access logs and provide very detailed web server performance statistics. This module is able to parse 200k+ + rows in less than half a second. +- [MySQL](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/mysql/): Collect database global, + replication, and per-user statistics. +- [Redis](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/redis): Monitor database status by + reading the server's response to the `INFO` command. +- [Apache](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/apache/): Collect Apache web server + performance metrics via the `server-status?auto` endpoint. +- [Nginx](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/nginx/): Monitor web server status + information by gathering metrics via `ngx_http_stub_status_module`. +- [Postgres](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/postgres): Collect database health + and performance metrics. +- [ElasticSearch](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/elasticsearch): Collect search + engine performance and health statistics. Optionally collects per-index metrics. +- [PHP-FPM](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/phpfpm/): Collect application summary + and processes health metrics by scraping the status page (`/status?full`). + +Our [supported collectors list](/collectors/COLLECTORS.md#service-and-application-collectors) shows all Netdata's +application metrics collectors, including those for containers/k8s clusters. + +## Collect metrics from applications running on Windows + +Netdata is fully capable of collecting and visualizing metrics from applications running on Windows systems. The only +caveat is that you must [install Netdata](/docs/get-started.mdx) on a separate system or a compatible VM because there +is no native Windows version of the Netdata Agent. + +Once you have Netdata running on that separate system, you can follow the [enable and configure +doc](/docs/collect/enable-configure.md) to tell the collector to look for exposed metrics on the Windows system's IP +address or hostname, plus the applicable port. + +For example, you have a MySQL database with a root password of `my-secret-pw` running on a Windows system with the IP +address 203.0.113.0. you can configure the [MySQL +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/mysql) to look at `203.0.113.0:3306`: + +```yml +jobs: + - name: local + dsn: root:my-secret-pw@tcp(203.0.113.0:3306)/ +``` + +This same logic applies to any application in our [supported collectors +list](/collectors/COLLECTORS.md#service-and-application-collectors) that can run on Windows. + +## What's next? + +If you haven't yet seen the [supported collectors list](/collectors/COLLECTORS.md) give it a once-over for any +additional applications you may want to monitor using Netdata's native collectors, or the [generic Prometheus +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/prometheus). + +Collecting all the available metrics on your nodes, and across your entire infrastructure, is just one piece of the +puzzle. Next, learn more about Netdata's famous real-time visualizations by [seeing an overview of your +infrastructure](/docs/visualize/overview-infrastructure.md) using Netdata Cloud. + + diff --git a/docs/collect/container-metrics.md b/docs/collect/container-metrics.md new file mode 100644 index 0000000..5d14536 --- /dev/null +++ b/docs/collect/container-metrics.md @@ -0,0 +1,99 @@ + + +# Collect container metrics with Netdata + +Thanks to close integration with Linux cgroups and the virtual files it maintains under `/sys/fs/cgroup`, Netdata can +monitor the health, status, and resource utilization of many different types of Linux containers. + +Netdata uses [cgroups.plugin](/collectors/cgroups.plugin/README.md) to poll `/sys/fs/cgroup` and convert the raw data +into human-readable metrics and meaningful visualizations. Through cgroups, Netdata is compatible with **all Linux +containers**, such as Docker, LXC, LXD, Libvirt, systemd-nspawn, and more. Read more about [Docker-specific +monitoring](#collect-docker-metrics) below. + +Netdata also has robust **Kubernetes monitoring** support thanks to a +[Helmchart](/packaging/installer/methods/kubernetes.md) to automate deployment, collectors for k8s agent services, and +robust [service discovery](https://github.com/netdata/agent-service-discovery/#service-discovery) to monitor the +services running inside of pods in your k8s cluster. Read more about [Kubernetes +monitoring](#collect-kubernetes-metrics) below. + +A handful of additional collectors gather metrics from container-related services, such as +[dockerd](/collectors/python.d.plugin/dockerd/README.md) or [Docker +Engine](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/docker_engine/). You can find all +container collectors in our supported collectors list under the +[containers/VMs](/collectors/COLLECTORS.md#containers-and-vms) and +[Kubernetes](/collectors/COLLECTORS.md#containers-and-vms) headings. + +## Collect Docker metrics + +Netdata has robust Docker monitoring thanks to the aforementioned +[cgroups.plugin](/collectors/cgroups.plugin/README.md). By polling cgroups every second, Netdata can produce meaningful +visualizations about the CPU, memory, disk, and network utilization of all running containers on the host system with +zero configuration. + +Netdata also collects metrics from applications running inside of Docker containers. For example, if you create a MySQL +database container using `docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag`, it exposes +metrics on port 3306. You can configure the [MySQL +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/mysql) to look at `127.0.0.0:3306` for +MySQL metrics: + +```yml +jobs: + - name: local + dsn: root:my-secret-pw@tcp(127.0.0.1:3306)/ +``` + +Netdata then collects metrics from the container itself, but also dozens [MySQL-specific +metrics](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/mysql#charts) as well. + +### Collect metrics from applications running in Docker containers + +You could use this technique to monitor an entire infrastructure of Docker containers. The same [enable and +configure](/docs/collect/enable-configure.md) procedures apply whether an application runs on the host system or inside +a container. You may need to configure the target endpoint if it's not the application's default. + +Netdata can even [run in a Docker container](/packaging/docker/README.md) itself, and then collect metrics about the +host system, its own container with cgroups, and any applications you want to monitor. + +See our [application metrics doc](/docs/collect/application-metrics.md) for details about Netdata's application metrics +collection capabilities. + +## Collect Kubernetes metrics + +We already have a few complementary tools and collectors for monitoring the many layers of a Kubernetes cluster, +_entirely for free_. These methods work together to help you troubleshoot performance or availability issues across +your k8s infrastructure. + +- A [Helm chart](https://github.com/netdata/helmchart), which bootstraps a Netdata Agent pod on every node in your + cluster, plus an additional parent pod for storing metrics and managing alarm notifications. +- A [service discovery plugin](https://github.com/netdata/agent-service-discovery), which discovers and creates + configuration files for [compatible + applications](https://github.com/netdata/helmchart#service-discovery-and-supported-services) and any endpoints + covered by our [generic Prometheus + collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/prometheus). With these + configuration files, Netdata collects metrics from any compatible applications as they run _inside_ of a pod. + Service discovery happens without manual intervention as pods are created, destroyed, or moved between nodes. +- A [Kubelet collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/k8s_kubelet), which runs + on each node in a k8s cluster to monitor the number of pods/containers, the volume of operations on each container, + and more. +- A [kube-proxy collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/k8s_kubeproxy), which + also runs on each node and monitors latency and the volume of HTTP requests to the proxy. +- A [cgroups collector](/collectors/cgroups.plugin/README.md), which collects CPU, memory, and bandwidth metrics for + each container running on your k8s cluster. + +For a holistic view of Netdata's Kubernetes monitoring capabilities, see our guide: [_Monitor a Kubernetes (k8s) cluster +with Netdata_](https://learn.netdata.cloud/guides/monitor/kubernetes-k8s-netdata). + +## What's next? + +Netdata is capable of collecting metrics from hundreds of applications, such as web servers, databases, messaging +brokers, and more. See more in the [application metrics doc](/docs/collect/application-metrics.md). + +If you already have all the information you need about collecting metrics, move into Netdata's meaningful visualizations +with [seeing an overview of your infrastructure](/docs/visualize/overview-infrastructure.md) using Netdata Cloud. + + diff --git a/docs/collect/enable-configure.md b/docs/collect/enable-configure.md new file mode 100644 index 0000000..19e680c --- /dev/null +++ b/docs/collect/enable-configure.md @@ -0,0 +1,68 @@ + + +# Enable or configure a collector + +When Netdata starts up, each collector searches for exposed metrics on the default endpoint established by that service +or application's standard installation procedure. For example, the [Nginx +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/nginx) searches at +`http://127.0.0.1/stub_status` for exposed metrics in the correct format. If an Nginx web server is running and exposes +metrics on that endpoint, the collector begins gathering them. + +However, not every node or infrastructure uses standard ports, paths, files, or naming conventions. You may need to +enable or configure a collector to gather all available metrics from your systems, containers, or applications. + +## Enable a collector or its orchestrator + +You can enable/disable collectors individually, or enable/disable entire orchestrators, using their configuration files. +For example, you can change the behavior of the Go orchestrator, or any of its collectors, by editing `go.d.conf`. + +Use `edit-config` from your [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory) to open +the orchestrator primary configuration file: + +```bash +cd /etc/netdata +sudo ./edit-config go.d.conf +``` + +Within this file, you can either disable the orchestrator entirely (`enabled: yes`), or find a specific collector and +enable/disable it with `yes` and `no` settings. Uncomment any line you change to ensure the Netdata daemon reads it on +start. + +After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +## Configure a collector + +First, [find the collector](/collectors/COLLECTORS.md) you want to edit and open its documentation. Some software has +collectors written in multiple languages. In these cases, you should always pick the collector written in Go. + +Use `edit-config` from your [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory) to open a +collector's configuration file. For example, edit the Nginx collector with the following: + +```bash +./edit-config go.d/nginx.conf +``` + +Each configuration file describes every available option and offers examples to help you tweak Netdata's settings +according to your needs. In addition, every collector's documentation shows the exact command you need to run to +configure that collector. Uncomment any line you change to ensure the collector's orchestrator or the Netdata daemon +read it on start. + +After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +## What's next? + +Read high-level overviews on how Netdata collects [system metrics](/docs/collect/system-metrics.md), [container +metrics](/docs/collect/container-metrics.md), and [application metrics](/docs/collect/application-metrics.md). + +If you're already collecting all metrics from your systems, containers, and applications, it's time to move into +Netdata's visualization features. [See an overview of your infrastructure](/docs/visualize/overview-infrastructure.md) +using Netdata Cloud, or learn how to [interact with dashboards and +charts](/docs/visualize/interact-dashboards-charts.md). + + diff --git a/docs/collect/how-collectors-work.md b/docs/collect/how-collectors-work.md new file mode 100644 index 0000000..07e3485 --- /dev/null +++ b/docs/collect/how-collectors-work.md @@ -0,0 +1,78 @@ + + +# How Netdata's metrics collectors work + +When Netdata starts, and with zero configuration, it auto-detects thousands of data sources and immediately collects +per-second metrics. + +Netdata can immediately collect metrics from these endpoints thanks to 300+ **collectors**, which all come pre-installed +when you [install Netdata](/docs/get-started.mdx). + +Every collector has two primary jobs: + +- Look for exposed metrics at a pre- or user-defined endpoint. +- Gather exposed metrics and use additional logic to build meaningful, interactive visualizations. + +If the collector finds compatible metrics exposed on the configured endpoint, it begins a per-second collection job. The +Netdata Agent gathers these metrics, sends them to the [database engine for +storage](/docs/store/change-metrics-storage.md), and immediately [visualizes them +meaningfully](/docs/visualize/interact-dashboards-charts.md) on dashboards. + +Each collector comes with a pre-defined configuration that matches the default setup for that application. This endpoint +can be a URL and port, a socket, a file, a web page, and more. + +For example, the [Nginx collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/nginx) searches +at `http://127.0.0.1/stub_status`, which is the default endpoint for exposing Nginx metrics. The [web log collector for +Nginx or Apache](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog) searches at +`/var/log/nginx/access.log` and `/var/log/apache2/access.log`, respectively, both of which are standard locations for +access log files on Linux systems. + +The endpoint is user-configurable, as are many other specifics of what a given collector does. + +## What can Netdata collect? + +To quickly find your answer, see our [list of supported collectors](/collectors/COLLECTORS.md). + +Generally, Netdata's collectors can be grouped into three types: + +- [Systems](/docs/collect/system-metrics.md): Monitor CPU, memory, disk, networking, systemd, eBPF, and much more. + Every metric exposed by `/proc`, `/sys`, and other Linux kernel sources. +- [Containers](/docs/collect/container-metrics.md): Gather metrics from container agents, like `dockerd` or `kubectl`, + along with the resource usage of containers and the applications they run. +- [Applications](/docs/collect/application-metrics.md): Collect per-second metrics from web servers, databases, logs, + message brokers, APM tools, email servers, and much more. + +## Collector architecture and terminology + +**Collector** is a catch-all term for any Netdata process that gathers metrics from an endpoint. + +While we use _collector_ most often in documentation, release notes, and educational content, you may encounter other +terms related to collecting metrics. + +- **Modules** are a type of collector. +- **Orchestrators** are external plugins that run and manage one or more modules. They run as independent processes. + The Go orchestrator is in active development. + - [go.d.plugin](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/): An orchestrator for data + collection modules written in `go`. + - [python.d.plugin](/collectors/python.d.plugin/README.md): An orchestrator for data collection modules written in + `python` v2/v3. + - [charts.d.plugin](/collectors/charts.d.plugin/README.md): An orchestrator for data collection modules written in + `bash` v4+. +- **External plugins** gather metrics from external processes, such as a webserver or database, and run as independent + processes that communicate with the Netdata daemon via pipes. +- **Internal plugins** gather metrics from `/proc`, `/sys`, and other Linux kernel sources. They are written in `C`, + and run as threads within the Netdata daemon. + +## What's next? + +[Enable or configure a collector](/docs/collect/enable-configure.md) if the default settings are not compatible with +your infrastructure. + +See our [collectors reference](/collectors/REFERENCE.md) for detailed information on Netdata's collector architecture, +troubleshooting a collector, developing a custom collector, and more. + + diff --git a/docs/collect/system-metrics.md b/docs/collect/system-metrics.md new file mode 100644 index 0000000..ecd8dad --- /dev/null +++ b/docs/collect/system-metrics.md @@ -0,0 +1,60 @@ + + +# Collect system metrics with Netdata + +Netdata collects thousands of metrics directly from the operating systems of physical and virtual systems, IoT/edge +devices, and [containers](/docs/collect/container-metrics.md) with zero configuration. + +To gather system metrics, Netdata uses roughly a dozen plugins, each of which has one or more collectors for very +specific metrics exposed by the host. The system metrics Netdata users interact with most for health monitoring and +performance troubleshooting are collected and visualized by `proc.plugin`, `cgroups.plugin`, and `ebpf.plugin`. + +[**proc.plugin**](/collectors/proc.plugin/README.md) gathers metrics from the `/proc` and `/sys` folders in Linux +systems, along with a few other endpoints, and is responsible for the bulk of the system metrics collected and +visualized by Netdata. It collects CPU, memory, disks, load, networking, mount points, and more with zero configuration. +It even allows Netdata to monitor its own resource utilization! + +[**cgroups.plugin**](/collectors/cgroups.plugin/README.md) collects rich metrics about containers and virtual machines +using the virtual files under `/sys/fs/cgroup`. By reading cgroups, Netdata can instantly collect resource utilization +metrics for systemd services, all containers (Docker, LXC, LXD, Libvirt, systemd-nspawn), and more. Learn more in the +[collecting container metrics](/docs/collect/container-metrics.md) doc. + +[**ebpf.plugin**](/collectors/ebpf.plugin/README.md): Netdata's extended Berkeley Packet Filter (eBPF) collector +monitors Linux kernel-level metrics for file descriptors, virtual filesystem IO, and process management. You can use our +eBPF collector to analyze how and when a process accesses files, when it makes system calls, whether it leaks memory or +creating zombie processes, and more. + +While the above plugins and associated collectors are the most important for system metrics, there are many others. You +can find all system collectors in our [supported collectors list](/collectors/COLLECTORS.md#system-collectors). + +## Collect Windows system metrics + +Netdata is also capable of monitoring Windows systems. The [WMI +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/wmi) integrates with +[windows_exporter](https://github.com/prometheus-community/windows_exporter), a small Go-based binary that you can run +on Windows systems. The WMI collector then gathers metrics from an endpoint created by windows_exporter, for more +details see [the requirements](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/wmi#requirements). + +Next, [configure the WMI +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/wmi#configuration) to point to the URL +and port of your exposed endpoint. Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. You'll start seeing Windows system metrics, such as CPU +utilization, memory, bandwidth per NIC, number of processes, and much more. + +For information about collecting metrics from applications _running on Windows systems_, see the [application metrics +doc](/docs/collect/application-metrics.md#collect-metrics-from-applications-running-on-windows). + +## What's next? + +Because there's some overlap between system metrics and [container metrics](/docs/collect/container-metrics.md), you +should investigate Netdata's container compatibility if you use them heavily in your infrastructure. + +If you don't use containers, skip ahead to collecting [application metrics](/docs/collect/application-metrics.md) with +Netdata. + + diff --git a/docs/configure/common-changes.md b/docs/configure/common-changes.md new file mode 100644 index 0000000..93b12d2 --- /dev/null +++ b/docs/configure/common-changes.md @@ -0,0 +1,214 @@ + + +# Common configuration changes + +The Netdata Agent requires no configuration upon installation to collect thousands of per-second metrics from most +systems, containers, and applications, but there are hundreds of settings to tweak if you want to exercise more control +over your monitoring platform. + +This document assumes familiarity with using [`edit-config`](/docs/configure/nodes.md) from the Netdata config +directory. + +## Change dashboards and visualizations + +The Netdata Agent's [local dashboard](/web/gui/README.md), accessible at `http://NODE:19999` is highly configurable. If +you use Netdata Cloud for [infrastructure monitoring](/docs/quickstart/infrastructure.md), you will see many of these +changes reflected in those visualizations due to the way Netdata Cloud proxies metric data and metadata to your browser. + +### Increase the long-term metrics retention period + +Increase the values for the `page cache size` and `dbengine multihost disk space` settings in the [`[global]` +section](/daemon/config/README.md#global-section-options) of `netdata.conf`. + +```conf +[global] + page cache size = 128 # 128 MiB of memory for metrics storage + dbengine multihost disk space = 4096 # 4GiB of disk space for metrics storage +``` + +Read our doc on [increasing long-term metrics storage](/docs/store/change-metrics-storage.md) for details, including a +[calculator](/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics) +to help you determine the exact settings for your desired retention period. + +### Reduce the data collection frequency + +Change `update every` in the [`[global]` section](/daemon/config/README.md#global-section-options) of `netdata.conf` so +that it is greater than `1`. An `update every` of `5` means the Netdata Agent enforces a _minimum_ collection frequency +of 5 seconds. + +```conf +[global] + update every = 5 +``` + +Every collector and plugin has its own `update every` setting, which you can also change in the `go.d.conf`, +`python.d.conf` or `charts.d.conf` files, or in individual collector configuration files. If the `update +every` for an individual collector is less than the global, the Netdata Agent uses the global setting. See the [enable +or configure a collector](/docs/collect/enable-configure.md) doc for details. + +### Disable a collector or plugin + +Turn off entire plugins in the [`[plugins]` section](/daemon/config/README.md#plugins-section-options) of +`netdata.conf`. + +To disable specific collectors, open `go.d.conf`, `python.d.conf` or `charts.d.conf` and find the line +for that specific module. Uncomment the line and change its value to `no`. + +## Modify alarms and notifications + +Netdata's health monitoring watchdog uses hundreds of preconfigured health entities, with intelligent thresholds, to +generate warning and critical alarms for most production systems and their applications without configuration. However, +each alarm and notification method is completely customizable. + +### Add a new alarm + +To create a new alarm configuration file, initiate an empty file, with a filename that ends in `.conf`, in the +`health.d/` directory. The Netdata Agent loads any valid alarm configuration file ending in `.conf` in that directory. +Next, edit the new file with `edit-config`. For example, with a file called `example-alarm.conf`. + +```bash +sudo touch health.d/example-alarm.conf +sudo ./edit-config health.d/example-alarm.conf +``` + +Or, append your new alarm to an existing file by editing a relevant existing file in the `health.d/` directory. + +Read more about [configuring alarms](/docs/monitor/configure-alarms.md) to get started, and see the [health monitoring +reference](/health/REFERENCE.md) for a full listing of options available in health entities. + +### Configure a specific alarm + +Tweak existing alarms by editing files in the `health.d/` directory. For example, edit `health.d/cpu.conf` to change how +the Agent responds to anomalies related to CPU utilization. + +To see which configuration file you need to edit to configure a specific alarm, [view your active +alarms](/docs/monitor/view-active-alarms.md) in Netdata Cloud or the local Agent dashboard and look for the **source** +line. For example, it might read `source 4@/usr/lib/netdata/conf.d/health.d/cpu.conf`. + +Because the source path contains `health.d/cpu.conf`, run `sudo edit-config health.d/cpu.conf` to configure that alarm. + +### Disable a specific alarm + +Open the configuration file for that alarm and set the `to` line to `silent`. + +```conf +template: disk_fill_rate + on: disk.space + lookup: max -1s at -30m unaligned of avail + calc: ($this - $avail) / (30 * 60) + every: 15s + to: silent +``` + +### Turn of all alarms and notifications + +Set `enabled` to `no` in the [`[health]` section](/daemon/config/README.md#health-section-options) section of +`netdata.conf`. + +### Enable alarm notifications + +Open `health_alarm_notify.conf` for editing. First, read the [enabling +notifications](/docs/monitor/enable-notifications.md#netdata-agent) doc for an example of the process using Slack, then +click on the link to your preferred notification method to find documentation for that specific endpoint. + +## Improve node security + +While the Netdata Agent is both [open and secure by design](https://www.netdata.cloud/blog/netdata-agent-dashboard/), we +recommend every user take some action to administer and secure their nodes. + +Learn more about a few of the following changes in the [node security doc](/docs/configure/secure-nodes.md). + +### Disable the local Agent dashboard (`http://NODE:19999`) + +If you use Netdata Cloud to visualize metrics, stream metrics to a parent node, or otherwise don't need the local Agent +dashboard, disabling it reduces the Agent's resource utilization and improves security. + +Change the `mode` setting to `none` in the [`[web]` section](/web/server/README.md#configuration) of `netdata.conf`. + +```conf +[web] + mode = none +``` + +### Use access lists to restrict access to specific assets + +Allow access from only specific IP addresses, ranges of IP addresses, or hostnames using [access +lists](/web/server/README.md#access-lists) and [simple patterns](/libnetdata/simple_pattern/README.md). + +See a quickstart to access lists in the [node security +doc](/docs/configure/secure-nodes.md#restrict-access-to-the-local-dashboard). + +### Stop sending anonymous statistics to Google Analytics + +Create a file called `.opt-out-from-anonymous-statistics` inside of your Netdata config directory to immediately stop +the statistics script. + +```bash +sudo touch .opt-out-from-anonymous-statistics +``` + +Learn more about [why we collect anonymous statistics](/docs/anonymous-statistics.md). + +### Change the IP address/port Netdata listens to + +Change the `default port` setting in the `[web]` section to a port other than `19999`. + +```conf +[web] + default port = 39999 +``` + +Use the `bind to` setting to the ports other assets, such as the [running `netdata.conf` +configuration](/docs/configure/nodes.md#see-an-agents-running-configuration), API, or streaming requests listen to. + +## Reduce resource usage + +Read our [performance optimization guide](/docs/guides/configure/performance.md) for a long list of specific changes +that can reduce the Netdata Agent's CPU/memory footprint and IO requirements. + +## Organize nodes with host labels + +Beginning with v1.20, Netdata accepts user-defined **host labels**. These labels are sent during streaming, exporting, +and as metadata to Netdata Cloud, and help you organize the metrics coming from complex infrastructure. Host labels are +defined in the section `[host labels]`. + +For a quick introduction, read the [host label guide](/docs/guides/using-host-labels.md). + +The following restrictions apply to host label names: + +- Names cannot start with `_`, but it can be present in other parts of the name. +- Names only accept alphabet letters, numbers, dots, and dashes. + +The policy for values is more flexible, but you can not use exclamation marks (`!`), whitespaces (` `), single quotes +(`'`), double quotes (`"`), or asterisks (`*`), because they are used to compare label values in health alarms and +templates. + +## What's next? + +If you haven't already, learn how to [secure your nodes](/docs/configure/secure-nodes.md). + +As mentioned at the top, there are plenty of other + +You can also take what you've learned about node configuration to tweak the Agent's behavior or enable new features: + +- [Enable new collectors](/docs/collect/enable-configure.md) or tweak their behavior. +- [Configure existing health alarms](/docs/monitor/configure-alarms.md) or create new ones. +- [Enable notifications](/docs/monitor/enable-notifications.md) to receive updates about the health of your + infrastructure. +- Change [the long-term metrics retention period](/docs/store/change-metrics-storage.md) using the database engine. + +### Related reference documentation + +- [Netdata Agent · Daemon](/health/README.md) +- [Netdata Agent · Daemon configuration](/daemon/config/README.md) +- [Netdata Agent · Web server](/web/server/README.md) +- [Netdata Agent · Local Agent dashboard](/web/gui/README.md) +- [Netdata Agent · Health monitoring](/health/REFERENCE.md) +- [Netdata Agent · Notifications](/health/notifications/README.md) +- [Netdata Agent · Simple patterns](/libnetdata/simple_pattern/README.md) + +[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fconfigure%2Fcommon-changes&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) diff --git a/docs/configure/nodes.md b/docs/configure/nodes.md new file mode 100644 index 0000000..841419a --- /dev/null +++ b/docs/configure/nodes.md @@ -0,0 +1,170 @@ + + +# Configure the Netdata Agent + +Netdata's zero-configuration collection, storage, and visualization features work for many users, infrastructures, and +use cases, but there are some situations where you might want to configure the Netdata Agent running on your node(s), +which can be a physical or virtual machine (VM), container, cloud deployment, or edge/IoT device. + +For example, you might want to increase metrics retention, configure a collector based on your infrastructure's unique +setup, or secure the local dashboard by restricting it to only connections from `localhost`. + +Whatever the reason, Netdata users should know how to configure individual nodes to act decisively if an incident, +anomaly, or change in infrastructure affects how their Agents should perform. + +## The Netdata config directory + +On most Linux systems, using our [recommended one-line +installation](/docs/get-started.mdx#install-on-linux-with-one-line-installer), the **Netdata config +directory** is `/etc/netdata/`. The config directory contains several configuration files with the `.conf` extension, a +few directories, and a shell script named `edit-config`. + +> Some operating systems will use `/opt/netdata/etc/netdata/` as the config directory. If you're not sure where yours +> is, navigate to `http://NODE:19999/netdata.conf` in your browser, replacing `NODE` with the IP address or hostname of +> your node, and find the `# config directory = ` setting. The value listed is the config directory for your system. + +All of Netdata's documentation assumes that your config directory is at `/etc/netdata`, and that you're running any +scripts from inside that directory. + +## Netdata's configuration files + +Upon installation, the Netdata config directory contains a few files and directories. It's okay if you don't see all +these files in your own Netdata config directory, as the next section describes how to edit any that might not already +exist. + +- `netdata.conf` is the main configuration file. This is where you'll find most configuration options. Read descriptions + for each in the [daemon config](/daemon/config/README.md) doc. +- `edit-config` is a shell script used for [editing configuration files](#use-edit-config-to-edit-configuration-files). +- Various configuration files ending in `.conf` for [configuring plugins or + collectors](/docs/collect/enable-configure.md#enable-a-collector-or-its-orchestrator) behave. Examples: `go.d.conf`, + `python.d.conf`, and `ebpf.d.conf`. +- Various directories ending in `.d`, which contain other configuration files, each ending in `.conf`, for [configuring + specific collectors](/docs/collect/enable-configure.md#configure-a-collector). +- `apps_groups.conf` is a configuration file for changing how applications/processes are grouped when viewing the + **Application** charts from [`apps.plugin`](/collectors/apps.plugin/README.md) or + [`ebpf.plugin`](/collectors/ebpf.plugin/README.md). +- `health.d/` is a directory that contains [health configuration files](/docs/monitor/configure-alarms.md). +- `health_alarm_notify.conf` enables and configures [alarm notifications](/docs/monitor/enable-notifications.md). +- `statsd.d/` is a directory for configuring Netdata's [statsd collector](/collectors/statsd.plugin/README.md). +- `stream.conf` configures [parent-child streaming](/streaming/README.md) between separate nodes running the Agent. +- `.environment` is a hidden file that describes the environment in which the Netdata Agent is installed, including the + `PATH` and any installation options. Useful for [reinstalling](/packaging/installer/REINSTALL.md) or + [uninstalling](/packaging/installer/UNINSTALL.md) the Agent. + +The Netdata config directory also contains one symlink: + +- `orig` is a symbolic link to the directory `/usr/lib/netdata/conf.d`, which contains stock configuration files. Stock + versions are copied into the config directory when opened with `edit-config`. _Do not edit the files in + `/usr/lib/netdata/conf.d`, as they are overwritten by updates to the Netdata Agent._ + +## Configure a Netdata docker container + +See [configure agent containers](/packaging/docker/README.md#configure-agent-containers). + +## Use `edit-config` to edit configuration files + +The **recommended way to easily and safely edit Netdata's configuration** is with the `edit-config` script. This script +opens existing Netdata configuration files using your system's `$EDITOR`. If the file doesn't yet exist in your config +directory, the script copies the stock version from `/usr/lib/netdata/conf.d` and opens it for editing. + +Run `edit-config` without any options to see details on its usage and a list of all the configuration files you can +edit. + +```bash +./edit-config +USAGE: + ./edit-config FILENAME + + Copy and edit the stock config file named: FILENAME + if FILENAME is already copied, it will be edited as-is. + + The EDITOR shell variable is used to define the editor to be used. + + Stock config files at: '/usr/lib/netdata/conf.d' + User config files at: '/etc/netdata' + + Available files in '/usr/lib/netdata/conf.d' to copy and edit: + +./apps_groups.conf ./health.d/phpfpm.conf +./aws_kinesis.conf ./health.d/pihole.conf +./charts.d/ap.conf ./health.d/portcheck.conf +./charts.d/apcupsd.conf ./health.d/postgres.conf +... +``` + +To edit `netdata.conf`, run `./edit-config netdata.conf`. You may need to elevate your privileges with `sudo` or another +method for `edit-config` to write into the config directory. Use your `$EDITOR`, make your changes, and save the file. + +> `edit-config` uses the `EDITOR` environment variable on your system to edit the file. On many systems, that is +> defaulted to `vim` or `nano`. Use `export EDITOR=` to change this temporarily, or edit your shell configuration file +> to change to permanently. + +After you make your changes, you need to [restart the Agent](/docs/configure/start-stop-restart.md) with `sudo systemctl +restart netdata` or the appropriate method for your system. + +Here's an example of editing the node's hostname, which appears in both the local dashboard and in Netdata Cloud. + +![Animated GIF of editing the hostname option in +netdata.conf](https://user-images.githubusercontent.com/1153921/80994808-1c065300-8df2-11ea-81af-d28dc3ba27c8.gif) + +### Other configuration files + +You can edit any Netdata configuration file using `edit-config`. A few examples: + +```bash +./edit-config apps_groups.conf +./edit-config ebpf.d.conf +./edit-config health.d/load.conf +./edit-config go.d/prometheus.conf +``` + +The documentation for each of Netdata's components explains which file(s) to edit to achieve the desired behavior. + +## See an Agent's running configuration + +On start, the Netdata Agent daemon attempts to load `netdata.conf`. If that file is missing, incomplete, or contains +invalid settings, the daemon attempts to run sane defaults instead. In other words, the state of `netdata.conf` on your +filesystem may be different from the state of the Netdata Agent itself. + +To see the _running configuration_, navigate to `http://NODE:19999/netdata.conf` in your browser, replacing `NODE` with +the IP address or hostname of your node. The file displayed here is exactly the settings running live in the Netdata +Agent. + +If you're having issues with configuring the Agent, apply the running configuration to `netdata.conf` by downloading the +file to the Netdata config directory. Use `sudo` to elevate privileges. + +```bash +wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf +# or +curl -o /etc/netdata/netdata.conf http://NODE:19999/netdata.conf +``` + +## What's next? + +Learn more about [starting, stopping, or restarting](/docs/configure/start-stop-restart.md) the Netdata daemon to apply +configuration changes. + +Apply some [common configuration changes](/docs/configure/common-changes.md) to quickly tweak the Agent's behavior. + +[Add security to your node](/docs/configure/secure-nodes.md) with what you've learned about the Netdata config directory +and `edit-config`. We put together a few security best practices based on how you use the Netdata. + +You can also take what you've learned about node configuration to enable or enhance features: + +- [Enable new collectors](/docs/collect/enable-configure.md) or tweak their behavior. +- [Configure existing health alarms](/docs/monitor/configure-alarms.md) or create new ones. +- [Enable notifications](/docs/monitor/enable-notifications.md) to receive updates about the health of your + infrastructure. +- Change [the long-term metrics retention period](/docs/store/change-metrics-storage.md) using the database engine. + +### Related reference documentation + +- [Netdata Agent · Daemon](/daemon/README.md) +- [Netdata Agent · Health monitoring](/health/README.md) +- [Netdata Agent · Notifications](/health/notifications/README.md) + +[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fconfigure%2Fnodes&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) diff --git a/docs/configure/secure-nodes.md b/docs/configure/secure-nodes.md new file mode 100644 index 0000000..02057ab --- /dev/null +++ b/docs/configure/secure-nodes.md @@ -0,0 +1,123 @@ + + +# Secure your nodes + +Upon installation, the Netdata Agent serves the **local dashboard** at port `19999`. If the node is accessible to the +internet at large, anyone can access the dashboard and your node's metrics at `http://NODE:19999`. We made this decision +so that the local dashboard was immediately accessible to users, and so that we don't dictate how professionals set up +and secure their infrastructures. + +Despite this design decision, your [data](/docs/netdata-security.md#your-data-is-safe-with-netdata) and your +[systems](/docs/netdata-security.md#your-systems-are-safe-with-netdata) are safe with Netdata. Netdata is read-only, +cannot do anything other than present metrics, and runs without special/`sudo` privileges. Also, the local dashboard +only exposes chart metadata and metric values, not raw data. + +While Netdata is secure by design, we believe you should [protect your +nodes](/docs/netdata-security.md#why-netdata-should-be-protected). If left accessible to the internet at large, the +local dashboard could reveal sensitive information about your infrastructure. For example, an attacker can view which +applications you run (databases, webservers, and so on), or see every user account on a node. + +Instead of dictating how to secure your infrastructure, we give you many options to establish security best practices +that align with your goals and your organization's standards. + +- [Disable the local dashboard](#disable-the-local-dashboard): **Simplest and recommended method** for those who have + added nodes to Netdata Cloud and view dashboards and metrics there. +- [Restrict access to the local dashboard](#restrict-access-to-the-local-dashboard): Allow local dashboard access from + only certain IP addresses, such as a trusted static IP or connections from behind a management LAN. Full support for + Netdata Cloud. +- [Use a reverse proxy](#use-a-reverse-proxy): Password-protect a local dashboard and enable TLS to secure it. Full + support for Netdata Cloud. + +## Disable the local dashboard + +This is the _recommended method for those who have connected their nodes to Netdata Cloud_ and prefer viewing real-time +metrics using the War Room Overview, Nodes view, and Cloud dashboards. + +You can disable the local dashboard (and API) but retain the encrypted Agent-Cloud link ([ACLK](/aclk/README.md)) that +allows you to stream metrics on demand from your nodes via the Netdata Cloud interface. This change mitigates all +concerns about revealing metrics and system design to the internet at large, while keeping all the functionality you +need to view metrics and troubleshoot issues with Netdata Cloud. + +Open `netdata.conf` with `./edit-config netdata.conf`. Scroll down to the `[web]` section, and find the `mode = +static-threaded` setting, and change it to `none`. + +```conf +[web] + mode = none +``` + +Save and close the editor, then [restart your Agent](/docs/configure/start-stop-restart.md) using `sudo systemctl +restart netdata`. If you try to visit the local dashboard to `http://NODE:19999` again, the connection will fail because +that node no longer serves its local dashboard. + +> See the [configuration basics doc](/docs/configure/nodes.md) for details on how to find `netdata.conf` and use +> `edit-config`. + +## Restrict access to the local dashboard + +If you want to keep using the local dashboard, but don't want it exposed to the internet, you can restrict access with +[access lists](/web/server/README.md#access-lists). This method also fully retains the ability to stream metrics +on-demand through Netdata Cloud. + +The `allow connections from` setting helps you allow only certain IP addresses or FQDN/hostnames, such as a trusted +static IP, only `localhost`, or connections from behind a management LAN. + +By default, this setting is `localhost *`. This setting allows connections from `localhost` in addition to _all_ +connections, using the `*` wildcard. You can change this setting using Netdata's [simple +patterns](/libnetdata/simple_pattern/README.md). + +```conf +[web] + # Allow only localhost connections + allow connections from = localhost + + # Allow only from management LAN running on `10.X.X.X` + allow connections from = 10.* + + # Allow connections only from a specific FQDN/hostname + allow connections from = example* +``` + +The `allow connections from` setting is global and restricts access to the dashboard, badges, streaming, API, and +`netdata.conf`, but you can also set each of those access lists more granularly if you choose: + +```conf +[web] + allow connections from = localhost * + allow dashboard from = localhost * + allow badges from = * + allow streaming from = * + allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.* + allow management from = localhost +``` + +See the [web server](/web/server/README.md#access-lists) docs for additional details about access lists. You can take +access lists one step further by [enabling SSL](/web/server/README.md#enabling-tls-support) to encrypt data from local +dashboard in transit. The connection to Netdata Cloud is always secured with TLS. + +## Use a reverse proxy + +You can also put Netdata behind a reverse proxy for additional security while retaining the functionality of both the +local dashboard and Netdata Cloud dashboards. You can use a reverse proxy to password-protect the local dashboard and +enable HTTPS to encrypt metadata and metric values in transit. + +We recommend Nginx, as it's what we use for our [demo server](https://london.my-netdata.io/), and we have a guide +dedicated to [running Netdata behind Nginx](/docs/Running-behind-nginx.md). + +We also have guides for [Apache](/docs/Running-behind-apache.md), [Lighttpd](/docs/Running-behind-lighttpd.md), +[HAProxy](/docs/Running-behind-haproxy.md), and [Caddy](/docs/Running-behind-caddy.md). + +## What's next? + +Read about [Netdata's security design](/docs/netdata-security.md) and our [blog +post](https://www.netdata.cloud/blog/netdata-agent-dashboard/) about why the local Agent dashboard is both open and +secure by design. + +Next up, learn about [collectors](/docs/collect/how-collectors-work.md) to ensure you're gathering every essential +metric about your node, its applications, and your infrastructure at large. + +[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fconfigure%2Fsecure-nodesa&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) diff --git a/docs/configure/start-stop-restart.md b/docs/configure/start-stop-restart.md new file mode 100644 index 0000000..4967fff --- /dev/null +++ b/docs/configure/start-stop-restart.md @@ -0,0 +1,98 @@ + + +# Start, stop, or restart the Netdata Agent + +When you install the Netdata Agent, the [daemon](/daemon/README.md) is configured to start at boot and stop and +restart/shutdown. + +You will most often need to _restart_ the Agent to load new or editing configuration files. [Health +configuration](#reload-health-configuration) files are the only exception, as they can be reloaded without restarting +the entire Agent. + +Stopping or restarting the Netdata Agent will cause gaps in stored metrics until the `netdata` process initiates +collectors and the database engine. + +## Using `systemctl`, `service`, or `init.d` + +This is the recommended way to start, stop, or restart the Netdata daemon. + +- To **start** Netdata, run `sudo systemctl start netdata`. +- To **stop** Netdata, run `sudo systemctl stop netdata`. +- To **restart** Netdata, run `sudo systemctl restart netdata`. + +If the above commands fail, or you know that you're using a non-systemd system, try using the `service` command: + +- **service**: `sudo service netdata start`, `sudo service netdata stop`, `sudo service netdata restart` + +## Using `netdata` + +Use the `netdata` command, typically located at `/usr/sbin/netdata`, to start the Netdata daemon. + +```bash +sudo netdata +``` + +If you start the daemon this way, close it with `sudo killall netdata`. + +## Using `netdatacli` + +The Netdata Agent also comes with a [CLI tool](/cli/README.md) capable of performing shutdowns. Start the Agent back up +using your preferred method listed above. + +```bash +sudo netdatacli shutdown-agent +``` + +## Reload health configuration + +You do not need to restart the Netdata Agent between changes to health configuration files, such as specific health +entities. Instead, use [`netdatacli`](#using-netdatacli) and the `reload-health` option to prevent gaps in metrics +collection. + +```bash +sudo netdatacli reload-health +``` + +If `netdatacli` doesn't work on your system, send a `SIGUSR2` signal to the daemon, which reloads health configuration +without restarting the entire process. + +```bash +killall -USR2 netdata +``` + +## Force stop stalled or unresponsive `netdata` processes + +In rare cases, the Netdata Agent may stall or not properly close sockets, preventing a new process from starting. In +these cases, try the following three commands: + +```bash +sudo systemctl stop netdata +sudo killall netdata +ps aux| grep netdata +``` + +The output of `ps aux` should show no `netdata` or associated processes running. You can now start the Netdata Agent +again with `service netdata start`, or the appropriate method for your system. + +## What's next? + +Learn more about [securing the Netdata Agent](/docs/configure/secure-nodes.md). + +You can also use the restart/reload methods described above to enable new features: + +- [Enable new collectors](/docs/collect/enable-configure.md) or tweak their behavior. +- [Configure existing health alarms](/docs/monitor/configure-alarms.md) or create new ones. +- [Enable notifications](/docs/monitor/enable-notifications.md) to receive updates about the health of your + infrastructure. +- Change [the long-term metrics retention period](/docs/store/change-metrics-storage.md) using the database engine. + +### Related reference documentation + +- [Netdata Agent · Daemon](/daemon/README.md) +- [Netdata Agent · Netdata CLI](/cli/README.md) + +[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fconfigure%2Fstart-stop-restart&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) diff --git a/docs/contributing/contributing-documentation.md b/docs/contributing/contributing-documentation.md new file mode 100644 index 0000000..68b861d --- /dev/null +++ b/docs/contributing/contributing-documentation.md @@ -0,0 +1,109 @@ + + +# Contributing to documentation + +We welcome contributions to Netdata's already extensive documentation. + +We store documentation related to the open-source Netdata Agent inside of the [`netdata/netdata` +repository](https://github.com/netdata/netdata) on GitHub. Documentation related to Netdata Cloud is stored in a private +repository and is not currently open to community contributions. + +The Netdata team aggregates and publishes all documentation at [learn.netdata.cloud](https://learn.netdata.cloud/) using +[Docusaurus](https://v2.docusaurus.io/) in a private GitHub repository. + +## Before you get started + +Anyone interested in contributing to documentation should first read the [Netdata style +guide](/docs/contributing/style-guide.md) and the [Netdata Community Code of Conduct](https://learn.netdata.cloud/contribute/code-of-conduct). + +Netdata's documentation uses Markdown syntax. If you're not familiar with Markdown, read the [Mastering +Markdown](https://guides.github.com/features/mastering-markdown/) guide from GitHub for the basics on creating +paragraphs, styled text, lists, tables, and more. + +### Netdata's documentation structure + +Netdata's documentation is separated into four sections. + +- **Netdata**: Documents based on the actions users want to take, and solutions to their problems, such both the Netdata + Agent and Netdata Cloud. + - Stored in various subfolders of the [`/docs` folder](https://github.com/netdata/netdata/tree/master/docs) within the + `netdata/netdata` repository: `/docs/collect`, `/docs/configure`, `/docs/export`, `/docs/get`, `/docs/monitor`, + `/docs/overview`, `/docs/quickstart`, `/docs/store`, and `/docs/visualize`. + - Published at [`https://learn.netdata.cloud/docs`](https://learn.netdata.cloud/docs). +- **Netdata Agent reference**: Reference documentation for the open-source Netdata Agent. + - Stored in various `.md` files within the `netdata/netdata` repository alongside the code responsible for that + feature. For example, the database engine's reference documentation is at `/database/engine/README.md`. + - Published under the **Reference** section in the Netdata Learn sidebar. +- **Netdata Cloud reference**: Reference documentation for the closed-source Netdata Cloud web application. + - Stored in a private GitHub repository and not editable by the community. + - Published at [`https://learn.netdata.cloud/docs/cloud`](https://learn.netdata.cloud/docs/cloud). +- **Guides**: Solutions-based articles for users who want instructions on completing a specific complex task using the + Netdata Agent and/or Netdata Cloud. + - Stored in the [`/docs/guides` folder](https://github.com/netdata/netdata/tree/master/docs/guides) within the + `netdata/netdata` repository. Organized into subfolders that roughly correlate with the core Netdata documentation. + - Published at [`https://learn.netdata.cloud/guides`](https://learn.netdata.cloud/guides). + +Generally speaking, if you want to contribute to the reference documentation for a specific Netdata Agent feature, find +the appropriate `.md` file co-located with that feature. If you want to contribute documentation that spans features or +products, or has no direct correlation with the existing directory structure, place it in the `/docs` folder within +`netdata/netdata`. + +## How to contribute + +The easiest way to contribute to Netdata's documentation is to edit a file directly on GitHub. This is perfect for small +fixes to a single document, such as fixing a typo or clarifying a confusing sentence. + +Click on the **Edit this page** button on any published document on [Netdata Learn](https://learn.netdata.cloud). Each +page has two of these buttons: One beneath the table of contents, and another at the end of the document, which take you +to GitHub's code editor. Make your suggested changes, keeping [Netdata style guide](/docs/contributing/style-guide.md) +in mind, and use *Preview changes** button to ensure your Markdown syntax works as expected. + +Under the **Commit changes** header, write descriptive title for your requested change. Click the **Commit changes** +button to initiate your pull request (PR). + +Jump down to our instructions on [PRs](#making-a-pull-request) for your next steps. + +### Edit locally + +Editing documentation locally is the preferred method for complex changes that span multiple documents or change the +documentation's style or structure. + +Create a fork of the Netdata Agent repository by visit the [Netdata repository](https://github.com/netdata/netdata) and +clicking on the **Fork** button. + +![Screenshot of forking the Netdata +repository](https://user-images.githubusercontent.com/1153921/59873572-25f5a380-9351-11e9-92a4-a681fe4a2ed9.png) + +GitHub will ask you where you want to clone the repository. When finished, you end up at the index of your forked +Netdata Agent repository. Clone your fork to your local machine: + +```bash +git clone https://github.com/YOUR-GITHUB-USERNAME/netdata.git +``` + +Create a new branch using `git checkout -b BRANCH-NAME`. Use your favorite text editor to make your changes, keeping the +[Netdata style guide](/docs/contributing/style-guide.md) in mind. Add, commit, and push changes to your fork. When +you're finished, visit the [Netdata Agent Pull requests](https://github.com/netdata/netdata/pulls) to create a new pull +request based on the changes you made in the new branch of your fork. + +## Making a pull request + +Pull requests (PRs) should be concise and informative. See our [PR guidelines](https://learn.netdata.cloud/contribute/handbook#pr-guidelines) for +specifics. + +- The title must follow the [imperative mood](https://en.wikipedia.org/wiki/Imperative_mood) and be no more than ~50 + characters. +- The description should explain what was changed and why. Verify that you tested any code or processes that you are + trying to change. + +The Netdata team will review your PR and assesses it for correctness, conciseness, and overall quality. We may point to +specific sections and ask for additional information or other fixes. + +After merging your PR, the Netdata team rebuilds the [documentation site](https://learn.netdata.cloud) to publish the +changed documentation. + + diff --git a/docs/contributing/style-guide.md b/docs/contributing/style-guide.md new file mode 100644 index 0000000..5ff6116 --- /dev/null +++ b/docs/contributing/style-guide.md @@ -0,0 +1,492 @@ + + +# Netdata style guide + +The _Netdata style guide_ establishes editorial guidelines for any writing produced by the Netdata team or the Netdata +community, including documentation, articles, in-product UX copy, and more. Both internal Netdata teams and external +contributors to any of Netdata's open-source projects should reference and adhere to this style guide as much as +possible. + +Netdata's writing should **empower** and **educate**. You want to help people understand Netdata's value, encourage them +to learn more, and ultimately use Netdata's products to democratize monitoring in their organizations. To achieve these +goals, your writing should be: + +- **Clear**. Use simple words and sentences. Use strong, direct, and active language that encourages readers to action. +- **Concise**. Provide solutions and answers as quickly as possible. Give users the information they need right now, + along with opportunities to learn more. +- **Universal**. Think of yourself as a guide giving a tour of Netdata's products, features, and capabilities to a + diverse group of users. Write to reach the widest possible audience. + +You can achieve these goals by reading and adhering to the principles outlined below. + +## Voice and tone + +One way we write empowering, educational content is by using a consistent voice and an appropriate tone. + +_Voice_ is like your personality, which doesn't really change day to day. + +_Tone_ is how you express your personality. Your expression changes based on your attitude or mood, or based on who +you're around. In writing, your reflect tone in your word choice, punctuation, sentence structure, or even the use of +emoji. + +The same idea about voice and tone applies to organizations, too. Our voice shouldn't change much between two pieces of +content, no matter who wrote each, but the tone might be quite different based on who we think is reading. + +For example, a [blog post](https://www.netdata.cloud/blog/) and a [press release](https://www.netdata.cloud/news/) +should have a similar voice, despite most often being written by different people. However, blog posts are relaxed and +witty, while press releases are focused and academic. You won't see any emoji in a press release. + +### Voice + +Netdata's voice is authentic, passionate, playful, and respectful. + +- **Authentic** writing is honest and fact-driven. Focus on Netdata's strength while accurately communicating what + Netdata can and cannot do, and emphasize technical accuracy over hard sells and marketing jargon. +- **Passionate** writing is strong and direct. Be a champion for the product or feature you're writing about, and let + your unique personality and writing style shine. +- **Playful** writing is friendly, thoughtful, and engaging. Don't take yourself too seriously, as long as it's not at + the expense of Netdata or any of its users. +- **Respectful** writing treats people the way you want to be treated. Prioritize giving solutions and answers as + quickly as possible. + +### Tone + +Netdata's tone is fun and playful, but clarity and conciseness comes first. We also tend to be informal, and aren't +afraid of a playful joke or two. + +While we have general standards for voice and tone, we do want every individual's unique writing style to reflect in +published content. + +## Universal communication + +Netdata is a global company in every sense, with employees, contributors, and users from around the world. We strive to +communicate in a way that is clear and easily understood by everyone. + +Here are some guidelines, pointers, and questions to be aware of as you write to ensure your writing is universal. Some +of these are expanded into individual sections in the [language, grammar, and +mechanics](#language-grammar-and-mechanics) section below. + +- Would this language make sense to someone who doesn't work here? +- Could someone quickly scan this document and understand the material? +- Create an information hierarchy with key information presented first and clearly called out to improve scannability. +- Avoid directional language like "sidebar on the right of the page" or "header at the top of the page" since + presentation elements may adapt for devices. +- Use descriptive links rather than "click here" or "learn more". +- Include alt text for images and image links. +- Ensure any information contained within a graphic element is also available as plain text. +- Avoid idioms that may not be familiar to the user or that may not make sense when translated. +- Avoid local, cultural, or historical references that may be unfamiliar to users. +- Prioritize active, direct language. +- Avoid referring to someone's age unless it is directly relevant; likewise, avoid referring to people with age-related + descriptors like "young" or "elderly." +- Avoid disability-related idioms like "lame" or "falling on deaf ears." Don't refer to a person's disability unless + it’s directly relevant to what you're writing. +- Don't call groups of people "guys." Don't call women "girls." +- Avoid gendered terms in favor of neutral alternatives, like "server" instead of "waitress" and "businessperson" + instead of "businessman." +- When writing about a person, use their communicated pronouns. When in doubt, just ask or use their name. It's OK to + use "they" as a singular pronoun. + +> Some of these guidelines were adapted from MailChimp under the Creative Commons license. + +## Language, grammar, and mechanics + +To ensure Netdata's writing is clear, concise, and universal, we have established standards for language, grammar, and +certain writing mechanics. However, if you're writing about Netdata for an external publication, such as a guest blog +post, follow that publication's style guide or standards, while keeping the [preferred spelling of Netdata +terms](#netdata-specific-terms) in mind. + +### Active voice + +Active voice is more concise and easier to understand compared to passive voice. When using active voice, the subject of +the sentence is action. In passive voice, the subject is acted upon. A famous example of passive voice is the phrase +"mistakes were made." + +| | | +|-----------------|---------------------------------------------------------------------------------------------| +| Not recommended | When an alarm is triggered by a metric, a notification is sent by Netdata. | +| **Recommended** | When a metric triggers an alarm, Netdata sends a notification to your preferred endpoint. | + +### Second person + +Use the second person ("you") to give instructions or "talk" directly to users. + +In these situations, avoid "we," "I," "let's," and "us," particularly in documentation. The "you" pronoun can also be +implied, depending on your sentence structure. + +One valid exception is when a member of the Netdata team or community wants to write about said team or community. + +| | | +|--------------------------------|-------------------------------------------------------------------------------------------| +| Not recommended | To install Netdata, we should try the one-line installer... | +| **Recommended** | To install Netdata, you should try the one-line installer... | +| **Recommended**, implied "you" | To install Netdata, try the one-line installer... | + +### "Easy" or "simple" + +Using words that imply the complexity of a task or feature goes against our policy of [universal +communication](#universal-communication). If you claim that a task is easy and the reader struggles to complete it, you +may inadvertently discourage them. + +However, if you give users two options and want to relay that one option is genuinely less complex than another, be +specific about how and why. + +For example, don't write, "Netdata's one-line installer is the easiest way to install Netdata." Instead, you might want +to say, "Netdata's one-line installer requires fewer steps than manually installing from source." + +### Slang, metaphors, and jargon + +A particular word, phrase, or metaphor you're familiar with might not translate well to the other cultures featured +among Netdata's global community. We recommended you avoid slang or colloquialisms in your writing. + +In addition, don't use abbreviations that have not yet been defined in the content. See our section on +[abbreviations](#abbreviations-acronyms-and-initialisms) for additional guidance. + +If you must use industry jargon, such as "mean time to resolution," define the term as clearly and concisely as you can. + +> Netdata helps you reduce your organization's mean time to resolution (MTTR), which is the average time the responsible +> team requires to repair a system and resolve an ongoing incident. + +### Spelling + +While the Netdata team is mostly _not_ American, we still aspire to use American spelling whenever possible, as it is +the standard for the monitoring industry. + +See the [word list](#word-list) for spellings of specific words. + +### Capitalization + +Follow the general [English standards](https://owl.purdue.edu/owl/general_writing/mechanics/help_with_capitals.html) for +capitalization. In summary: + +- Capitalize the first word of every new sentence. +- Don't use uppercase for emphasis. (Netdata is the BEST!) +- Capitalize the names of brands, software, products, and companies according to their official guidelines. (Netdata, + Docker, Apache, NGINX) +- Avoid camel case (NetData) or all caps (NETDATA). + +Whenever you refer to the company Netdata, Inc., or the open-source monitoring agent the company develops, capitalize +**Netdata**. + +However, if you are referring to a process, user, or group on a Linux system, use lowercase and fence the word in an +inline code block: `` `netdata` ``. + +| | | +|-----------------|------------------------------------------------------------------------------------------------| +| Not recommended | The netdata agent, which spawns the netdata process, is actively maintained by netdata, inc. | +| **Recommended** | The Netdata Agent, which spawns the `netdata` process, is actively maintained by Netdata, Inc. | + +#### Capitalization of document titles and page headings + +Document titles and page headings should use sentence case. That means you should only capitalize the first word. + +If you need to use the name of a brand, software, product, and company, capitalize it according to their official +guidelines. + +Also, don't put a period (`.`) or colon (`:`) at the end of a title or header. + +| | | +|-----------------|-----------------------------------------------------------------------------------------------------| +| Not recommended | Getting Started Guide
Service Discovery and Auto-Detection:
Install netdata with docker | +| **Recommended** | Getting started guide
Service discovery and auto-detection
Install Netdata with Docker | + +### Abbreviations (acronyms and initialisms) + +Use abbreviations (including [acronyms and initialisms](https://www.dictionary.com/e/acronym-vs-abbreviation/)) in +documentation when one exists, when it's widely accepted within the monitoring/sysadmin community, and when it improves +the readability of a document. + +When introducing an abbreviation to a document for the first time, give the reader both the spelled-out version and the +shortened version at the same time. For example: + +> Use Netdata to monitor Extended Berkeley Packet Filter (eBPF) metrics in real-time. + +After you define an abbreviation, don't switch back and forth. Use only the abbreviation for the rest of the document. + +You can also use abbreviations in a document's title to keep the title short and relevant. If you do this, you should +still introduce the spelled-out name alongside the abbreviation as soon as possible. + +### Clause order + +When instructing users to take action, give them the context first. By placing the context in an initial clause at the +beginning of the sentence, users can immediately know if they want to read more, follow a link, or skip ahead. + +| | | +|-----------------|--------------------------------------------------------------------------------| +| Not recommended | Read the reference guide if you'd like to learn more about custom dashboards. | +| **Recommended** | If you'd like to learn more about custom dashboards, read the reference guide. | + +### Oxford comma + +The Oxford comma is the comma used after the second-to-last item in a list of three or more items. It appears just +before "and" or "or." + +| | | +|-----------------|------------------------------------------------------------------------------| +| Not recommended | Netdata can monitor RAM, disk I/O, MySQL queries per second and lm-sensors. | +| **Recommended** | Netdata can monitor RAM, disk I/O, MySQL queries per second, and lm-sensors. | + +### Future releases or features + +Do not mention future releases or upcoming features in writing unless they have been previously communicated via a +public roadmap. + +In particular, documentation must describe, as accurately as possible, the Netdata Agent _as of the [latest +commit](https://github.com/netdata/netdata/commits/master) in the GitHub repository_. For Netdata Cloud, documentation +must reflect the _current state of [production](https://app.netdata.cloud). + +### Informational links + +Every link should clearly state its destination. Don't use words like "here" to describe where a link will take your +reader. + +| | | +|-----------------|-------------------------------------------------------------------------------------------| +| Not recommended | To install Netdata, click [here](/packaging/installer/README.md). | +| **Recommended** | To install Netdata, read the [installation instructions](/packaging/installer/README.md). | + +Use links as often as required to provide necessary context. Blog posts and guides require less hyperlinks than +documentation. See the section on [linking between documentation](#linking-between-documentation) for guidance on the +Markdown syntax and path structure of inter-documentation links. + +### Contractions + +Contractions like "you'll" or "they're" are acceptable in most Netdata writing. They're both authentic and playful, and +reinforce the idea that you, as a writer, are guiding users through a particular idea, process, or feature. + +Contractions are generally not used in press releases or other media engagements. + +### Emoji + +Emoji can add fun and character to your writing, but should be used sparingly and only if it matches the content's tone +and desired audience. + +## Technical/Linux standards + +Configuration or maintenance of the Netdata Agent requires some system administration skills, such as navigating +directories, editing files, or starting/stopping/restarting services. Certain processes + +### Switching Linux users + +Netdata documentation often suggests that users switch from their normal user to the `netdata` user to run specific +commands. Use the following command to instruct users to make the switch: + +```bash +sudo su -s /bin/bash netdata +``` + +### Hostname/IP address of a node + +Use `NODE` instead of an actual or example IP address/hostname when referencing the process of navigating to a dashboard +or API endpoint in a browser. + +| | | +|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Not recommended | Navigate to `http://example.com:19999` in your browser to see Netdata's dashboard.
Navigate to `http://203.0.113.0:19999` in your browser to see Netdata's dashboard. | +| **Recommended** | Navigate to `http://NODE:19999` in your browser to see Netdata's dashboard. | + +If you worry that `NODE` doesn't provide enough context for the user, particularly in documentation or guides designed +for beginners, you can provide an explanation: + +> With the Netdata Agent running, visit `http://NODE:19999/api/v1/info` in your browser, replacing `NODE` with the IP +> address or hostname of your Agent. + +### Paths and running commands + +When instructing users to run a Netdata-specific command, don't assume the path to said command, because not every +Netdata Agent installation will have commands under the same paths. When applicable, help them navigate to the correct +path, providing a recommendation or instructions on how to view the running configuration, which includes the correct +paths. + +For example, the [configuration](/docs/configure/nodes.md) doc first teaches users how to find the Netdata config +directory and navigate to it, then runs commands from the `/etc/netdata` path so that the instructions are more +universal. + +Don't include full paths, beginning from the system's root (`/`), as these might not work on certain systems. + +| | | +|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Not recommended | Use `edit-config` to edit Netdata's configuration: `sudo /etc/netdata/edit-config netdata.conf`. | +| **Recommended** | Use `edit-config` to edit Netdata's configuration by first navigating to your [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory), which is typically at `/etc/netdata`, then running `sudo edit-config netdata.conf`. | + +### `sudo` + +Include `sudo` before a command if you believe most Netdata users will need to elevate privileges to run it. This makes +our writing more universal, and users on `sudo`-less systems are generally already aware that they need to run commands +differently. + +For example, most users need to use `sudo` with the `edit-config` script, because the Netdata config directory is owned +by the `netdata` user. Same goes for restarting the Netdata Agent with `systemctl`. + +| | | +|-----------------|----------------------------------------------------------------------------------------------------------------------------------------------| +| Not recommended | Run `edit-config netdata.conf` to configure the Netdata Agent.
Run `systemctl restart netdata` to restart the Netdata Agent. | +| **Recommended** | Run `sudo edit-config netdata.conf` to configure the Netdata Agent.
Run `sudo systemctl restart netdata` to restart the Netdata Agent. | + +## Markdown syntax + +Netdata's documentation uses Markdown syntax. + +If you're not familiar with Markdown, read the [Mastering +Markdown](https://guides.github.com/features/mastering-markdown/) guide from GitHub for the basics on creating +paragraphs, styled text, lists, tables, and more. + +The following sections describe situations in which a specific syntax is required. + +### Syntax standards (`remark-lint`) + +The Netdata team uses [`remark-lint`](https://github.com/remarkjs/remark-lint) for Markdown code styling. + +- Use a maximum of 120 characters per line. +- Begin headings with hashes, such as `# H1 heading`, `## H2 heading`, and so on. +- Use `_` for italics/emphasis. +- Use `**` for bold. +- Use dashes `-` to begin an unordered list, and put a single space after the dash. +- Tables should be padded so that pipes line up vertically with added whitespace. + +If you want to see all the settings, open the +[`remarkrc.js`](https://github.com/netdata/netdata/blob/master/.remarkrc.js) file in the `netdata/netdata` repository. + +### Frontmatter + +Every document must begin with frontmatter, followed by an H1 (`#`) heading. + +Unlike typical Markdown frontmatter, Netdata uses HTML comments (``) to begin and end the frontmatter block. +These HTML comments are later converted into typical frontmatter syntax when building [Netdata +Learn](https://learn.netdata.cloud). + +Frontmatter _must_ contain the following variables: + +- A `title` that quickly and distinctly describes the document's content. +- A `description` that elaborates on the purpose or goal of the document using no less than 100 characters and no more + than 155 characters. +- A `custom_edit_url` that links directly to the GitHub URL where another user could suggest additional changes to the + published document. + +Some documents, like the Ansible guide and others in the `/docs/guides` folder, require an `image` variable as well. In +this case, replace `/docs` with `/img/seo`, and then rebuild the remainder of the path to the document in question. End +the path with `.png`. A member of the Netdata team will assist in creating the image when publishing the content. + +For example, here is the frontmatter for the guide about [deploying the Netdata Agent with +Ansible](https://learn.netdata.cloud/guides/deploy/ansible). + +```markdown + + +# Deploy Netdata with Ansible + +... +``` + +Questions about frontmatter in documentation? [Ask on our community +forum](https://community.netdata.cloud/c/blog-posts-and-articles/6). + +### Linking between documentation + +Documentation should link to relevant pages whenever it's relevant and provides valuable context to the reader. + +Links should always reference the full path to the document, beginning at the root of the Netdata Agent repository +(`/`), and ending with the `.md` file extension. Avoid relative links or traversing up directories using `../`. + +For example, if you want to link to our node configuration document, link to `/docs/configure/nodes.md`. To reference +the guide for deploying the Netdata Agent with Ansible, link to `/docs/guides/deploy/ansible.md`. + +### References to UI elements + +When referencing a user interface (UI) element in Netdata, reference the label text of the link/button with Markdown's +(`**bold text**`) tag. + +```markdown +Click the **Sign in** button. +``` + +Avoid directional language whenever possible. Not every user can use instructions like "look at the top-left corner" to +find their way around an interface, and interfaces often change between devices. If you must use directional language, +try to supplement the text with an [image](#images). + +### Images + +Don't rely on images to convey features, ideas, or instructions. Accompany every image with descriptive alt text. + +In Markdown, use the standard image syntax, `![]()`, and place the alt text between the brackets `[]`. Here's an example +using our logo: + +```markdown +![The Netdata logo](/web/gui/static/img/netdata-logomark.svg) +``` + +Reference in-product text, code samples, and terminal output with actual text content, not screen captures or other +images. Place the text in an appropriate element, such as a blockquote or code block, so all users can parse the +information. + +### Syntax highlighting + +Our documentation site at [learn.netdata.cloud](https://learn.netdata.cloud) uses +[Prism](https://v2.docusaurus.io/docs/markdown-features#syntax-highlighting) for syntax highlighting. Netdata +documentation will use the following for the most part: `c`, `python`, `js`, `shell`, `markdown`, `bash`, `css`, `html`, +and `go`. If no language is specified, Prism tries to guess the language based on its content. + +Include the language directly after the three backticks (```` ``` ````) that start the code block. For highlighting C +code, for example: + +````c +```c +inline char *health_stock_config_dir(void) { + char buffer[FILENAME_MAX + 1]; + snprintfz(buffer, FILENAME_MAX, "%s/health.d", netdata_configured_stock_config_dir); + return config_get(CONFIG_SECTION_DIRECTORIES, "stock health config", buffer); +} +``` +```` + +And the prettified result: + +```c +inline char *health_stock_config_dir(void) { + char buffer[FILENAME_MAX + 1]; + snprintfz(buffer, FILENAME_MAX, "%s/health.d", netdata_configured_stock_config_dir); + return config_get(CONFIG_SECTION_DIRECTORIES, "stock health config", buffer); +} +``` + +Prism also supports titles and line highlighting. See the [Docusaurus +documentation](https://v2.docusaurus.io/docs/markdown-features#code-blocks) for more information. + +## Word list + +The following tables describe the standard spelling, capitalization, and usage of words found in Netdata's writing. + +### Netdata-specific terms + +| Term | Definition | +|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **claimed node** | A node that you've proved ownership of by completing the [connecting to Cloud process](/claim/README.md). The claimed node will then appear in your Space and any War Rooms you added it to. | +| **Netdata** | The company behind the open-source Netdata Agent and the Netdata Cloud web application. Never use _netdata_ or _NetData_.

In general, focus on the user's goals, actions, and solutions rather than what the company provides. For example, write _Learn more about enabling alarm notifications on your preferred platforms_ instead of _Netdata sends alarm notifications to your preferred platforms_. | +| **Netdata Agent** | The free and open source [monitoring agent](https://github.com/netdata/netdata) that you can install on all of your distributed systems, whether they're physical, virtual, containerized, ephemeral, and more. The Agent monitors systems running Linux, Docker, Kubernetes, macOS, FreeBSD, and more, and collects metrics from hundreds of popular services and applications. | +| **Netdata Cloud** | The web application hosted at [https://app.netdata.cloud](https://app.netdata.cloud) that helps you monitor an entire infrastructure of distributed systems in real time.

Never use _Cloud_ without the preceding _Netdata_ to avoid ambiguity. | +| **Netdata community** | Contributors to any of Netdata's [open-source projects](https://learn.netdata.cloud/contribute/projects), members of the [community forum](https://community.netdata.cloud/). | +| **Netdata community forum** | The Discourse-powered forum for feature requests, Netdata Cloud technical support, and conversations about Netdata's monitoring and troubleshooting products. | +| **node** | A system on which the Netdata Agent is installed. The system can be physical, virtual, in a Docker container, and more. Depending on your infrastructure, you may have one, dozens, or hundreds of nodes. Some nodes are _ephemeral_, in that they're created/destroyed automatically by an orchestrator service. | +| **Space** | The highest level container within Netdata Cloud for a user to organize their team members and nodes within their infrastructure. A Space likely represents an entire organization or a large team.

_Space_ is always capitalized. | +| **unreachable node** | A connected node with a disrupted [Agent-Cloud link](/aclk/README.md). Unreachable could mean the node no longer exists or is experiencing network connectivity issues with Cloud. | +| **visited node** | A node which has had its Agent dashboard directly visited by a user. A list of these is maintained on a per-user basis. | +| **War Room** | A smaller grouping of nodes where users can view key metrics in real-time and monitor the health of many nodes with their alarm status. War Rooms can be used to organize nodes in any way that makes sense for your infrastructure, such as by a service, purpose, physical location, and more.

_War Room_ is always capitalized. | + +### Other technical terms + +| Term | Definition | +|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **filesystem** | Use instead of _file system_. | +| **preconfigured** | The concept that many of Netdata's features come with sane defaults that users don't need to configure to find [immediate value](/docs/overview/why-netdata.md#simple-to-deploy). | +| **real time**/**real-time** | Use _real time_ as a noun phrase, most often with _in_: _Netdata collects metrics in real time_. Use _real-time_ as an adjective: _Netdata collects real-time metrics from hundreds of supported applications and services. | + + diff --git a/docs/dashboard/customize.mdx b/docs/dashboard/customize.mdx new file mode 100644 index 0000000..8f0b222 --- /dev/null +++ b/docs/dashboard/customize.mdx @@ -0,0 +1,93 @@ +--- +title: "Customize the standard dashboard" +description: "Netdata's preconfigured dashboard offers many customization options, such as choosing when charts are updated, your preferred theme, and custom text to document processes, and more." +type: how-to +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/customize.mdx +--- + +# Customize the standard dashboard + +While the [Netdata dashboard](/docs/dashboard/how-dashboard-works.mdx) comes preconfigured with hundreds of charts and +thousands of metrics, you may want to alter your experience based on a particular use case or preferences. + +## Dashboard settings + +To change dashboard settings, click the on the **settings** icon ![Import +icon](https://raw.githubusercontent.com/netdata/netdata-ui/98e31799c1ec0983f433537ff16d2ac2b0d994aa/src/components/icon/assets/gear.svg) +in the top panel. + +These settings only affect how the dashboard behaves in your browser. They take effect immediately and are permanently +saved to browser local storage (except the refresh on focus / always option). Some settings are applied immediately, and +others are only reflected after the dashboard is refreshed, which happens automatically. + +Here are a few popular settings: + +### Change chart legend position + +Find this setting under the **Visual** tab. By default, Netdata places the [legend of dimensions](/docs/dashboard/dimensions-contexts-families.mdx#dimension) _below_ charts. +Click this toggle to move the legend to the _right_ of charts. + + +### Change theme + +Find this setting under the **Visual** tab. Choose between Dark (the default) and White. + +## Customize the standard dashboard info + +Netdata stores information about individual charts in the `dashboard_info.js` file. This file includes section and +subsection headings, descriptions, colors, titles, tooltips, and other information for Netdata to render on the +dashboard. + +One common use case for customizing the standard dashboard is adding internal "documentation" a section or specific +chart that can then be read by anyone with access to that dashboard. + +For example, here is how `dashboard_info.js` defines the **System Overview** section. + +```javascript +netdataDashboard.menu = { + 'system': { + title: 'System Overview', + icon: '', + info: 'Overview of the key system metrics.' + }, +``` + +If you want to customize this information, use the example `dashboard_info_custom_example.js` as a starting point. +First, navigate to the web server's directory. If you're on a Linux system, this should be at `/usr/share/netdata/web/`. +Copy the example file, then ensure that its permissions match the rest of the web server, which is `netdata:netdata` by +default. + +```bash +cd /usr/share/netdata/web/ +sudo cp dashboard_info_custom_example.js your_dashboard_info_file.js +sudo chown netdata:netdata your_dashboard_info_file.js +``` + +Edit the file with customizations to the `title`, `icon`, and `info` fields. Replace the string after `fas fa-` with any +icon from [Font Awesome](https://fontawesome.com/cheatsheet) to customize the icons that appear throughout the +dashboard. + +Save the file, then navigate to your [Netdata config directory](/docs/configure/nodes.md) to edit `netdata.conf`. Add +the following line to the `[web]` section to tell Netdata where to find your custom configuration. + +```conf +[web] + custom dashboard_info.js = your_dashboard_info_file.js +``` + +Reload your browser tab to see your custom configuration. + +## What's next? + +If you're keen on continuing to customize your Netdata experience, check out our docs on [building new custom +dashboards](/web/gui/custom/README.md) with HTML, CSS, and JavaScript. + +### Further reading & related information + +- Dashboard + - [How the dashboard works](/docs/dashboard/how-dashboard-works.mdx) + - [Interact with charts](/docs/dashboard/interact-charts.mdx) + - [Chart dimensions, contexts, and families](/docs/dashboard/dimensions-contexts-families.mdx) + - [Select timeframes to visualize](/docs/dashboard/visualization-date-and-time-controls.mdx) + - [Import, export, and print a snapshot](/docs/dashboard/import-export-print-snapshot.mdx) + - **[Customize the standard dashboard](/docs/dashboard/customize.mdx)** diff --git a/docs/dashboard/dimensions-contexts-families.mdx b/docs/dashboard/dimensions-contexts-families.mdx new file mode 100644 index 0000000..0c6dd2a --- /dev/null +++ b/docs/dashboard/dimensions-contexts-families.mdx @@ -0,0 +1,96 @@ +--- +title: "Chart dimensions, contexts, and families" +description: "Netdata organizes charts into dimensions, contexts, and families to automatically and meaningfully organize thousands of metrics into interactive charts." +type: explanation +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/dimensions-contexts-families.mdx +--- + +# Chart dimensions, contexts, and families + +While Netdata's charts require no configuration and are [easy to interact with](/docs/dashboard/interact-charts.mdx), +they have a lot of underlying complexity. To meaningfully organize charts out of the box based on what's happening in +your nodes, Netdata uses the concepts of **dimensions**, **contexts**, and **families**. + +Understanding how these work will help you more easily navigate the dashboard, [write new +alarms](/docs/monitor/configure-alarms.md), or play around with the [API](/web/api/README.md). + +For a refresher on the anatomy of a chart, see [dashboards and charts](/docs/dashboard/how-dashboard-works.mdx). + +## Dimension + +A **dimension** is a value that gets shown on a chart. The value can be raw data or calculated values, such as the +average (the default), minimum, or maximum. These values can then be given any type of unit. For example, CPU +utilization is represented as a percentage, disk I/O as `MiB/s`, and available RAM as an absolute value in `MiB` or +`GiB`. + +Beneath every chart (or on the right-side if you configure the dashboard) is a legend of dimensions. When there are +multiple dimensions, you'll see a different entry in the legend for each dimension. + +The **Apps CPU Time** chart (with the [context](#context) `apps.cpu`), which visualizes CPU utilization of +different types of processes/services/applications on your node, always provides a vibrant example of a chart with +multiple dimensions. + +![An example apps.cpu chart with many +dimensions](https://user-images.githubusercontent.com/1153921/114207816-a5cb7400-9911-11eb-8800-06f60b745f9c.png) + +The chart shows 13 unique dimensions, such as `httpd` for the CPU utilization for web servers, `kernel` for anything +related to the Linux kernel, and so on. In your dashboard, these specific dimensions will almost certainly be different. + +Dimensions can be [hidden](/docs/dashboard/interact-charts.mdx#show-and-hide-dimensions) to help you focus your +attention. + +## Context + +A **context** is a way of grouping charts by the types of metrics collected and dimensions displayed. It's kind of like +a machine-readable naming and organization scheme. + +For example, the **Apps CPU Time** has the context `apps.cpu`. A little further down on the dashboard is a similar +chart, **Apps Real Memory (w/o shared)** with the context `apps.mem`. The `apps` portion of the context is the **type**, +whereas anything after the `.` is specified either by the chart's developer or by the [**family**](#family). + +By default, a chart's type affects where it fits in the menu, while its family creates submenus. + +Netdata also relies on contexts for [alarm configuration](/docs/monitor/configure-alarms.md) (the [`on` +line](/health/REFERENCE.md#alarm-line-on)). + +## Family + +**Families** are a _single instance_ of a hardware or software resource that needs to be displayed separately from +similar instances. + +For example, let's look at the **Disks** section, which contains a number of charts with contexts like `disk.io`, +`disk.ops`, `disk.backlog`, and `disk.util`. If your node has multiple disk drives at `sda` and `sdb`, Netdata creates +a separate family for each. + +Netdata now merges the contexts and families to create charts that are grouped by family, following a +`[context].[family]` naming scheme, so that you can see the `disk.io` and `disk.ops` charts for `sda` right next to each +other. + +Given the four example contexts, and two families of `sda` and `sdb`, Netdata will create the following charts and their +names: + +| Context | `sda` family | `sdb` family | +| :------------- | ------------------ | ------------------ | +| `disk.io` | `disk_io.sda` | `disk_io.sdb` | +| `disk.ops` | `disk_ops.sda` | `disk_ops.sdb` | +| `disk.backlog` | `disk_backlog.sda` | `disk_backlog.sdb` | +| `disk.util` | `disk_util.sda` | `disk_util.sdb` | + +## What's next? + +With an understanding of a chart's dimensions, context, and family, you're now ready to dig even deeper into Netdata's +dashboard. We recommend looking into [using the timeframe selector](/docs/dashboard/visualization-date-and-time-controls.mdx). + +If you feel comfortable with the [dashboard](/docs/dashboard/how-dashboard-works.mdx) and interacting with charts, we +recommend learning about [configuration](/docs/configure/nodes.md). While Netdata doesn't _require_ a complicated setup +process or a query language to create charts, there are a lot of ways to tweak the experience to match your needs. + +### Further reading & related information + +- Dashboard + - [How the dashboard works](/docs/dashboard/how-dashboard-works.mdx) + - [Interact with charts](/docs/dashboard/interact-charts.mdx) + - **[Chart dimensions, contexts, and families](/docs/dashboard/dimensions-contexts-families.mdx)** + - [Select timeframes to visualize](/docs/dashboard/visualization-date-and-time-controls.mdx) + - [Import, export, and print a snapshot](/docs/dashboard/import-export-print-snapshot.mdx) + - [Customize the standard dashboard](/docs/dashboard/customize.mdx) diff --git a/docs/dashboard/how-dashboard-works.mdx b/docs/dashboard/how-dashboard-works.mdx new file mode 100644 index 0000000..315614d --- /dev/null +++ b/docs/dashboard/how-dashboard-works.mdx @@ -0,0 +1,112 @@ +--- +title: "How the dashboard works" +description: "Learn how to navigate Netdata's preconfigured dashboard to get started exploring, visualizing, and troubleshooting in real time." +type: explanation +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/how-dashboard-works.mdx +--- + +# How the dashboard works + +Because Netdata is a monitoring and _troubleshooting_ platform, a dashboard with real-time, meaningful, and +context-aware charts is essential. + +As soon as you [install Netdata](/docs/get-started.mdx), it autodetects hardware, OS, containers, services, and +applications running on your node and builds a dashboard on a single, scrollable webpage. This page features hundreds of +charts, which are preconfigured to save you time from learning a query language, all stacked on top of one another. This +vertical rhythm is designed to encourage exploration and help you visually identify connections between the metrics +visualized in different charts. + +It's essential to understand the core concepts and features of Netdata's dashboard if you want to maximize your Netdata +experience right after installation. + +## Open the dashboard + +Access Netdata's dashboard by navigating to `http://NODE:19999` in your browser, replacing `NODE` with either +`localhost` or the hostname/IP address of a remote node. + +![Animated GIF of navigating to the +dashboard](https://user-images.githubusercontent.com/1153921/80825153-abaec600-8b94-11ea-8b17-1b770a2abaa9.gif) + +Many features of the internal web server that serves the dashboard are [configurable](/web/server/README.md), including +the listen port, enforced TLS, and even disabling the dashboard altogether. + +## Sections and menus + +As mentioned in the introduction, Netdata automatically organizes all the metrics it collects from your node, and places +them into **sections** of closely related charts. + +The first section on any dashboard is the **System Overview**, followed by **CPUs**, **Memory**, and so on. + +These sections populate the **menu**, which is on the right-hand side of the dashboard. Instead of manually scrolling up +and down to explore the dashboard, it's generally faster to click on the relevant menu item to jump to that position on +the dashboard. + +Many menu items also contain a **submenu**, with links to additional categories. For example, the **Disks** section is often separated into multiple groups based on the number of disk drives/partitions on your node, which are also known as a family. + +![Animated GIF of using Netdata's menus and +submenus](https://user-images.githubusercontent.com/1153921/80832425-7c528600-8ba1-11ea-8140-d0a17a62009b.gif) + +## Charts + +Every **chart** in the Netdata dashboard is [fully interactive](/docs/dashboard/interact-charts.mdx). Netdata +synchronizes your interactions to help you understand exactly how a node behaved in any timeframe, whether that's +seconds or days. + +A chart is an individual, interactive, always-updating graphic displaying one or more collected/calculated metrics, +which are generated by [collectors](/docs/collect/how-collectors-work.md). + +![Animated GIF of the standard Netdata dashboard being manipulated and synchronizing +charts](https://user-images.githubusercontent.com/1153921/80839230-b034a800-8baf-11ea-9cb2-99c1e10f0f85.gif) + +Hover over any chart to temporarily pause it and see the exact metrics values presented as different dimensions. Click +or tap to stop the chart from automatically updating with new metrics, thereby locking it to a single timeframe. +Double-click it to resume auto-updating. + +Let's cover two of the most important ways to interact with charts: panning through time and zooming. + +To pan through time, **click and hold** (or touch and hold) on any chart, then **drag your mouse** (or finger) to the +left or right. Drag to the right to pan backward through time, or drag to the left to pan forward in time. Think of it +like pushing the current timeframe off the screen to see what came before or after. + +To zoom, press and hold `Shift`, then use your mouse's scroll wheel, or a two-finger pinch if you're using a touchpad. + +See [interact with charts](/docs/dashboard/interact-charts.mdx) for all the possible ways to interact with the charts on +your dashboard. + +## Alarms + +Many of the preconfigured charts on the Netdata dashboard also come with preconfigured alarms. Netdata sends three +primary alarm states via alarms: `CLEAR`, `WARNING`, and `CRITICAL`. If an alarm moves from a `CLEAR` state to either +`WARNING` or `CRITICAL`, Netdata creates a notification to let you know exactly what's going on. There are [other alarm +states](/health/REFERENCE.md#alarm-statuses) as well. + +The easiest way to see alarms is by clicking on the alarm icon ![Alarms +icon](https://raw.githubusercontent.com/netdata/netdata-ui/98e31799c1ec0983f433537ff16d2ac2b0d994aa/src/components/icon/assets/alarm.svg) +in the top panel to open the alarms panel, which shows you all the active alarms. The other **All** tab shows every +active alarm, and the **Log** tab shows a historical record of exactly when alarms triggered and to which state. + +![Animated GIF of looking at raised alarms and the alarm +log](https://user-images.githubusercontent.com/1153921/80842482-8c289500-8bb6-11ea-9791-600cfdbe82ce.gif) + +Learn more about [viewing active alarms](/docs/monitor/view-active-alarms.md), [configuring +alarms](/docs/monitor/configure-alarms.md), or [enabling a new notification +method](/docs/monitor/enable-notifications.md). + +## What's next? + +Learn more about [interacting with charts](/docs/dashboard/interact-charts.mdx) to quickly pan through time, zoom, and +show/hide dimensions to best understand the state of your node in any timeframe. A complete understanding of [chart +dimensions, contexts, and families](/docs/dashboard/dimensions-contexts-families.mdx) will also help with how Netdata +organizes its dashboard and operates [alarms](/docs/monitor/configure-alarms.md). + +### Further reading & related information + +- Dashboard + - **[How the dashboard works](/docs/dashboard/how-dashboard-works.mdx)** + - [Interact with charts](/docs/dashboard/interact-charts.mdx) + - [Chart dimensions, contexts, and families](/docs/dashboard/dimensions-contexts-families.mdx) + - [Select timeframes to visualize](/docs/dashboard/visualization-date-and-time-controls.mdx) + - [Import, export, and print a snapshot](/docs/dashboard/import-export-print-snapshot.mdx) + - [Customize the standard dashboard](/docs/dashboard/customize.mdx) +- [HTTP API](/web/api/README.md) +- [Custom dashboards](/web/gui/custom/README.md) diff --git a/docs/dashboard/import-export-print-snapshot.mdx b/docs/dashboard/import-export-print-snapshot.mdx new file mode 100644 index 0000000..df43feb --- /dev/null +++ b/docs/dashboard/import-export-print-snapshot.mdx @@ -0,0 +1,83 @@ +--- +title: "Import, export, and print a snapshot" +description: "Snapshots can be incredibly useful for diagnosing anomalies after they've already happened, and are interoperable with any other node running Netdata." +type: how-to +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/import-export-print-snapshot.mdx +--- + +# Import, export, and print snapshots + +Netdata can export snapshots of the contents of your dashboard at a given time, which you can then import into any other +node running Netdata. Or, you can create a print-ready version of your dashboard to save to PDF or actually print to +paper. + +Snapshots can be incredibly useful for diagnosing anomalies after they've already happened. Let's say Netdata triggered a warning alarm while you were asleep. In the morning, you can [select the +timeframe](/docs/dashboard/visualization-date-and-time-controls.mdx) when the alarm triggered, export a snapshot, and send it to a + +colleague for further analysis. + +Or, send the Netdata team a snapshot of your dashboard when [filing a bug +report](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml) on +GitHub. + +![The export, import, and print +buttons](https://user-images.githubusercontent.com/1153921/114218399-360fb600-991e-11eb-8dea-fabd2bffc5b3.gif) + +## Import a snapshot + +To import a snapshot, click on the **import** icon ![Import +icon](https://raw.githubusercontent.com/netdata/netdata-ui/98e31799c1ec0983f433537ff16d2ac2b0d994aa/src/components/icon/assets/upload.svg) +in the top panel. + +Select the Netdata snapshot file to import. Once the file is loaded, the modal updates with information about the +snapshot and the system from which it was taken. Click **Import** to begin to process. + +Netdata takes the data embedded inside the snapshot and re-creates a static replica on your dashboard. When the import +finishes, you're free to move around and examine the charts. + +Some caveats and tips to keep in mind: + +- Only metrics in the export timeframe are available to you. If you zoom out or pan through time, you'll see the + beginning and end of the snapshot. +- Charts won't update with new information, as you're looking at a static replica, not the live dashboard. +- The import is only temporary. Reload your browser tab to return to your node's real-time dashboard. + +## Export a snapshot + +To export a snapshot, first pan/zoom any chart to an appropriate _visible timeframe_. The export snapshot will only +contain the metrics you see in charts, so choose the most relevant timeframe. + +Next, click on the **export** icon ![Export +icon](https://raw.githubusercontent.com/netdata/netdata-ui/98e31799c1ec0983f433537ff16d2ac2b0d994aa/src/components/icon/assets/download.svg) +in the top panel. + +Select the metrics resolution to export. The default is 1-second, equal to how often Netdata collects and stores +metrics. Lowering the resolution will reduce the number of data points, and thus the snapshot's overall size. + +Edit the snapshot file name and select your desired compression method. Click on **Export**. When the export is +complete, your browser will prompt you to save the `.snapshot` file to your machine. + +## Print a snapshot + +To print a snapshot, click on the **print** icon ![Import +icon](https://raw.githubusercontent.com/netdata/netdata-ui/98e31799c1ec0983f433537ff16d2ac2b0d994aa/src/components/icon/assets/print.svg) +in the top panel. + +When you click **Print**, Netdata opens a new window to render every chart. This might take some time. When finished, +Netdata opens a browser print dialog for you to save to PDF or print. + +## What's next? + +Now that you understand snapshots, now is a good time to delve deeper into some of the dashboard's lesser-known +features, such as [customization](/docs/dashboard/customize.mdx) or [building new, custom +dashboards](/web/gui/custom/README.md). + +### Further reading & related information + +- Dashboard + - [How the dashboard works](/docs/dashboard/how-dashboard-works.mdx) + - [Interact with charts](/docs/dashboard/interact-charts.mdx) + - [Chart dimensions, contexts, and families](/docs/dashboard/dimensions-contexts-families.mdx) + - [Select timeframes to visualize](/docs/dashboard/visualization-date-and-time-controls.mdx) + - **[Import, export, and print a snapshot](/docs/dashboard/import-export-print-snapshot.mdx)** + - [Customize the standard dashboard](/docs/dashboard/customize.mdx) \ No newline at end of file diff --git a/docs/dashboard/interact-charts.mdx b/docs/dashboard/interact-charts.mdx new file mode 100644 index 0000000..fe0f05e --- /dev/null +++ b/docs/dashboard/interact-charts.mdx @@ -0,0 +1,197 @@ +--- +title: "Interact with charts" +description: "Learn how to pan, zoom, select, and customize Netdata's preconfigured charts to help you troubleshooting with real-time, per-second metrics data." +type: how-to +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/interact-charts.mdx +--- + +# Interact with charts + +> ⚠️ There is a new version of charts that is currently **only** available on [Netdata Cloud](https://learn.netdata.cloud/docs/cloud/visualize/interact-new-charts). We didn't +> want to keep this valuable feature from you, so after we get this into your hands on the Cloud, we will collect and implement your feedback to make sure we are providing the best possible version of the feature on the Netdata Agent dashboard as quickly as possible. + +While charts that update every second with new metrics are helpful for understanding the immediate state of a node, deep +troubleshooting and root cause analysis begins by manipulating the default charts. To help you troubleshoot, Netdata +synchronizes every chart every time you interact with one of them. + +Here's what synchronization looks like: + +![Animated GIF of the standard Netdata dashboard being manipulated and synchronizing +charts](https://user-images.githubusercontent.com/1153921/80839230-b034a800-8baf-11ea-9cb2-99c1e10f0f85.gif) + +Once you understand all the interactions available to you, you'll be able to quickly move around the dashboard, search +for anomalies, and find root causes using per-second metrics. + +## Pause or stop + +| Interaction | Keyboard/mouse | Touchpad/touchscreen | +| :---------------- | :------------- | :------------------- | +| **Pause** a chart | `hover` | `n/a` | +| **Stop** a chart | `click` | `tap` | + +By hovering over any chart, you temporarily pause it so that you can hover over a specific timeframe and see the exact +values presented as dimensions. Click on the chart to lock it to this timeframe, which is useful if you want to jump to +a different chart to look for possible correlations. + +![Animated GIF of hovering over a chart to see +values](https://user-images.githubusercontent.com/1153921/62968279-9227dd00-bdbf-11e9-9112-1d21444d0f31.gif) + +## Pan + +| Interaction | Keyboard/mouse | Touchpad/touchscreen | +| :---------- | :------------- | :------------------- | +| **Pan** | `click + drag` | `swipe` | + +Drag your mouse/finger to the right to pan backward through time, or drag to the left to pan forward in time. Think of +it like pushing the current timeframe off the screen to see what came before or after. + +## Zoom + +| Interaction | Keyboard/mouse | Touchpad/touchscreen | +| :------------------------------- | :-------------------------- | :--------------------------------------------------- | +| **Zoom** in or out | `Shift + mouse scrollwheel` | `two-finger pinch`
`Shift + two-finger scroll` | +| **Zoom** to a specific timeframe | `Shift + mouse selection` | `n/a` | + +Zooming in helps you see metrics with maximum granularity, which is useful when you're trying to diagnose the root cause +of an anomaly or outage. Zooming out lets you see metrics within the larger context, such as the last hour, day, or +week, which is useful in understanding what "normal" looks like, or to identify long-term trends, like a slow creep in +memory usage. + +## Select + +| Interaction | Keyboard/mouse | Touchpad/touchscreen | +| :------------------------------ | :-------------------------------------------------------- | :------------------- | +| **Select** a specific timeframe | `Alt + mouse selection` or `⌘ + mouse selection` (macOS) | `n/a` | + +Selecting timeframes is useful when you see an interesting spike or change in a chart and want to investigate further. + +Select a timeframe, then move to different charts/sections of the dashboard. Each chart shows the same selection to help +you immediately identify the timeframe and look for correlations. + +## Reset a chart to its default state + +| Interaction | Keyboard/mouse | Touchpad/touchscreen | +| :---------------- | :------------- | :------------------- | +| **Reset** a chart | `double-click` | `n/a` | + +Double-check on a chart to restore it to the default auto-updating state, with a timeframe based on your browser +viewport. + +## Resize + +Click-and-drag the icon on the bottom-right corner of any chart. To restore the chart to its original height, +double-click the same icon. + +![Animated GIF of resizing a chart and resetting it to the default +height](https://user-images.githubusercontent.com/1153921/80842459-7d41e280-8bb6-11ea-9488-1bc29f94d7f2.gif) + +## Show and hide dimensions + +| Interaction | Keyboard/mouse | Touchpad/touchscreen | +| :------------------------------------- | :-------------- | :------------------- | +| **Show one** dimension and hide others | `click` | `tap` | +| **Toggle (show/hide)** one dimension | `Shift + click` | `n/a` | + +Hiding dimensions simplifies the chart and can help you better discover exactly which aspect of your system might be +behaving strangely. + +## See the context + +Hover your mouse over the date that appears just beneath the chart itself. A tooltip will tell you the context for that +chart. Below, the context is `apps.cpu`. + +![See a chart's +context](https://user-images.githubusercontent.com/1153921/114212924-39ec0a00-9917-11eb-9a9e-7e171057b3fd.gif) + +## See the resolution and update frequency + +Hover your mouse over the timestamp just to the right of the date. `resolution` is the number of seconds between each +"tick" in the chart. `collection every` is how often Netdata collects and stores that metric. + +If the `resolution` value is higher than `collection every`, such as `resolution 5 secs, collected every 1 sec`, this +means that each tick is calculating represents the average values across a 5-second period. You can zoom in to increase +the resolution to `resolution 1 sec` to see the exact values. + +## Chart controls + +Many of the above interactions can also be triggered using the icons on the bottom-right corner of every chart. They +are, respectively, `Pan Left`, `Reset`, `Pan Right`, `Zoom In`, and `Zoom Out`. + +## Chart label filtering + +The chart label filtering feature supports grouping by and filtering each chart based on labels (key/value pairs) applicable to the context and provides fine-grain capability on slicing the data and metrics. + +All metrics collected get "tagged" with labels and values, thus providing a powerful way of slicing and visualizing all metrics related to the infrastructure. + +The chart label filtering is currently enabled on: + +- All charts on the **Overview** tab +- Custom dashboards + +![Chart filtering on Overview tab chart](https://user-images.githubusercontent.com/88642300/193084084-01074495-c826-4519-a09f-d210f7e3e6be.png) +![Chart filtering on Custom dashboard](https://user-images.githubusercontent.com/88642300/193084172-358dfded-c318-4d9f-b6e2-46a8fc33030b.png) + +The top panel on each chart displays the various filters and grouping options selected on the specific chart. These filters are specific for each chart and need to be manually configured on each chart. + +Additionally, the charts can be saved to a custom dashboard, new or existing, with the selected filters from the overview screen. + +![Chart filtering saved on custom dashboard](https://user-images.githubusercontent.com/88642300/193084225-1b65984e-566c-4815-8bc1-a2781d3564bd.png) + +## Custom labels for Collectors + +In addition to the default labels associated with a collector and metrics context (you can identify them by seeing which ones have an underscore as a prefix), there is now a new feature enabled to create custom labels. These custom labels may be needed to group your jobs or instances into various categories. + +These custom labels can be configured within your go.d plugins by simply associating a label key/value pair, as in the following eaxmple. + +```conf +jobs: + - name: example_1 + someOption: someValue + labels: + label1: value1 + label2: value2 + - name: example_2 + someOption: someValue + labels: + label3: value3 + label4: value4 +``` + +For instance, you may be running multiple Postgres database instances within an infrastructure. Some of these may be associated with testing environments, some with staging and some with production environments. You can now associate each Postgres job / instance with a custom label. The “group by” and filtering options will then allow you to associate individual jobs by specific labels. + +```conf +jobs: + - name: local + dsn: 'postgres://postgres:postgres@127.0.0.1:5432/postgres' + collect_databases_matching: '*' + labels: + instance_type: production + ``` + ![Group by individual job labels one](https://user-images.githubusercontent.com/88642300/193084580-49df500a-ddfb-45bb-a209-3c7a904ee9e0.png) + ![group by individual job labels two](https://user-images.githubusercontent.com/88642300/193084624-6d9848d0-9400-4e34-9cd4-78e50c784cc0.png) + +### Future Work + +We already have [configurable host labels](https://learn.netdata.cloud/guides/using-host-labels) as well, which currently can’t be used to filter or group your metrics. We intend to provide the same capabilities described here with host labels, among other capabilities on other areas of the app as well + +## What's next? + +We recommend you read up on the differences between [chart dimensions, contexts, and +families](/docs/dashboard/dimensions-contexts-families.mdx) to complete your understanding of how Netdata organizes its +dashboards. Another valuable way to interact with charts is to use the [timeframe +selector](/docs/dashboard/visualization-date-and-time-controls.mdx), which helps you visualize specific moments of historical metrics. + +If you feel comfortable with the [dashboard](/docs/dashboard/how-dashboard-works.mdx) and interacting with charts, we +recommend moving on to learning about [configuration](/docs/configure/nodes.md). While Netdata doesn't _require_ a +complicated setup process or a query language to create charts, there are a lot of ways to tweak the experience to match +your needs. + +### Further reading & related information + +- Dashboard + - [How the dashboard works](/docs/dashboard/how-dashboard-works.mdx) + - [Netdata Cloud · Interact with new charts](https://learn.netdata.cloud/docs/cloud/visualize/interact-new-charts) + - [Chart dimensions, contexts, and families](/docs/dashboard/dimensions-contexts-families.mdx) + - [Select timeframes to visualize](/docs/dashboard/visualization-date-and-time-controls.mdx) + - [Import, export, and print a snapshot](/docs/dashboard/import-export-print-snapshot.mdx) + - [Customize the standard dashboard](/docs/dashboard/customize.mdx) diff --git a/docs/dashboard/reference-web-server.mdx b/docs/dashboard/reference-web-server.mdx new file mode 100644 index 0000000..55e761a --- /dev/null +++ b/docs/dashboard/reference-web-server.mdx @@ -0,0 +1,278 @@ +--- +title: "Web server reference" +description: "The Netdata Agent's local static-threaded web server serves dashboards and real-time visualizations with security and DDoS protection." +type: reference +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/reference-web-server.mdx +--- + +# Web server reference + +The Netdata web server is `static-threaded`, with a fixed, configurable number of threads. + +All the threads are concurrently listening for web requests on the same sockets, and the kernel distributes the incoming +requests to them. Each thread uses non-blocking I/O so it can serve any number of web requests in parallel. + +This web server respects the `keep-alive` HTTP header to serve multiple HTTP requests via the same connection. + +## Configuration + +From within your Netdata config directory (typically `/etc/netdata`), [use `edit-config`](/docs/configure/nodes.md) to +open `netdata.conf`. + +``` +sudo ./edit-config netdata.conf +``` + +Scroll down to the `[web]` section to find the following settings. + +## Settings + +| Setting | Default | Description | +|:-------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `ssl key` | `/etc/netdata/ssl/key.pem` | Declare the location of an SSL key to [enable HTTPS](#enable-httpstls-support). | +| `ssl certificate` | `/etc/netdata/ssl/cert.pem` | Declare the location of an SSL certificate to [enable HTTPS](#enable-httpstls-support). | +| `tls version` | `1.3` | Choose which TLS version to use. While all versions are allowed (`1` or `1.0`, `1.1`, `1.2` and `1.3`), we recommend `1.3` for the most secure encryption. If left blank, Netdata uses the highest available protocol version on your system. | +| `tls ciphers` | `none` | Choose which TLS cipher to use. Options include `TLS_AES_256_GCM_SHA384`, `TLS_CHACHA20_POLY1305_SHA256`, and `TLS_AES_128_GCM_SHA256`. If left blank, Netdata uses the default cipher list for that protocol provided by your TLS implementation. | +| `ses max window` | `15` | See [single exponential smoothing](/web/api/queries/ses/README.md). | +| `des max window` | `15` | See [double exponential smoothing](/web/api/queries/des/README.md). | +| `mode` | `static-threaded` | Turns on (`static-threaded` or off (`none`) the static-threaded web server. See the [example](#disable-the-web-server) to turn off the web server and disable the dashboard. | +| `listen backlog` | `4096` | The port backlog. Check `man 2 listen`. | +| `default port` | `19999` | The listen port for the static web server. | +| `web files owner` | `netdata` | The user that owns the web static files. Netdata will refuse to serve a file that is not owned by this user, even if it has read access to that file. If the user given is not found, Netdata will only serve files owned by user given in `run as user`. | +| `web files group` | `netdata` | If this is set, Netdata will check if the file is owned by this group and refuse to serve the file if it's not. | +| `disconnect idle clients after seconds` | `60` | The time in seconds to disconnect web clients after being totally idle. | +| `timeout for first request` | `60` | How long to wait for a client to send a request before closing the socket. Prevents slow request attacks. | +| `accept a streaming request every seconds` | `0` | Can be used to set a limit on how often a parent node will accept streaming requests from child nodes in a [streaming and replication setup](/streaming/README.md). | +| `respect do not track policy` | `no` | If set to `yes`, Netdata will respect the user's browser preferences for [Do Not Track](https://www.eff.org/issues/do-not-track) (DNT) and storing cookies. If DNT is _enabled_ in the browser, and this option is set to `yes`, users will not be able to sign in to Netdata Cloud via their local Agent dashboard, and their node will not connect to any [registry](/registry/README.md). For certain browsers, users must disable DNT and change this option to `yes` for full functionality. | +| `x-frame-options response header` | ` ` | Avoid [clickjacking attacks](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options), by ensuring that the content is not embedded into other sites. | +| `allow connections from` | `localhost *` | Declare which IP addresses or full-qualified domain names (FQDNs) are allowed to connect to the web server, including the [dashboard](/docs/dashboard/interact-charts.mdx) or [HTTP API](/web/api/README.md). This is a global setting with higher priority to any of the ones below. | +| `allow connections by dns` | `heuristic` | See the [access list examples](#access-lists) for details on using `allow` settings. | +| `allow dashboard from` | `localhost *` | | +| `allow dashboard by dns` | `heuristic` | | +| `allow badges from` | `*` | | +| `allow badges by dns` | `heuristic` | | +| `allow streaming from` | `*` | | +| `allow streaming by dns` | `heuristic` | | +| `allow netdata.conf` | `localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.* UNKNOWN` | | +| `allow netdata.conf by dns` | `no` | | +| `allow management from` | `localhost` | | +| `allow management by dns` | `heuristic` | | +| `enable gzip compression` | `yes` | When set to `yes`, Netdata web responses will be GZIP compressed, if the web client accepts such responses. | +| `gzip compression strategy` | `default` | Valid settings are `default`, `filtered`, `huffman only`, `rle` and `fixed`. | +| `gzip compression level` | `3` | Valid settings are 1 (fastest) to 9 (best ratio). | +| `web server threads` | ` ` | How many processor threads the web server is allowed. The default is system-specific, the minimum of `6` or the number of CPU cores. | +| `web server max sockets` | ` ` | Available sockets. The default is system-specific, automatically adjusted to 50% of the max number of open files Netdata is allowed to use (via `/etc/security/limits.conf` or systemd), to allow enough file descriptors to be available for data collection. | +| `custom dashboard_info.js` | ` ` | Specifies the location of a custom `dashboard.js` file. See [customizing the standard dashboard](/docs/dashboard/customize.mdx#customize-the-standard-dashboard) for details. | + +## Examples + +### Disable the web server + +Disable the web server by editing `netdata.conf` and setting: + +``` +[web] + mode = none +``` + +### Change the number of threads + +Control the number of threads and sockets with the following settings: + +``` +[web] + web server threads = 4 + web server max sockets = 512 +``` + +### Binding Netdata to multiple ports + +Netdata can bind to multiple IPs and ports, offering access to different services on each. Up to 100 sockets can be used (increase it at compile time with `CFLAGS="-DMAX_LISTEN_FDS=200" ./netdata-installer.sh ...`). + +The ports to bind are controlled via `[web].bind to`, like this: + +``` +[web] + default port = 19999 + bind to = 127.0.0.1=dashboard^SSL=optional 10.1.1.1:19998=management|netdata.conf hostname:19997=badges [::]:19996=streaming^SSL=force localhost:19995=registry *:http=dashboard unix:/run/netdata/netdata.sock +``` + +Using the above, Netdata will bind to: + +- IPv4 127.0.0.1 at port 19999 (port was used from `default port`). Only the UI (dashboard) and the read API will be accessible on this port. Both HTTP and HTTPS requests will be accepted. +- IPv4 10.1.1.1 at port 19998. The management API and `netdata.conf` will be accessible on this port. +- All the IPs `hostname` resolves to (both IPv4 and IPv6 depending on the resolved IPs) at port 19997. Only badges will be accessible on this port. +- All IPv6 IPs at port 19996. Only metric streaming requests from other Netdata agents will be accepted on this port. Only encrypted streams will be allowed (i.e. child nodes also need to be [configured for TLS](/streaming/README.md). +- All the IPs `localhost` resolves to (both IPv4 and IPv6 depending the resolved IPs) at port 19996. This port will only accept registry API requests. +- All IPv4 and IPv6 IPs at port `http` as set in `/etc/services`. Only the UI (dashboard) and the read API will be accessible on this port. +- Unix domain socket `/run/netdata/netdata.sock`. All requests are serviceable on this socket. Note that in some OSs like Fedora, every service sees a different `/tmp`, so don't create a Unix socket under `/tmp`. `/run` or `/var/run` is suggested. + +The option `[web].default port` is used when an entries in `[web].bind to` do not specify a port. + +Note that the access permissions specified with the `=request type|request type|...` format are available from version 1.12 onwards. +As shown in the example above, these permissions are optional, with the default being to permit all request types on the specified port. +The request types are strings identical to the `allow X from` directives of the access lists, i.e. `dashboard`, `streaming`, `registry`, `netdata.conf`, `badges` and `management`. +The access lists themselves and the general setting `allow connections from` in the next section are applied regardless of the ports that are configured to provide these services. +The API requests are serviced as follows: + +- `dashboard` gives access to the UI, the read API and badges API calls. +- `badges` gives access only to the badges API calls. +- `management` gives access only to the management API calls. + +### Enable HTTPS/TLS support + +Since v1.16.0, Netdata supports encrypted HTTP connections to the web server, plus encryption of streaming data to a +parent from its child nodes, via the TLS protocol. + +Inbound unix socket connections are unaffected, regardless of the TLS settings. + +> While Netdata uses Transport Layer Security (TLS) 1.2 to encrypt communications rather than the obsolete SSL protocol, +> it's still common practice to refer to encrypted web connections as `SSL`. Many vendors, like Nginx and even Netdata +> itself, use `SSL` in configuration files, whereas documentation will always refer to encrypted communications as `TLS` +> or `TLS/SSL`. + +To enable TLS, provide the path to your certificate and private key in the `[web]` section of `netdata.conf`: + +```conf +[web] + ssl key = /etc/netdata/ssl/key.pem + ssl certificate = /etc/netdata/ssl/cert.pem +``` + +Both files must be readable by the `netdata` user. If either of these files do not exist or are unreadable, Netdata will fall back to HTTP. For a parent-child connection, only the parent needs these settings. + +For test purposes, generate self-signed certificates with the following command: + +```bash +openssl req -newkey rsa:2048 -nodes -sha512 -x509 -days 365 -keyout key.pem -out cert.pem +``` + +> If you use 4096 bits for your key and the certificate, Netdata will need more CPU to process the communication. +> `rsa4096` can be up to 4 times slower than `rsa2048`, so we recommend using 2048 bits. Verify the difference +> by running: +> +> ```sh +> openssl speed rsa2048 rsa4096 +> ``` + +### Select TLS version + +Beginning with version `v1.21.0`, specify the TLS version and the ciphers that you want to use: + +```conf +[web] + tls version = 1.3 + tls ciphers = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256 +``` + +If you do not specify these options, Netdata will use the highest available protocol version on your system and the default cipher list for that protocol provided by your TLS implementation. + +#### TLS/SSL enforcement + +When the certificates are defined and unless any other options are provided, a Netdata server will: + +- Redirect all incoming HTTP web server requests to HTTPS. Applies to the dashboard, the API, `netdata.conf` and badges. +- Allow incoming child connections to use both unencrypted and encrypted communications for streaming. + +To change this behavior, you need to modify the `bind to` setting in the `[web]` section of `netdata.conf`. At the end of each port definition, append `^SSL=force` or `^SSL=optional`. What happens with these settings differs, depending on whether the port is used for HTTP/S requests, or for streaming. + +| SSL setting | HTTP requests|HTTPS requests|Unencrypted Streams|Encrypted Streams| +|:---------:|:-----------:|:------------:|:-----------------:|:----------------| +| none | Redirected to HTTPS|Accepted|Accepted|Accepted| +| `force`| Redirected to HTTPS|Accepted|Denied|Accepted| +| `optional`| Accepted|Accepted|Accepted|Accepted| + +Example: + +``` +[web] + bind to = *=dashboard|registry|badges|management|streaming|netdata.conf^SSL=force +``` + +For information how to configure the child to use TLS, check [securing the communication](/streaming/README.md#securing-streaming-communications) in the streaming documentation. There you will find additional details on the expected behavior for client and server nodes, when their respective TLS options are enabled. + +When we define the use of SSL in a Netdata agent for different ports, Netdata will apply the behavior specified on each port. For example, using the configuration line below: + +``` +[web] + bind to = *=dashboard|registry|badges|management|streaming|netdata.conf^SSL=force *:20000=netdata.conf^SSL=optional *:20001=dashboard|registry +``` + +Netdata will: + +- Force all HTTP requests to the default port to be redirected to HTTPS (same port). +- Refuse unencrypted streaming connections from child nodes on the default port. +- Allow both HTTP and HTTPS requests to port 20000 for `netdata.conf` +- Force HTTP requests to port 20001 to be redirected to HTTPS (same port). Only allow requests for the dashboard, the read API and the registry on port 20001. + +#### TLS/SSL errors + +When you start using Netdata with TLS, you may find errors in the Netdata log, which is stored at `/var/log/netdata/error.log` by default. + +Most of the time, these errors are due to incompatibilities between your browser's options related to TLS/SSL protocols and Netdata's internal configuration. The most common error is `error:00000006:lib(0):func(0):EVP lib`. + +In the near future, Netdata will allow our users to change the internal configuration to avoid similar errors. Until then, we're recommending only the most common and safe encryption protocols listed above. + +### Access lists + +Netdata supports access lists in `netdata.conf`: + +``` +[web] + allow connections from = localhost * + allow dashboard from = localhost * + allow badges from = * + allow streaming from = * + allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.* + allow management from = localhost +``` + +`*` does string matches on the IPs or FQDNs of the clients. + +- `allow connections from` matches anyone that connects on the Netdata port(s). + So, if someone is not allowed, it will be connected and disconnected immediately, without reading even + a single byte from its connection. This is a global setting with higher priority to any of the ones below. + +- `allow dashboard from` receives the request and examines if it is a static dashboard file or an API call the + dashboards do. + +- `allow badges from` checks if the API request is for a badge. Badges are not matched by `allow dashboard from`. + +- `allow streaming from` checks if the child willing to stream metrics to this Netdata is allowed. + This can be controlled per API KEY and MACHINE GUID in `stream.conf`. + The setting in `netdata.conf` is checked before the ones in `stream.conf`. + +- `allow netdata.conf from` checks the IP to allow `http://netdata.host:19999/netdata.conf`. + The IPs listed are all the private IPv4 addresses, including link local IPv6 addresses. Keep in mind that connections to Netdata API ports are filtered by `allow connections from`. So, IPs allowed by `allow netdata.conf from` should also be allowed by `allow connections from`. + +- `allow management from` checks the IPs to allow API management calls. Management via the API is currently supported for [health](/web/api/health/README.md#health-management-api) + +In order to check the FQDN of the connection without opening the Netdata agent to DNS-spoofing, a reverse-dns record +must be setup for the connecting host. At connection time the reverse-dns of the peer IP address is resolved, and +a forward DNS resolution is made to validate the IP address against the name-pattern. + +Please note that this process can be expensive on a machine that is serving many connections. Each access list has an +associated configuration option to turn off DNS-based patterns completely to avoid incurring this cost at run-time: + +``` + allow connections by dns = heuristic + allow dashboard by dns = heuristic + allow badges by dns = heuristic + allow streaming by dns = heuristic + allow netdata.conf by dns = no + allow management by dns = heuristic +``` + +The three possible values for each of these options are `yes`, `no` and `heuristic`. The `heuristic` option disables +the check when the pattern only contains IPv4/IPv6 addresses or `localhost`, and enables it when wildcards are +present that may match DNS FQDNs. + +## DDoS protection + +If you publish your Netdata web server to the internet, you may want to apply some protection against DDoS: + +1. Use the `static-threaded` web server (it is the default) +2. Use reasonable `[web].web server max sockets` (the default is) +3. Don't use all your CPU cores for Netdata (lower `[web].web server threads`) +4. Run the `netdata` process with a low process scheduling priority (the default is the lowest) +5. If possible, proxy Netdata via a full featured web server (Nginx, Apache, etc) diff --git a/docs/dashboard/visualization-date-and-time-controls.mdx b/docs/dashboard/visualization-date-and-time-controls.mdx new file mode 100644 index 0000000..fe3e510 --- /dev/null +++ b/docs/dashboard/visualization-date-and-time-controls.mdx @@ -0,0 +1,121 @@ +--- +title: "Visualization date and time controls" +description: "Netdata's dashboard features powerful date visualization controls that include a time control (play, pause, force play), a timezone selector and a rich date and timeframe selector, with useful defaults and rich customization, to help you narrow your focus when troubleshooting issues or anomalies." +type: how-to +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/visualization-date-and-time-controls.mdx +--- + +# Visualization date and time controls + +## Date and time selector + +### Pick timeframes to visualize + +While [panning through time and zooming in/out](/docs/dashboard/interact-charts.mdx) from charts it is helpful when +you're looking a recent history, or want to do granular troubleshooting, what if you want to see metrics from 6 hours +ago? Or 6 days? + +Netdata's dashboard features a **timeframe selector** to help you visualize specific timeframes in a few helpful ways. +By default, it shows a certain number of minutes of historical metrics based on the your browser's viewport to ensure +it's always showing per-second granularity. + +#### Open the timeframe selector + +To visualize a new timeframe, you need to open the picker, which appears just above the menu, near the top-right cover +of the dashboard. + +![The timeframe selector in the local Agent +dashboard](https://user-images.githubusercontent.com/1153921/101507784-2c585080-3934-11eb-9d6e-eff30b8553e4.png) + +The **Clear** button resets the dashboard back to its default state based on your browser viewport, and **Apply** closes +the picker and shifts all charts to the selected timeframe. + +#### Use the Quick Selector + +Click any of the following options in the **Quick Selector** to choose a commonly-used timeframe. + +- Last 5 minutes +- Last 15 minutes +- Last 2 hours +- Last 6 hours +- Last 12 hours + +Click **Apply** to see metrics from your selected timeframe. + +#### Choose a specific interval + +Beneath the Quick Selector is an input field and dropdown you use in combination to select a specific timeframe of +minutes, hours, days, or months. Enter a number and choose the appropriate unit of time, then click **Apply**. + +#### Choose multiple days + +Use the calendar to select multiple days. Click on a date to begin the timeframe selection, then an ending date. The +timeframe begins at noon on the beginning and end dates. Click **Apply** to see your selected multi-day timeframe. + +## Time controls + +The time control provides you the following options: **Play**, **Pause** and **Force Play**. +* **Play** - the content of the page will be automatically refreshed while this is in the foreground +* **Pause** - the content of the page isn't refreshed due to a manual request to pause it or, for example, when your investigating data on a +chart (cursor is on top of a chart) +* **Force Play** - the content of the page will be automatically refreshed even if this is in the background + +With this, we aim to bring more clarity and allow you to distinguish if the content you are looking at is live or historical and also allow you + to always refresh the content of the page when the tabs are in the background. + +Main use cases for **Force Play**: +* You use a terminal or deployment tools to do changes in your infra and want to see immediately, Netdata is in the background, displaying the impact +of these changes +* You want to have Netdata on the background, example displayed on a TV, to constantly see metrics through dashboards or to watch the alert +status + +![The time control with Play, Pause and +Force Play](https://user-images.githubusercontent.com/82235632/129206460-03c47d0d-1a5b-428a-b972-473718b74bdb.png) + +## Timezone selector + +With the timezone selector, you have the ability to change the timezone on Netdata Cloud. More often than not teams are +distributed in different timezones and they need to collaborate. + +Our goal is to make it easier for you and your teams to troubleshoot based on your timezone preference and communicate easily +with varying timezones and timeframes without the need to be concerned about their specificity. + +![Timezon selector](https://user-images.githubusercontent.com/82235632/129209528-bc1d572d-4582-4142-aace-918287849499.png) + +When you change the timezone all the date and time fields will be updated to be displayed according to the specified timezone, this goes from +charts to alerts information and across the Netdata Cloud. + +## Caveats and considerations + +**Longer timeframes will decrease metrics granularity**. At the default timeframe, based on your browser viewport, each +"tick" on charts represents one second. If you select a timeframe of 6 hours, each tick represents the _average_ value +across a larger period of time. + +**You can only see metrics as far back in history as your metrics retention policy allows**. Netdata uses an internal +time-series database (TSDB) to store as many metrics as it can within a specific amount of disk space. The default +storage is 256 MiB, which should be enough for 1-3 days of historical metrics. If you navigate back to a timeframe +beyond stored historical metrics, you'll see this message: + +![Screenshot of reaching the end of historical metrics +storage](https://user-images.githubusercontent.com/1153921/114207597-63a23280-9911-11eb-863d-4d2f75b030b4.png) + +At any time, [configure the internal TSDB's storage capacity](/docs/store/change-metrics-storage.md) to expand your +depth of historical metrics. + +## What's next? + +One useful next step after selecting a timeframe is [exporting the +metrics](/docs/dashboard/import-export-print-snapshot.mdx) into a snapshot file, which can then be shared and imported +into any other Netdata dashboard. + +There are also many ways to [customize](/docs/dashboard/customize.mdx) the standard dashboard experience, from changing +the theme to editing the text that accompanies every section of charts. + +## Further reading & related information + +- Dashboard + - [How the dashboard works](/docs/dashboard/how-dashboard-works.mdx) + - [Interact with charts](/docs/dashboard/interact-charts.mdx) + - [Chart dimensions, contexts, and families](/docs/dashboard/dimensions-contexts-families.mdx) + - [Import, export, and print a snapshot](/docs/dashboard/import-export-print-snapshot.mdx) + - [Customize the standard dashboard](/docs/dashboard/customize.mdx) diff --git a/docs/export/enable-connector.md b/docs/export/enable-connector.md new file mode 100644 index 0000000..a914a11 --- /dev/null +++ b/docs/export/enable-connector.md @@ -0,0 +1,95 @@ + + +# Enable an exporting connector + +Now that you found the right connector for your [external time-series +database](/docs/export/external-databases.md#supported-databases), you can now enable the exporting engine and the +connector itself. We'll walk through the process of enabling the exporting engine itself, followed by two examples using +the OpenTSDB and Graphite connectors. + +> When you enable the exporting engine and a connector, the Netdata Agent exports metrics _beginning from the time you +> restart its process_, not the entire [database of long-term metrics](/docs/store/change-metrics-storage.md). + +Once you understand the process of enabling a connector, you can translate that knowledge to any other connector. + +## Enable the exporting engine + +Use `edit-config` from your [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory) to open +`exporting.conf`: + +```bash +sudo ./edit-config exporting.conf +``` + +Enable the exporting engine itself by setting `enabled` to `yes`: + +```conf +[exporting:global] + enabled = yes +``` + +Save the file but keep it open, as you will edit it again to enable specific connectors. + +## Example: Enable the OpenTSDB connector + +Use the following configuration as a starting point. Copy and paste it into `exporting.conf`. + +```conf +[opentsdb:http:my_opentsdb_http_instance] + enabled = yes + destination = localhost:4242 +``` + +Replace `my_opentsdb_http_instance` with an instance name of your choice, and change the `destination` setting to the IP +address or hostname of your OpenTSDB database. + +Restart your Agent with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to begin exporting to your OpenTSDB database. The +Netdata Agent exports metrics _beginning from the time the process starts_, and because it exports as metrics are +collected, you should start seeing data in your external database after only a few seconds. + +Any further configuration is optional, based on your needs and the configuration of your OpenTSDB database. See the +[OpenTSDB connector doc](/exporting/opentsdb/README.md) and [exporting engine +reference](/exporting/README.md#configuration) for details. + +## Example: Enable the Graphite connector + +Use the following configuration as a starting point. Copy and paste it into `exporting.conf`. + +```conf +[graphite:my_graphite_instance] + enabled = yes + destination = 203.0.113.0:2003 +``` + +Replace `my_graphite_instance` with an instance name of your choice, and change the `destination` setting to the IP +address or hostname of your Graphite-supported database. + +Restart your Agent with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to begin exporting to your Graphite-supported database. +Because the Agent exports metrics as they're collected, you should start seeing data in your external database after +only a few seconds. + +Any further configuration is optional, based on your needs and the configuration of your Graphite-supported database. +See [exporting engine reference](/exporting/README.md#configuration) for details. + +## What's next? + +If you want to further configure your exporting connectors, see the [exporting engine +reference](/exporting/README.md#configuration). + +For a comprehensive example of using the Graphite connector, read our guide: [_Export and visualize Netdata metrics in +Graphite_](/docs/guides/export/export-netdata-metrics-graphite.md). Or, start [using host +labels](/docs/guides/using-host-labels.md) on exported metrics. + +### Related reference documentation + +- [Exporting engine reference](/exporting/README.md) +- [OpenTSDB connector](/exporting/opentsdb/README.md) +- [Graphite connector](/exporting/graphite/README.md) + + diff --git a/docs/export/external-databases.md b/docs/export/external-databases.md new file mode 100644 index 0000000..a542e8e --- /dev/null +++ b/docs/export/external-databases.md @@ -0,0 +1,89 @@ + + +# Export metrics to external time-series databases + +Netdata allows you to export metrics to external time-series databases with the [exporting +engine](/exporting/README.md). This system uses a number of **connectors** to initiate connections to [more than +thirty](#supported-databases) supported databases, including InfluxDB, Prometheus, Graphite, ElasticSearch, and much +more. + +The exporting engine resamples Netdata's thousands of per-second metrics at a user-configurable interval, and can export +metrics to multiple time-series databases simultaneously. + +Based on your needs and resources you allocated to your external time-series database, you can configure the interval +that metrics are exported or export only certain charts with filtering. You can also choose whether metrics are exported +as-collected, a normalized average, or the sum/volume of metrics values over the configured interval. + +Exporting is an important part of Netdata's effort to be [interoperable](/docs/overview/netdata-monitoring-stack.md) +with other monitoring software. You can use an external time-series database for long-term metrics retention, further +analysis, or correlation with other tools, such as application tracing. + +## Supported databases + +Netdata supports exporting metrics to the following databases through several +[connectors](/exporting/README.md#features). Once you find the connector that works for your database, open its +documentation and the [enabling a connector](/docs/export/enable-connector.md) doc for details on enabling it. + +- **AppOptics**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **AWS Kinesis**: [AWS Kinesis Data Streams](/exporting/aws_kinesis/README.md) +- **Azure Data Explorer**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Azure Event Hubs**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Blueflood**: [Graphite](/exporting/graphite/README.md) +- **Chronix**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Cortex**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **CrateDB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **ElasticSearch**: [Graphite](/exporting/graphite/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **Gnocchi**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Google BigQuery**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Google Cloud Pub/Sub**: [Google Cloud Pub/Sub Service](/exporting/pubsub/README.md) +- **Graphite**: [Graphite](/exporting/graphite/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **InfluxDB**: [Graphite](/exporting/graphite/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **IRONdb**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **JSON**: [JSON document databases](/exporting/json/README.md) +- **Kafka**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **KairosDB**: [Graphite](/exporting/graphite/README.md), [OpenTSDB](/exporting/opentsdb/README.md) +- **M3DB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **MetricFire**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **MongoDB**: [MongoDB](/exporting/mongodb/README.md) +- **New Relic**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **OpenTSDB**: [OpenTSDB](/exporting/opentsdb/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **PostgreSQL**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) + via [PostgreSQL Prometheus Adapter](https://github.com/CrunchyData/postgresql-prometheus-adapter) +- **Prometheus**: [Prometheus scraper](/exporting/prometheus/README.md) +- **TimescaleDB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md), + [netdata-timescale-relay](/exporting/TIMESCALE.md) +- **QuasarDB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **SignalFx**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Splunk**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **TiKV**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Thanos**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **VictoriaMetrics**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Wavefront**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) + +Can't find your preferred external time-series database? Ask our [community](https://community.netdata.cloud/) for +solutions, or file an [issue on +GitHub](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml). + +## What's next? + +We recommend you read our document on [enabling a connector](/docs/export/enable-connector.md) to learn about the +process and discover important configuration options. If you would rather skip ahead, click on any of the above links to +connectors for their reference documentation, which outline any prerequisites to install for that connector, along with +connector-specific configuration options. + +Read about one possible use case for exporting metrics in our guide: [_Export and visualize Netdata metrics in +Graphite_](/docs/guides/export/export-netdata-metrics-graphite.md). + +### Related reference documentation + +- [Exporting engine reference](/exporting/README.md) + + diff --git a/docs/get-started.mdx b/docs/get-started.mdx new file mode 100644 index 0000000..892baa0 --- /dev/null +++ b/docs/get-started.mdx @@ -0,0 +1,107 @@ +--- +title: "Get started with Netdata" +description: "Download and install the open-source Netdata monitoring agent on physical/virtual servers, Linux (Ubuntu/Debian/CentOS/etc), Docker, Kubernetes, and many others, often with one command." +type: how-to +sidebar_label: "Get started" +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/get-started.mdx +--- + +import { OneLineInstallWget } from '@site/src/components/OneLineInstall/' +import { Install, InstallBox } from '@site/src/components/Install/' + +# Get started with Netdata + +Netdata is a free and open-source (FOSS) monitoring agent that collects thousands of hardware and software metrics from +any physical or virtual system (we call them _nodes_). These metrics are organized in an easy-to-use and -navigate interface. + +Together with [Netdata Cloud](https://learn.netdata.cloud/docs/cloud), you can monitor your entire infrastructure in +real time and troubleshoot problems that threaten the health of your nodes. + +Netdata runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices. It +runs on Linux distributions (Ubuntu, Debian, CentOS, and more), container/microservice platforms (Kubernetes clusters, +Docker), and many other operating systems (FreeBSD, macOS), with no `sudo` required. + +## Install on Linux with one-line installer + +The **recommended** way to install Netdata on a Linux node (physical, virtual, container, IoT) is our one-line +[kickstart script](/packaging/installer/methods/kickstart.md). This script automatically installs dependencies and +builds Netdata from its source code. + +Copy the script, paste it into your node's terminal, and hit `Enter` to begin the installation process. + + + +Jump down to [what's next](#whats-next) to learn how to view your new dashboard and take your next steps monitoring and +troubleshooting with Netdata. + +## Other installation options + + + + + + + + + + + +## What's next? + +To start using Netdata, open a browser and navigate to `http://NODE:19999`, replacing `NODE` with either `localhost` or +the hostname/IP address of a remote node. + +Where you go from here is based on your use case, immediate needs, and experience with monitoring and troubleshooting. + +### Dashboard + +Learn more about [how the dashboard works](/docs/dashboard/how-dashboard-works.mdx), or dive directly into the many ways +to [interact with charts](/docs/dashboard/interact-charts.mdx). + +### Configuration + +Discover the recommended way to [configure Netdata's settings or behavior](/docs/configure/nodes.md) using our built-in +`edit-config` script, then apply that knowledge to mission-critical tweaks, such as [changing how long Netdata stores +metrics](/docs/store/change-metrics-storage.md). + +### Data collection + +If Netdata didn't autodetect all the hardware, containers, services, or applications running on your node, you should +learn more about [how data collectors work](/docs/collect/how-collectors-work.md). If there's a [supported +collector](/collectors/COLLECTORS.md) for metrics you need, [configure the collector](/docs/collect/enable-configure.md) +or read about its requirements to configure your endpoint to publish metrics in the correct format and endpoint. + +### Alarms & notifications + +Netdata comes with hundreds of preconfigured alarms, designed by our monitoring gurus in parallel with our open-source +community, but you may want to [edit alarms](/docs/monitor/configure-alarms.md) or [enable +notifications](/docs/monitor/enable-notifications.md) to customize your Netdata experience. + +### Need to monitor multiple nodes in one place? + +For robust multi-node monitoring from a single interface, consider [Netdata +Cloud](https://learn.netdata.cloud/docs/cloud), which streams, aggregates, and visualizes metrics from any number of +nodes. It's all the same out-of-the-box, zero-configuration functionality of the open-source monitoring agent, but for +any number of distributed nodes, _entirely for free_. + +There is an alternative for those who aren't interested in using Netdata Cloud, albeit with some required configuration. +Each node can [stream](/streaming/README.md) its metrics to any other node, and the default +[registry](/registry/README.md) is configurable to create a private "network" of Netdata dashboards. diff --git a/docs/guides/collect-apache-nginx-web-logs.md b/docs/guides/collect-apache-nginx-web-logs.md new file mode 100644 index 0000000..a75a4b1 --- /dev/null +++ b/docs/guides/collect-apache-nginx-web-logs.md @@ -0,0 +1,127 @@ + + +# Monitor Nginx or Apache web server log files with Netdata + +Log files have been a critical resource for developers and system administrators who want to understand the health and +performance of their web servers, and Netdata is taking important steps to make them even more valuable. + +By parsing web server log files with Netdata, and seeing the volume of redirects, requests, or server errors over time, +you can better understand what's happening on your infrastructure. Too many bad requests? Maybe a recent deploy missed a +few small SVG icons. Too many requests? Time to batten down the hatches—it's a DDoS. + +You can use the [LTSV log format](http://ltsv.org/), track TLS and cipher usage, and the whole parser is faster than +ever. In one test on a system with SSD storage, the collector consistently parsed the logs for 200,000 requests in +200ms, using ~30% of a single core. + +The [web_log](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog/) collector is currently compatible +with [Nginx](https://nginx.org/en/) and [Apache](https://httpd.apache.org/). + +This guide will walk you through using the new Go-based web log collector to turn the logs these web servers +constantly write to into real-time insights into your infrastructure. + +## Set up your web servers + +As with all data sources, Netdata can auto-detect Nginx or Apache servers if you installed them using their standard +installation procedures. + +Almost all web server installations will need _no_ configuration to start collecting metrics. As long as your web server +has readable access log file, you can configure the web log plugin to access and parse it. + +## Custom configuration of the web log collector + +The web log collector's default configuration comes with a few example jobs that should cover most Linux distributions +and their default locations for log files: + +```yaml +# [ JOBS ] +jobs: +# NGINX +# debian, arch + - name: nginx + path: /var/log/nginx/access.log + +# gentoo + - name: nginx + path: /var/log/nginx/localhost.access_log + +# APACHE +# debian + - name: apache + path: /var/log/apache2/access.log + +# gentoo + - name: apache + path: /var/log/apache2/access_log + +# arch + - name: apache + path: /var/log/httpd/access_log + +# debian + - name: apache_vhosts + path: /var/log/apache2/other_vhosts_access.log + +# GUNICORN + - name: gunicorn + path: /var/log/gunicorn/access.log + + - name: gunicorn + path: /var/log/gunicorn/gunicorn-access.log +``` + +However, if your log files were not auto-detected, it might be because they are in a different location. Try the default +`web_log.conf` file. + +```bash +./edit-config go.d/web_log.conf +``` + +To create a new custom configuration, you need to set the `path` parameter to point to your web server's access log +file. You can give it a `name` as well, and set the `log_type` to `auto`. + +```yaml +jobs: + - name: example + path: /path/to/file.log + log_type: auto +``` + +Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. Netdata should pick up your web server's access log and +begin showing real-time charts! + +### Custom log formats and fields + +The web log collector is capable of parsing custom Nginx and Apache log formats and presenting them as charts, but we'll +leave that topic for a separate guide. + +We do have [extensive +documentation](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog/#custom-log-format) on how +to build custom parsing for Nginx and Apache logs. + +## Tweak web log collector alarms + +Over time, we've created some default alarms for web log monitoring. These alarms are designed to work only when your +web server is receiving more than 120 requests per minute. Otherwise, there's simply not enough data to make conclusions +about what is "too few" or "too many." + +- [web log alarms](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/web_log.conf). + +You can also edit this file directly with `edit-config`: + +```bash +./edit-config health.d/weblog.conf +``` + +For more information about editing the defaults or writing new alarm entities, see our [health monitoring +documentation](/health/README.md). + +## What's next? + +Now that you have web log collection up and running, we recommend you take a look at the collector's [documentation](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog/) for some ideas of how you can turn these rather "boring" logs into powerful real-time tools for keeping your servers happy. + +Don't forget to give GitHub user [Wing924](https://github.com/Wing924) a big 👍 for his hard work in starting up the Go +refactoring effort. diff --git a/docs/guides/collect-unbound-metrics.md b/docs/guides/collect-unbound-metrics.md new file mode 100644 index 0000000..8edcab1 --- /dev/null +++ b/docs/guides/collect-unbound-metrics.md @@ -0,0 +1,140 @@ + + +# Monitor Unbound DNS servers with Netdata + +[Unbound](https://nlnetlabs.nl/projects/unbound/about/) is a "validating, recursive, caching DNS resolver" from NLNet +Labs. In v1.19 of Netdata, we release a completely refactored collector for collecting real-time metrics from Unbound +servers and displaying them in Netdata dashboards. + +Unbound runs on FreeBSD, OpenBSD, NetBSD, macOS, Linux, and Windows, and supports DNS-over-TLS, which ensures that DNS +queries and answers are all encrypted with TLS. In theory, that should reduce the risk of eavesdropping or +man-in-the-middle attacks when communicating to DNS servers. + +This guide will show you how to collect dozens of essential metrics from your Unbound servers with minimal +configuration. + +## Set up your Unbound installation + +As with all data sources, Netdata can auto-detect Unbound servers if you installed them using the standard installation +procedure. + +Regardless of whether you're connecting to a local or remote Unbound server, you need to be able to access the server's +`remote-control` interface via an IP address, FQDN, or Unix socket. + +To set up the `remote-control` interface, you can use `unbound-control`. First, run `unbound-control-setup` to generate +the TLS key files that will encrypt connections to the remote interface. Then add the following to the end of your +`unbound.conf` configuration file. See the [Unbound +documentation](https://nlnetlabs.nl/documentation/unbound/howto-setup/#setup-remote-control) for more details on using +`unbound-control`, such as how to handle situations when Unbound is run under a unique user. + +```conf +# enable remote-control +remote-control: + control-enable: yes +``` + +Next, make your `unbound.conf`, `unbound_control.key`, and `unbound_control.pem` files readable by Netdata using [access +control lists](https://wiki.archlinux.org/index.php/Access_Control_Lists) (ACL). + +```bash +sudo setfacl -m user:netdata:r unbound.conf +sudo setfacl -m user:netdata:r unbound_control.key +sudo setfacl -m user:netdata:r unbound_control.pem +``` + +Finally, take note whether you're using Unbound in _cumulative_ or _non-cumulative_ mode. This will become relevant when +configuring the collector. + +## Configure the Unbound collector + +You may not need to do any more configuration to have Netdata collect your Unbound metrics. + +If you followed the steps above to enable `remote-control` and make your Unbound files readable by Netdata, that should +be enough. Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. You should see Unbound metrics in your Netdata +dashboard! + +![Some charts showing Unbound metrics in real-time](https://user-images.githubusercontent.com/1153921/69659974-93160f00-103c-11ea-88e6-27e9efcf8c0d.png) + +If that failed, you will need to manually configure `unbound.conf`. See the next section for details. + +### Manual setup for a local Unbound server + +To configure Netdata's Unbound collector module, navigate to your Netdata configuration directory (typically at +`/etc/netdata/`) and use `edit-config` to initialize and edit your Unbound configuration file. + +```bash +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ +sudo ./edit-config go.d/unbound.conf +``` + +The file contains all the global and job-related parameters. The `name` setting is required, and two Unbound servers +can't have the same name. + +> It is important you know whether your Unbound server is running in cumulative or non-cumulative mode, as a conflict +> between modes will create incorrect charts. + +Here are two examples for local Unbound servers, which may work based on your unique setup: + +```yaml +jobs: + - name: local + address: 127.0.0.1:8953 + cumulative: no + use_tls: yes + tls_skip_verify: yes + tls_cert: /path/to/unbound_control.pem + tls_key: /path/to/unbound_control.key + + - name: local + address: 127.0.0.1:8953 + cumulative: yes + use_tls: no +``` + +Netdata will attempt to read `unbound.conf` to get the appropriate `address`, `cumulative`, `use_tls`, `tls_cert`, and +`tls_key` parameters. + +Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +### Manual setup for a remote Unbound server + +Collecting metrics from remote Unbound servers requires manual configuration. There are too many possibilities to cover +all remote connections here, but the [default `unbound.conf` +file](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/unbound.conf) contains a few useful examples: + +```yaml +jobs: + - name: remote + address: 203.0.113.10:8953 + use_tls: no + + - name: remote_cumulative + address: 203.0.113.11:8953 + use_tls: no + cumulative: yes + + - name: remote + address: 203.0.113.10:8953 + cumulative: yes + use_tls: yes + tls_cert: /etc/unbound/unbound_control.pem + tls_key: /etc/unbound/unbound_control.key +``` + +To see all the available options, see the default [unbound.conf +file](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/unbound.conf). + +## What's next? + +Now that you're collecting metrics from your Unbound servers, let us know how it's working for you! There's always room +for improvement or refinement based on real-world use cases. Feel free to [file an +issue](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml) with your +thoughts. + + diff --git a/docs/guides/configure/performance.md b/docs/guides/configure/performance.md new file mode 100644 index 0000000..cb52a11 --- /dev/null +++ b/docs/guides/configure/performance.md @@ -0,0 +1,277 @@ + + +# How to optimize the Netdata Agent's performance + +We designed the Netdata Agent to be incredibly lightweight, even when it's collecting a few thousand dimensions every +second and visualizing that data into hundreds of charts. When properly configured for a production node, the Agent +itself should never use more than 1% of a single CPU core, roughly 50-100 MiB of RAM, and minimal disk I/O to collect, +store, and visualize all this data. + +We take this scalability seriously. We have one user [running +Netdata](https://github.com/netdata/netdata/issues/1323#issuecomment-266427841) on a system with 144 cores and 288 +threads. Despite collecting 100,000 metrics every second, the Agent still only uses 9% CPU utilization on a +single core. + +But not everyone has such powerful systems at their disposal. For example, you might run the Agent on a cloud VM with +only 512 MiB of RAM, or an IoT device like a [Raspberry Pi](/docs/guides/monitor/pi-hole-raspberry-pi.md). In these +cases, reducing Netdata's footprint beyond its already diminutive size can pay big dividends, giving your services more +horsepower while still monitoring the health and the performance of the node, OS, hardware, and applications. + +The default settings of the Netdata Agent are not optimized for performance, but for a simple standalone setup. We want +the first install to give you something you can run without any configuration. Most of the settings and options are +enabled, since we want you to experience the full thing. + + +## Prerequisites + +- A node running the Netdata Agent. +- Familiarity with configuring the Netdata Agent with `edit-config`. + +If you're not familiar with how to configure the Netdata Agent, read our [node configuration +doc](/docs/configure/nodes.md) before continuing with this guide. This guide assumes familiarity with the Netdata config +directory, using `edit-config`, and the process of uncommenting/editing various settings in `netdata.conf` and other +configuration files. + +## What affects Netdata's performance? + +Netdata's performance is primarily affected by **data collection/retention** and **clients accessing data**. + +You can configure almost all aspects of data collection/retention, and certain aspects of clients accessing data. For +example, you can't control how many users might be viewing a local Agent dashboard, [viewing an +infrastructure](/docs/visualize/overview-infrastructure.md) in real-time with Netdata Cloud, or running [Metric +Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations). + +The Netdata Agent runs with the lowest possible [process scheduling +policy](/daemon/README.md#netdata-process-scheduling-policy), which is `nice 19`, and uses the `idle` process scheduler. +Together, these settings ensure that the Agent only gets CPU resources when the node has CPU resources to space. If the +node reaches 100% CPU utilization, the Agent is stopped first to ensure your applications get any available resources. +In addition, under heavy load, collectors that require disk I/O may stop and show gaps in charts. + +Let's walk through the best ways to improve the Netdata Agent's performance. + +## Reduce collection frequency + +The fastest way to improve the Agent's resource utilization is to reduce how often it collects metrics. + +### Global + +If you don't need per-second metrics, or if the Netdata Agent uses a lot of CPU even when no one is viewing that node's +dashboard, configure the Agent to collect metrics less often. + +Open `netdata.conf` and edit the `update every` setting. The default is `1`, meaning that the Agent collects metrics +every second. + +If you change this to `2`, Netdata enforces a minimum `update every` setting of 2 seconds, and collects metrics every +other second, which will effectively halve CPU utilization. Set this to `5` or `10` to collect metrics every 5 or 10 +seconds, respectively. + +```conf +[global] + update every = 5 +``` + +### Specific plugin or collector + +Every collector and plugin has its own `update every` setting, which you can also change in the `go.d.conf`, +`python.d.conf`, or `charts.d.conf` files, or in individual collector configuration files. If the `update +every` for an individual collector is less than the global, the Netdata Agent uses the global setting. See the [enable +or configure a collector](/docs/collect/enable-configure.md) doc for details. + +To reduce the frequency of an [internal +plugin/collector](/docs/collect/how-collectors-work.md#collector-architecture-and-terminology), open `netdata.conf` and +find the appropriate section. For example, to reduce the frequency of the `apps` plugin, which collects and visualizes +metrics on application resource utilization: + +```conf +[plugin:apps] + update every = 5 +``` + +To [configure an individual collector](/docs/collect/enable-configure.md), open its specific configuration file with +`edit-config` and look for the `update_every` setting. For example, to reduce the frequency of the `nginx` collector, +run `sudo ./edit-config go.d/nginx.conf`: + +```conf +# [ GLOBAL ] +update_every: 10 +``` + +## Disable unneeded plugins or collectors + +If you know that you don't need an [entire plugin or a specific +collector](/docs/collect/how-collectors-work.md#collector-architecture-and-terminology), you can disable any of them. +Keep in mind that if a plugin/collector has nothing to do, it simply shuts down and does not consume system resources. +You will only improve the Agent's performance by disabling plugins/collectors that are actively collecting metrics. + +Open `netdata.conf` and scroll down to the `[plugins]` section. To disable any plugin, uncomment it and set the value to +`no`. For example, to explicitly keep the `proc` and `go.d` plugins enabled while disabling `python.d` and `charts.d`. + +```conf +[plugins] + proc = yes + python.d = no + charts.d = no + go.d = yes +``` + +Disable specific collectors by opening their respective plugin configuration files, uncommenting the line for the +collector, and setting its value to `no`. + +```bash +sudo ./edit-config go.d.conf +sudo ./edit-config python.d.conf +sudo ./edit-config charts.d.conf +``` + +For example, to disable a few Python collectors: + +```conf +modules: + apache: no + dockerd: no + fail2ban: no +``` + +## Lower memory usage for metrics retention + +Reduce the disk space that the [database engine](/database/engine/README.md) uses to retain metrics by editing +the `dbengine multihost disk space` option in `netdata.conf`. The default value is `256`, but can be set to a minimum of +`64`. By reducing the disk space allocation, Netdata also needs to store less metadata in the node's memory. + +The `page cache size` option also directly impacts Netdata's memory usage, but has a minimum value of `32`. + +Reducing the value of `dbengine multihost disk space` does slim down Netdata's resource usage, but it also reduces how +long Netdata retains metrics. Find the right balance of performance and metrics retention by using the [dbengine +calculator](/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics). + +All the settings are found in the `[global]` section of `netdata.conf`: + +```conf +[db] + memory mode = dbengine + page cache size = 32 + dbengine multihost disk space = 256 +``` + +To save even more memory, you can disable the dbengine and reduce retention to just 30 minutes, as shown below: + +```conf +[db] + storage tiers = 1 + mode = alloc + retention = 1800 +``` + +Metric retention is not important in certain use cases, such as: + - Data collection nodes stream collected metrics collected to a centralization point. + - Data collection nodes export their metrics to another time series DB, or are scraped by Prometheus + - Netdata installed only during incidents, to get richer information. +In such cases, you may not want to use the dbengine at all and instead opt for memory mode +`memory mode = alloc` or `memory mode = none`. + +## Disable machine learning + +Automated anomaly detection may be a powerful tool, but we recommend it to only be enabled on Netdata parents +that sit outside your production infrastructure, or if you have cpu and memory to spare. You can disable ML +with the following: + +```conf +[ml] + enabled = no +``` + +## Run Netdata behind Nginx + +A dedicated web server like Nginx provides far more robustness than the Agent's internal [web server](/web/README.md). +Nginx can handle more concurrent connections, reuse idle connections, and use fast gzip compression to reduce payloads. + +For details on installing Nginx as a proxy for the local Agent dashboard, see our [Nginx +doc](/docs/Running-behind-nginx.md). + +After you complete Nginx setup according to the doc linked above, we recommend setting `keepalive` to `1024`, and using +gzip compression with the following options in the `location /` block: + +```conf + location / { + ... + gzip on; + gzip_proxied any; + gzip_types *; + } +``` + +Finally, edit `netdata.conf` with the following settings: + +```conf +[global] + bind socket to IP = 127.0.0.1 + disconnect idle web clients after seconds = 3600 + enable web responses gzip compression = no +``` + +## Disable/lower gzip compression for the dashboard + +If you choose not to run the Agent behind Nginx, you can disable or lower the Agent's web server's gzip compression. +While gzip compression does reduce the size of the HTML/CSS/JS payload, it does use additional CPU while a user is +looking at the local Agent dashboard. + +To disable gzip compression, open `netdata.conf` and find the `[web]` section: + +```conf +[web] + enable gzip compression = no +``` + +Or to lower the default compression level: + +```conf +[web] + enable gzip compression = yes + gzip compression level = 1 +``` + +## Disable logs + +If you installation is working correctly, and you're not actively auditing Netdata's logs, disable them in +`netdata.conf`. + +```conf +[logs] + debug log = none + error log = none + access log = none +``` + +## Disable health checks + +If you are streaming metrics to parent nodes, we recommend you run your health checks on the parent, for all the metrics collected +by the children nodes. This saves resources on the children and makes it easier to configure or disable alerts and agent notifications. + +The parents by default run health checks for each child, as long as it is connected (the details are in `stream.conf`). +On the child nodes you should add to `netdata.conf` the following: + +```conf +[health] + enabled = no +``` + +## What's next? + +We hope this guide helped you better understand how to optimize the performance of the Netdata Agent. + +Now that your Agent is running smoothly, we recommend you [secure your nodes](/docs/configure/nodes.md) if you haven't +already. + +Next, dive into some of Netdata's more complex features, such as configuring its health watchdog or exporting metrics to +an external time-series database. + +- [Interact with dashboards and charts](/docs/visualize/interact-dashboards-charts.md) +- [Configure health alarms](/docs/monitor/configure-alarms.md) +- [Export metrics to external time-series databases](/docs/export/external-databases.md) + +[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fguides%2Fconfigure%2Fperformance.md&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) diff --git a/docs/guides/deploy/ansible.md b/docs/guides/deploy/ansible.md new file mode 100644 index 0000000..35c9460 --- /dev/null +++ b/docs/guides/deploy/ansible.md @@ -0,0 +1,176 @@ + + +# Deploy Netdata with Ansible + +Netdata's [one-line kickstart](/docs/get-started.mdx) is zero-configuration, highly adaptable, and compatible with tons +of different operating systems and Linux distributions. You can use it on bare metal, VMs, containers, and everything +in-between. + +But what if you're trying to bootstrap an infrastructure monitoring solution as quickly as possible? What if you need to +deploy Netdata across an entire infrastructure with many nodes? What if you want to make this deployment reliable, +repeatable, and idempotent? What if you want to write and deploy your infrastructure or cloud monitoring system like +code? + +Enter [Ansible](https://ansible.com), a popular system provisioning, configuration management, and infrastructure as +code (IaC) tool. Ansible uses **playbooks** to glue many standardized operations together with a simple syntax, then run +those operations over standard and secure SSH connections. There's no agent to install on the remote system, so all you +have to worry about is your application and your monitoring software. + +Ansible has some competition from the likes of [Puppet](https://puppet.com/) or [Chef](https://www.chef.io/), but the +most valuable feature about Ansible is **idempotent**. From the [Ansible +glossary](https://docs.ansible.com/ansible/latest/reference_appendices/glossary.html) + +> An operation is idempotent if the result of performing it once is exactly the same as the result of performing it +> repeatedly without any intervening actions. + +Idempotency means you can run an Ansible playbook against your nodes any number of times without affecting how they +operate. When you deploy Netdata with Ansible, you're also deploying _monitoring as code_. + +In this guide, we'll walk through the process of using an [Ansible +playbook](https://github.com/netdata/community/tree/main/netdata-agent-deployment/ansible-quickstart) to automatically +deploy the Netdata Agent to any number of distributed nodes, manage the configuration of each node, and connect them to +your Netdata Cloud account. You'll go from some unmonitored nodes to a infrastructure monitoring solution in a matter of +minutes. + +## Prerequisites + +- A Netdata Cloud account. [Sign in and create one](https://app.netdata.cloud) if you don't have one already. +- An administration system with [Ansible](https://www.ansible.com/) installed. +- One or more nodes that your administration system can access via [SSH public + keys](https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key) (preferably password-less). + +## Download and configure the playbook + +First, download the +[playbook](https://github.com/netdata/community/tree/main/netdata-agent-deployment/ansible-quickstart), move it to the +current directory, and remove the rest of the cloned repository, as it's not required for using the Ansible playbook. + +```bash +git clone https://github.com/netdata/community.git +mv community/netdata-agent-deployment/ansible-quickstart . +rm -rf community +``` + +Or if you don't want to clone the entire repository, use the [gitzip browser extension](https://gitzip.org/) to get the netdata-agent-deployment directory as a zip file. + +Next, `cd` into the Ansible directory. + +```bash +cd ansible-quickstart +``` + +### Edit the `hosts` file + +The `hosts` file contains a list of IP addresses or hostnames that Ansible will try to run the playbook against. The +`hosts` file that comes with the repository contains two example IP addresses, which you should replace according to the +IP address/hostname of your nodes. + +```conf +203.0.113.0 hostname=node-01 +203.0.113.1 hostname=node-02 +``` + +You can also set the `hostname` variable, which appears both on the local Agent dashboard and Netdata Cloud, or you can +omit the `hostname=` string entirely to use the system's default hostname. + +#### Set the login user (optional) + +If you SSH into your nodes as a user other than `root`, you need to configure `hosts` according to those user names. Use +the `ansible_user` variable to set the login user. For example: + +```conf +203.0.113.0 hostname=ansible-01 ansible_user=example +``` + +#### Set your SSH key (optional) + +If you use an SSH key other than `~/.ssh/id_rsa` for logging into your nodes, you can set that on a per-node basis in +the `hosts` file with the `ansible_ssh_private_key_file` variable. For example, to log into a Lightsail instance using +two different SSH keys supplied by AWS. + +```conf +203.0.113.0 hostname=ansible-01 ansible_ssh_private_key_file=~/.ssh/LightsailDefaultKey-us-west-2.pem +203.0.113.1 hostname=ansible-02 ansible_ssh_private_key_file=~/.ssh/LightsailDefaultKey-us-east-1.pem +``` + +### Edit the `vars/main.yml` file + +In order to connect your node(s) to your Space in Netdata Cloud, and see all their metrics in real-time in [composite +charts](/docs/visualize/overview-infrastructure.md) or perform [Metric +Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations), you need to set the `claim_token` +and `claim_room` variables. + +To find your `claim_token` and `claim_room`, go to Netdata Cloud, then click on your Space's name in the top navigation, +then click on **Manage your Space**. Click on the **Nodes** tab in the panel that appears, which displays a script with +`token` and `room` strings. + +![Animated GIF of finding the claiming script and the token and room +strings](https://user-images.githubusercontent.com/1153921/98740235-f4c3ac00-2367-11eb-8ffd-e9ab0f04c463.gif) + +Copy those strings into the `claim_token` and `claim_rooms` variables. + +```yml +claim_token: XXXXX +claim_rooms: XXXXX +``` + +Change the `dbengine_multihost_disk_space` if you want to change the metrics retention policy by allocating more or less +disk space for storing metrics. The default is 2048 Mib, or 2 GiB. + +Because we're connecting this node to Netdata Cloud, and will view its dashboards there instead of via the IP address or +hostname of the node, the playbook disables that local dashboard by setting `web_mode` to `none`. This gives a small +security boost by not allowing any unwanted access to the local dashboard. + +You can read more about this decision, or other ways you might lock down the local dashboard, in our [node security +doc](https://learn.netdata.cloud/docs/configure/secure-nodes). + +> Curious about why Netdata's dashboard is open by default? Read our [blog +> post](https://www.netdata.cloud/blog/netdata-agent-dashboard/) on that zero-configuration design decision. + +## Run the playbook + +Time to run the playbook from your administration system: + +```bash +ansible-playbook -i hosts tasks/main.yml +``` + +Ansible first connects to your node(s) via SSH, then [collects +facts](https://docs.ansible.com/ansible/latest/user_guide/playbooks_vars_facts.html#ansible-facts) about the system. +This playbook doesn't use these facts, but you could expand it to provision specific types of systems based on the +makeup of your infrastructure. + +Next, Ansible makes changes to each node according to the `tasks` defined in the playbook, and +[returns](https://docs.ansible.com/ansible/latest/reference_appendices/common_return_values.html#changed) whether each +task results in a changed, failure, or was skipped entirely. + +The task to install Netdata will take a few minutes per node, so be patient! Once the playbook reaches the connect to Cloud +task, your nodes start populating your Space in Netdata Cloud. + +## What's next? + +Go use Netdata! + +If you need a bit more guidance for how you can use Netdata for health monitoring and performance troubleshooting, see +our [documentation](https://learn.netdata.cloud/docs). It's designed like a comprehensive guide, based on what you might +want to do with Netdata, so use those categories to dive in. + +Some of the best places to start: + +- [Enable or configure a collector](/docs/collect/enable-configure.md) +- [Supported collectors list](/collectors/COLLECTORS.md) +- [See an overview of your infrastructure](/docs/visualize/overview-infrastructure.md) +- [Interact with dashboards and charts](/docs/visualize/interact-dashboards-charts.md) +- [Change how long Netdata stores metrics](/docs/store/change-metrics-storage.md) + +We're looking for more deployment and configuration management strategies, whether via Ansible or other +provisioning/infrastructure as code software, such as Chef or Puppet, in our [community +repo](https://github.com/netdata/community). Anyone is able to fork the repo and submit a PR, either to improve this +playbook, extend it, or create an entirely new experience for deploying Netdata across entire infrastructure. + + diff --git a/docs/guides/export/export-netdata-metrics-graphite.md b/docs/guides/export/export-netdata-metrics-graphite.md new file mode 100644 index 0000000..dd742e4 --- /dev/null +++ b/docs/guides/export/export-netdata-metrics-graphite.md @@ -0,0 +1,183 @@ + +import { OneLineInstallWget } from '@site/src/components/OneLineInstall/' + +# Export and visualize Netdata metrics in Graphite + +Collecting metrics is an essential part of monitoring any application, service, or infrastructure, but it's not the +final step for any developer, sysadmin, SRE, or DevOps engineer who's keeping an eye on things. To take meaningful +action on these metrics, you may need to develop a stack of monitoring tools that work in parallel to help you diagnose +anomalies and discover root causes faster. + +We designed Netdata with interoperability in mind. The Agent collects thousands of metrics every second, and then what +you do with them is up to you. You can [store metrics in the database engine](/docs/guides/longer-metrics-storage.md), +or send them to another time series database for long-term storage or further analysis using Netdata's [exporting +engine](/docs/export/external-databases.md). + +In this guide, we'll show you how to export Netdata metrics to [Graphite](https://graphiteapp.org/) for long-term +storage and further analysis. Graphite is a free open-source software (FOSS) tool that collects graphs numeric +time-series data, such as all the metrics collected by the Netdata Agent itself. Using Netdata and Graphite together, +you get more visibility into the health and performance of your entire infrastructure. + +![A custom dashboard in Grafana with Netdata +metrics](https://user-images.githubusercontent.com/1153921/83903855-b8828480-a713-11ea-8edb-927ba521599b.png) + +Let's get started. + +## Install the Netdata Agent + +If you don't have the Netdata Agent installed already, visit the [installation guide](/packaging/installer/README.md) +for the recommended instructions for your system. In most cases, you can use the one-line installation script: + + + +Once installation finishes, open your browser and navigate to `http://NODE:19999`, replacing `NODE` with the IP address +or hostname of your system, to find the Agent dashboard. + +## Install Graphite via Docker + +For this guide, we'll install Graphite using Docker. See the [Docker documentation](https://docs.docker.com/get-docker/) +for details if you don't yet have it installed on your system. + +> If you already have Graphite installed, skip this step. If you want to install via a different method, see the +> [Graphite installation docs](https://graphite.readthedocs.io/en/latest/install.html), with the caveat that some +> configuration settings may be different. + +Start up the Graphite image with `docker run`. + +```bash +docker run -d \ + --name graphite \ + --restart=always \ + -p 80:80 \ + -p 2003-2004:2003-2004 \ + -p 2023-2024:2023-2024 \ + -p 8125:8125/udp \ + -p 8126:8126 \ + graphiteapp/graphite-statsd +``` + +Open your browser and navigate to `http://NODE`, to see the Graphite interface. Nothing yet, but we'll fix that soon +enough. + +![An empty Graphite +dashboard](https://user-images.githubusercontent.com/1153921/83798958-ea371500-a659-11ea-8403-d46f77a05b78.png) + +## Enable the Graphite exporting connector + +You're now ready to begin exporting Netdata metrics to Graphite. + +Begin by using `edit-config` to open the `exporting.conf` file. + +```bash +cd /etc/netdata # Replace this path with your Netdata config directory +sudo ./edit-config exporting.conf +``` + +If you haven't already, enable the exporting engine by setting `enabled` to `yes` in the `[exporting:global]` section. + +```conf +[exporting:global] + enabled = yes +``` + +Next, configure the connector. Find the `[graphite:my_graphite_instance]` example section and uncomment the line. +Replace `my_graphite_instance` with a name of your choice. Let's go with `[graphite:netdata]`. Set `enabled` to `yes` +and uncomment the line. Your configuration should now look like this: + +```conf +[graphite:netdata] + enabled = yes + # destination = localhost + # data source = average + # prefix = netdata + # hostname = my_hostname + # update every = 10 + # buffer on failures = 10 + # timeout ms = 20000 + # send names instead of ids = yes + # send charts matching = * + # send hosts matching = localhost * +``` + +Set the `destination` setting to `localhost:2003`. By default, the Docker image for Graphite listens on port `2003` for +incoming metrics. If you installed Graphite a different way, or tweaked the `docker run` command, you may need to change +the port accordingly. + +```conf +[graphite:netdata] + enabled = yes + destination = localhost:2003 + ... +``` + +We'll not worry about the rest of the settings for now. Restart the Agent using `sudo systemctl restart netdata`, or the +[appropriate method](/docs/configure/start-stop-restart.md) for your system, to spin up the exporting engine. + +## See and organize Netdata metrics in Graphite + +Head back to the Graphite interface again, then click on the **Dashboard** link to get started with Netdata's exported +metrics. You can also navigate directly to `http://NODE/dashboard`. + +Let's switch the interface to help you understand which metrics Netdata is exporting to Graphite. Click on **Dashboard** +and **Configure UI**, then choose the **Tree** option. Refresh your browser to change the UI. + +![Change the Graphite +UI](https://user-images.githubusercontent.com/1153921/83798697-77c63500-a659-11ea-8ed5-5e274953c871.png) + +You should now see a tree of available contexts, including one that matches the hostname of the Agent exporting metrics. +In this example, the Agent's hostname is `arcturus`. + +Let's add some system CPU charts so you can monitor the long-term health of your system. Click through the tree to find +**hostname → system → cpu** metrics, then click on the **user** context. A chart with metrics from that context appears +in the dashboard. Add a few other system CPU charts to flesh things out. + +Next, let's combine one or two of these charts. Click and drag one chart onto the other, and wait until the green **Drop +to merge** dialog appears. Release to merge the charts. + +![Merging charts in +Graphite](https://user-images.githubusercontent.com/1153921/83817628-1bbfd880-a67a-11ea-81bc-05efc639b6ce.png) + +Finally, save your dashboard. Click **Dashboard**, then **Save As**, then choose a name. Your dashboard is now saved. + +Of course, this is just the beginning of the customization you can do with Graphite. You can change the time range, +share your dashboard with others, or use the composer to customize the size and appearance of specific charts. Learn +more about adding, modifying, and combining graphs in the [Graphite +docs](https://graphite.readthedocs.io/en/latest/dashboard.html). + +## Monitor the exporting engine + +As soon as the exporting engine begins, Netdata begins reporting metrics about the system's health and performance. + +![Graphs for monitoring the exporting +engine](https://user-images.githubusercontent.com/1153921/83800787-e5c02b80-a65c-11ea-865a-c447d2ce4cbb.png) + +You can use these charts to verify that Netdata is properly exporting metrics to Graphite. You can even add these +exporting charts to your Graphite dashboard! + +### Add exporting charts to Netdata Cloud + +You can also show these exporting engine metrics on Netdata Cloud. If you don't have an account already, go [sign +in](https://app.netdata.cloud) and get started for free. If you need some help along the way, read the [get started with +Cloud guide](https://learn.netdata.cloud/docs/cloud/get-started). + +Add more metrics to a War Room's Nodes view by clicking on the **Add metric** button, then typing `exporting` into the +context field. Choose the exporting contexts you want to add, then click **Add**. You'll see these charts alongside any +others you've customized in Netdata Cloud. + +![Exporting engine metrics in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/83902769-db139e00-a711-11ea-828e-aa7e32b04c75.png) + +## What's next? + +What you do with your exported metrics is entirely up to you, but as you might have seen in the Graphite connector +configuration block, there are many other ways to tweak and customize which metrics you export to Graphite and how +often. + +For full details about each configuration option and what it does, see the [exporting reference +guide](/exporting/README.md). + + diff --git a/docs/guides/longer-metrics-storage.md b/docs/guides/longer-metrics-storage.md new file mode 100644 index 0000000..8ccd958 --- /dev/null +++ b/docs/guides/longer-metrics-storage.md @@ -0,0 +1,158 @@ + + +# Netdata Longer Metrics Retention + +Metrics retention affects 3 parameters on the operation of a Netdata Agent: + +1. The disk space required to store the metrics. +2. The memory the Netdata Agent will require to have that retention available for queries. +3. The CPU resources that will be required to query longer time-frames. + +As retention increases, the resources required to support that retention increase too. + +Since Netdata Agents usually run at the edge, inside production systems, Netdata Agent **parents** should be considered. When having a **parent - child** setup, the child (the Netdata Agent running on a production system) delegates all its functions, including longer metrics retention and querying, to the parent node that can dedicate more resources to this task. A single Netdata Agent parent can centralize multiple children Netdata Agents (dozens, hundreds, or even thousands depending on its available resources). + + +## Ephemerality of metrics + +The ephemerality of metrics plays an important role in retention. In environments where metrics stop being collected and new metrics are constantly being generated, we are interested about 2 parameters: + +1. The **expected concurrent number of metrics** as an average for the lifetime of the database. + This affects mainly the storage requirements. + +2. The **expected total number of unique metrics** for the lifetime of the database. + This affects mainly the memory requirements for having all these metrics indexed and available to be queried. + +## Granularity of metrics + +The granularity of metrics (the frequency they are collected and stored, i.e. their resolution) is significantly affecting retention. + +Lowering the granularity from per second to every two seconds, will double their retention and half the CPU requirements of the Netdata Agent, without affecting disk space or memory requirements. + +## Which database mode to use + +Netdata Agents support multiple database modes. + +The default mode `[db].mode = dbengine` has been designed to scale for longer retentions. + +The other available database modes are designed to minimize resource utilization and should usually be considered on **parent - child** setups at the children side. + +So, + +* On a single node setup, use `[db].mode = dbengine` to increase retention. +* On a **parent - child** setup, use `[db].mode = dbengine` on the parent to increase retention and a more resource efficient mode (like `save`, `ram` or `none`) for the child to minimize resources utilization. + +To use `dbengine`, set this in `netdata.conf` (it is the default): + +``` +[db] + mode = dbengine +``` + +## Tiering + +`dbengine` supports tiering. Tiering allows having up to 3 versions of the data: + +1. Tier 0 is the high resolution data. +2. Tier 1 is the first tier that samples data every 60 data collections of Tier 0. +3. Tier 2 is the second tier that samples data every 3600 data collections of Tier 0 (60 of Tier 1). + +To enable tiering set `[db].storage tiers` in `netdata.conf` (the default is 1, to enable only Tier 0): + +``` +[db] + mode = dbengine + storage tiers = 3 +``` + +## Disk space requirements + +Netdata Agents require about 1 bytes on disk per database point on Tier 0 and 4 times more on higher tiers (Tier 1 and 2). They require 4 times more storage per point compared to Tier 0, because for every point higher tiers store `min`, `max`, `sum`, `count` and `anomaly rate` (the values are 5, but they require 4 times the storage because `count` and `anomaly rate` are 16-bit integers). The `average` is calculated on the fly at query time using `sum / count`. + +### Tier 0 - per second for a week + +For 2000 metrics, collected every second and retained for a week, Tier 0 needs: 1 byte x 2000 metrics x 3600 secs per hour x 24 hours per day x 7 days per week = 1100MB. + +The setting to control this is in `netdata.conf`: + +``` +[db] + mode = dbengine + + # per second data collection + update every = 1 + + # enable only Tier 0 + storage tiers = 1 + + # Tier 0, per second data for a week + dbengine multihost disk space MB = 1100 +``` + +By setting it to `1100` and restarting the Netdata Agent, this node will start maintaining about a week of data. But pay attention to the number of metrics. If you have more than 2000 metrics on a node, or you need more that a week of high resolution metrics, you may need to adjust this setting accordingly. + +### Tier 1 - per minute for a month + +Tier 1 is by default sampling the data every 60 points of Tier 0. If Tier 0 is per second, then Tier 1 is per minute. + +Tier 1 needs 4 times more storage per point compared to Tier 0. So, for 2000 metrics, with per minute resolution, retained for a month, Tier 1 needs: 4 bytes x 2000 metrics x 60 minutes per hour x 24 hours per day x 30 days per month = 330MB. + +Do this in `netdata.conf`: + +``` +[db] + mode = dbengine + + # per second data collection + update every = 1 + + # enable only Tier 0 and Tier 1 + storage tiers = 2 + + # Tier 0, per second data for a week + dbengine multihost disk space MB = 1100 + + # Tier 1, per minute data for a month + dbengine tier 1 multihost disk space MB = 330 +``` + +Once `netdata.conf` is edited, the Netdata Agent needs to be restarted for the changes to take effect. + +### Tier 2 - per hour for a year + +Tier 2 is by default sampling data every 3600 points of Tier 0 (60 of Tier 1). If Tier 0 is per second, then Tier 2 is per hour. + +The storage requirements are the same to Tier 1. + +For 2000 metrics, with per hour resolution, retained for a year, Tier 2 needs: 4 bytes x 2000 metrics x 24 hours per day x 365 days per year = 67MB. + +Do this in `netdata.conf`: + +``` +[db] + mode = dbengine + + # per second data collection + update every = 1 + + # enable only Tier 0 and Tier 1 + storage tiers = 3 + + # Tier 0, per second data for a week + dbengine multihost disk space MB = 1100 + + # Tier 1, per minute data for a month + dbengine tier 1 multihost disk space MB = 330 + + # Tier 2, per hour data for a year + dbengine tier 2 multihost disk space MB = 67 +``` + +Once `netdata.conf` is edited, the Netdata Agent needs to be restarted for the changes to take effect. + + + diff --git a/docs/guides/monitor-cockroachdb.md b/docs/guides/monitor-cockroachdb.md new file mode 100644 index 0000000..46dd253 --- /dev/null +++ b/docs/guides/monitor-cockroachdb.md @@ -0,0 +1,136 @@ + + +# Monitor CockroachDB metrics with Netdata + +[CockroachDB](https://github.com/cockroachdb/cockroach) is an open-source project that brings SQL databases into +scalable, disaster-resilient cloud deployments. Thanks to a [new CockroachDB +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/cockroachdb/) released in +[v1.20](https://blog.netdata.cloud/posts/release-1.20/), you can now monitor any number of CockroachDB databases with +maximum granularity using Netdata. Collect more than 50 unique metrics and put them on interactive visualizations +designed for better visual anomaly detection. + +Netdata itself uses CockroachDB as part of its Netdata Cloud infrastructure, so we're happy to introduce this new +collector and help others get started with it straight away. + +Let's dive in and walk through the process of monitoring CockroachDB metrics with Netdata. + +## What's in this guide + +- [Configure the CockroachDB collector](#configure-the-cockroachdb-collector) + - [Manual setup for a local CockroachDB database](#manual-setup-for-a-local-cockroachdb-database) +- [Tweak CockroachDB alarms](#tweak-cockroachdb-alarms) + +## Configure the CockroachDB collector + +Because _all_ of Netdata's collectors can auto-detect the services they monitor, you _shouldn't_ need to worry about +configuring CockroachDB. Netdata only needs to regularly query the database's `_status/vars` page to gather metrics and +display them on the dashboard. + +If your CockroachDB instance is accessible through `http://localhost:8080/` or `http://127.0.0.1:8080`, your setup is +complete. Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, and refresh your browser. You should see CockroachDB +metrics in your Netdata dashboard! + +
+ CPU utilization charts from a CockroachDB database monitored by Netdata +
CPU utilization charts from a CockroachDB database monitored by Netdata
+
+ +> Note: Netdata collects metrics from CockroachDB every 10 seconds, instead of our usual 1 second, because CockroachDB +> only updates `_status/vars` every 10 seconds. You can't change this setting in CockroachDB. + +If you don't see CockroachDB charts, you may need to configure the collector manually. + +### Manual setup for a local CockroachDB database + +To configure Netdata's CockroachDB collector, navigate to your Netdata configuration directory (typically at +`/etc/netdata/`) and use `edit-config` to initialize and edit your CockroachDB configuration file. + +```bash +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ +./edit-config go.d/cockroachdb.conf +``` + +Scroll down to the `[JOBS]` section at the bottom of the file. You will see the two default jobs there, which you can +edit, or create a new job with any of the parameters listed above in the file. Both the `name` and `url` values are +required, and everything else is optional. + +For a production cluster, you'll use either an IP address or the system's hostname. Be sure that your remote system +allows TCP communication on port 8080, or whichever port you have configured CockroachDB's [Admin +UI](https://www.cockroachlabs.com/docs/stable/monitoring-and-alerting.html#prometheus-endpoint) to listen on. + +```yaml +# [ JOBS ] +jobs: + - name: remote + url: http://203.0.113.0:8080/_status/vars + + - name: remote_hostname + url: http://cockroachdb.example.com:8080/_status/vars +``` + +For a secure cluster, use `https` in the `url` field instead. + +```yaml +# [ JOBS ] +jobs: + - name: remote + url: https://203.0.113.0:8080/_status/vars + tls_skip_verify: yes # If your certificate is self-signed + + - name: remote_hostname + url: https://cockroachdb.example.com:8080/_status/vars + tls_skip_verify: yes # If your certificate is self-signed +``` + +You can add as many jobs as you'd like based on how many CockroachDB databases you have—Netdata will create separate +charts for each job. Once you've edited `cockroachdb.conf` according to the needs of your infrastructure, restart +Netdata to see your new charts. + +
+ Charts showing a node failure during a simulated test +
Charts showing a node failure during a simulated test
+
+ +## Tweak CockroachDB alarms + +This release also includes eight pre-configured alarms for live nodes, such as whether the node is live, storage +capacity, issues with replication, and the number of SQL connections/statements. See [health.d/cockroachdb.conf on +GitHub](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/cockroachdb.conf) for details. + +You can also edit these files directly with `edit-config`: + +```bash +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ +./edit-config health.d/cockroachdb.conf # You may need to use `sudo` for write privileges +``` + +For more information about editing the defaults or writing new alarm entities, see our health monitoring [quickstart +guide](/health/QUICKSTART.md). + +## What's next? + +Now that you're collecting metrics from your CockroachDB databases, let us know how it's working for you! There's always +room for improvement or refinement based on real-world use cases. Feel free to [file an +issue](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml) with your +thoughts. + +Also, be sure to check out these useful resources: + +- [Netdata's CockroachDB + documentation](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/cockroachdb/) +- [Netdata's CockroachDB + configuration](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/cockroachdb.conf) +- [Netdata's CockroachDB + alarms](https://github.com/netdata/netdata/blob/29d9b5e51603792ee27ef5a21f1de0ba8e130158/health/health.d/cockroachdb.conf) +- [CockroachDB homepage](https://www.cockroachlabs.com/product/) +- [CockroachDB documentation](https://www.cockroachlabs.com/docs/stable/) +- [`_status/vars` endpoint + docs](https://www.cockroachlabs.com/docs/stable/monitoring-and-alerting.html#prometheus-endpoint) +- [Monitor CockroachDB with + Prometheus](https://www.cockroachlabs.com/docs/stable/monitor-cockroachdb-with-prometheus.html) + + diff --git a/docs/guides/monitor-hadoop-cluster.md b/docs/guides/monitor-hadoop-cluster.md new file mode 100644 index 0000000..62403f8 --- /dev/null +++ b/docs/guides/monitor-hadoop-cluster.md @@ -0,0 +1,203 @@ + + +# Monitor a Hadoop cluster with Netdata + +Hadoop is an [Apache project](https://hadoop.apache.org/) is a framework for processing large sets of data across a +distributed cluster of systems. + +And while Hadoop is designed to be a highly-available and fault-tolerant service, those who operate a Hadoop cluster +will want to monitor the health and performance of their [Hadoop Distributed File System +(HDFS)](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) and [Zookeeper](https://zookeeper.apache.org/) +implementations. + +Netdata comes with built-in and pre-configured support for monitoring both HDFS and Zookeeper. + +This guide assumes you have a Hadoop cluster, with HDFS and Zookeeper, running already. If you don't, please follow +the [official Hadoop +instructions](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html) or an +alternative, like the guide available from +[DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-in-stand-alone-mode-on-ubuntu-18-04). + +For more specifics on the collection modules used in this guide, read the respective pages in our documentation: + +- [HDFS](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/hdfs) +- [Zookeeper](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/zookeeper) + +## Set up your HDFS and Zookeeper installations + +As with all data sources, Netdata can auto-detect HDFS and Zookeeper nodes if you installed them using the standard +installation procedure. + +For Netdata to collect HDFS metrics, it needs to be able to access the node's `/jmx` endpoint. You can test whether an +JMX endpoint is accessible by using `curl HDFS-IP:PORT/jmx`. For a NameNode, you should see output similar to the +following: + +```json +{ + "beans" : [ { + "name" : "Hadoop:service=NameNode,name=JvmMetrics", + "modelerType" : "JvmMetrics", + "MemNonHeapUsedM" : 65.67851, + "MemNonHeapCommittedM" : 67.3125, + "MemNonHeapMaxM" : -1.0, + "MemHeapUsedM" : 154.46341, + "MemHeapCommittedM" : 215.0, + "MemHeapMaxM" : 843.0, + "MemMaxM" : 843.0, + "GcCount" : 15, + "GcTimeMillis" : 305, + "GcNumWarnThresholdExceeded" : 0, + "GcNumInfoThresholdExceeded" : 0, + "GcTotalExtraSleepTime" : 92, + "ThreadsNew" : 0, + "ThreadsRunnable" : 6, + "ThreadsBlocked" : 0, + "ThreadsWaiting" : 7, + "ThreadsTimedWaiting" : 34, + "ThreadsTerminated" : 0, + "LogFatal" : 0, + "LogError" : 0, + "LogWarn" : 2, + "LogInfo" : 348 + }, + { ... } + ] +} +``` + +The JSON result for a DataNode's `/jmx` endpoint is slightly different: + +```json +{ + "beans" : [ { + "name" : "Hadoop:service=DataNode,name=DataNodeActivity-dev-slave-01.dev.local-9866", + "modelerType" : "DataNodeActivity-dev-slave-01.dev.local-9866", + "tag.SessionId" : null, + "tag.Context" : "dfs", + "tag.Hostname" : "dev-slave-01.dev.local", + "BytesWritten" : 500960407, + "TotalWriteTime" : 463, + "BytesRead" : 80689178, + "TotalReadTime" : 41203, + "BlocksWritten" : 16, + "BlocksRead" : 16, + "BlocksReplicated" : 4, + ... + }, + { ... } + ] +} +``` + +If Netdata can't access the `/jmx` endpoint for either a NameNode or DataNode, it will not be able to auto-detect and +collect metrics from your HDFS implementation. + +Zookeeper auto-detection relies on an accessible client port and a allow-listed `mntr` command. For more details on +`mntr`, see Zookeeper's documentation on [cluster +options](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_clusterOptions) and [Zookeeper +commands](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkCommands). + +## Configure the HDFS and Zookeeper modules + +To configure Netdata's HDFS module, navigate to your Netdata directory (typically at `/etc/netdata/`) and use +`edit-config` to initialize and edit your HDFS configuration file. + +```bash +cd /etc/netdata/ +sudo ./edit-config go.d/hdfs.conf +``` + +At the bottom of the file, you will see two example jobs, both of which are commented out: + +```yaml +# [ JOBS ] +#jobs: +# - name: namenode +# url: http://127.0.0.1:9870/jmx +# +# - name: datanode +# url: http://127.0.0.1:9864/jmx +``` + +Uncomment these lines and edit the `url` value(s) according to your setup. Now's the time to add any other configuration +details, which you can find inside of the `hdfs.conf` file itself. Most production implementations will require TLS +certificates. + +The result for a simple HDFS setup, running entirely on `localhost` and without certificate authentication, might look +like this: + +```yaml +# [ JOBS ] +jobs: + - name: namenode + url: http://127.0.0.1:9870/jmx + + - name: datanode + url: http://127.0.0.1:9864/jmx +``` + +At this point, Netdata should be configured to collect metrics from your HDFS servers. Let's move on to Zookeeper. + +Next, use `edit-config` again to initialize/edit your `zookeeper.conf` file. + +```bash +cd /etc/netdata/ +sudo ./edit-config go.d/zookeeper.conf +``` + +As with the `hdfs.conf` file, head to the bottom, uncomment the example jobs, and tweak the `address` values according +to your setup. Again, you may need to add additional configuration options, like TLS certificates. + +```yaml +jobs: + - name : local + address : 127.0.0.1:2181 + + - name : remote + address : 203.0.113.10:2182 +``` + +Finally, [restart Netdata](/docs/configure/start-stop-restart.md). + +```sh +sudo systemctl restart netdata +``` + +Upon restart, Netdata should recognize your HDFS/Zookeeper servers, enable the HDFS and Zookeeper modules, and begin +showing real-time metrics for both in your Netdata dashboard. 🎉 + +## Configuring HDFS and Zookeeper alarms + +The Netdata community helped us create sane defaults for alarms related to both HDFS and Zookeeper. You may want to +investigate these to ensure they work well with your Hadoop implementation. + +- [HDFS alarms](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/hdfs.conf) +- [Zookeeper alarms](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/zookeeper.conf) + +You can also access/edit these files directly with `edit-config`: + +```bash +sudo /etc/netdata/edit-config health.d/hdfs.conf +sudo /etc/netdata/edit-config health.d/zookeeper.conf +``` + +For more information about editing the defaults or writing new alarm entities, see our [health monitoring +documentation](/health/README.md). + +## What's next? + +If you're having issues with Netdata auto-detecting your HDFS/Zookeeper servers, or want to help improve how Netdata +collects or presents metrics from these services, feel free to [file an +issue](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml). + +- Read up on the [HDFS configuration + file](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/hdfs.conf) to understand how to configure + global options or per-job options, such as username/password, TLS certificates, timeouts, and more. +- Read up on the [Zookeeper configuration + file](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/zookeeper.conf) to understand how to configure + global options or per-job options, timeouts, TLS certificates, and more. + + diff --git a/docs/guides/monitor/anomaly-detection-python.md b/docs/guides/monitor/anomaly-detection-python.md new file mode 100644 index 0000000..ad8398c --- /dev/null +++ b/docs/guides/monitor/anomaly-detection-python.md @@ -0,0 +1,189 @@ + + +# Detect anomalies in systems and applications + +Beginning with v1.27, the [open-source Netdata Agent](https://github.com/netdata/netdata) is capable of unsupervised +[anomaly detection](https://en.wikipedia.org/wiki/Anomaly_detection) with machine learning (ML). As with all things +Netdata, the anomalies collector comes with preconfigured alarms and instant visualizations that require no query +languages or organizing metrics. You configure the collector to look at specific charts, and it handles the rest. + +Netdata's implementation uses a handful of functions in the [Python Outlier Detection (PyOD) +library](https://github.com/yzhao062/pyod/tree/master), which periodically runs a `train` function that learns what +"normal" looks like on your node and creates an ML model for each chart, then utilizes the +[`predict_proba()`](https://pyod.readthedocs.io/en/latest/api_cc.html#pyod.models.base.BaseDetector.predict_proba) and +[`predict()`](https://pyod.readthedocs.io/en/latest/api_cc.html#pyod.models.base.BaseDetector.predict) PyOD functions to +quantify how anomalous certain charts are. + +All these metrics and alarms are available for centralized monitoring in [Netdata Cloud](https://app.netdata.cloud). If +you choose to sign up for Netdata Cloud and [connect your nodes](/claim/README.md), you will have the ability to run +tailored anomaly detection on every node in your infrastructure, regardless of its purpose or workload. + +In this guide, you'll learn how to set up the anomalies collector to instantly detect anomalies in an Nginx web server +and/or the node that hosts it, which will give you the tools to configure parallel unsupervised monitors for any +application in your infrastructure. Let's get started. + +![Example anomaly detection with an Nginx web +server](https://user-images.githubusercontent.com/1153921/103586700-da5b0a00-4ea2-11eb-944e-46edd3f83e3a.png) + +## Prerequisites + +- A node running the Netdata Agent. If you don't yet have that, [get Netdata](/docs/get-started.mdx). +- A Netdata Cloud account. [Sign up](https://app.netdata.cloud) if you don't have one already. +- Familiarity with configuring the Netdata Agent with [`edit-config`](/docs/configure/nodes.md). +- _Optional_: An Nginx web server running on the same node to follow the example configuration steps. + +## Install required Python packages + +The anomalies collector uses a few Python packages, available with `pip3`, to run ML training. It requires +[`numba`](http://numba.pydata.org/), [`scikit-learn`](https://scikit-learn.org/stable/), +[`pyod`](https://pyod.readthedocs.io/en/latest/), in addition to +[`netdata-pandas`](https://github.com/netdata/netdata-pandas), which is a package built by the Netdata team to pull data +from a Netdata Agent's API into a [Pandas](https://pandas.pydata.org/). Read more about `netdata-pandas` on its [package +repo](https://github.com/netdata/netdata-pandas) or in Netdata's [community +repo](https://github.com/netdata/community/tree/main/netdata-agent-api/netdata-pandas). + +```bash +# Become the netdata user +sudo su -s /bin/bash netdata + +# Install required packages for the netdata user +pip3 install --user netdata-pandas==0.0.38 numba==0.50.1 scikit-learn==0.23.2 pyod==0.8.3 +``` + +> If the `pip3` command fails, you need to install it. For example, on an Ubuntu system, use `sudo apt install +> python3-pip`. + +Use `exit` to become your normal user again. + +## Enable the anomalies collector + +Navigate to your [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory) and use `edit-config` +to open the `python.d.conf` file. + +```bash +sudo ./edit-config python.d.conf +``` + +In `python.d.conf` file, search for the `anomalies` line. If the line exists, set the value to `yes`. Add the line +yourself if it doesn't already exist. Either way, the final result should look like: + +```conf +anomalies: yes +``` + +[Restart the Agent](/docs/configure/start-stop-restart.md) with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to start up the anomalies collector. By default, the +model training process runs every 30 minutes, and uses the previous 4 hours of metrics to establish a baseline for +health and performance across the default included charts. + +> 💡 The anomaly collector may need 30-60 seconds to finish its initial training and have enough data to start +> generating anomaly scores. You may need to refresh your browser tab for the **Anomalies** section to appear in menus +> on both the local Agent dashboard or Netdata Cloud. + +## Configure the anomalies collector + +Open `python.d/anomalies.conf` with `edit-conf`. + +```bash +sudo ./edit-config python.d/anomalies.conf +``` + +The file contains many user-configurable settings with sane defaults. Here are some important settings that don't +involve tweaking the behavior of the ML training itself. + +- `charts_regex`: Which charts to train models for and run anomaly detection on, with each chart getting a separate + model. +- `charts_to_exclude`: Specific charts, selected by the regex in `charts_regex`, to exclude. +- `train_every_n`: How often to train the ML models. +- `train_n_secs`: The number of historical observations to train each model on. The default is 4 hours, but if your node + doesn't have historical metrics going back that far, consider [changing the metrics retention + policy](/docs/store/change-metrics-storage.md) or reducing this window. +- `custom_models`: A way to define custom models that you want anomaly probabilities for, including multi-node or + streaming setups. + +> ⚠️ Setting `charts_regex` with many charts or `train_n_secs` to a very large number will have an impact on the +> resources and time required to train a model for every chart. The actual performance implications depend on the +> resources available on your node. If you plan on changing these settings beyond the default, or what's mentioned in +> this guide, make incremental changes to observe the performance impact. Considering `train_max_n` to cap the number of +> observations actually used to train on. + +### Run anomaly detection on Nginx and log file metrics + +As mentioned above, this guide uses an Nginx web server to demonstrate how the anomalies collector works. You must +configure the collector to monitor charts from the +[Nginx](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/nginx) and [web +log](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog) collectors. + +`charts_regex` allows for some basic regex, such as wildcards (`*`) to match all contexts with a certain pattern. For +example, `system\..*` matches with any chart with a context that begins with `system.`, and ends in any number of other +characters (`.*`). Note the escape character (`\`) around the first period to capture a period character exactly, and +not any character. + +Change `charts_regex` in `anomalies.conf` to the following: + +```conf + charts_regex: 'system\..*|nginx_local\..*|web_log_nginx\..*|apps.cpu|apps.mem' +``` + +This value tells the anomaly collector to train against every `system.` chart, every `nginx_local` chart, every +`web_log_nginx` chart, and specifically the `apps.cpu` and `apps.mem` charts. + +![The anomalies collector chart with many +dimensions](https://user-images.githubusercontent.com/1153921/102813877-db5e4880-4386-11eb-8040-d7a1d7a476bb.png) + +### Remove some metrics from anomaly detection + +As you can see in the above screenshot, this node is now looking for anomalies in many places. The result is a single +`anomalies_local.probability` chart with more than twenty dimensions, some of which the dashboard hides at the bottom of +a scrollable area. In addition, training and analyzing the anomaly collector on many charts might require more CPU +utilization that you're willing to give. + +First, explicitly declare which `system.` charts to monitor rather than of all of them using regex (`system\..*`). + +```conf + charts_regex: 'system\.cpu|system\.load|system\.io|system\.net|system\.ram|nginx_local\..*|web_log_nginx\..*|apps.cpu|apps.mem' +``` + +Next, remove some charts with the `charts_to_exclude` setting. For this example, using an Nginx web server, focus on the +volume of requests/responses, not, for example, which type of 4xx response a user might receive. + +```conf + charts_to_exclude: 'web_log_nginx.excluded_requests,web_log_nginx.responses_by_status_code_class,web_log_nginx.status_code_class_2xx_responses,web_log_nginx.status_code_class_4xx_responses,web_log_nginx.current_poll_uniq_clients,web_log_nginx.requests_by_http_method,web_log_nginx.requests_by_http_version,web_log_nginx.requests_by_ip_proto' +``` + +![The anomalies collector with less +dimensions](https://user-images.githubusercontent.com/1153921/102820642-d69f9180-4392-11eb-91c5-d3d166d40105.png) + +Apply the ideas behind the collector's regex and exclude settings to any other +[system](/docs/collect/system-metrics.md), [container](/docs/collect/container-metrics.md), or +[application](/docs/collect/application-metrics.md) metrics you want to detect anomalies for. + +## What's next? + +Now that you know how to set up unsupervised anomaly detection in the Netdata Agent, using an Nginx web server as an +example, it's time to apply that knowledge to other mission-critical parts of your infrastructure. If you're not sure +what to monitor next, check out our list of [collectors](/collectors/COLLECTORS.md) to see what kind of metrics Netdata +can collect from your systems, containers, and applications. + +Keep on moving to [part 2](/docs/guides/monitor/visualize-monitor-anomalies.md), which covers the charts and alarms +Netdata creates for unsupervised anomaly detection. + +For a different troubleshooting experience, try out the [Metric +Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations) feature in Netdata Cloud. Metric +Correlations helps you perform faster root cause analysis by narrowing a dashboard to only the charts most likely to be +related to an anomaly. + +### Related reference documentation + +- [Netdata Agent · Anomalies collector](/collectors/python.d.plugin/anomalies/README.md) +- [Netdata Agent · Nginx collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/nginx) +- [Netdata Agent · web log collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog) +- [Netdata Cloud · Metric Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations) diff --git a/docs/guides/monitor/anomaly-detection.md b/docs/guides/monitor/anomaly-detection.md new file mode 100644 index 0000000..e98c5c0 --- /dev/null +++ b/docs/guides/monitor/anomaly-detection.md @@ -0,0 +1,75 @@ + + + + +## Overview + +As of [`v1.32.0`](https://github.com/netdata/netdata/releases/tag/v1.32.0), Netdata comes with some ML powered [anomaly detection](https://en.wikipedia.org/wiki/Anomaly_detection) capabilities built into it and available to use out of the box, with zero configuration required (ML was enabled by default in `v1.35.0-29-nightly` in [this PR](https://github.com/netdata/netdata/pull/13158), previously it required a one line config change). + +This means that in addition to collecting raw value metrics, the Netdata agent will also produce an [`anomaly-bit`](https://learn.netdata.cloud/docs/agent/ml#anomaly-bit---100--anomalous-0--normal) every second which will be `100` when recent raw metric values are considered anomalous by Netdata and `0` when they look normal. Once we aggregate beyond one second intervals this aggregated `anomaly-bit` becomes an ["anomaly rate"](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate---averageanomaly-bit). + +To be as concrete as possible, the below api call shows how to access the raw anomaly bit of the `system.cpu` chart from the [london.my-netdata.io](https://london.my-netdata.io) Netdata demo server. Passing `options=anomaly-bit` returns the anomay bit instead of the raw metric value. + +``` +https://london.my-netdata.io/api/v1/data?chart=system.cpu&options=anomaly-bit +``` + +If we aggregate the above to just 1 point by adding `points=1` we get an "[Anomaly Rate](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate---averageanomaly-bit)": + +``` +https://london.my-netdata.io/api/v1/data?chart=system.cpu&options=anomaly-bit&points=1 +``` + +The fundamentals of Netdata's anomaly detection approach and implmentation are covered in lots more detail in the [agent ML documentation](https://learn.netdata.cloud/docs/agent/ml). + +This guide will explain how to get started using these ML based anomaly detection capabilities within Netdata. + +## Anomaly Advisor + +The [Anomaly Advisor](https://learn.netdata.cloud/docs/cloud/insights/anomaly-advisor) is the flagship anomaly detection feature within Netdata. In the "Anomalies" tab of Netdata you will see an overall "Anomaly Rate" chart that aggregates node level anomaly rate for all nodes in a space. The aim of this chart is to make it easy to quickly spot periods of time where the overall "[node anomaly rate](https://learn.netdata.cloud/docs/agent/ml#node-anomaly-rate)" is evelated in some unusual way and for what node or nodes this relates to. + +![image](https://user-images.githubusercontent.com/2178292/175928290-490dd8b9-9c55-4724-927e-e145cb1cc837.png) + +Once an area on the Anomaly Rate chart is highlighted netdata will append a "heatmap" to the bottom of the screen that shows which metrics were more anomalous in the highlighted timeframe. Each row in the heatmap consists of an anomaly rate sparkline graph that can be expanded to reveal the raw underlying metric chart for that dimension. + +![image](https://user-images.githubusercontent.com/2178292/175929162-02c8fe69-cc4f-4cf4-9b3a-a5e559a6feca.png) + +## Embedded Anomaly Rate Charts + +Charts in both the [Overview](https://learn.netdata.cloud/docs/cloud/visualize/overview) and [single node dashboard](https://learn.netdata.cloud/docs/cloud/visualize/overview#jump-to-single-node-dashboards) tabs also expose the underlying anomaly rates for each dimension so users can easily see if the raw metrics are considered anomalous or not by Netdata. + +Pressing the anomalies icon (next to the information icon in the chart header) will expand the anomaly rate chart to make it easy to see how the anomaly rate for any individual dimension corresponds to the raw underlying data. In the example below we can see that the spike in `system.pgpgio|in` corresponded in the anomaly rate for that dimension jumping to 100% for a small period of time until the spike passed. + +![image](https://user-images.githubusercontent.com/2178292/175933078-5dd951ff-7709-4bb9-b4be-34199afb3945.png) + +## Anomaly Rate Based Alerts + +It is possible to use the `anomaly-bit` when defining traditional Alerts within netdata. The `anomaly-bit` is just another `options` parameter that can be passed as part of an [alarm line lookup](https://learn.netdata.cloud/docs/agent/health/reference#alarm-line-lookup). + +You can see some example ML based alert configurations below: + +- [Anomaly rate based CPU dimensions alarm](https://learn.netdata.cloud/docs/agent/health/reference#example-8---anomaly-rate-based-cpu-dimensions-alarm) +- [Anomaly rate based CPU chart alarm](https://learn.netdata.cloud/docs/agent/health/reference#example-9---anomaly-rate-based-cpu-chart-alarm) +- [Anomaly rate based node level alarm](https://learn.netdata.cloud/docs/agent/health/reference#example-10---anomaly-rate-based-node-level-alarm) +- More examples in the [`/health/health.d/ml.conf`](https://github.com/netdata/netdata/blob/master/health/health.d/ml.conf) file that ships with the agent. + +## Learn More + +Check out the resources below to learn more about how Netdata is approaching ML: + +- [Agent ML documentation](https://learn.netdata.cloud/docs/agent/ml). +- [Anomaly Advisor documentation](https://learn.netdata.cloud/docs/cloud/insights/anomaly-advisor). +- [Metric Correlations documentation](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations). +- Anomaly Advisor [launch blog post](https://www.netdata.cloud/blog/introducing-anomaly-advisor-unsupervised-anomaly-detection-in-netdata/). +- Netdata Approach to ML [blog post](https://www.netdata.cloud/blog/our-approach-to-machine-learning/). +- `areal/ml` related [GitHub Discussions](https://github.com/netdata/netdata/discussions?discussions_q=label%3Aarea%2Fml). +- Netdata Machine Learning Meetup [deck](https://docs.google.com/presentation/d/1rfSxktg2av2k-eMwMbjN0tXeo76KC33iBaxerYinovs/edit?usp=sharing) and [YouTube recording](https://www.youtube.com/watch?v=eJGWZHVQdNU). +- Netdata Anomaly Advisor [YouTube Playlist](https://youtube.com/playlist?list=PL-P-gAHfL2KPeUcCKmNHXC-LX-FfdO43j). diff --git a/docs/guides/monitor/dimension-templates.md b/docs/guides/monitor/dimension-templates.md new file mode 100644 index 0000000..5391273 --- /dev/null +++ b/docs/guides/monitor/dimension-templates.md @@ -0,0 +1,176 @@ + + +# Use dimension templates to create dynamic alarms + +Your ability to monitor the health of your systems and applications relies on your ability to create and maintain +the best set of alarms for your particular needs. + +In v1.18 of Netdata, we introduced **dimension templates** for alarms, which simplifies the process of writing [alarm +entities](/health/REFERENCE.md#health-entity-reference) for charts with many dimensions. + +Dimension templates can condense many individual entities into one—no more copy-pasting one entity and changing the +`alarm`/`template` and `lookup` lines for each dimension you'd like to monitor. + +They are, however, an advanced health monitoring feature. For more basic instructions on creating your first alarm, +check out our [health monitoring documentation](/health/README.md), which also includes +[examples](/health/REFERENCE.md#example-alarms). + +## The fundamentals of `foreach` + +Our dimension templates update creates a new `foreach` parameter to the existing [`lookup` +line](/health/REFERENCE.md#alarm-line-lookup). This is where the magic happens. + +You use the `foreach` parameter to specify which dimensions you want to monitor with this single alarm. You can separate +them with a comma (`,`) or a pipe (`|`). You can also use a [Netdata simple pattern](/libnetdata/simple_pattern/README.md) +to create many alarms with a regex-like syntax. + +The `foreach` parameter _has_ to be the last parameter in your `lookup` line, and if you have both `of` and `foreach` in +the same `lookup` line, Netdata will ignore the `of` parameter and use `foreach` instead. + +Let's get into some examples so you can see how the new parameter works. + +> ⚠️ The following entities are examples to showcase the functionality and syntax of dimension templates. They are not +> meant to be run as-is on production systems. + +## Condensing entities with `foreach` + +Let's say you want to monitor the `system`, `user`, and `nice` dimensions in your system's overall CPU utilization. +Before dimension templates, you would need the following three entities: + +```yaml + alarm: cpu_system + on: system.cpu +lookup: average -10m percentage of system + every: 1m + warn: $this > 50 + crit: $this > 80 + + alarm: cpu_user + on: system.cpu +lookup: average -10m percentage of user + every: 1m + warn: $this > 50 + crit: $this > 80 + + alarm: cpu_nice + on: system.cpu +lookup: average -10m percentage of nice + every: 1m + warn: $this > 50 + crit: $this > 80 +``` + +With dimension templates, you can condense these into a single alarm. Take note of the `alarm` and `lookup` lines. + +```yaml + alarm: cpu_template + on: system.cpu +lookup: average -10m percentage foreach system,user,nice + every: 1m + warn: $this > 50 + crit: $this > 80 +``` + +The `alarm` line specifies the naming scheme Netdata will use. You can use whatever naming scheme you'd like, with `.` +and `_` being the only allowed symbols. + +The `lookup` line has changed from `of` to `foreach`, and we're now passing three dimensions. + +In this example, Netdata will create three alarms with the names `cpu_template_system`, `cpu_template_user`, and +`cpu_template_nice`. Every minute, each alarm will use the same database query to calculate the average CPU usage for +the `system`, `user`, and `nice` dimensions over the last 10 minutes and send out alarms if necessary. + +You can find these three alarms active by clicking on the **Alarms** button in the top navigation, and then clicking on +the **All** tab and scrolling to the **system - cpu** collapsible section. + +![Three new alarms created from the dimension template](https://user-images.githubusercontent.com/1153921/66218994-29523800-e67f-11e9-9bcb-9bca23e2c554.png) + +Let's look at some other examples of how `foreach` works so you can best apply it in your configurations. + +### Using a Netdata simple pattern in `foreach` + +In the last example, we used `foreach system,user,nice` to create three distinct alarms using dimension templates. But +what if you want to quickly create alarms for _all_ the dimensions of a given chart? + +Use a [simple pattern](/libnetdata/simple_pattern/README.md)! One example of a simple pattern is a single wildcard +(`*`). + +Instead of monitoring system CPU usage, let's monitor per-application CPU usage using the `apps.cpu` chart. Passing a +wildcard as the simple pattern tells Netdata to create a separate alarm for _every_ process on your system: + +```yaml + alarm: app_cpu + on: apps.cpu +lookup: average -10m percentage foreach * + every: 1m + warn: $this > 50 + crit: $this > 80 +``` + +This entity will now create alarms for every dimension in the `apps.cpu` chart. Given that most `apps.cpu` charts have +10 or more dimensions, using the wildcard ensures you catch every CPU-hogging process. + +To learn more about how to use simple patterns with dimension templates, see our [simple patterns +documentation](/libnetdata/simple_pattern/README.md). + +## Using `foreach` with alarm templates + +Dimension templates also work with [alarm templates](/health/REFERENCE.md#alarm-line-alarm-or-template). Alarm +templates help you create alarms for all the charts with a given context—for example, all the cores of your system's +CPU. + +By combining the two, you can create dozens of individual alarms with a single template entity. Here's how you would +create alarms for the `system`, `user`, and `nice` dimensions for every chart in the `cpu.cpu` context—or, in other +words, every CPU core. + +```yaml +template: cpu_template + on: cpu.cpu + lookup: average -10m percentage foreach system,user,nice + every: 1m + warn: $this > 50 + crit: $this > 80 +``` + +On a system with a 6-core, 12-thread Ryzen 5 1600 CPU, this one entity creates alarms on the following charts and +dimensions: + +- `cpu.cpu0` + - `cpu_template_user` + - `cpu_template_system` + - `cpu_template_nice` +- `cpu.cpu1` + - `cpu_template_user` + - `cpu_template_system` + - `cpu_template_nice` +- `cpu.cpu2` + - `cpu_template_user` + - `cpu_template_system` + - `cpu_template_nice` +- ... +- `cpu.cpu11` + - `cpu_template_user` + - `cpu_template_system` + - `cpu_template_nice` + +And how just a few of those dimension template-generated alarms look like in the Netdata dashboard. + +![A few of the created alarms in the Netdata dashboard](https://user-images.githubusercontent.com/1153921/66219669-708cf880-e680-11e9-8b3a-7bfe178fa28b.png) + +All in all, this single entity creates 36 individual alarms. Much easier than writing 36 separate entities in your +health configuration files! + +## What's next? + +We hope you're excited about the possibilities of using dimension templates! Maybe they'll inspire you to build new +alarms that will help you better monitor the health of your systems. + +Or, at the very least, simplify your configuration files. + +For information about other advanced features in Netdata's health monitoring toolkit, check out our [health +documentation](/health/README.md). And if you have some cool alarms you built using dimension templates, + + diff --git a/docs/guides/monitor/kubernetes-k8s-netdata.md b/docs/guides/monitor/kubernetes-k8s-netdata.md new file mode 100644 index 0000000..5cfefe8 --- /dev/null +++ b/docs/guides/monitor/kubernetes-k8s-netdata.md @@ -0,0 +1,254 @@ + + +# Kubernetes monitoring with Netdata: Overview and visualizations + +At Netdata, we've built Kubernetes monitoring tools that add visibility without complexity while also helping you +actively troubleshoot anomalies or outages. This guide walks you through each of the visualizations and offers best +practices on how to use them to start Kubernetes monitoring in a matter of minutes, not hours or days. + +Netdata's Kubernetes monitoring solution uses a handful of [complementary tools and +collectors](#related-reference-documentation) for peeling back the many complex layers of a Kubernetes cluster, +_entirely for free_. These methods work together to give you every metric you need to troubleshoot performance or +availability issues across your Kubernetes infrastructure. + +## Challenge + +While Kubernetes (k8s) might simplify the way you deploy, scale, and load-balance your applications, not all clusters +come with "batteries included" when it comes to monitoring. Doubly so for a monitoring stack that helps you actively +troubleshoot issues with your cluster. + +Some k8s providers, like GKE (Google Kubernetes Engine), do deploy clusters bundled with monitoring capabilities, such +as Google Stackdriver Monitoring. However, these pre-configured solutions might not offer the depth of metrics, +customization, or integration with your preferred alerting methods. + +Without this visibility, it's like you built an entire house and _then_ smashed your way through the finished walls to +add windows. + +## Solution + +In this tutorial, you'll learn how to navigate Netdata's Kubernetes monitoring features, using +[robot-shop](https://github.com/instana/robot-shop) as an example deployment. Deploying robot-shop is purely optional. +You can also follow along with your own Kubernetes deployment if you choose. While the metrics might be different, the +navigation and best practices are the same for every cluster. + +## What you need to get started + +To follow this tutorial, you need: + +- A free Netdata Cloud account. [Sign up](https://app.netdata.cloud/sign-up?cloudRoute=/spaces) if you don't have one + already. +- A working cluster running Kubernetes v1.9 or newer, with a Netdata deployment and connected parent/child nodes. See + our [Kubernetes deployment process](/packaging/installer/methods/kubernetes.md) for details on deployment and + conneting to Cloud. +- The [`kubectl`](https://kubernetes.io/docs/reference/kubectl/overview/) command line tool, within [one minor version + difference](https://kubernetes.io/docs/tasks/tools/install-kubectl/#before-you-begin) of your cluster, on an + administrative system. +- The [Helm package manager](https://helm.sh/) v3.0.0 or newer on the same administrative system. + +### Install the `robot-shop` demo (optional) + +Begin by downloading the robot-shop code and using `helm` to create a new deployment. + +```bash +git clone git@github.com:instana/robot-shop.git +cd robot-shop/K8s/helm +kubectl create ns robot-shop +helm install robot-shop --namespace robot-shop . +``` + +Running `kubectl get pods` shows both the Netdata and robot-shop deployments. + +```bash +kubectl get pods --all-namespaces +NAMESPACE NAME READY STATUS RESTARTS AGE +default netdata-child-29f9c 2/2 Running 0 10m +default netdata-child-8xphf 2/2 Running 0 10m +default netdata-child-jdvds 2/2 Running 0 11m +default netdata-parent-554c755b7d-qzrx4 1/1 Running 0 11m +kube-system aws-node-jnjv8 1/1 Running 0 17m +kube-system aws-node-svzdb 1/1 Running 0 17m +kube-system aws-node-ts6n2 1/1 Running 0 17m +kube-system coredns-559b5db75d-f58hp 1/1 Running 0 22h +kube-system coredns-559b5db75d-tkzj2 1/1 Running 0 22h +kube-system kube-proxy-9p9cd 1/1 Running 0 17m +kube-system kube-proxy-lt9ss 1/1 Running 0 17m +kube-system kube-proxy-n75t9 1/1 Running 0 17m +robot-shop cart-b4bbc8fff-t57js 1/1 Running 0 14m +robot-shop catalogue-8b5f66c98-mr85z 1/1 Running 0 14m +robot-shop dispatch-67d955c7d8-lnr44 1/1 Running 0 14m +robot-shop mongodb-7f65d86c-dsslc 1/1 Running 0 14m +robot-shop mysql-764c4c5fc7-kkbnf 1/1 Running 0 14m +robot-shop payment-67c87cb7d-5krxv 1/1 Running 0 14m +robot-shop rabbitmq-5bb66bb6c9-6xr5b 1/1 Running 0 14m +robot-shop ratings-94fd9c75b-42wvh 1/1 Running 0 14m +robot-shop redis-0 0/1 Pending 0 14m +robot-shop shipping-7d69cb88b-w7hpj 1/1 Running 0 14m +robot-shop user-79c445b44b-hwnm9 1/1 Running 0 14m +robot-shop web-8bb887476-lkcjx 1/1 Running 0 14m +``` + +## Explore Netdata's Kubernetes monitoring charts + +The Netdata Helm chart deploys and enables everything you need for monitoring Kubernetes on every layer. Once you deploy +Netdata and connect your cluster's nodes, you're ready to check out the visualizations **with zero configuration**. + +To get started, [sign in](https://app.netdata.cloud/sign-in?cloudRoute=/spaces) to your Netdata Cloud account. Head over +to the War Room you connected your cluster to, if not **General**. + +Netdata Cloud is already visualizing your Kubernetes metrics, streamed in real-time from each node, in the +[Overview](https://learn.netdata.cloud/docs/cloud/visualize/overview): + +![Netdata's Kubernetes monitoring +dashboard](https://user-images.githubusercontent.com/1153921/109037415-eafc5500-7687-11eb-8773-9b95941e3328.png) + +Let's walk through monitoring each layer of a Kubernetes cluster using the Overview as our framework. + +## Cluster and node metrics + +The gauges and time-series charts you see right away in the Overview show aggregated metrics from every node in your +cluster. + +For example, the `apps.cpu` chart (in the **Applications** menu item), visualizes the CPU utilization of various +applications/services running on each of the nodes in your cluster. The **X Nodes** dropdown shows which nodes +contribute to the chart and links to jump a single-node dashboard for further investigation. + +![Per-application monitoring in a Kubernetes +cluster](https://user-images.githubusercontent.com/1153921/109042169-19c8fa00-768d-11eb-91a7-1a7afc41fea2.png) + +For example, the chart above shows a spike in the CPU utilization from `rabbitmq` every minute or so, along with a +baseline CPU utilization of 10-15% across the cluster. + +Read about the [Overview](https://learn.netdata.cloud/docs/cloud/visualize/overview) and some best practices on [viewing +an overview of your infrastructure](/docs/visualize/overview-infrastructure.md) for details on using composite charts to +drill down into per-node performance metrics. + +## Pod and container metrics + +Click on the **Kubernetes xxxxxxx...** section to jump down to Netdata Cloud's unique Kubernetes visualizations for view +real-time resource utilization metrics from your Kubernetes pods and containers. + +![Navigating to the Kubernetes monitoring +visualizations](https://user-images.githubusercontent.com/1153921/109049195-349f6c80-7695-11eb-8902-52a029dca77f.png) + +### Health map + +The first visualization is the [health map](https://learn.netdata.cloud/docs/cloud/visualize/kubernetes#health-map), +which places each container into its own box, then varies the intensity of their color to visualize the resource +utilization. By default, the health map shows the **average CPU utilization as a percentage of the configured limit** +for every container in your cluster. + +![The Kubernetes health map in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/109050085-3f0e3600-7696-11eb-988f-52cb187f53ea.png) + +Let's explore the most colorful box by hovering over it. + +![Hovering over a +container](https://user-images.githubusercontent.com/1153921/109049544-a8417980-7695-11eb-80a7-109b4a645a27.png) + +The **Context** tab shows `rabbitmq-5bb66bb6c9-6xr5b` as the container's image name, which means this container is +running a [RabbitMQ](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/rabbitmq) workload. + +Click the **Metrics** tab to see real-time metrics from that container. Unsurprisingly, it shows a spike in CPU +utilization at regular intervals. + +![Viewing real-time container +metrics](https://user-images.githubusercontent.com/1153921/109050482-aa580800-7696-11eb-9e3e-d3bdf0f3eff7.png) + +### Time-series charts + +Beneath the health map is a variety of time-series charts that help you visualize resource utilization over time, which +is useful for targeted troubleshooting. + +The default is to display metrics grouped by the `k8s_namespace` label, which shows resource utilization based on your +different namespaces. + +![Time-series Kubernetes monitoring in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/109075210-126a1680-76b6-11eb-918d-5acdcdac152d.png) + +Each composite chart has a [definition bar](https://learn.netdata.cloud/docs/cloud/visualize/overview#definition-bar) +for complete customization. For example, grouping the top chart by `k8s_container_name` reveals new information. + +![Changing time-series charts](https://user-images.githubusercontent.com/1153921/109075212-139b4380-76b6-11eb-836f-939482ae55fc.png) + +## Service metrics + +Netdata has a [service discovery plugin](https://github.com/netdata/agent-service-discovery), which discovers and +creates configuration files for [compatible +services](https://github.com/netdata/helmchart#service-discovery-and-supported-services) and any endpoints covered by +our [generic Prometheus collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/prometheus). +Netdata uses these files to collect metrics from any compatible application as they run _inside_ of a pod. Service +discovery happens without manual intervention as pods are created, destroyed, or moved between nodes. + +Service metrics show up on the Overview as well, beneath the **Kubernetes** section, and are labeled according to the +service in question. For example, the **RabbitMQ** section has numerous charts from the [`rabbitmq` +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/rabbitmq): + +![Finding service discovery +metrics](https://user-images.githubusercontent.com/1153921/109054511-2eac8a00-769b-11eb-97f1-da93acb4b5fe.png) + +> The robot-shop cluster has more supported services, such as MySQL, which are not visible with zero configuration. This +> is usually because of services running on non-default ports, using non-default names, or required passwords. Read up +> on [configuring service discovery](/packaging/installer/methods/kubernetes.md#configure-service-discovery) to collect +> more service metrics. + +Service metrics are essential to infrastructure monitoring, as they're the best indicator of the end-user experience, +and key signals for troubleshooting anomalies or issues. + +## Kubernetes components + +Netdata also automatically collects metrics from two essential Kubernetes processes. + +### kubelet + +The **k8s kubelet** section visualizes metrics from the Kubernetes agent responsible for managing every pod on a given +node. This also happens without any configuration thanks to the [kubelet +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/k8s_kubelet). + +Monitoring each node's kubelet can be invaluable when diagnosing issues with your Kubernetes cluster. For example, you +can see if the number of running containers/pods has dropped, which could signal a fault or crash in a particular +Kubernetes service or deployment (see `kubectl get services` or `kubectl get deployments` for more details). If the +number of pods increases, it may be because of something more benign, like another team member scaling up a +service with `kubectl scale`. + +You can also view charts for the Kubelet API server, the volume of runtime/Docker operations by type, +configuration-related errors, and the actual vs. desired numbers of volumes, plus a lot more. + +### kube-proxy + +The **k8s kube-proxy** section displays metrics about the network proxy that runs on each node in your Kubernetes +cluster. kube-proxy lets pods communicate with each other and accept sessions from outside your cluster. Its metrics are +collected by the [kube-proxy +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/k8s_kubeproxy). + +With Netdata, you can monitor how often your k8s proxies are syncing proxy rules between nodes. Dramatic changes in +these figures could indicate an anomaly in your cluster that's worthy of further investigation. + +## What's next? + +After reading this guide, you should now be able to monitor any Kubernetes cluster with Netdata, including nodes, pods, +containers, services, and more. + +With the health map, time-series charts, and the ability to drill down into individual nodes, you can see hundreds of +per-second metrics with zero configuration and less time remembering all the `kubectl` options. Netdata moves with your +cluster, automatically picking up new nodes or services as your infrastructure scales. And it's entirely free for +clusters of all sizes. + +### Related reference documentation + +- [Netdata Helm chart](https://github.com/netdata/helmchart) +- [Netdata service discovery](https://github.com/netdata/agent-service-discovery) +- [Netdata Agent · `kubelet` + collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/k8s_kubelet) +- [Netdata Agent · `kube-proxy` + collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/k8s_kubeproxy) +- [Netdata Agent · `cgroups.plugin`](/collectors/cgroups.plugin/README.md) + + diff --git a/docs/guides/monitor/lamp-stack.md b/docs/guides/monitor/lamp-stack.md new file mode 100644 index 0000000..29b35e1 --- /dev/null +++ b/docs/guides/monitor/lamp-stack.md @@ -0,0 +1,246 @@ + +import { OneLineInstallWget } from '@site/src/components/OneLineInstall/' + +# LAMP stack monitoring (Linux, Apache, MySQL, PHP) with Netdata + +The LAMP stack is the "hello world" for deploying dynamic web applications. It's fast, flexible, and reliable, which +means a developer or sysadmin won't go far in their career without interacting with the stack and its services. + +_LAMP_ is an acronym of the core services that make up the web application: **L**inux, **A**pache, **M**ySQL, and +**P**HP. + +- [Linux](https://en.wikipedia.org/wiki/Linux) is the operating system running the whole stack. +- [Apache](https://httpd.apache.org/) is a web server that responds to HTTP requests from users and returns web pages. +- [MySQL](https://www.mysql.com/) is a database that stores and returns information based on queries from the web + application. +- [PHP](https://www.php.net/) is a scripting language used to query the MySQL database and build new pages. + +LAMP stacks are the foundation for tons of end-user applications, with [Wordpress](https://wordpress.org/) being the +most popular. + +## Challenge + +You've already deployed a LAMP stack, either in testing or production. You want to monitor every service's performance +and availability to ensure the best possible experience for your end-users. You might also be particularly interested in +using a free, open-source monitoring tool. + +Depending on your monitoring experience, you may not even know what metrics you're looking for, much less how to build +dashboards using a query language. You need a robust monitoring experience that has the metrics you need without a ton +of required setup. + +## Solution + +In this tutorial, you'll set up robust LAMP stack monitoring with Netdata in just a few minutes. When you're done, +you'll have one dashboard to monitor every part of your web application, including each essential LAMP stack service. + +This dashboard updates every second with new metrics, and pairs those metrics up with preconfigured alarms to keep you +informed of any errors or odd behavior. + +## What you need to get started + +To follow this tutorial, you need: + +- A physical or virtual Linux system, which we'll call a _node_. +- A functional LAMP stack. There's plenty of tutorials for installing a LAMP stack, like [this + one](https://www.digitalocean.com/community/tutorials/how-to-install-linux-apache-mysql-php-lamp-stack-ubuntu-18-04) + from Digital Ocean. +- Optionally, a [Netdata Cloud](https://app.netdata.cloud/sign-up?cloudRoute=/spaces) account, which you can use to view + metrics from multiple nodes in one dashboard, and a whole lot more, for free. + +## Install the Netdata Agent + +If you don't have the free, open-source Netdata monitoring agent installed on your node yet, get started with a [single +kickstart command](/docs/get-started.mdx): + + + +The Netdata Agent is now collecting metrics from your node every second. You don't need to jump into the dashboard yet, +but if you're curious, open your favorite browser and navigate to `http://localhost:19999` or `http://NODE:19999`, +replacing `NODE` with the hostname or IP address of your system. + +## Enable hardware and Linux system monitoring + +There's nothing you need to do to enable [system monitoring](/docs/collect/system-metrics.md) and Linux monitoring with +the Netdata Agent, which autodetects metrics from CPUs, memory, disks, networking devices, and Linux processes like +systemd without any configuration. If you're using containers, Netdata automatically collects resource utilization +metrics from each using the [cgroups data collector](/collectors/cgroups.plugin/README.md). + +## Enable Apache monitoring + +Let's begin by configuring Apache to work with Netdata's [Apache data +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/apache). + +Actually, there's nothing for you to do to enable Apache monitoring with Netdata. + +Apache comes with `mod_status` enabled by default these days, and Netdata is smart enough to look for metrics at that +endpoint without you configuring it. Netdata is already collecting [`mod_status` +metrics](https://httpd.apache.org/docs/2.4/mod/mod_status.html), which is just _part_ of your web server monitoring. + +## Enable web log monitoring + +The Netdata Agent also comes with a [web log +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog), which reads Apache's access +log file, processes each line, and converts them into per-second metrics. On Debian systems, it reads the file at +`/var/log/apache2/access.log`. + +At installation, the Netdata Agent adds itself to the [`adm` +group](https://wiki.debian.org/SystemGroups#Groups_without_an_associated_user), which gives the `netdata` process the +right privileges to read Apache's log files. In other words, you don't need to do anything to enable Apache web log +monitoring. + +## Enable MySQL monitoring + +Because your MySQL database is password-protected, you do need to tell MySQL to allow the `netdata` user to connect to +without a password. Netdata's [MySQL data +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/mysql) collects metrics in _read-only_ +mode, without being able to alter or affect operations in any way. + +First, log into the MySQL shell. Then, run the following three commands, one at a time: + +```mysql +CREATE USER 'netdata'@'localhost'; +GRANT USAGE, REPLICATION CLIENT, PROCESS ON *.* TO 'netdata'@'localhost'; +FLUSH PRIVILEGES; +``` + +Run `sudo systemctl restart netdata`, or the [appropriate alternative for your +system](/docs/configure/start-stop-restart.md), to collect dozens of metrics every second for robust MySQL monitoring. + +## Enable PHP monitoring + +Unlike Apache or MySQL, PHP isn't a service that you can monitor directly, unless you instrument a PHP-based application +with [StatsD](/collectors/statsd.plugin/README.md). + +However, if you use [PHP-FPM](https://php-fpm.org/) in your LAMP stack, you can monitor that process with our [PHP-FPM +data collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/phpfpm). + +Open your PHP-FPM configuration for editing, replacing `7.4` with your version of PHP: + +```bash +sudo nano /etc/php/7.4/fpm/pool.d/www.conf +``` + +> Not sure what version of PHP you're using? Run `php -v`. + +Find the line that reads `;pm.status_path = /status` and remove the `;` so it looks like this: + +```conf +pm.status_path = /status +``` + +Next, add a new `/status` endpoint to Apache. Open the Apache configuration file you're using for your LAMP stack. + +```bash +sudo nano /etc/apache2/sites-available/your_lamp_stack.conf +``` + +Add the following to the end of the file, again replacing `7.4` with your version of PHP: + +```apache +ProxyPass "/status" "unix:/run/php/php7.4-fpm.sock|fcgi://localhost" +``` + +Save and close the file. Finally, restart the PHP-FPM, Apache, and Netdata processes. + +```bash +sudo systemctl restart php7.4-fpm.service +sudo systemctl restart apache2 +sudo systemctl restart netdata +``` + +As the Netdata Agent starts up again, it automatically connects to the new `127.0.0.1/status` page and collects +per-second PHP-FPM metrics to get you started with PHP monitoring. + +## View LAMP stack metrics + +If the Netdata Agent isn't already open in your browser, open a new tab and navigate to `http://localhost:19999` or +`http://NODE:19999`, replacing `NODE` with the hostname or IP address of your system. + +> If you [signed up](https://app.netdata.cloud/sign-up?cloudRoute=/spaces) for Netdata Cloud earlier, you can also view +> the exact same LAMP stack metrics there, plus additional features, like drag-and-drop custom dashboards. Be sure to +> [connecting your node](/claim/README.md) to start streaming metrics to your browser through Netdata Cloud. + +Netdata automatically organizes all metrics and charts onto a single page for easy navigation. Peek at gauges to see +overall system performance, then scroll down to see more. Click-and-drag with your mouse to pan _all_ charts back and +forth through different time intervals, or hold `SHIFT` and use the scrollwheel (or two-finger scroll) to zoom in and +out. Check out our doc on [interacting with charts](/docs/visualize/interact-dashboards-charts.md) for all the details. + +![The Netdata +dashboard](https://user-images.githubusercontent.com/1153921/109520555-98e17800-7a69-11eb-86ec-16f689da4527.png) + +The **System Overview** section, which you can also see in the right-hand menu, contains key hardware monitoring charts, +including CPU utilization, memory page faults, network monitoring, and much more. The **Applications** section shows you +exactly which Linux processes are using the most system resources. + +Next, let's check out LAMP-specific metrics. You should see four relevant sections: **Apache local**, **MySQL local**, +**PHP-FPM local**, and **web log apache**. Click on any of these to see metrics from each service in your LAMP stack. + +![LAMP stack monitoring in +Netdata](https://user-images.githubusercontent.com/1153921/109516332-49994880-7a65-11eb-807c-3cba045582e6.png) + +### Key LAMP stack monitoring charts + +Here's a quick reference for what charts you might want to focus on after setting up Netdata. + +| Chart name / context | Type | Why? | +|-------------------------------------------------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| System Load Average (`system.load`) | Hardware monitoring | A good baseline load average is `0.7`, while `1` (on a 1-core system, `2` on a 2-core system, and so on) means resources are "perfectly" utilized. Higher load indicates a bottleneck somewhere in your system. | +| System RAM (`system.ram`) | Hardware monitoring | Look at the `free` dimension. If that drops to `0`, your system will use swap memory and slow down. | +| Uptime (`apache_local.uptime`) | Apache monitoring | This chart should always be "climbing," indicating a continuous uptime. Investigate any drops back to `0`. | +| Requests By Type (`web_log_apache.requests_by_type`) | Apache monitoring | Check for increases in the `error` or `bad` dimensions, which could indicate users arriving at broken pages or PHP returning errors. | +| Queries (`mysql_local.queries`) | MySQL monitoring | Queries is the total number of queries (queries per second, QPS). Check this chart for sudden spikes or drops, which indicate either increases in traffic/demand or bottlenecks in hardware performance. | +| Active Connections (`mysql_local.connections_active`) | MySQL monitoring | If the `active` dimension nears the `limit`, your MySQL database will bottleneck responses. | +| Performance (phpfpm_local.performance) | PHP monitoring | The `slow requests` dimension lets you know if any requests exceed the configured `request_slowlog_timeout`. If so, users might be having a less-than-ideal experience. | + +## Get alarms for LAMP stack errors + +The Netdata Agent comes with hundreds of pre-configured alarms to help you keep tabs on your system, including 19 alarms +designed for smarter LAMP stack monitoring. + +Click the 🔔 icon in the top navigation to [see active alarms](/docs/monitor/view-active-alarms.md). The **Active** tabs +shows any alarms currently triggered, while the **All** tab displays a list of _every_ pre-configured alarm. The + +![An example of LAMP stack +alarms](https://user-images.githubusercontent.com/1153921/109524120-5883f900-7a6d-11eb-830e-0e7baaa28163.png) + +[Tweak alarms](/docs/monitor/configure-alarms.md) based on your infrastructure monitoring needs, and to see these alarms +in other places, like your inbox or a Slack channel, [enable a notification +method](/docs/monitor/enable-notifications.md). + +## What's next? + +You've now set up robust monitoring for your entire LAMP stack: Linux, Apache, MySQL, and PHP (-FPM, to be exact). These +metrics will help you keep tabs on the performance and availability of your web application and all its essential +services. The per-second metrics granularity means you have the most accurate information possible for troubleshooting +any LAMP-related issues. + +Another powerful way to monitor the availability of a LAMP stack is the [`httpcheck` +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck), which pings a web server at +a regular interval and tells you whether if and how quickly it's responding. The `response_match` option also lets you +monitor when the web server's response isn't what you expect it to be, which might happen if PHP-FPM crashes, for +example. + +The best way to use the `httpcheck` collector is from a separate node from the one running your LAMP stack, which is why +we're not covering it here, but it _does_ work in a single-node setup. Just don't expect it to tell you if your whole +node crashed. + +If you're planning on managing more than one node, or want to take advantage of advanced features, like finding the +source of issues faster with [Metric Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations), +[sign up](https://app.netdata.cloud/sign-up?cloudRoute=/spaces) for a free Netdata Cloud account. + +### Related reference documentation + +- [Netdata Agent · Get started](/docs/get-started.mdx) +- [Netdata Agent · Apache data collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/apache) +- [Netdata Agent · Web log collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog) +- [Netdata Agent · MySQL data collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/mysql) +- [Netdata Agent · PHP-FPM data collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/phpfpm) + diff --git a/docs/guides/monitor/pi-hole-raspberry-pi.md b/docs/guides/monitor/pi-hole-raspberry-pi.md new file mode 100644 index 0000000..1246d8b --- /dev/null +++ b/docs/guides/monitor/pi-hole-raspberry-pi.md @@ -0,0 +1,162 @@ + +import { OneLineInstallWget } from '@site/src/components/OneLineInstall/' + +# Monitor Pi-hole (and a Raspberry Pi) with Netdata + +Between intrusive ads, invasive trackers, and vicious malware, many techies and homelab enthusiasts are advancing their +networks' security and speed with a tiny computer and a powerful piece of software: [Pi-hole](https://pi-hole.net/). + +Pi-hole is a DNS sinkhole that prevents unwanted content from even reaching devices on your home network. It blocks ads +and malware at the network, instead of using extensions/add-ons for individual browsers, so you'll stop seeing ads in +some of the most intrusive places, like your smart TV. Pi-hole can even [improve your network's speed and reduce +bandwidth](https://discourse.pi-hole.net/t/will-pi-hole-slow-down-my-network/2048). + +Most Pi-hole users run it on a [Raspberry Pi](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) (hence the +name), a credit card-sized, super-capable computer that costs about $35. + +And to keep tabs on how both Pi-hole and the Raspberry Pi are working to protect your network, you can use the +open-source [Netdata monitoring agent](https://github.com/netdata/netdata). + +To get started, all you need is a [Raspberry Pi](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) with +Raspbian installed. This guide uses a Raspberry Pi 4 Model B and Raspbian GNU/Linux 10 (buster). This guide assumes +you're connecting to a Raspberry Pi remotely over SSH, but you could also complete all these steps on the system +directly using a keyboard, mouse, and monitor. + +## Why monitor Pi-hole and a Raspberry Pi with Netdata? + +Netdata helps you monitor and troubleshoot all kinds of devices and the applications they run, including IoT devices +like the Raspberry Pi and applications like Pi-hole. + +After a two-minute installation and with zero configuration, you'll be able to see all of Pi-hole's metrics, including +the volume of queries, connected clients, DNS queries per type, top clients, top blocked domains, and more. + +With Netdata installed, you can also monitor system metrics and any other applications you might be running. By default, +Netdata collects metrics on CPU usage, disk IO, bandwidth, per-application resource usage, and a ton more. With the +Raspberry Pi used for this guide, Netdata automatically collects about 1,500 metrics every second! + +![Real-time Pi-hole monitoring with +Netdata](https://user-images.githubusercontent.com/1153921/90447745-c8fe9600-e098-11ea-8a57-4f07339f002b.png) + +## Install Netdata + +Let's start by installing Netdata first so that it can start collecting system metrics as soon as possible for the most +possible historic data. + +> ⚠️ Don't install Netdata using `apt` and the default package available in Raspbian. The Netdata team does not maintain +> this package, and can't guarantee it works properly. + +On Raspberry Pis running Raspbian, the best way to install Netdata is our one-line kickstart script. This script asks +you to install dependencies, then compiles Netdata from source via [GitHub](https://github.com/netdata/netdata). + + + +Once installed on a Raspberry Pi 4 with no accessories, Netdata starts collecting roughly 1,500 metrics every second and +populates its dashboard with more than 250 charts. + +Open your browser of choice and navigate to `http://NODE:19999/`, replacing `NODE` with the IP address of your Raspberry +Pi. Not sure what that IP is? Try running `hostname -I | awk '{print $1}'` from the Pi itself. + +You'll see Netdata's dashboard and a few hundred real-time, +[interactive](https://learn.netdata.cloud/guides/step-by-step/step-02#interact-with-charts) charts. Feel free to +explore, but let's turn our attention to installing Pi-hole. + +## Install Pi-Hole + +Like Netdata, Pi-hole has a one-line script for simple installation. From your Raspberry Pi, run the following: + +```bash +curl -sSL https://install.pi-hole.net | bash +``` + +The installer will help you set up Pi-hole based on the topology of your network. Once finished, you should set up your +devices—or your router for system-wide sinkhole protection—to [use Pi-hole as their DNS +service](https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245). You've +finished setting up Pi-hole at this point. + +As far as configuring Netdata to monitor Pi-hole metrics, there's nothing you actually need to do. Netdata's [Pi-hole +collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/pihole) will autodetect the new service +running on your Raspberry Pi and immediately start collecting metrics every second. + +Restart Netdata with `sudo systemctl restart netdata`, which will then recognize that Pi-hole is running and start a +per-second collection job. When you refresh your Netdata dashboard or load it up again in a new tab, you'll see a new +entry in the menu for **Pi-hole** metrics. + +## Use Netdata to explore and monitor your Raspberry Pi and Pi-hole + +By the time you've reached this point in the guide, Netdata has already collected a ton of valuable data about your +Raspberry Pi, Pi-hole, and any other apps/services you might be running. Even a few minutes of collecting 1,500 metrics +per second adds up quickly. + +You can now use Netdata's synchronized charts to zoom, highlight, scrub through time, and discern how an anomaly in one +part of your system might affect another. + +![The Netdata dashboard in +action](https://user-images.githubusercontent.com/1153921/80827388-b9fee100-8b98-11ea-8f60-0d7824667cd3.gif) + +If you're completely new to Netdata, look at our [step-by-step guide](/docs/guides/step-by-step/step-00.md) for a +walkthrough of all its features. For a more expedited tour, see the [get started guide](/docs/get-started.mdx). + +### Enable temperature sensor monitoring + +You need to manually enable Netdata's built-in [temperature sensor +collector](https://learn.netdata.cloud/docs/agent/collectors/charts.d.plugin/sensors) to start collecting metrics. + +> Netdata uses a few plugins to manage its [collectors](/collectors/REFERENCE.md), each using a different language: Go, +> Python, Node.js, and Bash. While our Go collectors are undergoing the most active development, we still support the +> other languages. In this case, you need to enable a temperature sensor collector that's written in Bash. + +First, open the `charts.d.conf` file for editing. You should always use the `edit-config` script to edit Netdata's +configuration files, as it ensures your settings persist across updates to the Netdata Agent. + +```bash +cd /etc/netdata +sudo ./edit-config charts.d.conf +``` + +Uncomment the `sensors=force` line and save the file. Restart Netdata with `sudo systemctl restart netdata` to enable +Raspberry Pi temperature sensor monitoring. + +### Storing historical metrics on your Raspberry Pi + +By default, Netdata allocates 256 MiB in disk space to store historical metrics inside the [database +engine](/database/engine/README.md). On the Raspberry Pi used for this guide, Netdata collects 1,500 metrics every +second, which equates to storing 3.5 days worth of historical metrics. + +You can increase this allocation by editing `netdata.conf` and increasing the `dbengine multihost disk space` setting to +more than 256. + +```yaml +[global] + dbengine multihost disk space = 512 +``` + +Use our [database sizing +calculator](/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics) +and [guide on storing historical metrics](/docs/guides/longer-metrics-storage.md) to help you determine the right +setting for your Raspberry Pi. + +## What's next? + +Now that you're monitoring Pi-hole and your Raspberry Pi with Netdata, you can extend its capabilities even further, or +configure Netdata to more specific goals. + +Most importantly, you can always install additional services and instantly collect metrics from many of them with our +[300+ integrations](/collectors/COLLECTORS.md). + +- [Optimize performance](/docs/guides/configure/performance.md) using tweaks developed for IoT devices. +- [Stream Raspberry Pi metrics](/streaming/README.md) to a parent host for easy access or longer-term storage. +- [Tweak alarms](/health/QUICKSTART.md) for either Pi-hole or the health of your Raspberry Pi. +- [Export metrics to external databases](/exporting/README.md) with the exporting engine. + +Or, head over to [our guides](https://learn.netdata.cloud/guides/) for even more experiments and insights into +troubleshooting the health of your systems and services. + +If you have any questions about using Netdata to monitor your Raspberry Pi, Pi-hole, or any other applications, head on +over to our [community forum](https://community.netdata.cloud/). + + diff --git a/docs/guides/monitor/process.md b/docs/guides/monitor/process.md new file mode 100644 index 0000000..2f46d7a --- /dev/null +++ b/docs/guides/monitor/process.md @@ -0,0 +1,301 @@ + + +# Monitor any process in real-time with Netdata + +Netdata is more than a multitude of generic system-level metrics and visualizations. Instead of providing only a bird's +eye view of your system, leaving you to wonder exactly _what_ is taking up 99% CPU, Netdata also gives you visibility +into _every layer_ of your node. These additional layers give you context, and meaningful insights, into the true health +and performance of your infrastructure. + +One of these layers is the _process_. Every time a Linux system runs a program, it creates an independent process that +executes the program's instructions in parallel with anything else happening on the system. Linux systems track the +state and resource utilization of processes using the [`/proc` filesystem](https://en.wikipedia.org/wiki/Procfs), and +Netdata is designed to hook into those metrics to create meaningful visualizations out of the box. + +While there are a lot of existing command-line tools for tracking processes on Linux systems, such as `ps` or `top`, +only Netdata provides dozens of real-time charts, at both per-second and event frequency, without you having to write +SQL queries or know a bunch of arbitrary command-line flags. + +With Netdata's process monitoring, you can: + +- Benchmark/optimize performance of standard applications, like web servers or databases +- Benchmark/optimize performance of custom applications +- Troubleshoot CPU/memory/disk utilization issues (why is my system's CPU spiking right now?) +- Perform granular capacity planning based on the specific needs of your infrastructure +- Search for leaking file descriptors +- Investigate zombie processes + +... and much more. Let's get started. + +## Prerequisites + +- One or more Linux nodes running [Netdata](/docs/get-started.mdx). If you need more time to understand Netdata before + following this guide, see the [infrastructure](/docs/quickstart/infrastructure.md) or + [single-node](/docs/quickstart/single-node.md) monitoring quickstarts. +- A general understanding of how to [configure the Netdata Agent](/docs/configure/nodes.md) using `edit-config`. +- A Netdata Cloud account. [Sign up](https://app.netdata.cloud) if you don't have one already. + +## How does Netdata do process monitoring? + +The Netdata Agent already knows to look for hundreds of [standard applications that we support via +collectors](/collectors/COLLECTORS.md), and groups them based on their purpose. Let's say you want to monitor a MySQL +database using its process. The Netdata Agent already knows to look for processes with the string `mysqld` in their +name, along with a few others, and puts them into the `sql` group. This `sql` group then becomes a dimension in all +process-specific charts. + +The process and groups settings are used by two unique and powerful collectors. + +[**`apps.plugin`**](/collectors/apps.plugin/README.md) looks at the Linux process tree every second, much like `top` or +`ps fax`, and collects resource utilization information on every running process. It then automatically adds a layer of +meaningful visualization on top of these metrics, and creates per-process/application charts. + +[**`ebpf.plugin`**](/collectors/ebpf.plugin/README.md): Netdata's extended Berkeley Packet Filter (eBPF) collector +monitors Linux kernel-level metrics for file descriptors, virtual filesystem IO, and process management, and then hands +process-specific metrics over to `apps.plugin` for visualization. The eBPF collector also collects and visualizes +metrics on an _event frequency_, which means it captures every kernel interaction, and not just the volume of +interaction at every second in time. That's even more precise than Netdata's standard per-second granularity. + +### Per-process metrics and charts in Netdata + +With these collectors working in parallel, Netdata visualizes the following per-second metrics for _any_ process on your +Linux systems: + +- CPU utilization (`apps.cpu`) + - Total CPU usage + - User/system CPU usage (`apps.cpu_user`/`apps.cpu_system`) +- Disk I/O + - Physical reads/writes (`apps.preads`/`apps.pwrites`) + - Logical reads/writes (`apps.lreads`/`apps.lwrites`) + - Open unique files (if a file is found open multiple times, it is counted just once, `apps.files`) +- Memory + - Real Memory Used (non-shared, `apps.mem`) + - Virtual Memory Allocated (`apps.vmem`) + - Minor page faults (i.e. memory activity, `apps.minor_faults`) +- Processes + - Threads running (`apps.threads`) + - Processes running (`apps.processes`) + - Carried over uptime (since the last Netdata Agent restart, `apps.uptime`) + - Minimum uptime (`apps.uptime_min`) + - Average uptime (`apps.uptime_average`) + - Maximum uptime (`apps.uptime_max`) + - Pipes open (`apps.pipes`) +- Swap memory + - Swap memory used (`apps.swap`) + - Major page faults (i.e. swap activity, `apps.major_faults`) +- Network + - Sockets open (`apps.sockets`) +- eBPF file + - Number of calls to open files. (`apps.file_open`) + - Number of files closed. (`apps.file_closed`) + - Number of calls to open files that returned errors. + - Number of calls to close files that returned errors. +- eBPF syscall + - Number of calls to delete files. (`apps.file_deleted`) + - Number of calls to `vfs_write`. (`apps.vfs_write_call`) + - Number of calls to `vfs_read`. (`apps.vfs_read_call`) + - Number of bytes written with `vfs_write`. (`apps.vfs_write_bytes`) + - Number of bytes read with `vfs_read`. (`apps.vfs_read_bytes`) + - Number of calls to write a file that returned errors. + - Number of calls to read a file that returned errors. +- eBPF process + - Number of process created with `do_fork`. (`apps.process_create`) + - Number of threads created with `do_fork` or `__x86_64_sys_clone`, depending on your system's kernel version. (`apps.thread_create`) + - Number of times that a process called `do_exit`. (`apps.task_close`) +- eBPF net + - Number of bytes sent. (`apps.bandwidth_sent`) + - Number of bytes received. (`apps.bandwidth_recv`) + +As an example, here's the per-process CPU utilization chart, including a `sql` group/dimension. + +![A per-process CPU utilization chart in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/101217226-3a5d5700-363e-11eb-8610-aa1640aefb5d.png) + +## Configure the Netdata Agent to recognize a specific process + +To monitor any process, you need to make sure the Netdata Agent is aware of it. As mentioned above, the Agent is already +aware of hundreds of processes, and collects metrics from them automatically. + +But, if you want to change the grouping behavior, add an application that isn't yet supported in the Netdata Agent, or +monitor a custom application, you need to edit the `apps_groups.conf` configuration file. + +Navigate to your [Netdata config directory](/docs/configure/nodes.md) and use `edit-config` to edit the file. + +```bash +cd /etc/netdata # Replace this with your Netdata config directory if not at /etc/netdata. +sudo ./edit-config apps_groups.conf +``` + +Inside the file are lists of process names, oftentimes using wildcards (`*`), that the Netdata Agent looks for and +groups together. For example, the Netdata Agent looks for processes starting with `mysqld`, `mariad`, `postgres`, and +others, and groups them into `sql`. That makes sense, since all these processes are for SQL databases. + +```conf +sql: mysqld* mariad* postgres* postmaster* oracle_* ora_* sqlservr +``` + +These groups are then reflected as [dimensions](/web/README.md#dimensions) within Netdata's charts. + +![An example per-process CPU utilization chart in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/101369156-352e2100-3865-11eb-9f0d-b8fac162e034.png) + +See the following two sections for details based on your needs. If you don't need to configure `apps_groups.conf`, jump +down to [visualizing process metrics](#visualize-process-metrics). + +### Standard applications (web servers, databases, containers, and more) + +As explained above, the Netdata Agent is already aware of most standard applications you run on Linux nodes, and you +shouldn't need to configure it to discover them. + +However, if you're using multiple applications that the Netdata Agent groups together you may want to separate them for +more precise monitoring. If you're not running any other types of SQL databases on that node, you don't need to change +the grouping, since you know that any MySQL is the only process contributing to the `sql` group. + +Let's say you're using both MySQL and PostgreSQL databases on a single node, and want to monitor their processes +independently. Open the `apps_groups.conf` file as explained in the [section +above](#configure-the-netdata-agent-to-recognize-a-specific-process) and scroll down until you find the `database +servers` section. Create new groups for MySQL and PostgreSQL, and move their process queries into the unique groups. + +```conf +# ----------------------------------------------------------------------------- +# database servers + +mysql: mysqld* +postgres: postgres* +sql: mariad* postmaster* oracle_* ora_* sqlservr +``` + +Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to start collecting utilization metrics from your +application. Time to [visualize your process metrics](#visualize-process-metrics). + +### Custom applications + +Let's assume you have an application that runs on the process `custom-app`. To monitor eBPF metrics for that application +separate from any others, you need to create a new group in `apps_groups.conf` and associate that process name with it. + +Open the `apps_groups.conf` file as explained in the [section +above](#configure-the-netdata-agent-to-recognize-a-specific-process). Scroll down to `# NETDATA processes accounting`. +Above that, paste in the following text, which creates a new `custom-app` group with the `custom-app` process. Replace +`custom-app` with the name of your application's Linux process. `apps_groups.conf` should now look like this: + +```conf +... +# ----------------------------------------------------------------------------- +# Custom applications to monitor with apps.plugin and ebpf.plugin + +custom-app: custom-app + +# ----------------------------------------------------------------------------- +# NETDATA processes accounting +... +``` + +Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to start collecting utilization metrics from your +application. + +## Visualize process metrics + +Now that you're collecting metrics for your process, you'll want to visualize them using Netdata's real-time, +interactive charts. Find these visualizations in the same section regardless of whether you use [Netdata +Cloud](https://app.netdata.cloud) for infrastructure monitoring, or single-node monitoring with the local Agent's +dashboard at `http://localhost:19999`. + +If you need a refresher on all the available per-process charts, see the [above +list](#per-process-metrics-and-charts-in-netdata). + +### Using Netdata's application collector (`apps.plugin`) + +`apps.plugin` puts all of its charts under the **Applications** section of any Netdata dashboard. + +![Screenshot of the Applications section on a Netdata +dashboard](https://user-images.githubusercontent.com/1153921/101401172-2ceadb80-388f-11eb-9e9a-88443894c272.png) + +Let's continue with the MySQL example. We can create a [test +database](https://www.digitalocean.com/community/tutorials/how-to-measure-mysql-query-performance-with-mysqlslap) in +MySQL to generate load on the `mysql` process. + +`apps.plugin` immediately collects and visualizes this activity `apps.cpu` chart, which shows an increase in CPU +utilization from the `sql` group. There is a parallel increase in `apps.pwrites`, which visualizes writes to disk. + +![Per-application CPU utilization +metrics](https://user-images.githubusercontent.com/1153921/101409725-8527da80-389b-11eb-96e9-9f401535aafc.png) + +![Per-application disk writing +metrics](https://user-images.githubusercontent.com/1153921/101409728-85c07100-389b-11eb-83fd-d79dd1545b5a.png) + +Next, the `mysqlslap` utility queries the database to provide some benchmarking load on the MySQL database. It won't +look exactly like a production database executing lots of user queries, but it gives you an idea into the possibility of +these visualizations. + +```bash +sudo mysqlslap --user=sysadmin --password --host=localhost --concurrency=50 --iterations=10 --create-schema=employees --query="SELECT * FROM dept_emp;" --verbose +``` + +The following per-process disk utilization charts show spikes under the `sql` group at the same time `mysqlslap` was run +numerous times, with slightly different concurrency and query options. + +![Per-application disk +metrics](https://user-images.githubusercontent.com/1153921/101411810-d08fb800-389e-11eb-85b3-f3fa41f1f887.png) + +> 💡 Click on any dimension below a chart in Netdata Cloud (or to the right of a chart on a local Agent dashboard), to +> visualize only that dimension. This can be particularly useful in process monitoring to separate one process' +> utilization from the rest of the system. + +### Using Netdata's eBPF collector (`ebpf.plugin`) + +Netdata's eBPF collector puts its charts in two places. Of most importance to process monitoring are the **ebpf file**, +**ebpf syscall**, **ebpf process**, and **ebpf net** sub-sections under **Applications**, shown in the above screenshot. + +For example, running the above workload shows the entire "story" how MySQL interacts with the Linux kernel to open +processes/threads to handle a large number of SQL queries, then subsequently close the tasks as each query returns the +relevant data. + +![Per-process eBPF +charts](https://user-images.githubusercontent.com/1153921/101412395-c8844800-389f-11eb-86d2-20c8a0f7b3c0.png) + +`ebpf.plugin` visualizes additional eBPF metrics, which are system-wide and not per-process, under the **eBPF** section. + +## What's next? + +Now that you have `apps_groups.conf` configured correctly, and know where to find per-process visualizations throughout +Netdata's ecosystem, you can precisely monitor the health and performance of any process on your node using per-second +metrics. + +For even more in-depth troubleshooting, see our guide on [monitoring and debugging applications with +eBPF](/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md). + +If the process you're monitoring also has a [supported collector](/collectors/COLLECTORS.md), now is a great time to set +that up if it wasn't autodetected. With both process utilization and application-specific metrics, you should have every +piece of data needed to discover the root cause of an incident. See our [collector +setup](/docs/collect/enable-configure.md) doc for details. + +[Create new dashboards](/docs/visualize/create-dashboards.md) in Netdata Cloud using charts from `apps.plugin`, +`ebpf.plugin`, and application-specific collectors to build targeted dashboards for monitoring key processes across your +infrastructure. + +Try running [Metric Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations) on a node that's +running the process(es) you're monitoring. Even if nothing is going wrong at the moment, Netdata Cloud's embedded +intelligence helps you better understand how a MySQL database, for example, might influence a system's volume of memory +page faults. And when an incident is afoot, use Metric Correlations to reduce mean time to resolution (MTTR) and +cognitive load. + +If you want more specific metrics from your custom application, check out Netdata's [statsd +support](/collectors/statsd.plugin/README.md). With statd, you can send detailed metrics from your application to +Netdata and visualize them with per-second granularity. Netdata's statsd collector works with dozens of [statsd server +implementations](https://github.com/etsy/statsd/wiki#client-implementations), which work with most application +frameworks. + +### Related reference documentation + +- [Netdata Agent · `apps.plugin`](/collectors/apps.plugin/README.md) +- [Netdata Agent · `ebpf.plugin`](/collectors/ebpf.plugin/README.md) +- [Netdata Agent · Dashboards](/web/README.md#dimensions) +- [Netdata Agent · MySQL collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/mysql) + + diff --git a/docs/guides/monitor/raspberry-pi-anomaly-detection.md b/docs/guides/monitor/raspberry-pi-anomaly-detection.md new file mode 100644 index 0000000..73f57cd --- /dev/null +++ b/docs/guides/monitor/raspberry-pi-anomaly-detection.md @@ -0,0 +1,125 @@ +--- +title: "Unsupervised anomaly detection for Raspberry Pi monitoring" +description: "Use a low-overhead machine learning algorithm and an open-source monitoring tool to detect anomalous metrics on a Raspberry Pi." +image: /img/seo/guides/monitor/raspberry-pi-anomaly-detection.png +author: "Andy Maguire" +author_title: "Senior Machine Learning Engineer" +author_img: "/img/authors/andy-maguire.jpg" +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/guides/monitor/raspberry-pi-anomaly-detection.md +--- + +We love IoT and edge at Netdata, we also love machine learning. Even better if we can combine the two to ease the pain +of monitoring increasingly complex systems. + +We recently explored what might be involved in enabling our Python-based [anomalies +collector](/collectors/python.d.plugin/anomalies/README.md) on a Raspberry Pi. To our delight, it's actually quite +straightforward! + +Read on to learn all the steps and enable unsupervised anomaly detection on your on Raspberry Pi(s). + +> Spoiler: It's just a couple of extra commands that will make you feel like a pro. + +## What you need to get started + +- A Raspberry Pi running Raspbian, which we'll call a _node_. +- The [open-source Netdata](https://github.com/netdata/netdata) monitoring agent. If you don't have it installed on your + node yet, [get started now](/docs/get-started.mdx). + +## Install dependencies + +First make sure Netdata is using Python 3 when it runs Python-based data collectors. + +Next, open `netdata.conf` using [`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) +from within the [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory). Scroll down to the +`[plugin:python.d]` section to pass in the `-ppython3` command option. + +```conf +[plugin:python.d] + # update every = 1 + command options = -ppython3 +``` + +Next, install some of the underlying libraries used by the Python packages the collector depends upon. + +```bash +sudo apt install llvm-9 libatlas3-base libgfortran5 libatlas-base-dev +``` + +Now you're ready to install the Python packages used by the collector itself. First, become the `netdata` user. + +```bash +sudo su -s /bin/bash netdata +``` + +Then pass in the location to find `llvm` as an environment variable for `pip3`. + +```bash +LLVM_CONFIG=llvm-config-9 pip3 install --user llvmlite numpy==1.20.1 netdata-pandas==0.0.38 numba==0.50.1 scikit-learn==0.23.2 pyod==0.8.3 +``` + +## Enable the anomalies collector + +Now you're ready to enable the collector and [restart Netdata](/docs/configure/start-stop-restart.md). + +```bash +sudo ./edit-config python.d.conf +# set `anomalies: no` to `anomalies: yes` + +# restart netdata +sudo systemctl restart netdata +``` + +And that should be it! Wait a minute or two, refresh your Netdata dashboard, you should see the default anomalies +charts under the **Anomalies** section in the dashboard's menu. + +![Anomaly detection on the Raspberry +Pi](https://user-images.githubusercontent.com/1153921/110149717-9d749c00-7d9b-11eb-853c-e041a36f0a41.png) + +## Overhead on system + +Of course one of the most important considerations when trying to do anomaly detection at the edge (as opposed to in a +centralized cloud somewhere) is the resource utilization impact of running a monitoring tool. + +With the default configuration, the anomalies collector uses about 6.5% of CPU at each run. During the retraining step, +CPU utilization jumps to between 20-30% for a few seconds, but you can [configure +retraining](/collectors/python.d.plugin/anomalies/README.md#configuration) to happen less often if you wish. + +![CPU utilization of anomaly detection on the Raspberry +Pi](https://user-images.githubusercontent.com/1153921/110149718-9d749c00-7d9b-11eb-9af8-46e2032cd1d0.png) + +In terms of the runtime of the collector, it was averaging around 250ms during each prediction step, jumping to about +8-10 seconds during a retraining step. This jump equates only to a small gap in the anomaly charts for a few seconds. + +![Execution time of anomaly detection on the Raspberry +Pi](https://user-images.githubusercontent.com/1153921/110149715-9cdc0580-7d9b-11eb-826d-faf6f620621a.png) + +The last consideration then is the amount of RAM the collector needs to store both the models and some of the data +during training. By default, the anomalies collector, along with all other running Python-based collectors, uses about +100MB of system memory. + +![RAM utilization of anomaly detection on the Raspberry +Pi](https://user-images.githubusercontent.com/1153921/110149720-9e0d3280-7d9b-11eb-883d-b1d4d9b9b5e1.png) + +## What's next? + +So, all in all, with a small little bit of extra set up and a small overhead on the Pi itself, the anomalies collector +looks like a potentially useful addition to enable unsupervised anomaly detection on your Pi. + +See our two-part guide series for a more complete picture of configuring the anomalies collector, plus some best +practices on using the charts it automatically generates: + +- [_Detect anomalies in systems and applications_](/docs/guides/monitor/anomaly-detection-python.md) +- [_Monitor and visualize anomalies with Netdata_](/docs/guides/monitor/visualize-monitor-anomalies.md) + +If you're using your Raspberry Pi for other purposes, like blocking ads/trackers with Pi-hole, check out our companions +Pi guide: [_Monitor Pi-hole (and a Raspberry Pi) with Netdata_](/docs/guides/monitor/pi-hole-raspberry-pi.md). + +Once you've had a chance to give unsupervised anomaly detection a go, share your use cases and let us know of any +feedback on our [community forum](https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767). + +### Related reference documentation + +- [Netdata Agent · Get Netdata](/docs/get-started.mdx) +- [Netdata Agent · Anomalies collector](/collectors/python.d.plugin/anomalies/README.md) + + diff --git a/docs/guides/monitor/statsd.md b/docs/guides/monitor/statsd.md new file mode 100644 index 0000000..3e2f0f8 --- /dev/null +++ b/docs/guides/monitor/statsd.md @@ -0,0 +1,298 @@ + + +# StatsD Guide + +StatsD is a protocol and server implementation, first introduced at Etsy, to aggregate and summarize application metrics. With StatsD, applications are instrumented by developers using the libraries that already exist for the language, without caring about managing the data. The StatsD server is in charge of receiving the metrics, performing some simple processing on them, and then pushing them to the time-series database (TSDB) for long-term storage and visualization. + +Netdata is a fully-functional StatsD server and TSDB implementation, so you can instantly visualize metrics by simply sending them to Netdata using the built-in StatsD server. + +In this guide, we'll go through a scenario of visualizing our data in Netdata in a matter of seconds using [k6](https://k6.io), an open-source tool for automating load testing that outputs metrics to the StatsD format. + +Although we'll use k6 as the use-case, the same principles can be applied to every application that supports the StatsD protocol. Simply enable the StatsD output and point it to the node that runs Netdata, which is `localhost` in this case. + +In general, the process for creating a StatsD collector can be summarized in 2 steps: + +- Run an experiment by sending StatsD metrics to Netdata, without any prior configuration. This will create a chart per metric (called private charts) and will help you verify that everything works as expected from the application side of things. + - Make sure to reload the dashboard tab **after** you start sending data to Netdata. +- Create a configuration file for your app using [edit-config](/docs/configure/nodes.md): `sudo ./edit-config + statsd.d/myapp.conf` + - Each app will have it's own section in the right-hand menu. + +Now, let's see the above process in detail. + +## Prerequisites + +- A node with the [Netdata](/docs/get-started.mdx) installed. +- An application to instrument. For this guide, that will be [k6](https://k6.io/docs/getting-started/installation). + +## Understanding the metrics + +The real in instrumenting an application with StatsD for you is to decide what metrics you want to visualize and how you want them grouped. In other words, you need decide which metrics will be grouped in the same charts and how the charts will be grouped on Netdata's dashboard. + +Start with documentation for the particular application that you want to monitor (or the technological stack that you are using). In our case, the [k6 documentation](https://k6.io/docs/using-k6/metrics/) has a whole page dedicated to the metrics output by k6, along with descriptions. + +If you are using StatsD to monitor an existing application, you don't have much control over these metrics. For example, k6 has a type called `trend`, which is identical to timers and histograms. Thus, _k6 is clearly dictating_ which metrics can be used as histograms and simple gauges. + +On the other hand, if you are instrumenting your own code, you will need to not only decide what are the "things" that you want to measure, but also decide which StatsD metric type is the appropriate for each. + +## Use private charts to see all available metrics + +In Netdata, every metric will receive its own chart, called a `private chart`. Although in the final implementation this is something that we will disable, since it can create considerable noise (imagine having 100s of metrics), it’s very handy while building the configuration file. + +You can get a quick visual representation of the metrics and their type (e.g it’s a gauge, a timer, etc.). + +An important thing to notice is that StatsD has different types of metrics, as illustrated in the [Netdata documentation](https://learn.netdata.cloud/docs/agent/collectors/statsd.plugin#metrics-supported-by-netdata). Histograms and timers support mathematical operations to be performed on top of the baseline metric, like reporting the `average` of the value. + +Here are some examples of default private charts. You can see that the histogram private charts will visualize all the available operations. + +**Gauge private chart** + +![Gauge metric example](https://i.imgur.com/Sr5nJEV.png) + +**Histogram private chart** + +![Timer metric example](https://i.imgur.com/P4p0hvq.png) + +## Create a new StatsD configuration file + +Start by creating a new configuration file under the `statsd.d/` folder in the [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory). Use [`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) to create a new file called `k6.conf`. + +```bash= +sudo ./edit-config statsd.d/k6.conf +``` + +Copy the following configuration into your file as a starting point. + +```conf +[app] + name = k6 + metrics = k6* + private charts = yes + gaps when not collected = no + memory mode = dbengine +``` + +Next, you need is to understand how to organize metrics in Netdata’s StatsD. + +### Synthetic charts + +Netdata lets you group the metrics exposed by your instrumented application with _synthetic charts_. + +First, create a `[dictionary]` section to transform the names of the metrics into human-readable equivalents. `http_req_blocked`, `http_req_connecting`, `http_req_receiving`, and `http_reqs` are all metrics exposed by k6. + +``` +[dictionary] + http_req_blocked = Blocked HTTP Requests + http_req_connecting = Connecting HTTP Requests + http_req_receiving = Receiving HTTP Requests + http_reqs = Total HTTP requests +``` + +Continue this dictionary process with any other metrics you want to collect with Netdata. + +### Families and context + +Families and context are additional ways to group metrics. Families control the submenu at right-hand menu and it's a subcategory of the section. Given the metrics given by K6, we are organizing them in 2 major groups, or `families`: `k6 native metrics` and `http metrics`. + +Context is a second way to group metrics, when the metrics are of the same nature but different origin. In our case, if we ran several different load testing experiments side-by-side, we could define the same app, but different context (e.g `http_requests.experiment1`, `http_requests.experiment2`). + +Find more details about family and context in our [documentation](/web/README.md#families). + +### Dimension + +Now, having decided on how we are going to group the charts, we need to define how we are going to group metrics into different charts. This is particularly important, since we decide: + +- What metrics **not** to show, since they are not useful for our use-case. +- What metrics to consolidate into the same charts, so as to reduce noise and increase visual correlation. + +The dimension option has this syntax: `dimension = [pattern] METRIC NAME TYPE MULTIPLIER DIVIDER OPTIONS` + +- **pattern**: A keyword that tells the StatsD server the `METRIC` string is actually a [simple pattern].(/libnetdata/simple_pattern/README.md). We don't simple patterns in the example, but if we wanted to visualize all the `http_req` metrics, we could have a single dimension: `dimension = pattern 'k6.http_req*' last 1 1`. Find detailed examples with patterns in our [documentation](/collectors/statsd.plugin/README.md#dimension-patterns). +- **METRIC** The id of the metric as it comes from the client. You can easily find this in the private charts above, for example: `k6.http_req_connecting`. +- **NAME**: The name of the dimension. You can use the dictionary to expand this to something more human-readable. +- **TYPE**: + - For all charts: + - `events`: The number of events (data points) received by the StatsD server + - `last`: The last value that the server received + - For histograms and timers: + - `min`, `max`, `sum`, `average`, `percentile`, `median`, `stddev`: This is helpful if you want to see different representations of the same value. You can find an example at the `[iteration_duration]` above. Note that the baseline `metric` is the same, but the `name` of the dimension is different, since we use the baseline, but we perform a computation on it, creating a different final metric for visualization(dimension). +- **MULTIPLIER DIVIDER**: Handy if you want to convert Kilobytes to Megabytes or you want to give negative value. The second is handy for better visualization of send/receive. You can find an example at the **packets** submenu of the **IPv4 Networking Section**. + +> ❕ If you define a chart, run Netdata to visualize metrics, and then add or remove a dimension from that chart, this will result in a new chart with the same name, confusing Netdata. If you change the dimensions of the chart, please make sure to also change the `name` of that chart, since it serves as the `id` of that chart in Netdata's storage. (e.g http_req --> http_req_1). + +### Finalize your StatsD configuration file + +It's time to assemble all the pieces together and create the synthetic charts that will consist our application dashboard in Netdata. We can do it in a few simple steps: + +- Decide which metrics we want to use (we have viewed all of them as private charts). For example, we want to use `k6.http_requests`, `k6.vus`, etc. +- Decide how we want organize them in different synthetic charts. For example, we want `k6.http_requests`, `k6.vus` on their own, but `k6.http_req_blocked` and `k6.http_req_connecting` on the same chart. +- For each synthetic chart, we define a **unique** name and a human readable title. +- We decide at which `family` (submenu section) we want each synthetic chart to belong to. For example, here we have defined 2 families: `http requests`, `k6_metrics`. +- If we have multiple instances of the same metric, we can define different contexts, (Optional). +- We define a dimension according to the syntax we highlighted above. +- We define a type for each synthetic chart (line, area, stacked) +- We define the units for each synthetic chart. + +Following the above steps, we append to the `k6.conf` that we defined above, the following configuration: + +``` +[http_req_total] + name = http_req_total + title = Total HTTP Requests + family = http requests + context = k6.http_requests + dimension = k6.http_reqs http_reqs last 1 1 sum + type = line + units = requests/s + +[vus] + name = vus + title = Virtual Active Users + family = k6_metrics + dimension = k6.vus vus last 1 1 + dimension = k6.vus_max vus_max last 1 1 + type = line + unit = vus + +[iteration_duration] + name = iteration_duration_2 + title = Iteration duration + family = k6_metrics + dimension = k6.iteration_duration iteration_duration last 1 1 + dimension = k6.iteration_duration iteration_duration_max max 1 1 + dimension = k6.iteration_duration iteration_duration_min min 1 1 + dimension = k6.iteration_duration iteration_duration_avg avg 1 1 + type = line + unit = s + +[dropped_iterations] + name = dropped_iterations + title = Dropped Iterations + family = k6_metrics + dimension = k6.dropped_iterations dropped_iterations last 1 1 + units = iterations + type = line + +[data] + name = data + title = K6 Data + family = k6_metrics + dimension = k6.data_received data_received last 1 1 + dimension = k6.data_sent data_sent last -1 1 + units = kb/s + type = area + +[http_req_status] + name = http_req_status + title = HTTP Requests Status + family = http requests + dimension = k6.http_req_blocked http_req_blocked last 1 1 + dimension = k6.http_req_connecting http_req_connecting last 1 1 + units = ms + type = line + +[http_req_duration] + name = http_req_duration + title = HTTP requests duration + family = http requests + dimension = k6.http_req_sending http_req_sending last 1 1 + dimension = k6.http_req_waiting http_req_waiting last 1 1 + dimension = k6.http_req_receiving http_req_receiving last 1 1 + units = ms + type = stacked +``` + +> Take note that Netdata will report the rate for metrics and counters, even if k6 or another application sends an _absolute_ number. For example, k6 sends absolute HTTP requests with `http_reqs`, but Netdat visualizes that in `requests/second`. + +To enable this StatsD configuration, [restart Netdata](/docs/configure/start-stop-restart.md). + +## Final touches + +At this point, you have used StatsD to gather metrics for k6, creating a whole new section in your Netdata dashboard in the process. Moreover, you can further customize the icon of the particular section, as well as the description for each chart. + +To edit the section, please follow the Netdata [documentation](https://learn.netdata.cloud/docs/agent/web/gui#customizing-the-local-dashboard). + +While the following configuration will be placed in a new file, as the documentation suggests, it is instructing to use `dashboard_info.js` as a template. Open the file and see how the rest of sections and collectors have been defined. + +```javascript= +netdataDashboard.menu = { + 'k6': { + title: 'K6 Load Testing', + icon: '', + info: 'k6 is an open-source load testing tool and cloud service providing the best developer experience for API performance testing.' + }, + . + . + . +``` + +We can then add a description for each chart. Simply find the following section in `dashboard_info.js` to understand how a chart definitions are used: + +```javascript= +netdataDashboard.context = { + 'system.cpu': { + info: function (os) { + void (os); + return 'Total CPU utilization (all cores). 100% here means there is no CPU idle time at all. You can get per core usage at the CPUs section and per application usage at the Applications Monitoring section.' + + netdataDashboard.sparkline('
Keep an eye on iowait ', 'system.cpu', 'iowait', '%', '. If it is constantly high, your disks are a bottleneck and they slow your system down.') + + netdataDashboard.sparkline('
An important metric worth monitoring, is softirq ', 'system.cpu', 'softirq', '%', '. A constantly high percentage of softirq may indicate network driver issues.'); + }, + valueRange: "[0, 100]" + }, +``` + +Afterwards, you can open your `custom_dashboard_info.js`, as suggested in the documentation linked above, and add something like the following example: + +```javascript= +netdataDashboard.context = { + 'k6.http_req_duration': { + info: "Total time for the request. It's equal to http_req_sending + http_req_waiting + http_req_receiving (i.e. how long did the remote server take to process the request and respond, without the initial DNS lookup/connection times)" + }, + +``` +The chart is identified as ``.``. + +These descriptions can greatly help the Netdata user who is monitoring your application in the midst of an incident. + +The `info` field supports `html`, embedding useful links and instructions in the description. + +## Vendoring a new collector + +While we learned how to visualize any data source in Netdata using the StatsD protocol, we have also created a new collector. + +As long as you use the same underlying collector, every new `myapp.conf` file will create a new data source and dashboard section for Netdata. Netdata loads all the configuration files by default, but it will **not** create dashboard sections or charts, unless it starts receiving data for that particular data source. This means that we can now share our collector with the rest of the Netdata community. + +If you want to contribute or you need any help in developing your collector, we have a whole [Forum Category](https://community.netdata.cloud/c/agent-development/9) dedicated to contributing to the Netdata Agent. + +### Making a PR to the netdata/netdata repository + +- Make sure you follow the contributing guide and read our Code of Conduct +- Fork the netdata/netdata repository +- Place the configuration file inside `netdata/collectors/statsd.plugin` +- Add a reference in `netdata/collectors/statsd.plugin/Makefile.am`. For example, if we contribute the `k6.conf` file: +```Makefile +dist_statsdconfig_DATA = \ + example.conf \ + k6.conf \ + $(NULL) +``` + +## What's next? + +In this tutorial, you learned how to monitor an application using Netdata's StatsD implementation. + +Netdata allows you easily visualize any StatsD metric without any configuration, since it creates a private metric per chart by default. But to make your implementation more robust, you also learned how to group metrics by family and context, and create multiple dimensions. With these tools, you can quickly instrument any application with StatsD to monitor its performance and availability with per-second metrics. + +### Related reference documentation + +- [Netdata Agent · StatsD](/collectors/statsd.plugin/README.md) + + diff --git a/docs/guides/monitor/stop-notifications-alarms.md b/docs/guides/monitor/stop-notifications-alarms.md new file mode 100644 index 0000000..a8b73a8 --- /dev/null +++ b/docs/guides/monitor/stop-notifications-alarms.md @@ -0,0 +1,92 @@ + + +# Stop notifications for individual alarms + +In this short tutorial, you'll learn how to stop notifications for individual alarms in Netdata's health +monitoring system. We also refer to this process as _silencing_ the alarm. + +Why silence alarms? We designed Netdata's pre-configured alarms for production systems, so they might not be +relevant if you run Netdata on your laptop or a small virtual server. If they're not helpful, they can be a distraction +to real issues with health and performance. + +Silencing individual alarms is an excellent solution for situations where you're not interested in seeing a specific +alarm but don't want to disable a [notification system](/health/notifications/README.md) entirely. + +## Find the alarm configuration file + +To silence an alarm, you need to know where to find its configuration file. + +Let's use the `system.cpu` chart as an example. It's the first chart you'll see on most Netdata dashboards. + +To figure out which file you need to edit, open up Netdata's dashboard and, click the **Alarms** button at the top +of the dashboard, followed by clicking on the **All** tab. + +In this example, we're looking for the `system - cpu` entity, which, when opened, looks like this: + +![The system - cpu alarm +entity](https://user-images.githubusercontent.com/1153921/67034648-ebb4cc80-f0cc-11e9-9d49-1023629924f5.png) + +In the `source` row, you see that this chart is getting its configuration from +`4@/usr/lib/netdata/conf.d/health.d/cpu.conf`. The relevant part of begins at `health.d`: `health.d/cpu.conf`. That's +the file you need to edit if you want to silence this alarm. + +For more information about editing or referencing health configuration files on your system, see the [health +quickstart](/health/QUICKSTART.md#edit-health-configuration-files). + +## Edit the file to enable silencing + +To edit `health.d/cpu.conf`, use `edit-config` from inside of your Netdata configuration directory. + +```bash +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ +./edit-config health.d/cpu.conf +``` + +> You may need to use `sudo` or another method of elevating your privileges. + +The beginning of the file looks like this: + +```yaml +template: 10min_cpu_usage + on: system.cpu + os: linux + hosts: * + lookup: average -10m unaligned of user,system,softirq,irq,guest + units: % + every: 1m + warn: $this > (($status >= $WARNING) ? (75) : (85)) + crit: $this > (($status == $CRITICAL) ? (85) : (95)) + delay: down 15m multiplier 1.5 max 1h + info: average cpu utilization for the last 10 minutes (excluding iowait, nice and steal) + to: sysadmin +``` + +To silence this alarm, change `sysadmin` to `silent`. + +```yaml + to: silent +``` + +Use one of the available [methods](/health/QUICKSTART.md#reload-health-configuration) to reload your health configuration + and ensure you get no more notifications about that alarm**. + +You can add `to: silent` to any alarm you'd rather not bother you with notifications. + +## What's next? + +You should now know the fundamentals behind silencing any individual alarm in Netdata. + +To learn about _all_ of Netdata's health configuration possibilities, visit the [health reference +guide](/health/REFERENCE.md), or check out other [tutorials on health monitoring](/health/README.md#guides). + +Or, take better control over how you get notified about alarms via the [notification +system](/health/notifications/README.md). + +You can also use Netdata's [Health Management API](/web/api/health/README.md#health-management-api) to control health +checks and notifications while Netdata runs. With this API, you can disable health checks during a maintenance window or +backup process, for example. + + diff --git a/docs/guides/monitor/visualize-monitor-anomalies.md b/docs/guides/monitor/visualize-monitor-anomalies.md new file mode 100644 index 0000000..1f8c2c8 --- /dev/null +++ b/docs/guides/monitor/visualize-monitor-anomalies.md @@ -0,0 +1,142 @@ +--- +title: "Monitor and visualize anomalies with Netdata (part 2)" +description: "Using unsupervised anomaly detection and machine learning, get notified " +image: /img/seo/guides/monitor/visualize-monitor-anomalies.png +author: "Joel Hans" +author_title: "Editorial Director, Technical & Educational Resources" +author_img: "/img/authors/joel-hans.jpg" +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/guides/monitor/visualize-monitor-anomalies.md +--- + +Welcome to part 2 of our series of guides on using _unsupervised anomaly detection_ to detect issues with your systems, +containers, and applications using the open-source Netdata Agent. For an introduction to detecting anomalies and +monitoring associated metrics, see [part 1](/docs/guides/monitor/anomaly-detection-python.md), which covers prerequisites and +configuration basics. + +With anomaly detection in the Netdata Agent set up, you will now want to visualize and monitor which charts have +anomalous data, when, and where to look next. + +> 💡 In certain cases, the anomalies collector doesn't start immediately after restarting the Netdata Agent. If this +> happens, you won't see the dashboard section or the relevant [charts](#visualize-anomalies-in-charts) right away. Wait +> a minute or two, refresh, and look again. If the anomalies charts and alarms are still not present, investigate the +> error log with `less /var/log/netdata/error.log | grep anomalies`. + +## Test anomaly detection + +Time to see the Netdata Agent's unsupervised anomaly detection in action. To trigger anomalies on the Nginx web server, +use `ab`, otherwise known as [Apache Bench](https://httpd.apache.org/docs/2.4/programs/ab.html). Despite its name, it +works just as well with Nginx web servers. Install it on Ubuntu/Debian systems with `sudo apt install apache2-utils`. + +> 💡 If you haven't followed the guide's example of using Nginx, an easy way to test anomaly detection on your node is +> to use the `stress-ng` command, which is available on most Linux distributions. Run `stress-ng --cpu 0` to create CPU +> stress or `stress-ng --vm 0` for RAM stress. Each test will cause some "collateral damage," in that you may see CPU +> utilization rise when running the RAM test, and vice versa. + +The following test creates a minimum of 10,000,000 requests for Nginx to handle, with a maximum of 10 at any given time, +with a run time of 60 seconds. If your system can handle those 10,000,000 in less than 60 seconds, `ab` will keep +sending requests until the timer runs out. + +```bash +ab -k -c 10 -t 60 -n 10000000 http://127.0.0.1/ +``` + +Let's see how Netdata detects this anomalous behavior and propagates information to you through preconfigured alarms and +dashboards that automatically organize anomaly detection metrics into meaningful charts to help you begin root cause +analysis (RCA). + +## Monitor anomalies with alarms + +The anomalies collector creates two "classes" of alarms for each chart captured by the `charts_regex` setting. All these +alarms are preconfigured based on your [configuration in +`anomalies.conf`](/docs/guides/monitor/anomaly-detection-python.md#configure-the-anomalies-collector). With the `charts_regex` +and `charts_to_exclude` settings from [part 1](/docs/guides/monitor/anomaly-detection-python.md) of this guide series, the +Netdata Agent creates 32 alarms driven by unsupervised anomaly detection. + +The first class triggers warning alarms when the average anomaly probability for a given chart has stayed above 50% for +at least the last two minutes. + +![An example anomaly probability +alarm](https://user-images.githubusercontent.com/1153921/104225767-0a0a9480-5404-11eb-9bfd-e29592397203.png) + +The second class triggers warning alarms when the number of anomalies in the last two minutes hits 10 or higher. + +![An example anomaly count +alarm](https://user-images.githubusercontent.com/1153921/104225769-0aa32b00-5404-11eb-95f3-7309f9429fe1.png) + +If you see either of these alarms in Netdata Cloud, the local Agent dashboard, or on your preferred notification +platform, it's a safe bet that the node's current metrics have deviated from normal. That doesn't necessarily mean +there's a full-blown incident, depending on what application/service you're using anomaly detection on, but it's worth +further investigation. + +As you use the anomalies collector, you may find that the default settings provide too many or too few genuine alarms. +In this case, [configure the alarm](/docs/monitor/configure-alarms.md) with `sudo ./edit-config +health.d/anomalies.conf`. Take a look at the `lookup` line syntax in the [health +reference](/health/REFERENCE.md#alarm-line-lookup) to understand how the anomalies collector automatically creates +alarms for any dimension on the `anomalies_local.probability` and `anomalies_local.anomaly` charts. + +## Visualize anomalies in charts + +In either [Netdata Cloud](https://app.netdata.cloud) or the local Agent dashboard at `http://NODE:19999`, click on the +**Anomalies** [section](/web/gui/README.md#sections) to see the pair of anomaly detection charts, which are +preconfigured to visualize per-second anomaly metrics based on your [configuration in +`anomalies.conf`](/docs/guides/monitor/anomaly-detection-python.md#configure-the-anomalies-collector). + +These charts have the contexts `anomalies.probability` and `anomalies.anomaly`. Together, these charts +create meaningful visualizations for immediately recognizing not only that something is going wrong on your node, but +give context as to where to look next. + +The `anomalies_local.probability` chart shows the probability that the latest observed data is anomalous, based on the +trained model. The `anomalies_local.anomaly` chart visualizes 0→1 predictions based on whether the latest observed +data is anomalous based on the trained model. Both charts share the same dimensions, which you configured via +`charts_regex` and `charts_to_exclude` in [part 1](/docs/guides/monitor/anomaly-detection-python.md). + +In other words, the `probability` chart shows the amplitude of the anomaly, whereas the `anomaly` chart provides quick +yes/no context. + +![Two charts created by the anomalies +collector](https://user-images.githubusercontent.com/1153921/104226380-ef84eb00-5404-11eb-9faf-9e64c43b95ff.png) + +Before `08:32:00`, both charts show little in the way of verified anomalies. Based on the metrics the anomalies +collector has trained on, a certain percentage of anomaly probability score is normal, as seen in the +`web_log_nginx_requests_prob` dimension and a few others. What you're looking for is large deviations from the "noise" +in the `anomalies.probability` chart, or any increments to the `anomalies.anomaly` chart. + +Unsurprisingly, the stress test that began at `08:32:00` caused significant changes to these charts. The three +dimensions that immediately shot to 100% anomaly probability, and remained there during the test, were +`web_log_nginx.requests_prob`, `nginx_local.connections_accepted_handled_prob`, and `system.cpu_pressure_prob`. + +## Build an anomaly detection dashboard + +[Netdata Cloud](https://app.netdata.cloud) features a drag-and-drop [dashboard +editor](/docs/visualize/create-dashboards.md) that helps you create entirely new dashboards with charts targeted for +your specific applications. + +For example, here's a dashboard designed for visualizing anomalies present in an Nginx web server, including +documentation about why the dashboard exists and where to look next based on what you're seeing: + +![An example anomaly detection +dashboard](https://user-images.githubusercontent.com/1153921/104226915-c6188f00-5405-11eb-9bb4-559a18016fa7.png) + +Use the anomaly charts for instant visual identification of potential anomalies, and then Nginx-specific charts, in the +right column, to validate whether the probability and anomaly counters are showing a valid incident worth further +investigation using [Metric Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations) to narrow +the dashboard into only the charts relevant to what you're seeing from the anomalies collector. + +## What's next? + +Between this guide and [part 1](/docs/guides/monitor/anomaly-detection-python.md), which covered setup and configuration, you +now have a fundamental understanding of how unsupervised anomaly detection in Netdata works, from root cause to alarms +to preconfigured or custom dashboards. + +We'd love to hear your feedback on the anomalies collector. Hop over to the [community +forum](https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767), and let us know if you're already getting value from +unsupervised anomaly detection, or would like to see something added to it. You might even post a custom configuration +that works well for monitoring some other popular application, like MySQL, PostgreSQL, Redis, or anything else we +[support through collectors](/collectors/COLLECTORS.md). + +### Related reference documentation + +- [Netdata Agent · Anomalies collector](/collectors/python.d.plugin/anomalies/README.md) +- [Netdata Cloud · Build new dashboards](https://learn.netdata.cloud/docs/cloud/visualize/dashboards) + + diff --git a/docs/guides/python-collector.md b/docs/guides/python-collector.md new file mode 100644 index 0000000..920b9b9 --- /dev/null +++ b/docs/guides/python-collector.md @@ -0,0 +1,486 @@ + + +# Develop a custom data collector in Python + +The Netdata Agent uses [data collectors](/docs/collect/how-collectors-work.md) to fetch metrics from hundreds of system, +container, and service endpoints. While the Netdata team and community has built [powerful +collectors](/collectors/COLLECTORS.md) for most system, container, and service/application endpoints, there are plenty +of custom applications that can't be monitored by default. + +## Problem + +You have a custom application or infrastructure that you need to monitor, but no open-source monitoring tool offers a +prebuilt method for collecting your required metric data. + +## Solution + +In this tutorial, you'll learn how to leverage the [Python programming language](https://www.python.org/) to build a +custom data collector for the Netdata Agent. Follow along with your own dataset, using the techniques and best practices +covered here, or use the included examples for collecting and organizing either random or weather data. + +## What you need to get started + +- A physical or virtual Linux system, which we'll call a _node_. +- A working installation of the free and open-source [Netdata](/docs/get-started.mdx) monitoring agent. + +## Jobs and elements of a Python collector + +A Python collector for Netdata is a Python script that gathers data from an external source and transforms these data +into charts to be displayed by Netdata dashboard. The basic jobs of the plugin are: + +- Gather the data from the service/application. +- Create the required charts. +- Parse the data to extract or create the actual data to be represented. +- Assign the correct values to the charts +- Set the order for the charts to be displayed. +- Give the charts data to Netdata for visualization. + +The basic elements of a Netdata collector are: + +- `ORDER[]`: A list containing the charts to be displayed. +- `CHARTS{}`: A dictionary containing the details for the charts to be displayed. +- `data{}`: A dictionary containing the values to be displayed. +- `get_data()`: The basic function of the plugin which will return to Netdata the correct values. + +Let's walk through these jobs and elements as independent elements first, then apply them to example Python code. + +### Determine how to gather metrics data + +Netdata can collect data from any program that can print to stdout. Common input sources for collectors can be logfiles, +HTTP requests, executables, and more. While this tutorial will offer some example inputs, your custom application will +have different inputs and metrics. + +A great deal of the work in developing a Netdata collector is investigating the target application and understanding +which metrics it exposes and how to + +### Create charts + +For the data to be represented in the Netdata dashboard, you need to create charts. Charts (in general) are defined by +several characteristics: title, legend, units, type, and presented values. Each chart is represented as a dictionary +entry: + +```python +chart= { + "chart_name": + { + "options": [option_list], + "lines": [ + [dimension_list] + ] + } + } +``` + +Use the `options` field to set the chart's options, which is a list in the form `options: [name, title, units, family, +context, charttype]`, where: + +- `name`: The name of the chart. +- `title` : The title to be displayed in the chart. +- `units` : The units for this chart. +- `family`: An identifier used to group charts together (can be null). +- `context`: An identifier used to group contextually similar charts together. The best practice is to provide a context + that is `A.B`, with `A` being the name of the collector, and `B` being the name of the specific metric. +- `charttype`: Either `line`, `area`, or `stacked`. If null line is the default value. + +You can read more about `family` and `context` in the [web dashboard](/web/README.md#families) doc. + +Once the chart has been defined, you should define the dimensions of the chart. Dimensions are basically the metrics to +be represented in this chart and each chart can have more than one dimension. In order to define the dimensions, the +"lines" list should be filled in with the required dimensions. Each dimension is a list: + +`dimension: [id, name, algorithm, multiplier, divisor]` +- `id` : The id of the dimension. Mandatory unique field (string) required in order to set a value. +- `name`: The name to be presented in the chart. If null id will be used. +- `algorithm`: Can be absolute or incremental. If null absolute is used. Incremental shows the difference from the + previous value. +- `multiplier`: an integer value to divide the collected value, if null, 1 is used +- `divisor`: an integer value to divide the collected value, if null, 1 is used + +The multiplier/divisor fields are used in cases where the value to be displayed should be decimal since Netdata only +gathers integer values. + +### Parse the data to extract or create the actual data to be represented + +Once the data is received, your collector should process it in order to get the values required. If, for example, the +received data is a JSON string, you should parse the data to get the required data to be used for the charts. + +### Assign the correct values to the charts + +Once you have process your data and get the required values, you need to assign those values to the charts you created. +This is done using the `data` dictionary, which is in the form: + +`"data": {dimension_id: value }`, where: +- `dimension_id`: The id of a defined dimension in a created chart. +- `value`: The numerical value to associate with this dimension. + +### Set the order for the charts to be displayed + +Next, set the order of chart appearance with the `ORDER` list, which is in the form: + +`"ORDER": [chart_name_1,chart_name_2, …., chart_name_X]`, where: +- `chart_name_x`: is the chart name to be shown in X order. + +### Give the charts data to Netdata for visualization + +Our plugin should just rerun the data dictionary. If everything is set correctly the charts should be updated with the +correct values. + +## Framework classes + +The `python.d` plugin has a number of framework classes that can be used to speed up the development of your python +collector. Your class can inherit one of these framework classes, which have preconfigured methods. + +For example, the snippet below is from the [RabbitMQ +collector](https://github.com/netdata/netdata/blob/91f3268e9615edd393bd43de4ad8068111024cc9/collectors/python.d.plugin/rabbitmq/rabbitmq.chart.py#L273). +This collector uses an HTTP endpoint and uses the `UrlService` framework class, which only needs to define an HTTP +endpoint for data collection. + +```python +class Service(UrlService): + def __init__(self, configuration=None, name=None): + UrlService.__init__(self, configuration=configuration, name=name) + self.order = ORDER + self.definitions = CHARTS + self.url = '{0}://{1}:{2}'.format( + configuration.get('scheme', 'http'), + configuration.get('host', '127.0.0.1'), + configuration.get('port', 15672), + ) + self.node_name = str() + self.vhost = VhostStatsBuilder() + self.collected_vhosts = set() + self.collect_queues_metrics = configuration.get('collect_queues_metrics', False) + self.debug("collect_queues_metrics is {0}".format("enabled" if self.collect_queues_metrics else "disabled")) + if self.collect_queues_metrics: + self.queue = QueueStatsBuilder() + self.collected_queues = set() +``` + +In our use-case, we use the `SimpleService` framework, since there is no framework class that suits our needs. + +You can read more about the [framework classes](/collectors/python.d.plugin/README.md#how-to-write-a-new-module) from +the Netdata documentation. + +## An example collector using weather station data + +Let's build a custom Python collector for visualizing data from a weather monitoring station. + +### Determine how to gather metrics data + +This example assumes you can gather metrics data through HTTP requests to a web server, and that the data provided are +numeric values for temperature, humidity and pressure. It also assumes you can get the `min`, `max`, and `average` +values for these metrics. + +### Chart creation + +First, create a single chart that shows the latest temperature metric: + +```python +CHARTS = { + "temp_current": { + "options": ["my_temp", "Temperature", "Celsius", "TEMP", "weather_station.temperature", "line"], + "lines": [ + ["current_temp_id","current_temperature"] + ] + } +} +``` + +## Parse the data to extract or create the actual data to be represented + +A standard practice would be to either get the data on JSON format or transform them to JSON format. We use a dictionary +to give this format and issue random values to simulate received data. + +The following code iterates through the names of the expected values and creates a dictionary with the name of the value +as `key`, and a random value as `value`. + +```python + weather_data=dict() + weather_metrics=[ + "temp","av_temp","min_temp","max_temp", + "humid","av_humid","min_humid","max_humid", + "pressure","av_pressure","min_pressure","max_pressure", + ] + + def populate_data(self): + for metric in self.weather_metrics: + self.weather_data[metric]=random.randint(0,100) +``` + +### Assign the correct values to the charts + +Our chart has a dimension called `current_temp_id`, which should have the temperature value received. + +```python +data['current_temp_id'] = self.weather_data["temp"] +``` + +### Set the order for the charts to be displayed + +```python +ORDER = [ + "temp_current" +] +``` + +### Give the charts data to Netdata for visualization + +```python +return data +``` + +A snapshot of the chart created by this plugin: + +![A snapshot of the chart created by this plugin](https://i.imgur.com/2tR9KvF.png) + +Here's the current source code for the data collector: + +```python +# -*- coding: utf-8 -*- +# Description: howto weather station netdata python.d module +# Author: Panagiotis Papaioannou (papajohn-uop) +# SPDX-License-Identifier: GPL-3.0-or-later + +from bases.FrameworkServices.SimpleService import SimpleService + +import random + +NETDATA_UPDATE_EVERY=1 +priority = 90000 + +ORDER = [ + "temp_current" +] + +CHARTS = { + "temp_current": { + "options": ["my_temp", "Temperature", "Celsius", "TEMP", "weather_station.temperature", "line"], + "lines": [ + ["current_temperature"] + ] + } +} + +class Service(SimpleService): + def __init__(self, configuration=None, name=None): + SimpleService.__init__(self, configuration=configuration, name=name) + self.order = ORDER + self.definitions = CHARTS + #values to show at graphs + self.values=dict() + + @staticmethod + def check(): + return True + + weather_data=dict() + weather_metrics=[ + "temp","av_temp","min_temp","max_temp", + "humid","av_humid","min_humid","max_humid", + "pressure","av_pressure","min_pressure","max_pressure", + ] + + def logMe(self,msg): + self.debug(msg) + + def populate_data(self): + for metric in self.weather_metrics: + self.weather_data[metric]=random.randint(0,100) + + def get_data(self): + #The data dict is basically all the values to be represented + # The entries are in the format: { "dimension": value} + #And each "dimension" should belong to a chart. + data = dict() + + self.populate_data() + + data['current_temperature'] = self.weather_data["temp"] + + return data +``` + +## Add more charts to the existing weather station collector + +To enrich the example, add another chart the collector which to present the humidity metric. + +Add a new entry in the `CHARTS` dictionary with the definition for the new chart. + +```python +CHARTS = { + 'temp_current': { + 'options': ['my_temp', 'Temperature', 'Celsius', 'TEMP', 'weather_station.temperature', 'line'], + 'lines': [ + ['current_temperature'] + ] + }, + 'humid_current': { + 'options': ['my_humid', 'Humidity', '%', 'HUMIDITY', 'weather_station.humidity', 'line'], + 'lines': [ + ['current_humidity'] + ] + } +} +``` + +The data has already been created and parsed by the `weather_data=dict()` function, so you only need to populate the +`current_humidity` dimension `self.weather_data["humid"]`. + +```python + data['current_temperature'] = self.weather_data["temp"] + data['current_humidity'] = self.weather_data["humid"] +``` + +Next, put the new `humid_current` chart into the `ORDER` list: + +```python +ORDER = [ + 'temp_current', + 'humid_current' +] +``` + +[Restart Netdata](/docs/configure/start-stop-restart.md) with `sudo systemctl restart netdata` to see the new humidity +chart: + +![A snapshot of the modified chart](https://i.imgur.com/XOeCBmg.png) + +Next, time to add one more chart that visualizes the average, minimum, and maximum temperature values. + +Add a new entry in the `CHARTS` dictionary with the definition for the new chart. Since you want three values +represented in this this chart, add three dimensions. You should also use the same `FAMILY` value in the charts (`TEMP`) +so that those two charts are grouped together. + +```python +CHARTS = { + 'temp_current': { + 'options': ['my_temp', 'Temperature', 'Celsius', 'TEMP', 'weather_station.temperature', 'line'], + 'lines': [ + ['current_temperature'] + ] + }, + 'temp_stats': { + 'options': ['stats_temp', 'Temperature', 'Celsius', 'TEMP', 'weather_station.temperature_stats', 'line'], + 'lines': [ + ['min_temperature'], + ['max_temperature'], + ['avg_temperature'] + ] + }, + 'humid_current': { + 'options': ['my_humid', 'Humidity', '%', 'HUMIDITY', 'weather_station.humidity', 'line'], + 'lines': [ + ['current_humidity'] + ] + } + +} +``` + +As before, initiate new dimensions and add data to them: + +```python + data['current_temperature'] = self.weather_data["temp"] + data['min_temperature'] = self.weather_data["min_temp"] + data['max_temperature'] = self.weather_data["max_temp"] + data['avg_temperature`'] = self.weather_data["av_temp"] + data['current_humidity'] = self.weather_data["humid"] +``` + +Finally, set the order for the `temp_stats` chart: + +```python +ORDER = [ + 'temp_current', + ‘temp_stats’ + 'humid_current' +] +``` + +[Restart Netdata](/docs/configure/start-stop-restart.md) with `sudo systemctl restart netdata` to see the new +min/max/average temperature chart with multiple dimensions: + +![A snapshot of the modified chart](https://i.imgur.com/g7E8lnG.png) + +## Add a configuration file + +The last piece of the puzzle to create a fully robust Python collector is the configuration file. Python.d uses +configuration in [YAML](https://www.tutorialspoint.com/yaml/yaml_basics.htm) format and is used as follows: + +- Create a configuration file in the same directory as the `.chart.py`. Name it `.conf`. +- Define a `job`, which is an instance of the collector. It is useful when you want to collect data from different + sources with different attributes. For example, we could gather data from 2 different weather stations, which use + different temperature measures: Fahrenheit and Celsius. +- You can define many different jobs with the same name, but with different attributes. Netdata will try each job + serially and will stop at the first job that returns data. If multiple jobs have the same name, only one of them can + run. This enables you to define different "ways" to fetch data from a particular data source so that the collector has + more chances to work out-of-the-box. For example, if the data source supports both `HTTP` and `linux socket`, you can + define 2 jobs named `local`, with each using a different method. +- Check the `example` collector configuration file on + [GitHub](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/example/example.conf) to get a + sense of the structure. + +```yaml +weather_station_1: + name: 'Greece' + endpoint: 'https://endpoint_1.com' + port: 67 + type: 'celsius' +weather_station_2: + name: 'Florida USA' + endpoint: 'https://endpoint_2.com' + port: 67 + type: 'fahrenheit' +``` + +Next, access the above configuration variables in the `__init__` function: + +```python +def __init__(self, configuration=None, name=None): + SimpleService.__init__(self, configuration=configuration, name=name) + self.endpoint = self.configuration.get('endpoint', ) +``` + +Because you initiate the `framework class` (e.g `SimpleService.__init__`), the configuration will be available +throughout the whole `Service` class of your module, as `self.configuration`. Finally, note that the `configuration.get` +function takes 2 arguments, one with the name of the configuration field and one with a default value in case it doesn't +find the configuration field. This allows you to define sane defaults for your collector. + +Moreover, when creating the configuration file, create a large comment section that describes the configuration +variables and inform the user about the defaults. For example, take a look at the `example` collector on +[GitHub](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/example/example.conf). + +You can read more about the configuration file on the [`python.d.plugin` +documentation](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin). + +## What's next? + +Find the source code for the above examples on [GitHub](https://github.com/papajohn-uop/netdata). + +Now you are ready to start developing our Netdata python Collector and share it with the rest of the Netdata community. + +- If you need help while developing your collector, join our [Netdata + Community](https://community.netdata.cloud/c/agent-development/9) to chat about it. +- Follow the + [checklist](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin#pull-request-checklist-for-python-plugins) + to contribute the collector to the Netdata Agent [repository](https://github.com/netdata/netdata). +- Check out the [example](https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin/example) Python + collector, which is a minimal example collector you could also use as a starting point. Once comfortable with that, + then browse other [existing collectors](https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin) + that might have similarities to what you want to do. +- If you're developing a proof of concept (PoC), consider migrating the collector in Golang + ([go.d.plugin](https://github.com/netdata/go.d.plugin)) once you validate its value in production. Golang is more + performant, easier to maintain, and simpler for users since it doesn't require a particular runtime on the node to + execute (Python plugins require Python on the machine to be executed). Netdata uses Go as the platform of choice for + production-grade collectors. +- Celebrate! You have contributed to an open-source project with hundreds of thousands of users! + + diff --git a/docs/guides/step-by-step/step-00.md b/docs/guides/step-by-step/step-00.md new file mode 100644 index 0000000..9f0feca --- /dev/null +++ b/docs/guides/step-by-step/step-00.md @@ -0,0 +1,120 @@ + +import { OneLineInstallWget, OneLineInstallCurl } from '@site/src/components/OneLineInstall/' + +# The step-by-step Netdata guide + +Welcome to Netdata! We're glad you're interested in our health monitoring and performance troubleshooting system. + +Because Netdata is entirely open-source software, you can use it free of charge, whether you want to monitor one or ten +thousand systems! All our code is hosted on [GitHub](https://github.com/netdata/netdata). + +This guide is designed to help you understand what Netdata is, what it's capable of, and how it'll help you make +faster and more informed decisions about the health and performance of your systems and applications. If you're +completely new to Netdata, or have never tried health monitoring/performance troubleshooting systems before, this +guide is perfect for you. + +If you have monitoring experience, or would rather get straight into configuring Netdata to your needs, you can jump +straight into code and configurations with our [getting started guide](/docs/get-started.mdx). + +> This guide contains instructions for Netdata installed on a Linux system. Many of the instructions will work on +> other supported operating systems, like FreeBSD and macOS, but we can't make any guarantees. + +## Where to go if you need help + +No matter where you are in this Netdata guide, if you need help, head over to our [GitHub +repository](https://github.com/netdata/netdata/). That's where we collect questions from users, help fix their bugs, and +point people toward documentation that explains what they're having trouble with. + +Click on the **issues** tab to see all the conversations we're having with Netdata users. Use the search bar to find +previously-written advice for your specific problem, and if you don't see any results, hit the **New issue** button to +send us a question. + + +## Before we get started + +Let's make sure you have Netdata installed on your system! + +> If you already installed Netdata, feel free to skip to [Step 1: Netdata's building blocks](step-01.md). + +The easiest way to install Netdata on a Linux system is our `kickstart.sh` one-line installer. Run this on your system +and let it take care of the rest. + +This script will install Netdata from source, keep it up to date with nightly releases, connects to the Netdata +[registry](/registry/README.md), and sends [_anonymous statistics_](/docs/anonymous-statistics.md) about how you use +Netdata. We use this information to better understand how we can improve the Netdata experience for all our users. + +To install Netdata, run the following as your normal user: + + + +Or, if you have cURL but not wget (such as on macOS): + + + + +Once finished, you'll have Netdata installed, and you'll be set up to get _nightly updates_ to get the latest features, +improvements, and bugfixes. + +If this method doesn't work for you, or you want to use a different process, visit our [installation +documentation](/packaging/installer/README.md) for details. + +## Netdata fundamentals + +[Step 1. Netdata's building blocks](step-01.md) + +In this introductory step, we'll talk about the fundamental ideas, philosophies, and UX decisions behind Netdata. + +[Step 2. Get to know Netdata's dashboard](step-02.md) + +Visit Netdata's dashboard to explore, manipulate charts, and check out alarms. Get your first taste of visual anomaly +detection. + +[Step 3. Monitor more than one system with Netdata](step-03.md) + +While the dashboard lets you quickly move from one agent to another, Netdata Cloud is our SaaS solution for monitoring +the health of many systems. We'll cover its features and the benefits of using Netdata Cloud on top of the dashboard. + +[Step 4. The basics of configuring Netdata](step-04.md) + +While Netdata can monitor thousands of metrics in real-time without any configuration, you may _want_ to tweak some +settings based on your system's resources. + +## Intermediate steps + +[Step 5. Health monitoring alarms and notifications](step-05.md) + +Learn how to tune, silence, and write custom alarms. Then enable notifications so you never miss a change in health +status or performance anomaly. + +[Step 6. Collect metrics from more services and apps](step-06.md) + +Learn how to enable/disable collection plugins and configure a collection plugin job to add more charts to your Netdata +dashboard and begin monitoring more apps and services, like MySQL, Nginx, MongoDB, and hundreds more. + +[Step 7. Netdata's dashboard in depth](step-07.md) + +Now that you configured your Netdata monitoring agent to your exact needs, you'll dive back into metrics snapshots, +updates, and the dashboard's settings. + +## Advanced steps + +[Step 8. Building your first custom dashboard](step-08.md) + +Using simple HTML, CSS, and JavaScript, we'll build a custom dashboard that displays essential information in any format +you choose. You can even monitor many systems from a single HTML file. + +[Step 9. Long-term metrics storage](step-09.md) + +By default, Netdata can store lots of real-time metrics, but you can also tweak our custom database engine to your +heart's content. Want to take your Netdata metrics elsewhere? We're happy to help you archive data to Prometheus, +MongoDB, TimescaleDB, and others. + +[Step 10. Set up a proxy](step-10.md) + +Run Netdata behind an Nginx proxy to improve performance, and enable TLS/HTTPS for better security. + + diff --git a/docs/guides/step-by-step/step-01.md b/docs/guides/step-by-step/step-01.md new file mode 100644 index 0000000..f5430e3 --- /dev/null +++ b/docs/guides/step-by-step/step-01.md @@ -0,0 +1,156 @@ + + +# Step 1. Netdata's building blocks + +Netdata is a distributed and real-time _health monitoring and performance troubleshooting toolkit_ for monitoring your +systems and applications. + +Because the monitoring agent is highly-optimized, you can install it all your physical systems, containers, IoT devices, +and edge devices without disrupting their core function. + +By default, and without configuration, Netdata delivers real-time insights into everything happening on the system, from +CPU utilization to packet loss on every network device. Netdata can also auto-detect metrics from hundreds of your +favorite services and applications, like MySQL/MariaDB, Docker, Nginx, Apache, MongoDB, and more. + +All metrics are automatically-updated, providing interactive dashboards that allow you to dive in, discover anomalies, +and figure out the root cause analysis of any issue. + +Best of all, Netdata is entirely free, open-source software! Solo developers and enterprises with thousands of systems +can both use it free of charge. We're hosted on [GitHub](https://github.com/netdata/netdata). + +Want to learn about the history of Netdata, and what inspired our CEO to build it in the first place, and where we're +headed? Read Costa's comprehensive blog post: _[Redefining monitoring with Netdata (and how it came to +be)](https://blog.netdata.cloud/posts/redefining-monitoring-netdata/)_. + +## What you'll learn in this step + +In the first step of the Netdata guide, you'll learn about: + +- [Netdata's core features](#netdatas-core-features) +- [Why you should use Netdata](#why-you-should-use-netdata) +- [How Netdata has complementary systems, not competitors](#how-netdata-has-complementary-systems-not-competitors) + +Let's get started! + +## Netdata's core features + +Netdata has only been around for a few years, but it's a complex piece of software. Here are just some of the features +we'll cover throughout this guide. + +- A sophisticated **dashboard**, which we'll cover in [step 2](step-02.md). The real-time, highly-granular dashboard, + with hundreds of charts, is your main source of information about the health and performance of your systems/ + applications. We designed the dashboard with anomaly detection and quick analysis in mind. We'll return to + dashboard-related topics in both [step 7](step-07.md) and [step 8](step-08.md). +- **Long-term metrics storage** by default. With our new database engine, you can store days, weeks, or months of + per-second historical metrics. Or you can archive metrics to another database, like MongoDB or Prometheus. We'll + cover all these options in [step 9](step-09.md). +- **No configuration necessary**. Without any configuration, you'll get thousands of real-time metrics and hundreds of + alarms designed by our community of sysadmin experts. But you _can_ configure Netdata in a lot of ways, some of + which we'll cover in [step 4](step-04.md). +- **Distributed, per-system installation**. Instead of centralizing metrics in one location, you install Netdata on + _every_ system, and each system is responsible for its metrics. Having distributed agents reduces cost and lets + Netdata run on devices with little available resources, such as IoT and edge devices, without affecting their core + purpose. +- **Sophisticated health monitoring** to ensure you always know when an anomaly hits. In [step 5](step-05.md), we dive + into how you can tune alarms, write your own alarm, and enable two types of notifications. +- **High-speed, low-resource collectors** that allow you to collect thousands of metrics every second while using only + a fraction of your system's CPU resources and a few MiB of RAM. +- **Netdata Cloud** is our SaaS toolkit that helps Netdata users monitor the health and performance of entire + infrastructures, whether they are two or two thousand (or more!) systems. We'll cover Netdata Cloud in [step + 3](step-03.md). + +## Why you should use Netdata + +Because you care about the health and performance of your systems and applications, and all of the awesome features we +just mentioned. And it's free! + +All these may be valid reasons, but let's step back and talk about Netdata's _principles_ for health monitoring and +performance troubleshooting. We have a lot of [complementary +systems](#how-netdata-has-complementary-systems-not-competitors), and we think there's a good reason why Netdata should +always be your first choice when troubleshooting an anomaly. + +We built Netdata on four principles. + +### Per-second data collection + +Our first principle is per-second data collection for all metrics. + +That matters because you can't monitor a 2-second service-level agreement (SLA) with 10-second metrics. You can't detect +quick anomalies if your metrics don't show them. + +How do we solve this? By decentralizing monitoring. Each node is responsible for collecting metrics, triggering alarms, +and building dashboards locally, and we work hard to ensure it does each step (and others) with remarkable efficiency. +For example, Netdata can [collect 100,000 metrics](https://github.com/netdata/netdata/issues/1323) every second while +using only 9% of a single server-grade CPU core! + +By decentralizing monitoring and emphasizing speed at every turn, Netdata helps you scale your health monitoring and +performance troubleshooting to an infrastructure of every size. _And_ you get to keep per-second metrics in long-term +storage thanks to the database engine. + +### Unlimited metrics + +We believe all metrics are fundamentally important, and all metrics should be available to the user. + +If you don't collect _all_ the metrics a system creates, you're only seeing part of the story. It's like saying you've +read a book after skipping all but the last ten pages. You only know the ending, not everything that leads to it. + +Most monitoring solutions exist to poke you when there's a problem, and then tell you to use a dozen different console +tools to find the root cause. Netdata prefers to give you every piece of information you might need to understand why an +anomaly happened. + +### Meaningful presentation + +We want every piece of Netdata's dashboard not only to look good and update every second, but also provide context as to +what you're looking at and why it matters. + +The principle of meaningful presentation is fundamental to our dashboard's user experience (UX). We could have put +charts in a grid or hidden some behind tabs or buttons. We instead chose to stack them vertically, on a single page, so +you can visually see how, for example, a jump in disk usage can also increase system load. + +Here's an example of a system undergoing a disk stress test: + +![Screen Shot 2019-10-23 at 15 38 +32](https://user-images.githubusercontent.com/1153921/67439589-7f920700-f5ab-11e9-930d-fb0014900d90.png) + +> For the curious, here's the command: `stress-ng --fallocate 4 --fallocate-bytes 4g --timeout 1m --metrics --verify +> --times`! + +### Immediate results + +Finally, Netdata should be usable from the moment you install it. + +As we've talked about, and as you'll learn in the following nine steps, Netdata comes installed with: + +- Auto-detected metrics +- Human-readable units +- Metrics that are structured into charts, families, and contexts +- Automatically generated dashboards +- Charts designed for visual anomaly detection +- Hundreds of pre-configured alarms + +By standardizing your monitoring infrastructure, Netdata tries to make at least one part of your administrative tasks +easy! + +## How Netdata has complementary systems, not competitors + +We'll cover this quickly, as you're probably eager to get on with using Netdata itself. + +We don't want to lock you in to using Netdata by itself, and forever. By supporting [archiving to +external databases](/exporting/README.md) like Graphite, Prometheus, OpenTSDB, MongoDB, and others, you can use Netdata _in +conjunction_ with software that might seem like our competitors. + +We don't want to "wage war" with another monitoring solution, whether it's commercial, open-source, or anything in +between. We just want to give you all the metrics every second, and what you do with them next is your business, not +ours. Our mission is helping people create more extraordinary infrastructures! + +## What's next? + +We think it's imperative you understand why we built Netdata the way we did. But now that we have that behind us, let's +get right into that dashboard you've heard so much about. + +[Next: Get to know Netdata's dashboard →](step-02.md) + + diff --git a/docs/guides/step-by-step/step-02.md b/docs/guides/step-by-step/step-02.md new file mode 100644 index 0000000..4b802ff --- /dev/null +++ b/docs/guides/step-by-step/step-02.md @@ -0,0 +1,208 @@ + + +# Step 2. Get to know Netdata's dashboard + +Welcome to Netdata proper! Now that you understand how Netdata works, how it's built, and why we built it, you can start +working with the dashboard directly. + +This step-by-step guide assumes you've already installed Netdata on a system of yours. If you haven't yet, hop back over +to ["step 0"](step-00.md#before-we-get-started) for information about our one-line installer script. Or, view the +[installation docs](/packaging/installer/README.md) to learn more. Once you have Netdata installed, you can hop back +over here and dig in. + +## What you'll learn in this step + +In this step of the Netdata guide, you'll learn how to: + +- [Visit and explore the dashboard](#visit-and-explore-the-dashboard) +- [Explore available charts using menus](#explore-available-charts-using-menus) +- [Read the descriptions accompanying charts](#read-the-descriptions-accompanying-charts) +- [Interact with charts](#interact-with-charts) +- [See raised alarms and the alarm log](#see-raised-alarms-and-the-alarm-log) + +Let's get started! + +## Visit and explore the dashboard + +Netdata's dashboard is where you interact with your system's metrics. Time to open it up and start exploring. Open up +your browser of choice. + +Open up your web browser of choice and navigate to `http://NODE:19999`, replacing `NODE` with the IP address or hostname +of your Agent. If you're unsure, try `http://localhost:19999` first. Hit **Enter**. Welcome to Netdata! + +![Animated GIF of navigating to the +dashboard](https://user-images.githubusercontent.com/1153921/80825153-abaec600-8b94-11ea-8b17-1b770a2abaa9.gif) + +> From here on out in this guide, we'll refer to the address you use to view your dashboard as `NODE`. Be sure to +> replace it with either `localhost`, the IP address, or the hostname of your system. + +## Explore available charts using menus + +**Menus** are located on the right-hand side of the Netdata dashboard. You can use these to navigate to the +charts you're interested in. + +![Animated GIF of using the menus and +submenus](https://user-images.githubusercontent.com/1153921/80832425-7c528600-8ba1-11ea-8140-d0a17a62009b.gif) + +Netdata shows all its charts on a single page, so you can also scroll up and down using the mouse wheel, your +touchscreen/touchpad, or the scrollbar. + +Both menus and the items displayed beneath them, called **submenus**, are populated automatically by Netdata based on +what it's collecting. If you run Netdata on many different systems using different OS types or versions, the +menus and submenus may look a little different for each one. + +To learn more about menus, see our documentation about [navigating the standard +dashboard](/web/gui/README.md#metrics-menus). + +> ❗ By default, Netdata only creates and displays charts if the metrics are _not zero_. So, you may be missing some +> charts, menus, and submenus if those charts have zero metrics. You can change this by changing the **Which dimensions +> to show?** setting to **All**. In addition, if you start Netdata and immediately load the dashboard, not all +> charts/menus/submenus may be displayed, as some collectors can take a while to initialize. + +## Read the descriptions accompanying charts + +Many charts come with a short description of what dimensions the chart is displaying and why they matter. + +For example, here's the description that accompanies the **swap** chart. + +![Screenshot of the swap +description](https://user-images.githubusercontent.com/1153921/63452078-477b1600-c3fa-11e9-836b-2fc90fba8b4b.png) + +If you're new to health monitoring and performance troubleshooting, we recommend you spend some time reading these +descriptions and learning more at the pages linked above. + +## Understand charts, dimensions, families, and contexts + +A **chart** is an interactive visualization of one or more collected/calculated metrics. You can see the name (also +known as its unique ID) of a chart by looking at the top-left corner of a chart and finding the parenthesized text. On a +Linux system, one of the first charts on the dashboard will be the system CPU chart, with the name `system.cpu`: + +![Screenshot of the system CPU chart in the Netdata +dashboard](https://user-images.githubusercontent.com/1153921/67443082-43b16e80-f5b8-11e9-8d33-d6ee052c6678.png) + +A **dimension** is any value that gets shown on a chart. The value can be raw data or calculated values, such as +percentages, aggregates, and more. Most charts will have more than one dimension, in which case it will display each in +a different color. Here, a `system.cpu` chart is showing many dimensions, such as `user`, `system`, `softirq`, `irq`, +and more. + +![Screenshot of the dimensions shown in the system CPU chart in the Netdata +dashboard](https://user-images.githubusercontent.com/1153921/62721031-2bba4d80-b9c0-11e9-9dca-32403617ce72.png) + +A **family** is _one_ instance of a monitored hardware or software resource that needs to be monitored and displayed +separately from similar instances. For example, if your system has multiple partitions, Netdata will create different +families for `/`, `/boot`, `/home`, and so on. Same goes for entire disks, network devices, and more. + +![A number of families created for disk partitions](https://user-images.githubusercontent.com/1153921/67896952-a788e980-fb1a-11e9-880b-2dfb3945c8d6.png) + +A **context** groups several charts based on the types of metrics being collected and displayed. For example, the +**Disk** section often has many contexts: `disk.io`, `disk.ops`, `disk.backlog`, `disk.util`, and so on. Netdata uses +this context to create individual charts and then groups them by family. You can always see the context of any chart by +looking at its name or hovering over the chart's date. + +It's important to understand these differences, as Netdata uses charts, dimensions, families, and contexts to create +health alarms and configure collectors. To read even more about the differences between all these elements of the +dashboard, and how they affect other parts of Netdata, read our [dashboards +documentation](/web/README.md#charts-contexts-families). + +## Interact with charts + +We built Netdata to be a big sandbox for learning more about your systems and applications. Time to play! + +Netdata's charts are fully interactive. You can pan through historical metrics, zoom in and out, select specific +timeframes for further analysis, resize charts, and more. + +Best of all, Whenever you use a chart in this way, Netdata synchronizes all the other charts to match it. + +![Animated GIF of the standard Netdata dashboard being manipulated and synchronizing +charts](https://user-images.githubusercontent.com/1153921/81867875-3d6beb00-9526-11ea-94b8-388951e2e03d.gif) + +### Pan, zoom, highlight, and reset charts + +You can change how charts show their metrics in a few different ways, each of which have a few methods: + +| Change | Method #1 | Method #2 | Method #3 | +| ------------------------------------------------- | ----------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------- | +| **Reset** charts to default auto-refreshing state | `double click` | `double tap` (touchpad/touchscreen) | | +| **Select** a certain timeframe | `ALT` + `mouse selection` | `⌘` + `mouse selection` (macOS) | | +| **Pan** forward or back in time | `click and drag` | `touch and drag` (touchpad/touchscreen) | | +| **Zoom** to a specific timeframe | `SHIFT` + `mouse selection` | | | +| **Zoom** in/out | `SHIFT`/`ALT` + `mouse scrollwheel` | `SHIFT`/`ALT` + `two-finger pinch` (touchpad/touchscreen) | `SHIFT`/`ALT` + `two-finger scroll` (touchpad/touchscreen) | + +These interactions can also be triggered using the icons on the bottom-right corner of every chart. They are, +respectively, `Pan Left`, `Reset`, `Pan Right`, `Zoom In`, and `Zoom Out`. + +### Show and hide dimensions + +Each dimension can be hidden by clicking on it. Hiding dimensions simplifies the chart and can help you better discover +exactly which aspect of your system is behaving strangely. + +### Resize charts + +Additionally, resize charts by clicking-and-dragging the icon on the bottom-right corner of any chart. To restore the +chart to its original height, double-click the same icon. + +![Animated GIF of resizing a chart and resetting it to the default +height](https://user-images.githubusercontent.com/1153921/80842459-7d41e280-8bb6-11ea-9488-1bc29f94d7f2.gif) + +To learn more about other options and chart interactivity, read our [dashboard documentation](/web/README.md). + +## See raised alarms and the alarm log + +Aside from performance troubleshooting, the Agent helps you monitor the health of your systems and applications. That's +why every Netdata installation comes with dozens of pre-configured alarms that trigger alerts when your system starts +acting strangely. + +Find the **Alarms** button in the top navigation bring up a modal that shows currently raised alarms, all running +alarms, and the alarms log. + +Here is an example of a raised `system.cpu` alarm, followed by the full list and alarm log: + +![Animated GIF of looking at raised alarms and the alarm +log](https://user-images.githubusercontent.com/1153921/80842482-8c289500-8bb6-11ea-9791-600cfdbe82ce.gif) + +And a static screenshot of the raised CPU alarm: + +![Screenshot of a raised system CPU alarm](https://user-images.githubusercontent.com/1153921/80842330-2dfbb200-8bb6-11ea-8147-3cd366eb0f37.png) + +The alarm itself is named *system - cpu**, and its context is `system.cpu`. Beneath that is an auto-updating badge that +shows the latest value the chart that triggered the alarm. + +With the three icons beneath that and the **role** designation, you can: + +1. Scroll to the chart associated with this raised alarm. +2. Copy a link to the badge to your clipboard. +3. Copy the code to embed the badge onto another web page using an `` element. + +The table on the right-hand side displays information about the alarm's configuration. In above example, Netdata +triggers a warning alarm when CPU usage is between 75 and 85%, and a critical alarm when above 85%. It's a _little_ more +complicated than that, but we'll get into more complex health entity configurations in a later step. + +The `calculation` field is the equation used to calculate those percentages, and the `check every` field specifies how +often Netdata should be calculating these metrics to see if the alarm should remain triggered. + +The `execute` field tells Netdata how to notify you about this alarm, and the `source` field lets you know where you can +find the configuration file, if you'd like to edit its configuration. + +We'll cover alarm configuration in more detail later in the guide, so don't worry about it too much for now! Right +now, it's most important that you understand how to see alarms, and parse their details, if and when they appear on your +system. + +## What's next? + +In this step of the Netdata guide, you learned how to: + +- Visit the dashboard +- Explore available charts (using the right-side menu) +- Read the descriptions accompanying charts +- Interact with charts +- See raised alarms and the alarm log + +Next, you'll learn how to monitor multiple nodes through the dashboard. + +[Next: Monitor more than one system with Netdata →](step-03.md) + + diff --git a/docs/guides/step-by-step/step-03.md b/docs/guides/step-by-step/step-03.md new file mode 100644 index 0000000..c1d283b --- /dev/null +++ b/docs/guides/step-by-step/step-03.md @@ -0,0 +1,91 @@ + + +# Step 3. Monitor more than one system with Netdata + +The Netdata agent is _distributed_ by design. That means each agent operates independently from any other, collecting +and creating charts only for the system you installed it on. We made this decision a long time ago to [improve security +and performance](step-01.md). + +You might be thinking, "So, now I have to remember all these IP addresses, and type them into my browser +manually, to move from one system to another? Maybe I should just make a bunch of bookmarks. What's a few more tabs +on top of the hundred I have already?" + +We get it. That's why we built [Netdata Cloud](https://learn.netdata.cloud/docs/cloud/), which connects many distributed +agents for a seamless experience when monitoring an entire infrastructure of Netdata-monitored nodes. + +![Animated GIF of Netdata +Cloud](https://user-images.githubusercontent.com/1153921/80828986-1ebb3b00-8b9b-11ea-957f-2c8d0d009e44.gif) + +## What you'll learn in this step + +In this step of the Netdata guide, we'll talk about the following: + +- [Why you should use Netdata Cloud](#why-use-netdata-cloud) +- [Get started with Netdata Cloud](#get-started-with-netdata-cloud) +- [Navigate between dashboards with Visited Nodes](#navigate-between-dashboards-with-visited-nodes) + +## Why use Netdata Cloud? + +Our [Cloud documentation](https://learn.netdata.cloud/docs/cloud/) does a good job (we think!) of explaining why Cloud +gives you a ton of value at no cost: + +> Netdata Cloud gives you real-time visibility for your entire infrastructure. With Netdata Cloud, you can run all your +> distributed Agents in headless mode _and_ access the real-time metrics and insightful charts from their dashboards. +> View key metrics and active alarms at-a-glance, and then seamlessly dive into any of your distributed dashboards +> without leaving Cloud's centralized interface. + +You can add as many nodes and team members as you need, and as our free and open source Agent gets better with more +features, new collectors for more applications, and improved UI, so will Cloud. + +## Get started with Netdata Cloud + +Signing in, onboarding, and connecting your first nodes only takes a few minutes, and we have a [Get started with +Cloud](https://learn.netdata.cloud/docs/cloud/get-started) guide to help you walk through every step. + +Or, if you're feeling confident, dive right in. + +

Sign in to Cloud

+ +When you finish that guide, circle back to this step in the guide to learn how to use the Visited Nodes feature on +top of Cloud's centralized web interface. + +## Navigate between dashboards with Visited Nodes + +To add nodes to your visited nodes, you first need to navigate to that node's dashboard, then click the **Sign in** +button at the top of the dashboard. On the screen that appears, which states your node is requesting access to your +Netdata Cloud account, sign in with your preferred method. + +Cloud redirects you back to your node's dashboard, which is now connected to your Netdata Cloud account. You can now see the menu populated by a single visited node. + +![An Agent's dashboard with the Visited nodes +menu](https://user-images.githubusercontent.com/1153921/80830383-b6ba2400-8b9d-11ea-9eb2-379c7eccd22f.png) + +If you previously went through the Cloud onboarding process to create a Space and War Room, you will also see these +alongside your visited nodes. You can click on your Space or any of your War Rooms to navigate to Netdata Cloud and +continue monitoring your infrastructure from there. + +![A Agent's dashboard with the Visited nodes menu, plus Spaces and War +Rooms](https://user-images.githubusercontent.com/1153921/80830382-b6218d80-8b9d-11ea-869c-1170b95eeb4a.png) + +To add other visited nodes, navigate to their dashboard and sign in to Cloud by clicking on the **Sign in** button. This +process connects that node to your Cloud account and further populates the menu. + +Once you've added more than one node, you can use the menu to switch between various dashboards without remembering IP +addresses or hostnames or saving bookmarks for every node you want to monitor. + +![Switching between dashboards with Visited +nodes](https://user-images.githubusercontent.com/1153921/80831018-e158ac80-8b9e-11ea-882e-1d82cdc028cd.gif) + +## What's next? + +Now that you have a Netdata Cloud account with a connected node (or a few!) and can navigate between your dashboards with +Visited nodes, it's time to learn more about how you can configure Netdata to your liking. From there, you'll be able to +customize your Netdata experience to your exact infrastructure and the information you need. + +[Next: The basics of configuring Netdata →](step-04.md) + + diff --git a/docs/guides/step-by-step/step-04.md b/docs/guides/step-by-step/step-04.md new file mode 100644 index 0000000..37b4245 --- /dev/null +++ b/docs/guides/step-by-step/step-04.md @@ -0,0 +1,144 @@ + + +# Step 4. The basics of configuring Netdata + +Welcome to the fourth step of the Netdata guide. + +Since the beginning, we've covered the building blocks of Netdata, dashboard basics, and how you can monitor many +individual systems using many distributed Netdata agents. + +Next up: configuration. + +## What you'll learn in this step + +We'll talk about Netdata's default configuration, and then you'll learn how to do the following: + +- [Find your `netdata.conf` file](#find-your-netdataconf-file) +- [Use edit-config to open `netdata.conf`](#use-edit-config-to-open-netdataconf) +- [Navigate the structure of `netdata.conf`](#the-structure-of-netdataconf) +- [Edit your `netdata.conf` file](#edit-your-netdataconf-file) + +## Find your `netdata.conf` file + +Netdata primarily uses the `netdata.conf` file to configure its core functionality. `netdata.conf` resides within your +**Netdata config directory**. + +The location of that directory and `netdata.conf` depends on your operating system and the method you used to install +Netdata. + +The most reliable method of finding your Netdata config directory is loading your `netdata.conf` on your browser. Open a +tab and navigate to `http://HOST:19999/netdata.conf`. Your browser will load a text document that looks like this: + +![A netdata.conf file opened in the +browser](https://user-images.githubusercontent.com/1153921/68346763-344f1c80-00b2-11ea-9d1d-0ccac74d5558.png) + +Look for the line that begins with `# config directory = `. The text after that will be the path to your Netdata config +directory. + +In the system represented by the screenshot, the line reads: `config directory = /etc/netdata`. That means +`netdata.conf`, and all the other configuration files, can be found at `/etc/netdata`. + +> For more details on where your Netdata config directory is, take a look at our [installation +> instructions](/packaging/installer/README.md). + +For the rest of this guide, we'll assume you're editing files or running scripts from _within_ your **Netdata +configuration directory**. + +## Use edit-config to open `netdata.conf` + +Inside your Netdata config directory, there is a helper scripted called `edit-config`. This script will open existing +Netdata configuration files using a text editor. Or, if the configuration file doesn't yet exist, the script will copy +an example file to your Netdata config directory and then allow you to edit it before saving it. + +> `edit-config` will use the `EDITOR` environment variable on your system to edit the file. On many systems, that is +> defaulted to `vim` or `nano`. We highly recommend `nano` for beginners. To change this variable for the current +> session (it will revert to the default when you reboot), export a new value: `export EDITOR=nano`. Or, [make the +> change permanent](https://stackoverflow.com/questions/13046624/how-to-permanently-export-a-variable-in-linux). + +Let's give it a shot. Navigate to your Netdata config directory. To use `edit-config` on `netdata.conf`, you need to +have permissions to edit the file. On Linux/macOS systems, you can usually use `sudo` to elevate your permissions. + +```bash +cd /etc/netdata # Replace this path with your Netdata config directory, if different as found in the steps above +sudo ./edit-config netdata.conf +``` + +You should now see `netdata.conf` your editor! Let's walk through how the file is structured. + +## The structure of `netdata.conf` + +There are two main parts of the file to note: **sections** and **options**. + +The `netdata.conf` file is broken up into various **sections**, such as `[global]`, `[web]`, and `[registry]`. Each +section contains the configuration options for some core component of Netdata. + +Each section also contains many **options**. Options have a name and a value. With the option `config directory = +/etc/netdata`, `config directory` is the name, and `/etc/netdata` is the value. + +Most lines are **commented**, in that they start with a hash symbol (`#`), and the value is set to a sane default. To +tell Netdata that you'd like to change any option from its default value, you must **uncomment** it by removing that +hash. + +### Edit your `netdata.conf` file + +Let's try editing the options in `netdata.conf` to see how the process works. + +First, add a fake option to show you how Netdata loads its configuration files. Add a `test` option under the `[global]` +section and give it the value of `1`. + +```conf +[global] + test = 1 +``` + +Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +Now, open up your browser and navigate to `http://HOST:19999/netdata.conf`. You'll see that Netdata has recognized +that our fake option isn't valid and added a notice that Netdata will ignore it. + +Here's the process in GIF form! + +![Animated GIF of creating a fake option in +netdata.conf](https://user-images.githubusercontent.com/1153921/65470254-4422e200-de1f-11e9-9597-a97c89ee59b8.gif) + +Now, let's make a slightly more substantial edit to `netdata.conf`: change the Agent's name. + +If you edit the value of the `hostname` option, you can change the name of your Netdata Agent on the dashboard and a +handful of other places, like the Visited nodes menu _and_ Netdata Cloud. + +Use `edit-config` to change the `hostname` option to a name like `hello-world`. Be sure to uncomment it! + +```conf +[global] + hostname = hello-world +``` + +Once you're done, restart Netdata and refresh the dashboard. Say hello to your renamed agent! + +![Animated GIF of editing the hostname option in +netdata.conf](https://user-images.githubusercontent.com/1153921/80994808-1c065300-8df2-11ea-81af-d28dc3ba27c8.gif) + +Netdata has dozens upon dozens of options you can change. To see them all, read our [daemon +configuration](/daemon/config/README.md), or hop into our popular guide on [increasing long-term metrics +storage](/docs/guides/longer-metrics-storage.md). + +## What's next? + +At this point, you should be comfortable with getting to your Netdata directory, opening and editing `netdata.conf`, and +seeing your changes reflected in the dashboard. + +Netdata has many more configuration files that you might want to change, but we'll cover those in the following steps of +this guide. + +In the next step, we're going to cover one of Netdata's core functions: monitoring the health of your systems via alarms +and notifications. You'll learn how to disable alarms, create new ones, and push notifications to the system of your +choosing. + +[Next: Health monitoring alarms and notifications →](step-05.md) + + diff --git a/docs/guides/step-by-step/step-05.md b/docs/guides/step-by-step/step-05.md new file mode 100644 index 0000000..3cd8c5d --- /dev/null +++ b/docs/guides/step-by-step/step-05.md @@ -0,0 +1,350 @@ + + +# Step 5. Health monitoring alarms and notifications + +In the fifth step of the Netdata guide, we're introducing you to one of our core features: **health monitoring**. + +To accurately monitor the health of your systems and applications, you need to know _immediately_ when there's something +strange going on. Netdata's alarm and notification systems are essential to keeping you informed. + +Netdata comes with hundreds of pre-configured alarms that don't require configuration. They were designed by our +community of system administrators to cover the most important parts of production systems, so, in many cases, you won't +need to edit them. + +Luckily, Netdata's alarm and notification system are incredibly adaptable to your infrastructure's unique needs. + +## What you'll learn in this step + +We'll talk about Netdata's default configuration, and then you'll learn how to do the following: + +- [Tune Netdata's pre-configured alarms](#tune-netdatas-pre-configured-alarms) +- [Write your first health entity](#write-your-first-health-entity) +- [Enable Netdata's notification systems](#enable-netdatas-notification-systems) + +## Tune Netdata's pre-configured alarms + +First, let's tune an alarm that came pre-configured with your Netdata installation. + +The first chart you see on any Netdata dashboard is the `system.cpu` chart, which shows the system's CPU utilization +across all cores. To figure out which file you need to edit to tune this alarm, click the **Alarms** button at the top +of the dashboard, click on the **All** tab, and find the **system - cpu** alarm entity. + +![The system - cpu alarm +entity](https://user-images.githubusercontent.com/1153921/67034648-ebb4cc80-f0cc-11e9-9d49-1023629924f5.png) + +Look at the `source` row in the table. This means the `system.cpu` chart sources its health alarms from +`4@/usr/lib/netdata/conf.d/health.d/cpu.conf`. To tune these alarms, you'll need to edit the alarm file at +`health.d/cpu.conf`. Go to your [Netdata config directory](step-04.md#find-your-netdataconf-file) and use the +`edit-config` script. + +```bash +sudo ./edit-config health.d/cpu.conf +``` + +The first **health entity** in that file looks like this: + +```yaml +template: 10min_cpu_usage + on: system.cpu + os: linux + hosts: * + lookup: average -10m unaligned of user,system,softirq,irq,guest + units: % + every: 1m + warn: $this > (($status >= $WARNING) ? (75) : (85)) + crit: $this > (($status == $CRITICAL) ? (85) : (95)) + delay: down 15m multiplier 1.5 max 1h + info: average cpu utilization for the last 10 minutes (excluding iowait, nice and steal) + to: sysadmin +``` + +Let's say you want to tune this alarm to trigger warning and critical alarms at a lower CPU utilization. You can change +the `warn` and `crit` lines to the values of your choosing. For example: + +```yaml + warn: $this > (($status >= $WARNING) ? (60) : (75)) + crit: $this > (($status == $CRITICAL) ? (75) : (85)) +``` + +You _can_ restart Netdata with `sudo systemctl restart netdata`, to enable your tune, but you can also reload _only_ the +health monitoring component using one of the available [methods](/health/QUICKSTART.md#reload-health-configuration). + +You can also tune any other aspect of the default alarms. To better understand how each line in a health entity works, +read our [health documentation](/health/README.md). + +### Silence an individual alarm + +Many Netdata users don't need all the default alarms enabled. Instead of disabling any given alarm, or even _all_ +alarms, you can silence individual alarms by changing one line in a given health entity. Let's look at that +`health/cpu.conf` file again. + +```yaml +template: 10min_cpu_usage + on: system.cpu + os: linux + hosts: * + lookup: average -10m unaligned of user,system,softirq,irq,guest + units: % + every: 1m + warn: $this > (($status >= $WARNING) ? (75) : (85)) + crit: $this > (($status == $CRITICAL) ? (85) : (95)) + delay: down 15m multiplier 1.5 max 1h + info: average cpu utilization for the last 10 minutes (excluding iowait, nice and steal) + to: sysadmin +``` + +To silence this alarm, change `sysadmin` to `silent`. + +```yaml + to: silent +``` + +Use `netdatacli reload-health` to reload your health configuration. You can add `to: silent` to any alarm you'd rather not +bother you with notifications. + +## Write your first health entity + +The best way to understand how health entities work is building your own and experimenting with the options. To start, +let's build a health entity that triggers an alarm when system RAM usage goes above 80%. + +We will first create a new file inside of the `health.d/` directory. We'll name our file +`example.conf` for now. + +```bash +./edit-config health.d/example.conf +``` + +The first line in a health entity will be `alarm:`. This is how you name your entity. You can give it any name you +choose, but the only symbols allowed are `.` and `_`. Let's call the alarm `ram_usage`. + +```yaml + alarm: ram_usage +``` + +> You'll see some funky indentation in the lines coming up. Don't worry about it too much! Indentation is not important +> to how Netdata processes entities, and it will make sense when you're done. + +Next, you need to specify which chart this entity listens via the `on:` line. You're declaring that you want this alarm +to check metrics on the `system.ram` chart. + +```yaml + on: system.ram +``` + +Now comes the `lookup`. This line specifies what metrics the alarm is looking for, what duration of time it's looking +at, and how to process the metrics into a more usable format. + +```yaml +lookup: average -1m percentage of used +``` + +Let's take a moment to break this line down. + +- `average`: Calculate the average of all the metrics collected. +- `-1m`: Use metrics from 1 minute ago until now to calculate that average. +- `percentage`: Clarify that you want to calculate a percentage of RAM usage. +- `of used`: Specify which dimension (`used`) on the `system.ram` chart you want to monitor with this entity. + +In other words, you're taking 1 minute's worth of metrics from the `used` dimension on the `system.ram` chart, +calculating their average, and returning it as a percentage. + +You can move on to the `units` line, which lets Netdata know that we're working with a percentage and not an absolute +unit. + +```yaml + units: % +``` + +Next, the `every` line tells Netdata how often to perform the calculation you specified in the `lookup` line. For +certain alarms, you might want to use a shorter duration, which you can specify using values like `10s`. + +```yaml + every: 1m +``` + +We'll put the next two lines—`warn` and `crit`—together. In these lines, you declare at which percentage you want to +trigger a warning or critical alarm. Notice the variable `$this`, which is the value calculated by the `lookup` line. +These lines will trigger a warning if that average RAM usage goes above 80%, and a critical alert if it's above 90%. + +```yaml + warn: $this > 80 + crit: $this > 90 +``` + +> ❗ Most default Netdata alarms come with more complicated `warn` and `crit` lines. You may have noticed the line `warn: +> $this > (($status >= $WARNING) ? (75) : (85))` in one of the health entity examples above, which is an example of +> using the [conditional operator for hysteresis](/health/REFERENCE.md#special-use-of-the-conditional-operator). +> Hysteresis is used to keep Netdata from triggering a ton of alerts if the metric being tracked quickly goes above and +> then falls below the threshold. For this very simple example, we'll skip hysteresis, but recommend implementing it in +> your future health entities. + +Finish off with the `info` line, which creates a description of the alarm that will then appear in any +[notification](#enable-netdatas-notification-systems) you set up. This line is optional, but it has value—think of it as +documentation for a health entity! + +```yaml + info: The percentage of RAM being used by the system. +``` + +Here's what the entity looks like in full. Now you can see why we indented the lines, too. + +```yaml + alarm: ram_usage + on: system.ram +lookup: average -1m percentage of used + units: % + every: 1m + warn: $this > 80 + crit: $this > 90 + info: The percentage of RAM being used by the system. +``` + +What about what it looks like on the Netdata dashboard? + +![An active alert for the ram_usage alarm](https://user-images.githubusercontent.com/1153921/67056219-f89ee380-f0ff-11e9-8842-7dc210dd2908.png) + +If you'd like to try this alarm on your system, you can install a small program called +[stress](http://manpages.ubuntu.com/manpages/disco/en/man1/stress.1.html) to create a synthetic load. Use the command +below, and change the `8G` value to a number that's appropriate for the amount of RAM on your system. + +```bash +stress -m 1 --vm-bytes 8G --vm-keep +``` + +Netdata is capable of understanding much more complicated entities. To better understand how they work, read the [health +documentation](/health/README.md), look at some [examples](/health/REFERENCE.md#example-alarms), and open the files +containing the default entities on your system. + +## Enable Netdata's notification systems + +Health alarms, while great on their own, are pretty useless without some way of you knowing they've been triggered. +That's why Netdata comes with a notification system that supports more than a dozen services, such as email, Slack, +Discord, PagerDuty, Twilio, Amazon SNS, and much more. + +To see all the supported systems, visit our [notifications documentation](/health/notifications/README.md). + +We'll cover email and Slack notifications here, but with this knowledge you should be able to enable any other type of +notifications instead of or in addition to these. + +### Email notifications + +To use email notifications, you need `sendmail` or an equivalent installed on your system. Linux systems use `sendmail` +or similar programs to, unsurprisingly, send emails to any inbox. + +> Learn more about `sendmail` via its [documentation](http://www.postfix.org/sendmail.1.html). + +Edit the `health_alarm_notify.conf` file, which resides in your Netdata directory. + +```bash +sudo ./edit-config health_alarm_notify.conf +``` + +Look for the following lines: + +```conf +# if a role recipient is not configured, an email will be send to: +DEFAULT_RECIPIENT_EMAIL="root" +# to receive only critical alarms, set it to "root|critical" +``` + +Change the value of `DEFAULT_RECIPIENT_EMAIL` to the email address at which you'd like to receive notifications. + +```conf +# if a role recipient is not configured, an email will be sent to: +DEFAULT_RECIPIENT_EMAIL="me@example.com" +# to receive only critical alarms, set it to "root|critical" +``` + +Test email notifications system by first becoming the Netdata user and then asking Netdata to send a test alarm: + +```bash +sudo su -s /bin/bash netdata +/usr/libexec/netdata/plugins.d/alarm-notify.sh test +``` + +You should see output similar to this: + +```bash +# SENDING TEST WARNING ALARM TO ROLE: sysadmin +2019-10-17 18:23:38: alarm-notify.sh: INFO: sent email notification for: hostname test.chart.test_alarm is WARNING to 'me@example.com' +# OK + +# SENDING TEST CRITICAL ALARM TO ROLE: sysadmin +2019-10-17 18:23:38: alarm-notify.sh: INFO: sent email notification for: hostname test.chart.test_alarm is CRITICAL to 'me@example.com' +# OK + +# SENDING TEST CLEAR ALARM TO ROLE: sysadmin +2019-10-17 18:23:39: alarm-notify.sh: INFO: sent email notification for: hostname test.chart.test_alarm is CLEAR to 'me@example.com' +# OK +``` + +... and you should get three separate emails, one for each test alarm, in your inbox! (Be sure to check your spam +folder.) + +## Enable Slack notifications + +If you're one of the many who spend their workday getting pinged with GIFs by your colleagues, why not add Netdata +notifications to the mix? It's a great way to immediately see, collaborate around, and respond to anomalies in your +infrastructure. + +To get Slack notifications working, you first need to add an [incoming +webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) to the channel of your choice. Click the green **Add to +Slack** button, choose the channel, and click the **Add Incoming WebHooks Integration** button. + +On the following page, you'll receive a **Webhook URL**. That's what you'll need to configure Netdata, so keep it handy. + +Time to dive back into your `health_alarm_notify.conf` file: + +```bash +sudo ./edit-config health_alarm_notify.conf +``` + +Look for the `SLACK_WEBHOOK_URL=" "` line and add the incoming webhook URL you got from Slack: + +```conf +SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/XXXXXXXXXXXX" +``` + +A few lines down, edit the `DEFAULT_RECIPIENT_SLACK` line to contain a single hash `#` character. This instructs Netdata +to send a notification to the channel you configured with the incoming webhook. + +```conf +DEFAULT_RECIPIENT_SLACK="#" +``` + +Time to test the notifications again! + +```bash +sudo su -s /bin/bash netdata +/usr/libexec/netdata/plugins.d/alarm-notify.sh test +``` + +You should receive three notifications in your Slack channel. + +Congratulations! You're set up with two awesome ways to get notified about any change in the health of your systems or +applications. + +To further configure your email or Slack notification setup, or to enable other notification systems, check out the +following documentation: + +- [Email notifications](/health/notifications/email/README.md) +- [Slack notifications](/health/notifications/slack/README.md) +- [Netdata's notification system](/health/notifications/README.md) + +## What's next? + +In this step, you learned the fundamentals of Netdata's health monitoring tools: alarms and notifications. You should be +able to tune default alarms, silence them, and understand some of the basics of writing health entities. And, if you so +chose, you'll now have both email and Slack notifications enabled. + +You're coming along quick! + +Next up, we're going to cover how Netdata collects its metrics, and how you can get Netdata to collect real-time metrics +from hundreds of services with almost no configuration on your part. Onward! + +[Next: Collect metrics from more services and apps →](step-06.md) + + diff --git a/docs/guides/step-by-step/step-06.md b/docs/guides/step-by-step/step-06.md new file mode 100644 index 0000000..f04098f --- /dev/null +++ b/docs/guides/step-by-step/step-06.md @@ -0,0 +1,122 @@ + + +# Step 6. Collect metrics from more services and apps + +When Netdata _starts_, it auto-detects dozens of **data sources**, such as database servers, web servers, and more. + +To auto-detect and collect metrics from a source you just installed, you need to restart Netdata using `sudo systemctl +restart netdata`, or the [appropriate method](/docs/configure/start-stop-restart.md) for your system. + +However, auto-detection only works if you installed the source using its standard installation +procedure. If Netdata isn't collecting metrics after a restart, your source probably isn't configured +correctly. + +Check out the [collectors that come pre-installed with Netdata](/collectors/COLLECTORS.md) to find the module for the +source you want to monitor. + +## What you'll learn in this step + +We'll begin with an overview on Netdata's collector architecture, and then dive into the following: + +- [Netdata's collector architecture](#netdatas-collector-architecture) +- [Enable and disable plugins](#enable-and-disable-plugins) +- [Enable the Nginx collector as an example](#example-enable-the-nginx-collector) + +## Netdata's collector architecture + +Many Netdata users never have to configure collector or worry about which plugin orchestrator they want to use. + +But, if you want to configure collector or write a collector for your custom source, it's important to understand the +underlying architecture. + +By default, Netdata collects a lot of metrics every second using any number of discrete collector. Collectors, in turn, +are organized and manged by plugins. **Internal** plugins collect system metrics, **external** plugins collect +non-system metrics, and **orchestrator** plugins group individual collectors together based on the programming language +they were built in. + +These modules are primarily written in [Go](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/) (`go.d`) and +[Python](/collectors/python.d.plugin/README.md), although some use [Bash](/collectors/charts.d.plugin/README.md) +(`charts.d`). + +## Enable and disable plugins + +You don't need to explicitly enable plugins to auto-detect properly configured sources, but it's useful to know how to +enable or disable them. + +One reason you might want to _disable_ plugins is to improve Netdata's performance on low-resource systems, like +ephemeral nodes or edge devices. Disabling orchestrator plugins like `python.d` can save significant resources if you're +not using any of its data collector modules. + +You can enable or disable plugins in the `[plugin]` section of `netdata.conf`. This section features a list of all the +plugins with a boolean setting (`yes` or `no`) to enable or disable them. Be sure to uncomment the line by removing the +hash (`#`)! + +Enabled: + +```conf +[plugins] + # python.d = yes +``` + +Disabled: + +```conf +[plugins] + python.d = no +``` + +When you explicitly disable a plugin this way, it won't auto-collect metrics using its collectors. + +## Example: Enable the Nginx collector + +To help explain how the auto-detection process works, let's use an Nginx web server as an example. + +Even if you don't have Nginx installed on your system, we recommend you read through the following section so you can +apply the process to other data sources, such as Apache, Redis, Memcached, and more. + +The Nginx collector, which helps Netdata collect metrics from a running Nginx web server, is part of the +`python.d.plugin` external plugin _orchestrator_. + +In order for Netdata to auto-detect an Nginx web server, you need to enable `ngx_http_stub_status_module` and pass the +`stub_status` directive in the `location` block of your Nginx configuration file. + +You can confirm if the `stub_status` Nginx module is already enabled or not by using following command: + +```sh +nginx -V 2>&1 | grep -o with-http_stub_status_module +``` + +If this command returns nothing, you'll need to [enable this module](https://www.nginx.com/blog/monitoring-nginx/). + +Next, edit your `/etc/nginx/sites-enabled/default` file to include a `location` block with the following: + +```conf + location /stub_status { + stub_status; + } +``` + +Restart Netdata using `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, and Netdata will auto-detect metrics from your Nginx web +server! + +While not necessary for most auto-detection and collection purposes, you can also configure the Nginx collector itself +by editing its configuration file: + +```sh +./edit-config python.d/nginx.conf +``` + +After configuring any source, or changing the configuration files for their respective modules, always restart Netdata. + +## What's next? + +Now that you've learned the fundamentals behind configuring data sources for auto-detection, it's time to move back to +the dashboard to learn more about some of its more advanced features. + +[Next: Netdata's dashboard in depth →](step-07.md) + + diff --git a/docs/guides/step-by-step/step-07.md b/docs/guides/step-by-step/step-07.md new file mode 100644 index 0000000..17a02cd --- /dev/null +++ b/docs/guides/step-by-step/step-07.md @@ -0,0 +1,114 @@ + + +# Step 7. Netdata's dashboard in depth + +Welcome to the seventh step of the Netdata guide! + +This step of the guide aims to get you more familiar with the features of the dashboard not previously mentioned in +[step 2](/docs/guides/step-by-step/step-02.md). + +## What you'll learn in this step + +In this step of the Netdata guide, you'll learn how to: + +- [Change the dashboard's settings](#change-the-dashboards-settings) +- [Check if there's an update to Netdata](#check-if-theres-an-update-to-netdata) +- [Export and import a snapshot](#export-and-import-a-snapshot) + +Let's get started! + +## Change the dashboard's settings + +The settings area at the top of your Netdata dashboard houses browser settings. These settings do not affect the +operation of your Netdata server/daemon. They take effect immediately and are permanently saved to browser local storage +(except the refresh on focus / always option). + +You can see the **Performance**, **Synchronization**, **Visual**, and **Locale** tabs on the dashboard settings modal. + +![Animated GIF of opening the settings +modal](https://user-images.githubusercontent.com/1153921/80841197-c93f5800-8bb3-11ea-907d-85bfe23565e1.gif) + +To change any setting, click on the toggle button. We recommend you spend some time reading the descriptions for each setting to understand them before making changes. + +Pay particular attention to the following settings, as they have dramatic impacts on the performance and appearance of +your Netdata dashboard: + +- When to refresh the charts? +- How to handle hidden charts? +- Which chart refresh policy to use? +- Which theme to use? +- Do you need help? + +Some settings are applied immediately, and others are only reflected after you refresh the page. + +## Check if there's an update to Netdata + +You can always check if there is an update available from the **Update** area of your Netdata dashboard. + +![Opening the Agent's Update modal](https://user-images.githubusercontent.com/1153921/80829493-1adbe880-8b9c-11ea-9770-cc3b23a89414.gif) + +If an update is available, you'll see a modal similar to the one above. + +When you use the [automatic one-line installer script](/packaging/installer/README.md) attempt to update every day. If +you choose to update it manually, there are [several well-documented methods](/packaging/installer/UPDATE.md) to achieve +that. However, it is best practice for you to first go over the [changelog](/CHANGELOG.md). + +## Export and import a snapshot + +Netdata can export and import snapshots of the contents of your dashboard at a given time. Any Netdata agent can import +a snapshot created by any other Netdata agent. + +Snapshot files include all the information of the dashboard, including the URL of the origin server, its unique ID, and +chart data queries for the visible timeframe. While snapshots are not in real-time, and thus won't update with new +metrics, you can still pan, zoom, and highlight charts as you see fit. + +Snapshots can be incredibly useful for diagnosing anomalies after they've already happened. Let's say Netdata triggered +an alarm while you were sleeping. In the morning, you can look up the exact moment the alarm was raised, export a +snapshot, and send it to a colleague for further analysis. + +> ❗ Know how you shouldn't go around downloading software from suspicious-looking websites? Same policy goes for loading +> snapshots from untrusted or anonymous sources. Importing a snapshot loads quite a bit of data into your web browser, +> and so you should always err on the side of protecting your system. + +To export a snapshot, click on the **export** icon. + +![Animated GIF of opening the export +modal](https://user-images.githubusercontent.com/1153921/80993197-82d63d00-8def-11ea-88fa-98827814e930.gif) + +Edit the snapshot file name and select your desired compression method. Click on **Export**. + +When the export is complete, your browser will prompt you to save the `.snapshot` file to your machine. You can now +share this file with any other Netdata user via email, Slack, or even to help describe your Netdata experience when +[filing an issue](https://github.com/netdata/netdata/issues/new/choose) on GitHub. + +To import a snapshot, click on the **import** icon. + +![Animated GIF of opening the import +modal](https://user-images.githubusercontent.com/12263278/64901503-ee696f80-d691-11e9-9678-8d0e2a162402.gif) + +Select the Netdata snapshot file to import. Once the file is loaded, the dashboard will update with critical information +about the snapshot and the system from which it was taken. Click **import** to render it. + +Your Netdata dashboard will load data contained in the snapshot into charts. Because the snapshot only covers a certain +period, it won't update with new metrics. + +An imported snapshot is also temporary. If you reload your browser tab, Netdata will remove the snapshot data and +restore your real-time dashboard for your machine. + +## What's next? + +In this step of the Netdata guide, you learned how to: + +- Change the dashboard's settings +- Check if there's an update to Netdata +- Export or import a snapshot + +Next, you'll learn how to build your first custom dashboard! + +[Next: Build your first custom dashboard →](step-08.md) + + diff --git a/docs/guides/step-by-step/step-08.md b/docs/guides/step-by-step/step-08.md new file mode 100644 index 0000000..e9c0f90 --- /dev/null +++ b/docs/guides/step-by-step/step-08.md @@ -0,0 +1,395 @@ + + +# Step 8. Build your first custom dashboard + +In previous steps of the guide, you have learned how several sections of the Netdata dashboard worked. + +This step will show you how to set up a custom dashboard to fit your unique needs. If nothing else, Netdata is really, +really flexible. 🤸 + +## What you'll learn in this step + +In this step of the Netdata guide, you'll learn: + +- [Why you might want a custom dashboard](#why-should-i-create-a-custom-dashboard) +- [How to create and prepare your `custom-dashboard.html` file](#create-and-prepare-your-custom-dashboardhtml-file) +- [Where to add `dashboard.js` to your custom dashboard file](#add-dashboardjs-to-your-custom-dashboard-file) +- [How to add basic styling](#add-some-basic-styling) +- [How to add charts of different types, shapes, and sizes](#creating-your-dashboards-charts) + +Let's get on with it! + +## Why should I create a custom dashboard? + +Because it's cool! + +But there are way more reasons than that, most of which will prove more valuable to you. + +You could use custom dashboards to aggregate real-time data from multiple Netdata agents in one place. Or, you could put +all the charts with metrics collected from your custom application via `statsd` and perform application performance +monitoring from a single dashboard. You could even use a custom dashboard and a standalone web server to create an +enriched public status page for your service, and give your users something fun to look at while they're waiting for the +503 errors to clear up! + +Netdata's custom dashboarding capability is meant to be as flexible as your ideas. We hope you can take these +fundamental ideas and turn them into something amazing. + +## Create and prepare your `custom-dashboard.html` file + +By default, Netdata stores its web server files at `/usr/share/netdata/web`. As with finding the location of your +`netdata.conf` file, you can double-check this location by loading up `http://HOST:19999/netdata.conf` in your browser +and finding the value of the `web files directory` option. + +To create your custom dashboard, create a file at `/usr/share/netdata/web/custom-dashboard.html` and copy in the +following: + +```html + + + + My custom dashboard + + + + + + + + + + + + + +
+ +

My custom dashboard

+ + + +
+ + + +``` + +Try visiting `http://HOST:19999/custom-dashboard.html` in your browser. + +If you get a blank page with this text: `Access to file is not permitted: /usr/share/netdata/web/custom-dashboard.html`. +You can fix this error by changing the dashboard file's permissions to make it owned by the `netdata` user. + +```bash +sudo chown netdata:netdata /usr/share/netdata/web/custom-dashboard.html +``` + +Reload your browser, and you should see a blank page with the title: **Your custom dashboard**! + +## Add `dashboard.js` to your custom dashboard file + +You need to include the `dashboard.js` file of a Netdata agent to add Netdata charts. Add the following to the `` +of your custom dashboard page and change `HOST` according to your setup. + +```html + + +``` + +When you add `dashboard.js` to any web page, it loads several JavaScript and CSS files to create and style charts. It +also scans the page for elements that define charts, builds them, and refreshes with new metrics. + +> If you enabled SSL on your Netdata dashboard already, you'll need to use `https://` to grab the `dashboard.js` file. + +## Add some basic styling + +While not necessary, let's add some basic styling to make our dashboard look a little nicer. We're putting some +basic CSS into a ` + + +``` + +## Creating your dashboard's charts + +Time to create a chart! + +You need to create a `
` for each new chart. Each `
` element accepts a few `data-` attributes, some of which +are required and some of which are optional. + +Let's cover a few important ones. And while we do it, we'll create a custom dashboard that shows a few CPU-related +charts on a single page. + +### The chart unique ID (required) + +You need to specify the unique ID of a chart to show it on your custom dashboard. If you forgot how to find the unique +ID, head back over to [step 2](/docs/guides/step-by-step/step-02.md#understand-charts-dimensions-families-and-contexts) +for a re-introduction. + +You can then put this unique ID into a `
` element with the `data-netdata` attribute. Put this in the `` of +your custom dashboard file beneath the helpful comment. + +```html + + +
+ +

My custom dashboard

+ +
+ + +
+ +
+ +
+ + +``` + +Reload the page, and you should see a real-time `system.cpu` chart! + +... and a whole lot of white space. Let's fix that by adding a few more charts. + +```html + +
+
+
+
+``` + +![Custom dashboard with four charts +added](https://user-images.githubusercontent.com/1153921/67526566-e675f580-f669-11e9-8ff5-d1f21a84fb2b.png) + +### Set chart duration + +By default, these charts visualize 10 minutes of Netdata metrics. Let's get a little more granular on this dashboard. To +do so, add a new `data-after=""` attribute to each chart. + +`data-after` takes a _relative_ number of seconds from _now_. So, by putting `-300` as the value, you're asking the +custom dashboard to display the _last 5 minutes_ (`5m * 60s = 300s`) of data. + +```html + +
+
+
+
+
+
+
+
+``` + +### Set chart size + +You can set the size of any chart using the `data-height=""` and `data-width=""` attributes. These attributes can be +anything CSS accepts for width and height (e.g. percentages, pixels, em/rem, calc, and so on). + +Let's make the charts a little taller and allow them to fit side-by-side for a more compact view. Add +`data-height="200px"` and `data-width="50%"` to each chart. + +```html +
+
+
+
+``` + +Now we're getting somewhere! + +![A custom dashboard with four charts +side-by-side](https://user-images.githubusercontent.com/1153921/67526620-ff7ea680-f669-11e9-92d3-575665fc3a8e.png) + +## Final touches + +While we already have a perfectly workable dashboard, let's add some final touches to make it a little more pleasant on +the eyes. + +First, add some extra CSS to create some vertical whitespace between the top and bottom row of charts. + +```html + +``` + +To create horizontal whitespace, change the value of `data-width="50%"` to `data-width="calc(50% - 2rem)"`. + +```html +
+
+
+
+``` + +Told you the `data-width` and `data-height` attributes can take any CSS values! + +Prefer a dark theme? Add this to your `` _above_ where you added `dashboard.js`: + +```html + + + + +``` + +Refresh the dashboard to give your eyes a break from all that blue light! + +![A finished custom +dashboard](https://user-images.githubusercontent.com/1153921/67531221-a23d2200-f676-11e9-91fe-c2cf1c426bf9.png) + +## The final `custom-dashboard.html` + +In case you got lost along the way, here's the final version of the `custom-dashboard.html` file: + +```html + + + + My custom dashboard + + + + + + + + + + + + + + + + + + +
+ +

My custom dashboard

+ +
+ + +
+
+
+
+ +
+ +
+ + + +``` + +## What's next? + +In this guide, you learned the fundamentals of building a custom Netdata dashboard. You should now be able to add more +charts to your `custom-dashboard.html`, change the charts that are already there, and size them according to your needs. + +Of course, the custom dashboarding features covered here are just the beginning. Be sure to read up on our [custom +dashboard documentation](/web/gui/custom/README.md) for details on how you can use other chart libraries, pull metrics +from multiple Netdata agents, and choose which dimensions a given chart shows. + +Next, you'll learn how to store long-term historical metrics in Netdata! + +[Next: Long-term metrics storage →](/docs/guides/step-by-step/step-09.md) + + diff --git a/docs/guides/step-by-step/step-09.md b/docs/guides/step-by-step/step-09.md new file mode 100644 index 0000000..8aacd75 --- /dev/null +++ b/docs/guides/step-by-step/step-09.md @@ -0,0 +1,162 @@ + + +# Step 9. Long-term metrics storage + +By default, Netdata stores metrics in a custom database we call the [database engine](/database/engine/README.md), which +stores recent metrics in your system's RAM and "spills" historical metrics to disk. By using both RAM and disk, the +database engine helps you store a much larger dataset than the amount of RAM your system has. + +On a system that's collecting 2,000 metrics every second, the database engine's default configuration will store about +two day's worth of metrics in RAM and on disk. + +That's a lot of metrics. We're talking 345,600,000 individual data points. And the database engine does it with a tiny +a portion of the RAM available on most systems. + +To store _even more_ metrics, you have two options. First, you can tweak the database engine's options to expand the RAM +or disk it uses. Second, you can archive metrics to an external database. For that, we'll use MongoDB as examples. + +## What you'll learn in this step + +In this step of the Netdata guide, you'll learn how to: + +- [Tweak the database engine's settings](#tweak-the-database-engines-settings) +- [Archive metrics to an external database](#archive-metrics-to-an-external-database) + - [Use the MongoDB database](#archive-metrics-via-the-mongodb-exporting-connector) + +Let's get started! + +## Tweak the database engine's settings + +If you're using Netdata v1.18.0 or higher, and you haven't changed your `memory mode` settings before following this +guide, your Netdata agent is already using the database engine. + +Let's look at your `netdata.conf` file again. Under the `[global]` section, you'll find three connected options. + +```conf +[db] + # mode = dbengine + # dbengine page cache size MB = 32 + # dbengine disk space MB = 256 +``` + +The `memory mode` option is set, by default, to `dbengine`. `page cache size` determines the amount of RAM, in MiB, that +the database engine dedicates to caching the metrics it's collecting. `dbengine disk space` determines the amount of +disk space, in MiB, that the database engine will use to store these metrics once they've been "spilled" to disk.. + +You can uncomment and change either `page cache size` or `dbengine disk space` based on how much RAM and disk you want +the database engine to use. The higher those values, the more metrics Netdata will store. If you change them to 64 and +512, respectively, the database engine should store about four day's worth of data on a system collecting 2,000 metrics +every second. + +[**See our database engine calculator**](/docs/store/change-metrics-storage.md) to help you correctly set `dbengine disk +space` based on your needs. The calculator gives an accurate estimate based on how many child nodes you have, how many +metrics your Agent collects, and more. + +```conf +[db] + mode = dbengine + dbengine page cache size MB = 64 + dbengine disk space MB = 512 +``` + +After you've made your changes, restart Netdata using `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +To confirm the database engine is working, go to your Netdata dashboard and click on the **Netdata Monitoring** menu on +the right-hand side. You can find `dbengine` metrics after `queries`. + +![Image of the database engine reflected in the Netdata +Dashboard](https://user-images.githubusercontent.com/12263278/64781383-9c71fe00-d55a-11e9-962b-efd5558efbae.png) + +## Archive metrics to an external database + +You can archive all the metrics collected by Netdata to **external databases**. The supported databases and services +include Graphite, OpenTSDB, Prometheus, AWS Kinesis Data Streams, Google Cloud Pub/Sub, MongoDB, and the list is always +growing. + +As we said in [step 1](/docs/guides/step-by-step/step-01.md), we have only complimentary systems, not competitors! We're +happy to support these archiving methods and are always working to improve them. + +A lot of Netdata users archive their metrics to one of these databases for long-term storage or further analysis. Since +Netdata collects so many metrics every second, they can quickly overload small devices or even big servers that are +aggregating metrics streaming in from other Netdata agents. + +We even support resampling metrics during archiving. With resampling enabled, Netdata will archive only the average or +sum of every X seconds of metrics. This reduces the sheer amount of data, albeit with a little less accuracy. + +How you archive metrics, or if you archive metrics at all, is entirely up to you! But let's cover two easy archiving +methods, MongoDB and Prometheus remote write, to get you started. + +### Archive metrics via the MongoDB exporting connector + +Begin by installing MongoDB its dependencies via the correct package manager for your system. + +```bash +sudo apt-get install mongodb # Debian/Ubuntu +sudo dnf install mongodb # Fedora +sudo yum install mongodb # CentOS +``` + +Next, install the one essential dependency: v1.7.0 or higher of +[libmongoc](http://mongoc.org/libmongoc/current/installing.html). + +```bash +sudo apt-get install libmongoc-1.0-0 libmongoc-dev # Debian/Ubuntu +sudo dnf install mongo-c-driver mongo-c-driver-devel # Fedora +sudo yum install mongo-c-driver mongo-c-driver-devel # CentOS +``` + +Next, create a new MongoDB database and collection to store all these archived metrics. Use the `mongo` command to start +the MongoDB shell, and then execute the following command: + +```mongodb +use netdata +db.createCollection("netdata_metrics") +``` + +Next, Netdata needs to be [reinstalled](/packaging/installer/REINSTALL.md) in order to detect that the required +libraries to make this exporting connection exist. Since you most likely installed Netdata using the one-line installer +script, all you have to do is run that script again. Don't worry—any configuration changes you made along the way will +be retained! + +Now, from your Netdata config directory, initialize and edit a `exporting.conf` file to tell Netdata where to find the +database you just created. + +```sh +./edit-config exporting.conf +``` + +Add the following section to the file: + +```conf +[mongodb:my_mongo_instance] + enabled = yes + destination = mongodb://localhost + database = netdata + collection = netdata_metrics +``` + +Restart Netdata using `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to enable the MongoDB exporting connector. Click on the +**Netdata Monitoring** menu and check out the **exporting my mongo instance** sub-menu. You should start seeing these +charts fill up with data about the exporting process! + +![image](https://user-images.githubusercontent.com/1153921/70443852-25171200-1a56-11ea-8be3-494544b1c295.png) + +If you'd like to try connecting Netdata to another database, such as Prometheus or OpenTSDB, read our [exporting +documentation](/exporting/README.md). + +## What's next? + +You're getting close to the end! In this step, you learned how to make the most of the database engine, or archive +metrics to MongoDB for long-term storage. + +In the last step of this step-by-step guide, we'll put our sysadmin hat on and use Nginx to proxy traffic to and from +our Netdata dashboard. + +[Next: Set up a proxy →](/docs/guides/step-by-step/step-10.md) + + diff --git a/docs/guides/step-by-step/step-10.md b/docs/guides/step-by-step/step-10.md new file mode 100644 index 0000000..c9acf5a --- /dev/null +++ b/docs/guides/step-by-step/step-10.md @@ -0,0 +1,232 @@ + + +# Step 10. Set up a proxy + +You're almost through! At this point, you should be pretty familiar with now Netdata works and how to configure it to +your liking. + +In this step of the guide, we're going to add a proxy in front of Netdata. We're doing this for both improved +performance and security, so we highly recommend following these steps. Doubly so if you installed Netdata on a +publicly-accessible remote server. + +> ❗ If you installed Netdata on the machine you're currently using (e.g. on `localhost`), and have been accessing +> Netdata at `http://localhost:19999`, you can skip this step of the guide. In most cases, there is no benefit to +> setting up a proxy for a service running locally. + +> ❗❗ This guide requires more advanced administration skills than previous parts. If you're still working on your +> Linux administration skills, and would rather get back to Netdata, you might want to [skip this +> step](step-99.md) for now and return to it later. + +## What you'll learn in this step + +In this step of the Netdata guide, you'll learn: + +- [What a proxy is and the benefits of using one](#wait-whats-a-proxy) +- [How to connect Netdata to Nginx](#connect-netdata-to-nginx) +- [How to enable HTTPS in Nginx](#enable-https-in-nginx) +- [How to secure your Netdata dashboard with a password](#secure-your-netdata-dashboard-with-a-password) + +Let's dive in! + +## Wait. What's a proxy? + +A proxy is a middleman between the internet and a service you're running on your system. Traffic from the internet at +large enters your system through the proxy, which then routes it to the service. + +A proxy is often used to enable encrypted HTTPS connections with your browser, but they're also useful for load +balancing, performance, and password-protection. + +We'll use [Nginx](https://nginx.org/en/) for this step of the guide, but you can also use +[Caddy](https://caddyserver.com/) as a simple proxy if you prefer. + +## Required before you start + +You need three things to run a proxy using Nginx: + +- Nginx and Certbot installed on your system +- A fully qualified domain name +- A subdomain for Netdata that points to your system + +### Nginx and Certbot + +This step of the guide assumes you can install Nginx on your system. Here are the easiest methods to do so on Debian, +Ubuntu, Fedora, and CentOS systems. + +```bash +sudo apt-get install nginx # Debian/Ubuntu +sudo dnf install nginx # Fedora +sudo yum install nginx # CentOS +``` + +Check out [Nginx's installation +instructions](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/) for details on +other Linux distributions. + +Certbot is a tool to help you create and renew certificate+key pairs for your domain. Visit their +[instructions](https://certbot.eff.org/instructions) to get a detailed installation process for your operating system. + +### Fully qualified domain name + +The only other true prerequisite of using a proxy is a **fully qualified domain name** (FQDN). In other words, a domain +name like `example.com`, `netdata.cloud`, or `github.com`. + +If you don't have a domain name, you won't be able to use a proxy the way we'll describe here. + +Because we strongly recommend running Netdata behind a proxy, the cost of a domain name is worth the benefit. If you +don't have a preferred domain registrar, try [Google Domains](https://domains.google/), +[Cloudflare](https://www.cloudflare.com/products/registrar/), or [Namecheap](https://www.namecheap.com/). + +### Subdomain for Netdata + +Any of the three domain registrars mentioned above, and most registrars in general, will allow you to create new DNS +entries for your domain. + +To create a subdomain for Netdata, use your registrar's DNS settings to create an A record for a `netdata` subdomain. +Point the A record to the IP address of your system. + +Once finished with the steps below, you'll be able to access your dashboard at `http://netdata.example.com`. + +## Connect Netdata to Nginx + +The first part of enabling the proxy is to create a new server for Nginx. + +Use your favorite text editor to create a file at `/etc/nginx/sites-available/netdata`, copy in the following +configuration, and change the `server_name` line to match your domain. + +```nginx +upstream backend { + server 127.0.0.1:19999; + keepalive 64; +} + +server { + listen 80; + # uncomment the line if you want nginx to listen on IPv6 address + #listen [::]:80; + + # Change `example.com` to match your domain name. + server_name netdata.example.com; + + location / { + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_pass http://backend; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + } +} +``` + +Save and close the file. + +Test your configuration file by running `sudo nginx -t`. + +If that returns no errors, it's time to make your server available. Run the command to create a symbolic link in the +`sites-enabled` directory. + +```bash +sudo ln -s /etc/nginx/sites-available/netdata /etc/nginx/sites-enabled/netdata +``` + +Finally, restart Nginx to make your changes live. Open your browser and head to `http://netdata.example.com`. You should +see your proxied Netdata dashboard! + +## Enable HTTPS in Nginx + +All this proxying doesn't mean much if we can't take advantage of one of the biggest benefits: encrypted HTTPS +connections! Let's fix that. + +Certbot will automatically get a certificate, edit your Nginx configuration, and get HTTPS running in a single step. Run +the following: + +```bash +sudo certbot --nginx +``` + +> See this error after running `sudo certbot --nginx`? +> +> ``` +> Saving debug log to /var/log/letsencrypt/letsencrypt.log +> The requested nginx plugin does not appear to be installed` +> ``` +> +> You must install `python-certbot-nginx`. On Ubuntu or Debian systems, you can run `sudo apt-get install +> python-certbot-nginx` to download and install this package. + +You'll be prompted with a few questions. At the `Which names would you like to activate HTTPS for?` question, hit +`Enter`. Next comes this question: + +```bash +Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access. +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +1: No redirect - Make no further changes to the webserver configuration. +2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for +new sites, or if you're confident your site works on HTTPS. You can undo this +change by editing your web server's configuration. +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +``` + +You _do_ want to force HTTPS, so hit `2` and then `Enter`. Nginx will now ensure all attempts to access +`netdata.example.com` use HTTPS. + +Certbot will automatically renew your certificate whenever it's needed, so you're done configuring your proxy. Open your +browser again and navigate to `https://netdata.example.com`, and you'll land on an encrypted, proxied Netdata dashboard! + +## Secure your Netdata dashboard with a password + +Finally, let's take a moment to put your Netdata dashboard behind a password. This step is optional, but you might not +want _anyone_ to access the metrics in your proxied dashboard. + +Run the below command after changing `user` to the username you want to use to log in to your dashboard. + +```bash +sudo sh -c "echo -n 'user:' >> /etc/nginx/.htpasswd" +``` + +Then run this command to create a password: + +```bash +sudo sh -c "openssl passwd -apr1 >> /etc/nginx/.htpasswd" +``` + +You'll be prompted to create a password. Next, open your Nginx configuration file at +`/etc/nginx/sites-available/netdata` and add these two lines under `location / {`: + +```nginx + location / { + auth_basic "Restricted Content"; + auth_basic_user_file /etc/nginx/.htpasswd; + ... +``` + +Save, exit, and restart Nginx. Then try visiting your dashboard one last time. You'll see a prompt for the username and +password you just created. + +![Username/password +prompt](https://user-images.githubusercontent.com/1153921/67431031-5320bf80-f598-11e9-9573-f9f9912f1ef6.png) + +Your Netdata dashboard is now a touch more secure. + +## What's next? + +You're a real sysadmin now! + +If you want to configure your Nginx proxy further, check out the following: + +- [Running Netdata behind Nginx](/docs/Running-behind-nginx.md) +- [How to optimize Netdata's performance](/docs/guides/configure/performance.md) +- [Enabling TLS on Netdata's dashboard](/web/server/README.md#enabling-tls-support) + +And... you're _almost_ done with the Netdata guide. + +For some celebratory emoji and a clap on the back, head on over to our final step. + +[Next: The end. →](step-99.md) + + diff --git a/docs/guides/step-by-step/step-99.md b/docs/guides/step-by-step/step-99.md new file mode 100644 index 0000000..58902fe --- /dev/null +++ b/docs/guides/step-by-step/step-99.md @@ -0,0 +1,51 @@ + + +# Step ∞. You're finished! + +Congratulations. 🎉 + +You've completed the step-by-step Netdata guide. That means you're well on your way to becoming an expert in using +our toolkit for health monitoring and performance troubleshooting. + +But, perhaps more importantly, also that much closer to being an expert in the _fundamental skills behind health +monitoring and performance troubleshooting_, which you can take with you to any job or project. + +And that is the entire point of this guide, and Netdata's [documentation](https://learn.netdata.cloud) as a +whole—give you every resource possible to help you build faster, more resilient systems, services, and applications. + +Along the way, you learned how to: + +- Navigate Netdata's dashboard and visually detect anomalies using its charts. +- Monitor multiple systems using Netdata agents connected together with your browser and Netdata Cloud. +- Edit your `netdata.conf` file to tweak Netdata to your liking. +- Tune existing alarms and create entirely new ones, plus get notifications about alarms on your favorite services. +- Take advantage of Netdata's auto-detection capabilities to ensure your applications/services are monitored with + little to no configuration. +- Use advanced features within Netdata's dashboard. +- Build a custom dashboard using `dashboard.js`. +- Save more historical metrics with the database engine or archive metrics to MongoDB. +- Put Netdata behind a proxy to enable HTTPS and improve performance. + +Seems like a lot, right? Well, we hope it felt manageable and, yes, even _fun_. + +## What's next? + +Now that you're at the end of our step-by-step Netdata guide, the next steps are entirely up to you. In fact, you're +just at the beginning of your journey into health monitoring and performance troubleshooting. + +Our documentation exists to put every Netdata resource in front of you as easily and coherently as we possibly can. +Click around, search, and find new mountains to climb. + +If that feels like too much possibility to you, why not one of these options: + +- Share your experience with Netdata and this guide. Be sure to [@mention](https://twitter.com/linuxnetdata) us on + Twitter! +- Contribute to what we do. Browse our [open issues](https://github.com/netdata/netdata/issues) and check out out + [contributions doc](https://learn.netdata.cloud/contribute/) for ideas of how you can pitch in. + +We can't wait to see what you monitor next! Bon voyage! ⛵ + + diff --git a/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md b/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md new file mode 100644 index 0000000..3ebca54 --- /dev/null +++ b/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md @@ -0,0 +1,270 @@ + + +# Monitor, troubleshoot, and debug applications with eBPF metrics + +When trying to troubleshoot or debug a finicky application, there's no such thing as too much information. At Netdata, +we developed programs that connect to the [_extended Berkeley Packet Filter_ (eBPF) virtual +machine](/collectors/ebpf.plugin/README.md) to help you see exactly how specific applications are interacting with the +Linux kernel. With these charts, you can root out bugs, discover optimizations, diagnose memory leaks, and much more. + +This means you can see exactly how often, and in what volume, the application creates processes, opens files, writes to +filesystem using virtual filesystem (VFS) functions, and much more. Even better, the eBPF collector gathers metrics at +an _event frequency_, which is even faster than Netdata's beloved 1-second granularity. When you troubleshoot and debug +applications with eBPF, rest assured you miss not even the smallest meaningful event. + +Using this guide, you'll learn the fundamentals of setting up Netdata to give you kernel-level metrics from your +application so that you can monitor, troubleshoot, and debug to your heart's content. + +## Configure `apps.plugin` to recognize your custom application + +To start troubleshooting an application with eBPF metrics, you need to ensure your Netdata dashboard collects and +displays those metrics independent from any other process. + +You can use the `apps_groups.conf` file to configure which applications appear in charts generated by +[`apps.plugin`](/collectors/apps.plugin/README.md). Once you edit this file and create a new group for the application +you want to monitor, you can see how it's interacting with the Linux kernel via real-time eBPF metrics. + +Let's assume you have an application that runs on the process `custom-app`. To monitor eBPF metrics for that application +separate from any others, you need to create a new group in `apps_groups.conf` and associate that process name with it. + +Open the `apps_groups.conf` file in your Netdata configuration directory. + +```bash +cd /etc/netdata # Replace this path with your Netdata config directory +sudo ./edit-config apps_groups.conf +``` + +Scroll down past the explanatory comments and stop when you see `# NETDATA processes accounting`. Above that, paste in +the following text, which creates a new `dev` group with the `custom-app` process. Replace `custom-app` with the name of +your application's process name. + +Your file should now look like this: + +```conf +... +# ----------------------------------------------------------------------------- +# Custom applications to monitor with apps.plugin and ebpf.plugin + +dev: custom-app + +# ----------------------------------------------------------------------------- +# NETDATA processes accounting +... +``` + +Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to begin seeing metrics for this particular +group+process. You can also add additional processes to the same group. + +You can set up `apps_groups.conf` to more show more precise eBPF metrics for any application or service running on your +system, even if it's a standard package like Redis, Apache, or any other [application/service Netdata collects +from](/collectors/COLLECTORS.md). + +```conf +# ----------------------------------------------------------------------------- +# Custom applications to monitor with apps.plugin and ebpf.plugin + +dev: custom-app +database: *redis* +apache: *apache* + +# ----------------------------------------------------------------------------- +# NETDATA processes accounting +... +``` + +Now that you have `apps_groups.conf` set up to monitor your application/service, you can also set up the eBPF collector +to show other charts that will help you debug and troubleshoot how it interacts with the Linux kernel. + +## Configure the eBPF collector to monitor errors + +The eBPF collector has [two possible modes](/collectors/ebpf.plugin#ebpf-load-mode): `entry` and `return`. The default +is `entry`, and only monitors calls to kernel functions, but the `return` also monitors and charts _whether these calls +return in error_. + +Let's turn on the `return` mode for more granularity when debugging Firefox's behavior. + +```bash +cd /etc/netdata # Replace this path with your Netdata config directory +sudo ./edit-config ebpf.d.conf +``` + +Replace `entry` with `return`: + +```conf +[global] + ebpf load mode = return + disable apps = no + +[ebpf programs] + process = yes + network viewer = yes +``` + +Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +## Get familiar with per-application eBPF metrics and charts + +Visit the Netdata dashboard at `http://NODE:19999`, replacing `NODE` with the hostname or IP of the system you're using +to monitor this application. Scroll down to the **Applications** section. These charts now feature a `firefox` dimension +with metrics specific to that process. + +Pay particular attention to the charts in the **ebpf file**, **ebpf syscall**, **ebpf process**, and **ebpf net** +sub-sections. These charts are populated by low-level Linux kernel metrics thanks to eBPF, and showcase the volume of +calls to open/close files, call functions like `do_fork`, IO activity on the VFS, and much more. + +See the [eBPF collector documentation](/collectors/ebpf.plugin/README.md#integration-with-appsplugin) for the full list +of per-application charts. + +Let's show some examples of how you can first identify normal eBPF patterns, then use that knowledge to identify +anomalies in a few simulated scenarios. + +For example, the following screenshot shows the number of open files, failures to open files, and closed files on a +Debian 10 system. The first spike is from configuring/compiling a small C program, then from running Apache's `ab` tool +to benchmark an Apache web server. + +![An example of eBPF +charts](https://user-images.githubusercontent.com/1153921/85311677-a8380c80-b46a-11ea-9735-babaedc22fdb.png) + +In these charts, you can see first a spike in syscalls to open and close files from the configure/build process, +followed by a similar spike from the Apache benchmark. + +> 👋 Don't forget that you can view chart data directly via Netdata's API! +> +> For example, open your browser and navigate to `http://NODE:19999/api/v1/data?chart=apps.file_open`, replacing `NODE` +> with the IP address or hostname of your Agent. The API returns JSON of that chart's dimensions and metrics, which you +> can use in other operations. +> +> To see other charts, replace `apps.file_open` with the context of the chart you want to see data for. +> +> To see all the API options, visit our [Swagger +> documentation](https://editor.swagger.io/?url=https://raw.githubusercontent.com/netdata/netdata/master/web/api/netdata-swagger.yaml) +> and look under the **/data** section. + +## Troubleshoot and debug applications with eBPF + +The actual method of troubleshooting and debugging any application with Netdata's eBPF metrics depends on the +application, its place within your stack, and the type of issue you're trying to root cause. This guide won't be able to +explain how to troubleshoot _any_ application with eBPF metrics, but it should give you some ideas on how to start with +your own systems. + +The value of using Netdata to collect and visualize eBPF metrics is that you don't have to rely on existing (complex) +command line eBPF programs or, even worse, write your own eBPF program to get the information you need. + +Let's walk through some scenarios where you might find value in eBPF metrics. + +### Benchmark application performance + +You can use eBPF metrics to profile the performance of your applications, whether they're custom or a standard Linux +service, like a web server or database. + +For example, look at the charts below. The first spike represents running a Redis benchmark _without_ pipelining +(`redis-benchmark -n 1000000 -t set,get -q`). The second spike represents the same benchmark _with_ pipelining +(`redis-benchmark -n 1000000 -t set,get -q -P 16`). + +![Screenshot of eBPF metrics during a Redis +benchmark](https://user-images.githubusercontent.com/1153921/84916168-91607700-b072-11ea-8fec-b76df89315aa.png) + +The performance optimization is clear from the speed at which the benchmark finished (the horizontal length of the +spike) and the reduced write/read syscalls and bytes written to disk. + +You can run similar performance benchmarks against any application, view the results on a Linux kernel level, and +continuously improve the performance of your infrastructure. + +### Inspect for leaking file descriptors + +If your application runs fine and then only crashes after a few hours, leaking file descriptors may be to blame. + +Check the **Number of open files (apps.file_open)** and **Files closed (apps.file_closed)** for discrepancies. These +metrics should be more or less equal. If they diverge, with more open files than closed, your application may not be +closing file descriptors properly. + +See, for example, the volume of files opened and closed by `apps.plugin` itself. Because the eBPF collector is +monitoring these syscalls at an event level, you can see at any given second that the open and closed numbers as equal. + +This isn't to say Netdata is _perfect_, but at least `apps.plugin` doesn't have a file descriptor problem. + +![Screenshot of open and closed file +descriptors](https://user-images.githubusercontent.com/1153921/84816048-c57f5d80-afc8-11ea-9684-d2b923d5d2b2.png) + +### Pin down syscall failures + +If you enabled the eBPF collector's `return` mode as mentioned [in a previous +step](#configure-the-ebpf-collector-to-monitor-errors), you can view charts related to how often a given application's +syscalls return in failure. + +By understanding when these failures happen, and when, you might be able to diagnose a bug in your application. + +To diagnose potential issues with an application, look at the **Fails to open files (apps.file_open_error)**, **Fails to +close files (apps.file_close_error)**, **Fails to write (apps.vfs_write_error)**, and **Fails to read +(apps.vfs_read_error)** charts for failed syscalls coming from your application. If you see any, look to the surrounding +charts for anomalies at the same time frame, or correlate with other activity in the application or on the system to get +closer to the root cause. + +### Investigate zombie processes + +Look for the trio of **Process started (apps.process_create)**, **Threads started (apps.thread_create)**, and **Tasks +closed (apps.task_close)** charts to investigate situations where an application inadvertently leaves [zombie +processes](https://en.wikipedia.org/wiki/Zombie_process). + +These processes, which are terminated and don't use up system resources, can still cause issues if your system runs out +of available PIDs to allocate. + +For example, the chart below demonstrates a [zombie factory +program](https://www.refining-linux.org/archives/7-Dr.-Frankenlinux-or-how-to-create-zombie-processes.html) in action. + +![Screenshot of eBPF showing evidence of a zombie +process](https://user-images.githubusercontent.com/1153921/84831957-27e45800-afe1-11ea-9fe2-fdd910915366.png) + +Starting at 14:51:49, Netdata sees the `zombie` group creating one new process every second, but no closed tasks. This +continues for roughly 30 seconds, at which point the factory program was killed with `SIGINT`, which results in the 31 +closed tasks in the subsequent second. + +Zombie processes may not be catastrophic, but if you're developing an application on Linux, you should eliminate them. +If a service in your stack creates them, you should consider filing a bug report. + +## View eBPF metrics in Netdata Cloud + +You can also show per-application eBPF metrics in Netdata Cloud. This could be particularly useful if you're running the +same application on multiple systems and want to correlate how it performs on each target, or if you want to share your +findings with someone else on your team. + +If you don't already have a Netdata Cloud account, go [sign in](https://app.netdata.cloud) and get started for free. +Read the [get started with Cloud guide](https://learn.netdata.cloud/docs/cloud/get-started) for a walkthrough of +connecting nodes to and other fundamentals. + +Once you've added one or more nodes to a Space in Netdata Cloud, you can see aggregated eBPF metrics in the [Overview +dashboard](/docs/visualize/overview-infrastructure.md) under the same **Applications** or **eBPF** sections that you +find on the local Agent dashboard. Or, [create new dashboards](/docs/visualize/create-dashboards.md) using eBPF metrics +from any number of distributed nodes to see how your application interacts with multiple Linux kernels on multiple Linux +systems. + +Now that you can see eBPF metrics in Netdata Cloud, you can [invite your +team](https://learn.netdata.cloud/docs/cloud/manage/invite-your-team) and share your findings with others. + +## What's next? + +Debugging and troubleshooting an application takes a special combination of practice, experience, and sheer luck. With +Netdata's eBPF metrics to back you up, you can rest assured that you see every minute detail of how your application +interacts with the Linux kernel. + +If you're still trying to wrap your head around what we offer, be sure to read up on our accompanying documentation and +other resources on eBPF monitoring with Netdata: + +- [eBPF collector](/collectors/ebpf.plugin/README.md) +- [eBPF's integration with `apps.plugin`](/collectors/apps.plugin/README.md#integration-with-ebpf) +- [Linux eBPF monitoring with Netdata](https://www.netdata.cloud/blog/linux-ebpf-monitoring-with-netdata/) + +The scenarios described above are just the beginning when it comes to troubleshooting with eBPF metrics. We're excited +to explore others and see what our community dreams up. If you have other use cases, whether simulated or real-world, +we'd love to hear them: [info@netdata.cloud](mailto:info@netdata.cloud). + +Happy troubleshooting! + + diff --git a/docs/guides/troubleshoot/troubleshooting-agent-with-cloud-connection.md b/docs/guides/troubleshoot/troubleshooting-agent-with-cloud-connection.md new file mode 100644 index 0000000..3bb5ace --- /dev/null +++ b/docs/guides/troubleshoot/troubleshooting-agent-with-cloud-connection.md @@ -0,0 +1,117 @@ + + +# Troubleshoot Agent-Cloud connectivity issues + +When you are claiming a node, you might not be able to immediately see it online in Netdata Cloud. +This could be due to an error in the claiming process or a temporary outage of some services. + +We identified some scenarios that might cause this delay and possible actions you could take to overcome each situation. + +The most common explanation for the delay usually falls into one of the following three categories: + +- [The claiming process of the kickstart script was unsuccessful](#the-claiming-process-of-the-kickstart-script-was-unsuccessful) +- [Claiming on an older, deprecated version of the Agent](#claiming-on-an-older-deprecated-version-of-the-agent) +- [Network issues while connecting to the Cloud](#network-issues-while-connecting-to-the-cloud) + +## The claiming process of the kickstart script was unsuccessful + +Here, we will try to define some edge cases you might encounter when claiming a node. + +### The kickstart script auto-claimed the Agent but there was no error message displayed + +The kickstart script will install/update your Agent and then try to claim the node to the Cloud (if tokens are provided). To +complete the second part, the Agent must be running. In some platforms, the Netdata service cannot be enabled by default +and you must do it manually, using the following steps: + +1. Check if the Agent is running: + + ```bash + systemctl status netdata + ``` + + The expected output should contain info like this: + + ```bash + Active: active (running) since Wed 2022-07-06 12:25:02 EEST; 1h 40min ago + ``` + +2. Enable and start the Netdata Service. + + ```bash + systemctl enable netdata + systemctl start netdata + ``` + +3. Retry the kickstart claiming process. + +:::note + +In some cases a simple restart of the Agent can fix the issue. +Read more about [Starting, Stopping and Restarting the Agent](/docs/configure/start-stop-restart.md). + +::: + +## Claiming on an older, deprecated version of the Agent + +Make sure that you are using the latest version of Netdata if you are using the [Claiming script](https://learn.netdata.cloud/docs/agent/claim#claiming-script). + +With the introduction of our new architecture, Agents running versions lower than `v1.32.0` can face claiming problems, so we recommend you [update the Netdata Agent](https://learn.netdata.cloud/docs/agent/packaging/installer/update) to the latest stable version. + +## Network issues while connecting to the Cloud + +### Verify that your IP is whitelisted from Netdata Cloud + +Most of the nodes change IPs dynamically. It is possible that your current IP has been restricted from accessing `api.netdata.cloud` due to security concerns. + +To verify this: + +1. Check the Agent's `aclk-state`. + + ```bash + sudo netdatacli aclk-state | grep "Banned By Cloud" + ``` + + The output will contain a line indicating if the IP is banned from `api.netdata.cloud`: + + ```bash + Banned By Cloud: yes + ``` + +2. If your node's IP is banned, you can: + + - Contact our team to whitelist your IP by submitting a ticket in the [Netdata forum](https://community.netdata.cloud/) + - Change your node's IP + +### Make sure that your node has internet connectivity and can resolve network domains + +1. Try to reach a well known host: + + ```bash + ping 8.8.8.8 + ``` + +2. If you can reach external IPs, then check your domain resolution. + + ```bash + host api.netdata.cloud + ``` + + The expected output should be something like this: + + ```bash + api.netdata.cloud is an alias for main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com. + main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com has address 54.198.178.11 + main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com has address 44.207.131.212 + main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com has address 44.196.50.41 + ``` + + :::info + + There will be cases in which the firewall restricts network access. In those cases, you need to whitelist `api.netdata.cloud` and `mqtt.netdata.cloud` domains to be able to see your nodes in Netdata Cloud. + If you can't whitelist domains in your firewall, you can whitelist the IPs that the above command will produce, but keep in mind that they can change without any notice. + + ::: diff --git a/docs/guides/using-host-labels.md b/docs/guides/using-host-labels.md new file mode 100644 index 0000000..7a5381e --- /dev/null +++ b/docs/guides/using-host-labels.md @@ -0,0 +1,212 @@ + + +# Use host labels to organize systems, metrics, and alarms + +When you use Netdata to monitor and troubleshoot an entire infrastructure, whether that's dozens or hundreds of systems, +you need sophisticated ways of keeping everything organized. You need alarms that adapt to the system's purpose, or +whether the parent or child in a streaming setup. You need properly-labeled metrics archiving so you can sort, +correlate, and mash-up your data to your heart's content. You need to keep tabs on ephemeral Docker containers in a +Kubernetes cluster. + +You need **host labels**: a powerful new way of organizing your Netdata-monitored systems. We introduced host labels in +[v1.20 of Netdata](https://blog.netdata.cloud/posts/release-1.20/), and they come pre-configured out of the box. + +Let's take a peek into how to create host labels and apply them across a few of Netdata's features to give you more +organization power over your infrastructure. + +## Create unique host labels + +Host labels are defined in `netdata.conf`. To create host labels, open that file using `edit-config`. + +```bash +cd /etc/netdata # Replace this path with your Netdata config directory, if different +sudo ./edit-config netdata.conf +``` + +Create a new `[host labels]` section defining a new host label and its value for the system in question. Make sure not +to violate any of the [host label naming rules](/docs/configure/common-changes.md#organize-nodes-with-host-labels). + +```conf +[host labels] + type = webserver + location = us-seattle + installed = 20200218 +``` + +Once you've written a few host labels, you need to enable them. Instead of restarting the entire Netdata service, you +can reload labels using the helpful `netdatacli` tool: + +```bash +netdatacli reload-labels +``` + +Your host labels will now be enabled. You can double-check these by using `curl http://HOST-IP:19999/api/v1/info` to +read the status of your agent. For example, from a VPS system running Debian 10: + +```json +{ + ... + "host_labels": { + "_is_k8s_node": "false", + "_is_parent": "false", + "_virt_detection": "systemd-detect-virt", + "_container_detection": "none", + "_container": "unknown", + "_virtualization": "kvm", + "_architecture": "x86_64", + "_kernel_version": "4.19.0-6-amd64", + "_os_version": "10 (buster)", + "_os_name": "Debian GNU/Linux", + "type": "webserver", + "location": "seattle", + "installed": "20200218" + }, + ... +} +``` + +You may have noticed a handful of labels that begin with an underscore (`_`). These are automatic labels. + +### Automatic labels + +When Netdata starts, it captures relevant information about the system and converts them into automatically-generated +host labels. You can use these to logically organize your systems via health entities, exporting metrics, +parent-child status, and more. + +They capture the following: + +- Kernel version +- Operating system name and version +- CPU architecture, system cores, CPU frequency, RAM, and disk space +- Whether Netdata is running inside of a container, and if so, the OS and hardware details about the container's host +- Whether Netdata is running inside K8s node +- What virtualization layer the system runs on top of, if any +- Whether the system is a streaming parent or child + +If you want to organize your systems without manually creating host labels, try the automatic labels in some of the +features below. + +## Host labels in streaming + +You may have noticed the `_is_parent` and `_is_child` automatic labels from above. Host labels are also now +streamed from a child to its parent node, which concentrates an entire infrastructure's OS, hardware, container, +and virtualization information in one place: the parent. + +Now, if you'd like to remind yourself of how much RAM a certain child node has, you can access +`http://localhost:19999/host/CHILD_HOSTNAME/api/v1/info` and reference the automatically-generated host labels from the +child system. It's a vastly simplified way of accessing critical information about your infrastructure. + +> ⚠️ Because automatic labels for child nodes are accessible via API calls, and contain sensitive information like +> kernel and operating system versions, you should secure streaming connections with SSL. See the [streaming +> documentation](/streaming/README.md#securing-streaming-communications) for details. You may also want to use +> [access lists](/web/server/README.md#access-lists) or [expose the API only to LAN/localhost +> connections](/docs/netdata-security.md#expose-netdata-only-in-a-private-lan). + +You can also use `_is_parent`, `_is_child`, and any other host labels in both health entities and metrics +exporting. Speaking of which... + +## Host labels in health entities + +You can use host labels to logically organize your systems by their type, purpose, or location, and then apply specific +alarms to them. + +For example, let's use configuration example from earlier: + +```conf +[host labels] + type = webserver + location = us-seattle + installed = 20200218 +``` + +You could now create a new health entity (checking if disk space will run out soon) that applies only to any host +labeled `webserver`: + +```yaml + template: disk_fill_rate + on: disk.space + lookup: max -1s at -30m unaligned of avail + calc: ($this - $avail) / (30 * 60) + every: 15s + host labels: type = webserver +``` + +Or, by using one of the automatic labels, for only webserver systems running a specific OS: + +```yaml + host labels: _os_name = Debian* +``` + +In a streaming configuration where a parent node is triggering alarms for its child nodes, you could create health +entities that apply only to child nodes: + +```yaml + host labels: _is_child = true +``` + +Or when ephemeral Docker nodes are involved: + +```yaml + host labels: _container = docker +``` + +Of course, there are many more possibilities for intuitively organizing your systems with host labels. See the [health +documentation](/health/REFERENCE.md#alarm-line-host-labels) for more details, and then get creative! + +## Host labels in metrics exporting + +If you have enabled any metrics exporting via our experimental [exporters](/exporting/README.md), any new host +labels you created manually are sent to the destination database alongside metrics. You can change this behavior by +editing `exporting.conf`, and you can even send automatically-generated labels on with exported metrics. + +```conf +[exporting:global] +enabled = yes +send configured labels = yes +send automatic labels = no +``` + +You can also change this behavior per exporting connection: + +```conf +[opentsdb:my_instance3] +enabled = yes +destination = localhost:4242 +data source = sum +update every = 10 +send charts matching = system.cpu +send configured labels = no +send automatic labels = yes +``` + +By applying labels to exported metrics, you can more easily parse historical metrics with the labels applied. To learn +more about exporting, read the [documentation](/exporting/README.md). + +## What's next? + +Host labels are a brand-new feature to Netdata, and yet they've already propagated deeply into some of its core +functionality. We're just getting started with labels, and will keep the community apprised of additional functionality +as it's made available. You can also track [issue #6503](https://github.com/netdata/netdata/issues/6503), which is where +the Netdata team first kicked off this work. + +It should be noted that while the Netdata dashboard does not expose either user-configured or automatic host labels, API +queries _do_ showcase this information. As always, we recommend you secure Netdata + +- [Expose Netdata only in a private LAN](/docs/netdata-security.md#expose-netdata-only-in-a-private-lan) +- [Enable TLS/SSL for web/API requests](/web/server/README.md#enabling-tls-support) +- Put Netdata behind a proxy + - [Use an authenticating web server in proxy + mode](/docs/netdata-security.md#use-an-authenticating-web-server-in-proxy-mode) + - [Nginx proxy](/docs/Running-behind-nginx.md) + - [Apache proxy](/docs/Running-behind-apache.md) + - [Lighttpd](/docs/Running-behind-lighttpd.md) + - [Caddy](/docs/Running-behind-caddy.md) + +If you have issues or questions around using host labels, don't hesitate to [file an +issue](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml) on GitHub. We're +excited to make host labels even more valuable to our users, which we can only do with your input. + + diff --git a/docs/metrics-storage-management/enable-streaming.mdx b/docs/metrics-storage-management/enable-streaming.mdx new file mode 100644 index 0000000..a737b07 --- /dev/null +++ b/docs/metrics-storage-management/enable-streaming.mdx @@ -0,0 +1,151 @@ +--- +title: "Enable streaming between nodes" +description: "With metrics streaming enabled, you can not only replicate metrics data into a second database, but also view dashboards and trigger alarm notifications for multiple nodes in parallel." +type: how-to +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/enable-streaming.mdx +--- + +# Enable streaming between nodes + +The simplest streaming configuration is **replication**, in which a child node streams its metrics in real time to a +parent node, and both nodes retain metrics in their own databases. + +To configure replication, you need two nodes, each running Netdata. First you'll first enable streaming on your parent +node, then enable streaming on your child node. When you're finished, you'll be able to see the child node's metrics in +the parent node's dashboard, quickly switch between the two dashboards, and be able to serve [alarm +notifications](/docs/monitor/enable-notifications.md) from either or both nodes. + +## Enable streaming on the parent node + +First, log onto the node that will act as the parent. + +Run `uuidgen` to create a new API key, which is a randomly-generated machine GUID the Netdata Agent uses to identify +itself while initiating a streaming connection. Copy that into a separate text file for later use. + +> Find out how to [install `uuidgen`](https://command-not-found.com/uuidgen) on your node if you don't already have it. + +Next, open `stream.conf` using [`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) +from within the [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory). + +```bash +cd /etc/netdata +sudo ./edit-config stream.conf +``` + +Scroll down to the section beginning with `[API_KEY]`. Paste the API key you generated earlier between the brackets, so +that it looks like the following: + +```conf +[11111111-2222-3333-4444-555555555555] +``` + +Set `enabled` to `yes`, and `default memory mode` to `dbengine`. Leave all the other settings as their defaults. A +simplified version of the configuration, minus the commented lines, looks like the following: + +```conf +[11111111-2222-3333-4444-555555555555] + enabled = yes + default memory mode = dbengine +``` + +Save the file and close it, then restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +## Enable streaming on the child node + +Connect to your child node with SSH. + +Open `stream.conf` again. Scroll down to the `[stream]` section and set `enabled` to `yes`. Paste the IP address of your +parent node at the end of the `destination` line, and paste the API key generated on the parent node onto the `api key` +line. + +Leave all the other settings as their defaults. A simplified version of the configuration, minus the commented lines, +looks like the following: + +```conf +[stream] + enabled = yes + destination = 203.0.113.0 + api key = 11111111-2222-3333-4444-555555555555 +``` + +Save the file and close it, then restart Netdata with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system. + +## Enable TLS/SSL on streaming (optional) + +While encrypting the connection between your parent and child nodes is recommended for security, it's not required to +get started. If you're not interested in encryption, skip ahead to [view streamed +metrics](#view-streamed-metrics-in-netdatas-dashboard). + +In this example, we'll use self-signed certificates. + +On the **parent** node, use OpenSSL to create the key and certificate, then use `chown` to make the new files readable +by the `netdata` user. + +```bash +sudo openssl req -newkey rsa:2048 -nodes -sha512 -x509 -days 365 -keyout /etc/netdata/ssl/key.pem -out /etc/netdata/ssl/cert.pem +sudo chown netdata:netdata /etc/netdata/ssl/cert.pem /etc/netdata/ssl/key.pem +``` + +Next, enforce TLS/SSL on the web server. Open `netdata.conf`, scroll down to the `[web]` section, and look for the `bind +to` setting. Add `^SSL=force` to turn on TLS/SSL. See the [web server +reference](/web/server/README.md#enabling-tls-support) for other TLS/SSL options. + +```conf +[web] + bind to = *=dashboard|registry|badges|management|streaming|netdata.conf^SSL=force +``` + +Next, connect to the **child** node and open `stream.conf`. Add `:SSL` to the end of the existing `destination` setting +to connect to the parent using TLS/SSL. Uncomment the `ssl skip certificate verification` line to allow the use of +self-signed certificates. + +```conf +[stream] + enabled = yes + destination = 203.0.113.0:SSL + ssl skip certificate verification = yes + api key = 11111111-2222-3333-4444-555555555555 +``` + +Restart both the parent and child nodes with `sudo systemctl restart netdata`, or the [appropriate +method](/docs/configure/start-stop-restart.md) for your system, to stream encrypted metrics using TLS/SSL. + +## View streamed metrics in Netdata's dashboard + +At this point, the child node is streaming its metrics in real time to its parent. Open the local Agent dashboard for +the parent by navigating to `http://PARENT-NODE:19999` in your browser, replacing `PARENT-NODE` with its IP address or +hostname. + +This dashboard shows parent metrics. To see child metrics, open the left-hand sidebar with the hamburger icon +![Hamburger icon](https://raw.githubusercontent.com/netdata/netdata-ui/master/src/components/icon/assets/hamburger.svg) +in the top panel. Both nodes appear under the **Replicated Nodes** menu. Click on either of the links to switch between +separate parent and child dashboards. + +![Switching between parent and child +dashboards](https://user-images.githubusercontent.com/1153921/110043346-761ec000-7d04-11eb-8e58-77670ba39161.gif) + +The child dashboard is also available directly at `http://PARENT-NODE:19999/host/CHILD-HOSTNAME`, which in this example +is `http://203.0.113.0:19999/host/netdata-child`. + +## What's next? + +Now that you have a basic streaming setup with replication, you may want to tweak the configuration to eliminate the +child database, disable the child dashboard, or enable SSL on the streaming connection between the parent and child. + +See the [streaming reference +doc](/docs/metrics-storage-management/reference-streaming.mdx#examples) for details about +other possible configurations. + +When using Netdata's default TSDB (`dbengine`), the parent node maintains separate, parallel databases for itself and +every child node streaming to it. Each instance is sized identically based on the `dbengine multihost disk space` +setting in `netdata.conf`. See our doc on [changing metrics retention](/docs/store/change-metrics-storage.md) for +details. + +### Related information & further reading + +- Streaming + - [How Netdata streams metrics](/docs/metrics-storage-management/how-streaming-works.mdx) + - **[Enable streaming between nodes](/docs/metrics-storage-management/enable-streaming.mdx)** + - [Streaming reference](/docs/metrics-storage-management/reference-streaming.mdx) diff --git a/docs/metrics-storage-management/how-streaming-works.mdx b/docs/metrics-storage-management/how-streaming-works.mdx new file mode 100644 index 0000000..ecbce39 --- /dev/null +++ b/docs/metrics-storage-management/how-streaming-works.mdx @@ -0,0 +1,92 @@ +--- +title: "How metrics streaming works" +description: "Netdata's real-time streaming allows you to replicate metrics data across multiple nodes, or centralize all your metrics data into a single time-series database (TSDB)." +type: explanation +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/how-streaming-works.mdx +--- + +# How metrics streaming works + +Each node running Netdata can stream the metrics it collects, in real time, to another node. Streaming allows you to +replicate metrics data across multiple nodes, or centralize all your metrics data into a single time-series database +(TSDB). + +When one node streams metrics to another, the node receiving metrics can visualize them on the +[dashboard](/docs/visualize/interact-dashboards-charts.md), run health checks to [trigger +alarms](/docs/monitor/view-active-alarms.md) and [send notifications](/docs/monitor/enable-notifications.md), and +[export](/docs/export/external-databases.md) all metrics to an external TSDB. When Netdata streams metrics to another +Netdata, the receiving one is able to perform everything a Netdata instance is capable of. + +Streaming lets you decide exactly how you want to store and maintain metrics data. While we believe Netdata's +[distributed architecture](/docs/store/distributed-data-architecture.md) is ideal for speed and scale, streaming +provides centralization options for those who want to maintain only a single TSDB instance. + +## Streaming basics + +There are three types of nodes in Netdata's streaming ecosystem. + +- **Parent**: A node, running Netdata, that receives streamed metric data. +- **Child**: A node, running Netdata, that streams metric data to one or more parent. +- **Proxy**: A node, running Netdata, that receives metric data from a child and "forwards" them on to a + separate parent node. + +Netdata uses API keys, which are just random GUIDs, to authorize the communication between child and parent nodes. We +recommend using `uuidgen` for generating API keys, which can then be used across any number of streaming connections. +Or, you can generate unique API keys for each parent-child relationship. + +Once the parent node authorizes the child's API key, the child can start streaming metrics. + +It's important to note that the streaming connection uses TCP, UDP, or Unix sockets, _not HTTP_. To proxy streaming +metrics, you need to use a proxy that tunnels [OSI layer 4-7 +traffic](https://en.wikipedia.org/wiki/OSI_model#Layer_4:_Transport_Layer) without interfering with it, such as +[SOCKS](https://en.wikipedia.org/wiki/SOCKS) or Nginx's [TCP/UDP load +balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/). + +## Supported streaming configurations + +Netdata supports any combination of parent, child, and proxy nodes that you can imagine. Any node can act as both a +parent, child, or proxy at the same time, sending or receiving streaming metrics from any number of other nodes. + +Here are a few example streaming configurations: + +- **Headless collector**: + - Child `A`, _without_ a database or web dashboard, streams metrics to parent `B`. + - `A` metrics are only available via the local Agent dashboard for `B`. + - `B` generates alarms for `A`. +- **Replication**: + - Child `A`, _with_ a database and web dashboard, streams metrics to parent `B`. + - `A` metrics are available on both local Agent dashboards, and can be stored with the same or different metrics + retention policies. + - Both `A` and `B` generate alarms. +- **Proxy**: + - Child `A`, _with or without_ a database, sends metrics to proxy `C`, also _with or without_ a database. `C` sends + metrics to parent `B`. + - Any node with a database can generate alarms. + +## Viewing streamed metrics + +Parent nodes feature a **Replicated Nodes** section in the left-hand panel, which opens with the hamburger icon +![Hamburger icon](https://raw.githubusercontent.com/netdata/netdata-ui/master/src/components/icon/assets/hamburger.svg) +in the top navigation. The parent node, plus any child nodes, appear here. Click on any of the hostnames to switch +between parent and child dashboards, all served by the parent's [web server](/web/server/README.md). + +![Switching between +](https://user-images.githubusercontent.com/1153921/110043346-761ec000-7d04-11eb-8e58-77670ba39161.gif) + +Each child dashboard is also available directly at the following URL pattern: +`http://PARENT-NODE:19999/host/CHILD-HOSTNAME`. + +## What's next? + +Now that you understand the fundamentals of streaming metrics between nodes, go ahead and [enable +streaming](/docs/metrics-storage-management/enable-streaming.mdx) using a simple `parent-child` relationship. For all +the details, see the [streaming reference](/docs/metrics-storage-management/reference-streaming.mdx) doc. + +Take your streaming setup even further by [exporting metrics](/docs/export/external-databases.md) to an external TSDB. + +### Related information & further reading + +- Streaming + - **[How Netdata streams metrics](/docs/metrics-storage-management/how-streaming-works.mdx)** + - [Enable streaming between nodes](/docs/metrics-storage-management/enable-streaming.mdx) + - [Streaming reference](/docs/metrics-storage-management/reference-streaming.mdx) \ No newline at end of file diff --git a/docs/metrics-storage-management/reference-streaming.mdx b/docs/metrics-storage-management/reference-streaming.mdx new file mode 100644 index 0000000..c77ceb3 --- /dev/null +++ b/docs/metrics-storage-management/reference-streaming.mdx @@ -0,0 +1,486 @@ +--- +title: "Streaming reference" +description: "Each node running Netdata can stream the metrics it collects, in real time, to another node. See all of the available settings in this reference document." +type: reference +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/reference-streaming.mdx +--- + +# Streaming reference + +Each node running Netdata can stream the metrics it collects, in real time, to another node. To learn more, read about +[how streaming works](/docs/metrics-storage-management/how-streaming-works.mdx). + +For a quickstart guide for enabling a simple `parent-child` streaming relationship, see our [stream metrics between +nodes](/docs/metrics-storage-management/enable-streaming.mdx) doc. All other configuration options and scenarios are +covered in the sections below. + +## Configuration + +There are two files responsible for configuring Netdata's streaming capabilities: `stream.conf` and `netdata.conf`. + +From within your Netdata config directory (typically `/etc/netdata`), [use `edit-config`](/docs/configure/nodes.md) to +open either `stream.conf` or `netdata.conf`. + +``` +sudo ./edit-config stream.conf +sudo ./edit-config netdata.conf +``` + +## Settings + +As mentioned above, both `stream.conf` and `netdata.conf` contain settings relevant to streaming. + +### `stream.conf` + +The `stream.conf` file contains three sections. The `[stream]` section is for configuring child nodes. + +The `[API_KEY]` and `[MACHINE_GUID]` sections are both for configuring parent nodes, and share the same settings. +`[API_KEY]` settings affect every child node using that key, whereas `[MACHINE_GUID]` settings affect only the child +node with a matching GUID. + +The file `/var/lib/netdata/registry/netdata.public.unique.id` contains a random GUID that **uniquely identifies each +node**. This file is automatically generated by Netdata the first time it is started and remains unaltered forever. + +#### `[stream]` section + +| Setting | Default | Description | +| :---------------------------------------------- | :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enabled` | `no` | Whether this node streams metrics to any parent. Change to `yes` to enable streaming. | +| [`destination`](#destination) | ` ` | A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`. [Read more →](#destination) | +| `ssl skip certificate verification` | `yes` | If you want to accept self-signed or expired certificates, set to `yes` and uncomment. | +| `CApath` | `/etc/ssl/certs/` | The directory where known certificates are found. Defaults to OpenSSL's default path. | +| `CAfile` | `/etc/ssl/certs/cert.pem` | Add a parent node certificate to the list of known certificates in `CAPath`. | +| `api key` | ` ` | The `API_KEY` to use as the child node. | +| `timeout seconds` | `60` | The timeout to connect and send metrics to a parent. | +| `default port` | `19999` | The port to use if `destination` does not specify one. | +| [`send charts matching`](#send-charts-matching) | `*` | A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to filter which charts are streamed. [Read more →](#send-charts-matching) | +| `buffer size bytes` | `10485760` | The size of the buffer to use when sending metrics. The default `10485760` equals a buffer of 10MB, which is good for 60 seconds of data. Increase this if you expect latencies higher than that. The buffer is flushed on reconnect. | +| `reconnect delay seconds` | `5` | How long to wait until retrying to connect to the parent node. | +| `initial clock resync iterations` | `60` | Sync the clock of charts for how many seconds when starting. | + +### `[API_KEY]` and `[MACHINE_GUID]` sections + +| Setting | Default | Description | +| :---------------------------------------------- | :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enabled` | `no` | Whether this API KEY enabled or disabled. | +| [`allow from`](#allow-from) | `*` | A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) matching the IPs of nodes that will stream metrics using this API key. [Read more →](#allow-from) | +| `default history` | `3600` | The default amount of child metrics history to retain when using the `save`, `map`, or `ram` memory modes. | +| [`default memory mode`](#default-memory-mode) | `ram` | The [database](/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `map`, `save`, `ram`, or `none`. [Read more →](#default-memory-mode) | +| `health enabled by default` | `auto` | Whether alarms and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alarms when the child is connected. `yes` enables alarms always, and `no` disables alarms. | +| `default postpone alarms on connect seconds` | `60` | Postpone alarms and notifications for a period of time after the child connects. | +| `default proxy enabled` | ` ` | Route metrics through a proxy. | +| `default proxy destination` | ` ` | Space-separated list of `IP:PORT` for proxies. | +| `default proxy api key` | ` ` | The `API_KEY` of the proxy. | +| `default send charts matching` | `*` | See [`send charts matching`](#send-charts-matching). | + +#### `destination` + +A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using +the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`. + +- `PROTOCOL`: `tcp`, `udp`, or `unix`. (only tcp and unix are supported by parent nodes) +- `HOST`: A IPv4, IPv6 IP, or a hostname, or a unix domain socket path. IPv6 IPs should be given with brackets + `[ip:address]`. +- `INTERFACE` (IPv6 only): The network interface to use. +- `PORT`: The port number or service name (`/etc/services`) to use. +- `SSL`: To enable TLS/SSL encryption of the streaming connection. + +To enable TCP streaming to a parent node at `203.0.113.0` on port `20000` and with TLS/SSL encryption: + +```conf +[stream] + destination = tcp:203.0.113.0:20000:SSL +``` + +#### `send charts matching` + +A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to filter which charts are streamed. + +The default is a single wildcard `*`, which streams all charts. + +To send only a few charts, list them explicitly, or list a group using a wildcard. To send _only_ the `apps.cpu` chart +and charts with contexts beginning with `system.`: + +```conf +[stream] + send charts matching = apps.cpu system.* +``` + +To send all but a few charts, use `!` to create a negative match. To send _all_ charts _but_ `apps.cpu`: + +```conf +[stream] + send charts matching = !apps.cpu * +``` + +#### `allow from` + +A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) matching the IPs of nodes that +will stream metrics using this API key. The order is important, left to right, as the first positive or negative match is used. + +The default is `*`, which accepts all requests including the `API_KEY`. + +To allow from only a specific IP address: + +```conf +[API_KEY] + allow from = 203.0.113.10 +``` + +To allow all IPs starting with `10.*`, except `10.1.2.3`: + +```conf +[API_KEY] + allow from = !10.1.2.3 10.* +``` + +> If you set specific IP addresses here, and also use the `allow connections` setting in the `[web]` section of +> `netdata.conf`, be sure to add the IP address there so that it can access the API port. + +#### `default memory mode` + +The [database](/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `ram`, +`save`, `map`, or `none`. + +- `dbengine`: The default, recommended time-series database (TSDB) for Netdata. Stores recent metrics in memory, then + efficiently spills them to disk for long-term storage. +- `ram`: Stores metrics _only_ in memory, which means metrics are lost when Netdata stops or restarts. Ideal for + streaming configurations that use ephemeral nodes. +- `save`: Stores metrics in memory, but saves metrics to disk when Netdata stops or restarts, and loads historical + metrics on start. +- `map`: Stores metrics in memory-mapped files, like swap, with constant disk write. +- `none`: No database. + +When using `default memory mode = dbengine`, the parent node creates a separate instance of the TSDB to store metrics +from child nodes. The [size of _each_ instance is configurable](/docs/store/change-metrics-storage.md) with the `page +cache size` and `dbengine multihost disk space` settings in the `[global]` section in `netdata.conf`. + +### `netdata.conf` + +| Setting | Default | Description | +| :----------------------------------------- | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **`[global]` section** | | | +| `memory mode` | `dbengine` | Determines the [database type](/database/README.md) to be used on that node. Other options settings include `none`, `ram`, `save`, and `map`. `none` disables the database at this host. This also disables alarms and notifications, as those can't run without a database. | +| **`[web]` section** | | | +| `mode` | `static-threaded` | Determines the [web server](/web/server/README.md) type. The other option is `none`, which disables the dashboard, API, and registry. | +| `accept a streaming request every seconds` | `0` | Set a limit on how often a parent node accepts streaming requests from child nodes. `0` equals no limit. If this is set, you may see `... too busy to accept new streaming request. Will be allowed in X secs` in Netdata's `error.log`. | + +## Examples + +### Per-child settings + +While the `[API_KEY]` section applies settings for any child node using that key, you can also use per-child settings +with the `[MACHINE_GUID]` section. + +For example, the metrics streamed from only the child node with `MACHINE_GUID` are saved in memory, not using the +default `dbengine` as specified by the `API_KEY`, and alarms are disabled. + +```conf +[API_KEY] + enabled = yes + default memory mode = dbengine + health enabled by default = auto + allow from = * + +[MACHINE_GUID] + enabled = yes + memory mode = save + health enabled = no +``` + +### Securing streaming with TLS/SSL + +Netdata does not activate TLS encryption by default. To encrypt streaming connections, you first need to [enable TLS +support](/web/server/README.md#enabling-tls-support) on the parent. With encryption enabled on the receiving side, you +need to instruct the child to use TLS/SSL as well. On the child's `stream.conf`, configure the destination as follows: + +``` +[stream] + destination = host:port:SSL +``` + +The word `SSL` appended to the end of the destination tells the child that connections must be encrypted. + +> While Netdata uses Transport Layer Security (TLS) 1.2 to encrypt communications rather than the obsolete SSL protocol, +> it's still common practice to refer to encrypted web connections as `SSL`. Many vendors, like Nginx and even Netdata +> itself, use `SSL` in configuration files, whereas documentation will always refer to encrypted communications as `TLS` +> or `TLS/SSL`. + +#### Certificate verification + +When TLS/SSL is enabled on the child, the default behavior will be to not connect with the parent unless the server's +certificate can be verified via the default chain. In case you want to avoid this check, add the following to the +child's `stream.conf` file: + +``` +[stream] + ssl skip certificate verification = yes +``` + +#### Trusted certificate + +If you've enabled [certificate verification](#certificate-verification), you might see errors from the OpenSSL library +when there's a problem with checking the certificate chain (`X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY`). More +importantly, OpenSSL will reject self-signed certificates. + +Given these known issues, you have two options. If you trust your certificate, you can set the options `CApath` and +`CAfile` to inform Netdata where your certificates, and the certificate trusted file, are stored. + +For more details about these options, you can read about [verify +locations](https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_load_verify_locations.html). + +Before you changed your streaming configuration, you need to copy your trusted certificate to your child system and add +the certificate to OpenSSL's list. + +On most Linux distributions, the `update-ca-certificates` command searches inside the `/usr/share/ca-certificates` +directory for certificates. You should double-check by reading the `update-ca-certificate` manual (`man +update-ca-certificate`), and then change the directory in the below commands if needed. + +If you have `sudo` configured on your child system, you can use that to run the following commands. If not, you'll have +to log in as `root` to complete them. + +``` +# mkdir /usr/share/ca-certificates/netdata +# cp parent_cert.pem /usr/share/ca-certificates/netdata/parent_cert.crt +# chown -R netdata.netdata /usr/share/ca-certificates/netdata/ +``` + +First, you create a new directory to store your certificates for Netdata. Next, you need to change the extension on your +certificate from `.pem` to `.crt` so it's compatible with `update-ca-certificate`. Finally, you need to change +permissions so the user that runs Netdata can access the directory where you copied in your certificate. + +Next, edit the file `/etc/ca-certificates.conf` and add the following line: + +``` +netdata/parent_cert.crt +``` + +Now you update the list of certificates running the following, again either as `sudo` or `root`: + +``` +# update-ca-certificates +``` + +> Some Linux distributions have different methods of updating the certificate list. For more details, please read this +> guide on [adding trusted root certificates](https://github.com/Busindre/How-to-Add-trusted-root-certificates). + +Once you update your certificate list, you can set the stream parameters for Netdata to trust the parent certificate. +Open `stream.conf` for editing and change the following lines: + +``` +[stream] + CApath = /etc/ssl/certs/ + CAfile = /etc/ssl/certs/parent_cert.pem +``` + +With this configuration, the `CApath` option tells Netdata to search for trusted certificates inside `/etc/ssl/certs`. +The `CAfile` option specifies the Netdata parent certificate is located at `/etc/ssl/certs/parent_cert.pem`. With this +configuration, you can skip using the system's entire list of certificates and use Netdata's parent certificate instead. + +#### Expected behaviors + +With the introduction of TLS/SSL, the parent-child communication behaves as shown in the table below, depending on the +following configurations: + +- **Parent TLS (Yes/No)**: Whether the `[web]` section in `netdata.conf` has `ssl key` and `ssl certificate`. +- **Parent port TLS (-/force/optional)**: Depends on whether the `[web]` section `bind to` contains a `^SSL=force` or + `^SSL=optional` directive on the port(s) used for streaming. +- **Child TLS (Yes/No)**: Whether the destination in the child's `stream.conf` has `:SSL` at the end. +- **Child TLS Verification (yes/no)**: Value of the child's `stream.conf` `ssl skip certificate verification` + parameter (default is no). + +| Parent TLS enabled | Parent port SSL | Child TLS | Child SSL Ver. | Behavior | +| :----------------- | :--------------- | :-------- | :------------- | :--------------------------------------------------------------------------------------------------------------------------------------- | +| No | - | No | no | Legacy behavior. The parent-child stream is unencrypted. | +| Yes | force | No | no | The parent rejects the child connection. | +| Yes | -/optional | No | no | The parent-child stream is unencrypted (expected situation for legacy child nodes and newer parent nodes) | +| Yes | -/force/optional | Yes | no | The parent-child stream is encrypted, provided that the parent has a valid TLS/SSL certificate. Otherwise, the child refuses to connect. | +| Yes | -/force/optional | Yes | yes | The parent-child stream is encrypted. | + +### Proxy + +A proxy is a node that receives metrics from a child, then streams them onward to a parent. To configure a proxy, +configure it as a receiving and a sending Netdata at the same time. + +Netdata proxies may or may not maintain a database for the metrics passing through them. When they maintain a database, +they can also run health checks (alarms and notifications) for the remote host that is streaming the metrics. + +In the following example, the proxy receives metrics from a child node using the `API_KEY` of +`66666666-7777-8888-9999-000000000000`, then stores metrics using `dbengine`. It then uses the `API_KEY` of +`11111111-2222-3333-4444-555555555555` to proxy those same metrics on to a parent node at `203.0.113.0`. + +```conf +[stream] + enabled = yes + destination = 203.0.113.0 + api key = 11111111-2222-3333-4444-555555555555 + +[66666666-7777-8888-9999-000000000000] + enabled = yes + default memory mode = dbengine +``` + +### Ephemeral nodes + +Netdata can help you monitor ephemeral nodes, such as containers in an auto-scaling infrastructure, by always streaming +metrics to any number of permanently-running parent nodes. + +On the parent, set the following in `stream.conf`: + +```conf +[11111111-2222-3333-4444-555555555555] + # enable/disable this API key + enabled = yes + + # one hour of data for each of the child nodes + default history = 3600 + + # do not save child metrics on disk + default memory = ram + + # alarms checks, only while the child is connected + health enabled by default = auto +``` + +On the child nodes, set the following in `stream.conf`: + +```bash +[stream] + # stream metrics to another Netdata + enabled = yes + + # the IP and PORT of the parent + destination = 10.11.12.13:19999 + + # the API key to use + api key = 11111111-2222-3333-4444-555555555555 +``` + +In addition, edit `netdata.conf` on each child node to disable the database and alarms. + +```bash +[global] + # disable the local database + memory mode = none + +[health] + # disable health checks + enabled = no +``` + +## Troubleshooting + +Both parent and child nodes log information at `/var/log/netdata/error.log`. + +If the child manages to connect to the parent you will see something like (on the parent): + +``` +2017-03-09 09:38:52: netdata: INFO : STREAM [receive from [10.11.12.86]:38564]: new client connection. +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [10.11.12.86]:38564: receive thread created (task id 27721) +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: client willing to stream metrics for host 'xxx' with machine_guid '1234567-1976-11e6-ae19-7cdd9077342a': update every = 1, history = 3600, memory mode = ram, health auto +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: initializing communication... +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: receiving metrics... +``` + +and something like this on the child: + +``` +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: connecting... +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: initializing communication... +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: waiting response from remote netdata... +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: established communication - sending metrics... +``` + +The following sections describe the most common issues you might encounter when connecting parent and child nodes. + +### Slow connections between parent and child + +When you have a slow connection between parent and child, Netdata raises a few different errors. Most of the +errors will appear in the child's `error.log`. + +```bash +netdata ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM CHILD HOSTNAME [send to PARENT IP:PARENT PORT]: too many data pending - buffer is X bytes long, +Y unsent - we have sent Z bytes in total, W on this connection. Closing connection to flush the data. +``` + +On the parent side, you may see various error messages, most commonly the following: + +``` +netdata ERROR : STREAM_PARENT[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : read failed: end of file +``` + +Another common problem in slow connections is the child sending a partial message to the parent. In this case, the +parent will write the following to its `error.log`: + +``` +ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : sent command 'B' which is not known by netdata, for host 'HOSTNAME'. Disabling it. +``` + +In this example, `B` was part of a `BEGIN` message that was cut due to connection problems. + +Slow connections can also cause problems when the parent misses a message and then receives a command related to the +missed message. For example, a parent might miss a message containing the child's charts, and then doesn't know +what to do with the `SET` message that follows. When that happens, the parent will show a message like this: + +``` +ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : requested a SET on chart 'CHART NAME' of host 'HOSTNAME', without a dimension. Disabling it. +``` + +### Child cannot connect to parent + +When the child can't connect to a parent for any reason (misconfiguration, networking, firewalls, parent +down), you will see the following in the child's `error.log`. + +``` +ERROR : STREAM_SENDER[HOSTNAME] : Failed to connect to 'PARENT IP', port 'PARENT PORT' (errno 113, No route to host) +``` + +### 'Is this a Netdata?' + +This question can appear when Netdata starts the stream and receives an unexpected response. This error can appear when +the parent is using SSL and the child tries to connect using plain text. You will also see this message when +Netdata connects to another server that isn't Netdata. The complete error message will look like this: + +``` +ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM child HOSTNAME [send to PARENT HOSTNAME:PARENT PORT]: server is not replying properly (is it a netdata?). +``` + +### Stream charts wrong + +Chart data needs to be consistent between child and parent nodes. If there are differences between chart data on +a parent and a child, such as gaps in metrics collection, it most often means your child's `memory mode` +does not match the parent's. To learn more about the different ways Netdata can store metrics, and thus keep chart +data consistent, read our [memory mode documentation](/database/README.md). + +### Forbidding access + +You may see errors about "forbidding access" for a number of reasons. It could be because of a slow connection between +the parent and child nodes, but it could also be due to other failures. Look in your parent's `error.log` for errors +that look like this: + +``` +STREAM [receive from [child HOSTNAME]:child IP]: `MESSAGE`. Forbidding access." +``` + +`MESSAGE` will have one of the following patterns: + +- `request without KEY` : The message received is incomplete and the KEY value can be API, hostname, machine GUID. +- `API key 'VALUE' is not valid GUID`: The UUID received from child does not have the format defined in [RFC + 4122](https://tools.ietf.org/html/rfc4122) +- `machine GUID 'VALUE' is not GUID.`: This error with machine GUID is like the previous one. +- `API key 'VALUE' is not allowed`: This stream has a wrong API key. +- `API key 'VALUE' is not permitted from this IP`: The IP is not allowed to use STREAM with this parent. +- `machine GUID 'VALUE' is not allowed.`: The GUID that is trying to send stream is not allowed. +- `Machine GUID 'VALUE' is not permitted from this IP. `: The IP does not match the pattern or IP allowed to connect to + use stream. + +### Netdata could not create a stream + +The connection between parent and child is a stream. When the parent can't convert the initial connection into +a stream, it will write the following message inside `error.log`: + +``` +file descriptor given is not a valid stream +``` + +After logging this error, Netdata will close the stream. diff --git a/docs/monitor/configure-alarms.md b/docs/monitor/configure-alarms.md new file mode 100644 index 0000000..ac45811 --- /dev/null +++ b/docs/monitor/configure-alarms.md @@ -0,0 +1,148 @@ + + +# Configure health alarms + +Netdata's health watchdog is highly configurable, with support for dynamic thresholds, hysteresis, alarm templates, and +more. You can tweak any of the existing alarms based on your infrastructure's topology or specific monitoring needs, or +create new entities. + +You can use health alarms in conjunction with any of Netdata's [collectors](/docs/collect/how-collectors-work.md) (see +the [supported collector list](/collectors/COLLECTORS.md)) to monitor the health of your systems, containers, and +applications in real time. + +While you can see active alarms both on the local dashboard and Netdata Cloud, all health alarms are configured _per +node_ via individual Netdata Agents. If you want to deploy a new alarm across your +[infrastructure](/docs/quickstart/infrastructure.md), you must configure each node with the same health configuration +files. + +## Edit health configuration files + +All of Netdata's [health configuration files](/health/REFERENCE.md#health-configuration-files) are in Netdata's config +directory, inside the `health.d/` directory. Navigate to your [Netdata config directory](/docs/configure/nodes.md) and +use `edit-config` to make changes to any of these files. + +For example, to edit the `cpu.conf` health configuration file, run: + +```bash +sudo ./edit-config health.d/cpu.conf +``` + +Each health configuration file contains one or more health _entities_, which always begin with `alarm:` or `template:`. +For example, here is the first health entity in `health.d/cpu.conf`: + +```yaml +template: 10min_cpu_usage + on: system.cpu + os: linux + hosts: * + lookup: average -10m unaligned of user,system,softirq,irq,guest + units: % + every: 1m + warn: $this > (($status >= $WARNING) ? (75) : (85)) + crit: $this > (($status == $CRITICAL) ? (85) : (95)) + delay: down 15m multiplier 1.5 max 1h + info: average cpu utilization for the last 10 minutes (excluding iowait, nice and steal) + to: sysadmin +``` + +To tune this alarm to trigger warning and critical alarms at a lower CPU utilization, change the `warn` and `crit` lines +to the values of your choosing. For example: + +```yaml + warn: $this > (($status >= $WARNING) ? (60) : (75)) + crit: $this > (($status == $CRITICAL) ? (75) : (85)) +``` + +Save the file and [reload Netdata's health configuration](#reload-health-configuration) to make your changes live. + +### Silence an individual alarm + +Instead of disabling an alarm altogether, or even disabling _all_ alarms, you can silence individual alarms by changing +one line in a given health entity. To silence any single alarm, change the `to:` line in its entity to `silent`. + +```yaml + to: silent +``` + +## Write a new health entity + +While tuning existing alarms may work in some cases, you may need to write entirely new health entities based on how +your systems, containers, and applications work. + +Read Netdata's [health reference](/health/REFERENCE.md#health-entity-reference) for a full listing of the format, +syntax, and functionality of health entities. + +To write a new health entity into a new file, navigate to your [Netdata config directory](/docs/configure/nodes.md), +then use `touch` to create a new file in the `health.d/` directory. Use `edit-config` to start editing the file. + +As an example, let's create a `ram-usage.conf` file. + +```bash +sudo touch health.d/ram-usage.conf +sudo ./edit-config health.d/ram-usage.conf +``` + +For example, here is a health entity that triggers a warning alarm when a node's RAM usage rises above 80%, and a +critical alarm above 90%: + +```yaml + alarm: ram_usage + on: system.ram +lookup: average -1m percentage of used + units: % + every: 1m + warn: $this > 80 + crit: $this > 90 + info: The percentage of RAM being used by the system. +``` + +Let's look into each of the lines to see how they create a working health entity. + +- `alarm`: The name for your new entity. The name needs to follow these requirements: + - Any alphabet letter or number. + - The symbols `.` and `_`. + - Cannot be `chart name`, `dimension name`, `family name`, or `chart variable names`. +- `on`: Which chart the entity listens to. +- `lookup`: Which metrics the alarm monitors, the duration of time to monitor, and how to process the metrics into a + usable format. + - `average`: Calculate the average of all the metrics collected. + - `-1m`: Use metrics from 1 minute ago until now to calculate that average. + - `percentage`: Clarify that we're calculating a percentage of RAM usage. + - `of used`: Specify which dimension (`used`) on the `system.ram` chart you want to monitor with this entity. +- `units`: Use percentages rather than absolute units. +- `every`: How often to perform the `lookup` calculation to decide whether or not to trigger this alarm. +- `warn`/`crit`: The value at which Netdata should trigger a warning or critical alarm. This example uses simple + syntax, but most pre-configured health entities use + [hysteresis](/health/REFERENCE.md#special-use-of-the-conditional-operator) to avoid superfluous notifications. +- `info`: A description of the alarm, which will appear in the dashboard and notifications. + +In human-readable format: + +> This health entity, named **ram_usage**, watches the **system.ram** chart. It looks up the last **1 minute** of +> metrics from the **used** dimension and calculates the **average** of all those metrics in a **percentage** format, +> using a **% unit**. The entity performs this lookup **every minute**. +> +> If the average RAM usage percentage over the last 1 minute is **more than 80%**, the entity triggers a warning alarm. +> If the usage is **more than 90%**, the entity triggers a critical alarm. + +When you finish writing this new health entity, [reload Netdata's health configuration](#reload-health-configuration) to +see it live on the local dashboard or Netdata Cloud. + +## Reload health configuration + +To make any changes to your health configuration live, you must reload Netdata's health monitoring system. To do that +without restarting all of Netdata, run `netdatacli reload-health` or `killall -USR2 netdata`. + +## What's next? + +With your health entities configured properly, it's time to [enable +notifications](/docs/monitor/enable-notifications.md) to get notified whenever a node reaches a warning or critical +state. + +To build complex, dynamic alarms, read our guide on [dimension templates](/docs/guides/monitor/dimension-templates.md). + + diff --git a/docs/monitor/enable-notifications.md b/docs/monitor/enable-notifications.md new file mode 100644 index 0000000..438eef3 --- /dev/null +++ b/docs/monitor/enable-notifications.md @@ -0,0 +1,147 @@ + + +# Enable alarm notifications + +Netdata offers two ways to receive alarm notifications on external platforms. These methods work independently _or_ in +parallel, which means you can enable both at the same time to send alarm notifications to any number of endpoints. + +Both methods use a node's health alarms to generate the content of alarm notifications. Read the doc on [configuring +alarms](/docs/monitor/configure-alarms.md) to change the preconfigured thresholds or to create tailored alarms for your +infrastructure. + +Netdata Cloud offers [centralized alarm notifications](#netdata-cloud) via email, which leverages the health status +information already streamed to Netdata Cloud from connected nodes to send notifications to those who have enabled them. + +The Netdata Agent has a [notification system](#netdata-agent) that supports more than a dozen services, such as email, +Slack, PagerDuty, Twilio, Amazon SNS, Discord, and much more. + +For example, use centralized alarm notifications in Netdata Cloud for immediate, zero-configuration alarm notifications +for your team, then configure individual nodes send notifications to a PagerDuty endpoint for an automated incident +response process. + +## Netdata Cloud + +Netdata Cloud's [centralized alarm +notifications](https://learn.netdata.cloud/docs/cloud/alerts-notifications/notifications) is a zero-configuration way to +get notified when an anomaly or incident strikes any node or application in your infrastructure. The advantage of using +centralized alarm notifications from Netdata Cloud is that you don't have to worry about configuring each node in your +infrastructure. + +To enable centralized alarm notifications for a Space, click on **Manage Space** in the left-hand menu, then click on +the **Notifications** tab. Click the toggle switch next to **E-mail** to enable this notification method. + +Next, enable notifications on a user level by clicking on your profile icon, then **Profile** in the dropdown. The +**Notifications** tab reveals rich management settings, including the ability to enable/disable methods entirely or +choose what types of notifications to receive from each War Room. + +![Enabling and configuring alarm notifications in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/101936280-93c50900-3b9d-11eb-9ba0-d6927fa872b7.gif) + +See the [centralized alarm notifications](https://learn.netdata.cloud/docs/cloud/alerts-notifications/notifications) +reference doc for further details about what information is conveyed in an email notification, flood protection, and +more. + +## Netdata Agent + +The Netdata Agent's [notification system](/health/notifications/README.md) runs on every node and dispatches +notifications based on configured endpoints and roles. You can enable multiple endpoints on any one node _and_ use Agent +notifications in parallel with centralized alarm notifications in Netdata Cloud. + +> ❗ If you want to enable notifications from multiple nodes in your infrastructure, each running the Netdata Agent, you +> must configure each node individually. + +Below, we'll use [Slack notifications](#enable-slack-notifications) as an example of the process of enabling any +notification platform. + +### Supported notification endpoints + +- [**alerta.io**](/health/notifications/alerta/README.md) +- [**Amazon SNS**](/health/notifications/awssns/README.md) +- [**Custom endpoint**](/health/notifications/custom/README.md) +- [**Discord**](/health/notifications/discord/README.md) +- [**Dynatrace**](/health/notifications/dynatrace/README.md) +- [**Email**](/health/notifications/email/README.md) +- [**Flock**](/health/notifications/flock/README.md) +- [**Google Hangouts**](/health/notifications/hangouts/README.md) +- [**Gotify**](/health/notifications/gotify/README.md) +- [**IRC**](/health/notifications/irc/README.md) +- [**Kavenegar**](/health/notifications/kavenegar/README.md) +- [**Matrix**](/health/notifications/matrix/README.md) +- [**Messagebird**](/health/notifications/messagebird/README.md) +- [**Microsoft Teams**](/health/notifications/msteams/README.md) +- [**Netdata Agent dashboard**](/health/notifications/web/README.md) +- [**Opsgenie**](/health/notifications/opsgenie/README.md) +- [**PagerDuty**](/health/notifications/pagerduty/README.md) +- [**Prowl**](/health/notifications/prowl/README.md) +- [**PushBullet**](/health/notifications/pushbullet/README.md) +- [**PushOver**](/health/notifications/pushover/README.md) +- [**Rocket.Chat**](/health/notifications/rocketchat/README.md) +- [**Slack**](/health/notifications/slack/README.md) +- [**SMS Server Tools 3**](/health/notifications/smstools3/README.md) +- [**StackPulse**](/health/notifications/stackpulse/README.md) +- [**Syslog**](/health/notifications/syslog/README.md) +- [**Telegram**](/health/notifications/telegram/README.md) +- [**Twilio**](/health/notifications/twilio/README.md) + +### Enable Slack notifications + +First, [Add an incoming webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) in Slack for the channel where you +want to see alarm notifications from Netdata. Click the green **Add to Slack** button, choose the channel, and click the +**Add Incoming WebHooks Integration** button. + +On the following page, you'll receive a **Webhook URL**. That's what you'll need to configure Netdata, so keep it handy. + +Navigate to your [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory) and use `edit-config` to +open the `health_alarm_notify.conf` file: + +```bash +sudo ./edit-config health_alarm_notify.conf +``` + +Look for the `SLACK_WEBHOOK_URL=" "` line and add the incoming webhook URL you got from Slack: + +```conf +SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/XXXXXXXXXXXX" +``` + +A few lines down, edit the `DEFAULT_RECIPIENT_SLACK` line to contain a single hash `#` character. This instructs Netdata +to send a notification to the channel you configured with the incoming webhook. + +```conf +DEFAULT_RECIPIENT_SLACK="#" +``` + +To test Slack notifications, switch to the Netdata user. + +```bash +sudo su -s /bin/bash netdata +``` + +Next, run the `alarm-notify` script using the `test` option. + +```bash +/usr/libexec/netdata/plugins.d/alarm-notify.sh test +``` + +You should receive three notifications in your Slack channel for each health status change: `WARNING`, `CRITICAL`, and +`CLEAR`. + +See the [Agent Slack notifications](/health/notifications/slack/README.md) doc for more options and information. + +## What's next? + +Now that you have health entities configured to your infrastructure's needs and notifications to inform you of anomalies +or incidents, your health monitoring setup is complete. + +To make your dashboards most useful during root cause analysis, use Netdata's [distributed data +architecture](/docs/store/distributed-data-architecture.md) for the best-in-class performance and scalability. + +### Related reference documentation + +- [Netdata Cloud · Alarm notifications](https://learn.netdata.cloud/docs/cloud/alerts-notifications/notifications) +- [Netdata Agent · Notifications](/health/notifications/README.md) + diff --git a/docs/monitor/view-active-alarms.md b/docs/monitor/view-active-alarms.md new file mode 100644 index 0000000..be21826 --- /dev/null +++ b/docs/monitor/view-active-alarms.md @@ -0,0 +1,75 @@ + + +# View active health alarms + +Every Netdata Agent comes with hundreds of pre-installed health alarms designed to notify you when an anomaly or +performance issue affects your node or the applications it runs. + +## Netdata Cloud + +A War Room's [alarms indicator](https://learn.netdata.cloud/docs/cloud/war-rooms#indicators) displays the number of +active `critical` (red) and `warning` (yellow) alerts for the nodes in this War Room. Click on either the critical or +warning badges to open a pre-filtered modal displaying only those types of [active +alarms](https://learn.netdata.cloud/docs/cloud/alerts-notifications/view-active-alerts). + +![The Alarms panel in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/108564747-d2bfbb00-72c0-11eb-97b9-5863ad3324eb.png) + +The Alarms panel lists all active alarms for nodes within that War Room, and tells you which chart triggered the alarm, +what that chart's current value is, the alarm that triggered it, and when the alarm status first began. + +Use the input field in the Alarms panel to filter active alarms. You can sort by the node's name, alarm, status, chart +that triggered the alarm, or the operating system. Read more about the [filtering +syntax](https://learn.netdata.cloud/docs/cloud/war-rooms#node-filter) to build valuable filters for your infrastructure. + +Click on the 3-dot icon (`⋮`) to view active alarm information or navigate directly to the offending chart in that +node's Cloud dashboard with the **Go to chart** button. + +The active alarm information gives you details about the alarm that's been triggered. You can see the alarm's +configuration, how it calculates warning or critical alarms, and which configuration file you could edit on that node if +you want to tweak or disable the alarm to better suit your needs. + +![Active alarm details in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/108564813-f08d2000-72c0-11eb-80c8-b2af22a751fd.png) + +## Local Netdata Agent dashboard + +Find the alarms icon ![Alarms +icon](https://raw.githubusercontent.com/netdata/netdata-ui/98e31799c1ec0983f433537ff16d2ac2b0d994aa/src/components/icon/assets/alarm.svg) +in the top navigation to bring up a modal that shows currently raised alarms, all running alarms, and the alarms log. +Here is an example of a raised `system.cpu` alarm, followed by the full list and alarm log: + +![Animated GIF of looking at raised alarms and the alarm +log](https://user-images.githubusercontent.com/1153921/80842482-8c289500-8bb6-11ea-9791-600cfdbe82ce.gif) + +And a static screenshot of the raised CPU alarm: + +![Screenshot of a raised system CPU +alarm](https://user-images.githubusercontent.com/1153921/80842330-2dfbb200-8bb6-11ea-8147-3cd366eb0f37.png) + +The alarm itself is named **system - cpu**, and its context is `system.cpu`. Beneath that is an auto-updating badge that +shows the latest value of the chart that triggered the alarm. + +With the three icons beneath that and the **role** designation, you can: + +1. Scroll to the chart associated with this raised alarm. +2. Copy a link to the badge to your clipboard. +3. Copy the code to embed the badge onto another web page using an `` element. + +The table on the right-hand side displays information about the health entity that triggered the alarm, which you can +use as a reference to [configure alarms](/docs/monitor/configure-alarms.md). + +## What's next? + +With the information that appears on Netdata Cloud and the local dashboard about active alarms, you can [configure +alarms](/docs/monitor/configure-alarms.md) to match your infrastructure's needs or your team's goals. + +If you're happy with the pre-configured alarms, skip ahead to [enable +notifications](/docs/monitor/enable-notifications.md) to use Netdata Cloud's centralized alarm notifications and/or +per-node notifications to endpoints like Slack, PagerDuty, Twilio, and more. + + diff --git a/docs/netdata-for-IoT.md b/docs/netdata-for-IoT.md new file mode 100644 index 0000000..8d5bb21 --- /dev/null +++ b/docs/netdata-for-IoT.md @@ -0,0 +1,74 @@ + + +# Netdata for IoT + +![image1](https://cloud.githubusercontent.com/assets/2662304/14252446/11ae13c4-fa90-11e5-9d03-d93a3eb3317a.gif) + +> New to Netdata? Check its demo: **** +> +>[![User +>Base](https://registry.my-netdata.io/api/v1/badge.svg?chart=netdata.registry_entries&dimensions=persons&label=user%20base&units=null&value_color=blue&precision=0&v41)](https://registry.my-netdata.io/#netdata_registry) +>[![Monitored +>Servers](https://registry.my-netdata.io/api/v1/badge.svg?chart=netdata.registry_entries&dimensions=machines&label=servers%20monitored&units=null&value_color=orange&precision=0&v41)](https://registry.my-netdata.io/#netdata_registry) +>[![Sessions +>Served](https://registry.my-netdata.io/api/v1/badge.svg?chart=netdata.registry_sessions&label=sessions%20served&units=null&value_color=yellowgreen&precision=0&v41)](https://registry.my-netdata.io/#netdata_registry) +> +>[![New Users +>Today](https://registry.my-netdata.io/api/v1/badge.svg?chart=netdata.registry_entries&dimensions=persons&after=-86400&options=unaligned&group=incremental-sum&label=new%20users%20today&units=null&value_color=blue&precision=0&v40)](https://registry.my-netdata.io/#netdata_registry) +>[![New Machines +>Today](https://registry.my-netdata.io/api/v1/badge.svg?chart=netdata.registry_entries&dimensions=machines&group=incremental-sum&after=-86400&options=unaligned&label=servers%20added%20today&units=null&value_color=orange&precision=0&v40)](https://registry.my-netdata.io/#netdata_registry) +>[![Sessions +>Today](https://registry.my-netdata.io/api/v1/badge.svg?chart=netdata.registry_sessions&after=-86400&group=incremental-sum&options=unaligned&label=sessions%20served%20today&units=null&value_color=yellowgreen&precision=0&v40)](https://registry.my-netdata.io/#netdata_registry) + +--- + +Netdata is a [very efficient](/docs/guides/configure/performance.md) server performance monitoring solution. When running in server hardware, it can collect +thousands of system and application metrics **per second** with just 1% CPU utilization of a single core. Its web server +responds to most data requests in about **half a millisecond** making its web dashboards spontaneous, amazingly fast! + +Netdata can also be a very efficient real-time monitoring solution for **IoT devices** (RPIs, routers, media players, +wifi access points, industrial controllers and sensors of all kinds). Netdata will generally run everywhere a Linux +kernel runs (and it is glibc and [musl-libc](https://www.musl-libc.org/) friendly). + +You can use it as both a data collection agent (where you pull data using its API), for embedding its charts on other +web pages / consoles, but also for accessing it directly with your browser to view its dashboard. + +The Netdata web API already provides **reduce** functions allowing it to report **average** and **max** for any +timeframe. It can also respond in many formats including JSON, JSONP, CSV, HTML. Its API is also a **google charts** +provider so it can directly be used by google sheets, google charts, google widgets. + +![sensors](https://cloud.githubusercontent.com/assets/2662304/15339745/8be84540-1c8e-11e6-9e9a-106dea7539b6.gif) + +Although Netdata has been significantly optimized to lower the CPU and RAM resources it consumes, the plethora of data +collection plugins may be inappropriate for weak IoT devices. Please follow the [Netdata Agent performance +guide](/docs/guides/configure/performance.md) + +## Monitoring RPi temperature + +The python version of the sensors plugin uses `lm-sensors`. Unfortunately the temperature reading of RPi are not +supported by `lm-sensors`. + +Netdata also has a bash version of the sensors plugin that can read RPi temperatures. It is disabled by default to avoid +the conflicts with the python version. + +To enable it, run: + +```bash +cd /etc/netdata # Replace this path with your Netdata config directory +sudo ./edit-config charts.d.conf +``` + +and uncomment this line: + +```sh +sensors=force +``` + +Then restart Netdata. You will get this: + +![image](https://user-images.githubusercontent.com/2662304/29658868-23aa65ae-88c5-11e7-9dad-c159600db5cc.png) + + diff --git a/docs/netdata-security.md b/docs/netdata-security.md new file mode 100644 index 0000000..9bb26ad --- /dev/null +++ b/docs/netdata-security.md @@ -0,0 +1,229 @@ + + +# Security design + +We have given special attention to all aspects of Netdata, ensuring that everything throughout its operation is as secure as possible. Netdata has been designed with security in mind. + +**Table of Contents** + +1. [Your data is safe with Netdata](#your-data-is-safe-with-netdata) +2. [Your systems are safe with Netdata](#your-systems-are-safe-with-netdata) +3. [Netdata is read-only](#netdata-is-read-only) +4. [Netdata viewers authentication](#netdata-viewers-authentication) + * [Why Netdata should be protected](#why-netdata-should-be-protected) + * [Protect Netdata from the internet](#protect-netdata-from-the-internet) + * [Expose Netdata only in a private LAN](#expose-netdata-only-in-a-private-lan) + * [Use an authenticating web server in proxy mode](#use-an-authenticating-web-server-in-proxy-mode) + * [Other methods](#other-methods) +5. [Registry or how to not send any information to a third party server](#registry-or-how-to-not-send-any-information-to-a-third-party-server) + +## Your data is safe with Netdata + +Netdata collects raw data from many sources. For each source, Netdata uses a plugin that connects to the source (or reads the relative files produced by the source), receives raw data and processes them to calculate the metrics shown on Netdata dashboards. + +Even if Netdata plugins connect to your database server, or read your application log file to collect raw data, the product of this data collection process is always a number of **chart metadata and metric values** (summarized data for dashboard visualization). All Netdata plugins (internal to the Netdata daemon, and external ones written in any computer language), convert raw data collected into metrics, and only these metrics are stored in Netdata databases, sent to upstream Netdata servers, or archived to external time-series databases. + +> The **raw data** collected by Netdata, does not leave the host when collected. **The only data Netdata exposes are chart metadata and metric values.** + +This means that Netdata can safely be used in environments that require the highest level of data isolation (like PCI Level 1). + +## Your systems are safe with Netdata + +We are very proud that **the Netdata daemon runs as a normal system user, without any special privileges**. This is quite an achievement for a monitoring system that collects all kinds of system and application metrics. + +There are a few cases, however, that raw source data are only exposed to processes with escalated privileges. To support these cases, Netdata attempts to minimize and completely isolate the code that runs with escalated privileges. + +So, Netdata **plugins**, even those running with escalated capabilities or privileges, perform a **hard coded data collection job**. They do not accept commands from Netdata. The communication is strictly **unidirectional**: from the plugin towards the Netdata daemon. The original application data collected by each plugin do not leave the process they are collected, are not saved and are not transferred to the Netdata daemon. The communication from the plugins to the Netdata daemon includes only chart metadata and processed metric values. + +Child nodes use the same protocol when streaming metrics to their parent nodes. The raw data collected by the plugins of +child Netdata servers are **never leaving the host they are collected**. The only data appearing on the wire are chart +metadata and metric values. This communication is also **unidirectional**: child nodes never accept commands from +parent Netdata servers. + +## Netdata is read-only + +Netdata **dashboards are read-only**. Dashboard users can view and examine metrics collected by Netdata, but cannot instruct Netdata to do something other than present the already collected metrics. + +Netdata dashboards do not expose sensitive information. Business data of any kind, the kernel version, O/S version, application versions, host IPs, etc are not stored and are not exposed by Netdata on its dashboards. + +## Netdata viewers authentication + +Netdata is a monitoring system. It should be protected, the same way you protect all your admin apps. We assume Netdata will be installed privately, for your eyes only. + +### Why Netdata should be protected + +Viewers will be able to get some information about the system Netdata is running. This information is everything the dashboard provides. The dashboard includes a list of the services each system runs (the legends of the charts under the `Systemd Services` section), the applications running (the legends of the charts under the `Applications` section), the disks of the system and their names, the user accounts of the system that are running processes (the `Users` and `User Groups` section of the dashboard), the network interfaces and their names (not the IPs) and detailed information about the performance of the system and its applications. + +This information is not sensitive (meaning that it is not your business data), but **it is important for possible attackers**. It will give them clues on what to check, what to try and in the case of DDoS against your applications, they will know if they are doing it right or not. + +Also, viewers could use Netdata itself to stress your servers. Although the Netdata daemon runs unprivileged, with the minimum process priority (scheduling priority `idle` - lower than nice 19) and adjusts its OutOfMemory (OOM) score to 1000 (so that it will be first to be killed by the kernel if the system starves for memory), some pressure can be applied on your systems if someone attempts a DDoS against Netdata. + +### Protect Netdata from the internet + +Netdata is a distributed application. Most likely you will have many installations of it. Since it is distributed and you are expected to jump from server to server, there is very little usability to add authentication local on each Netdata. + +Until we add a distributed authentication method to Netdata, you have the following options: + +#### Expose Netdata only in a private LAN + +If your organisation has a private administration and management LAN, you can bind Netdata on this network interface on all your servers. This is done in `Netdata.conf` with these settings: + +``` +[web] + bind to = 10.1.1.1:19999 localhost:19999 +``` + +You can bind Netdata to multiple IPs and ports. If you use hostnames, Netdata will resolve them and use all the IPs (in the above example `localhost` usually resolves to both `127.0.0.1` and `::1`). + +**This is the best and the suggested way to protect Netdata**. Your systems **should** have a private administration and management LAN, so that all management tasks are performed without any possibility of them being exposed on the internet. + +For cloud based installations, if your cloud provider does not provide such a private LAN (or if you use multiple providers), you can create a virtual management and administration LAN with tools like `tincd` or `gvpe`. These tools create a mesh VPN allowing all servers to communicate securely and privately. Your administration stations join this mesh VPN to get access to management and administration tasks on all your cloud servers. + +For `gvpe` we have developed a [simple provisioning tool](https://github.com/netdata/netdata-demo-site/tree/master/gvpe) you may find handy (it includes statically compiled `gvpe` binaries for Linux and FreeBSD, and also a script to compile `gvpe` on your macOS system). We use this to create a management and administration LAN for all Netdata demo sites (spread all over the internet using multiple hosting providers). + +--- + +In Netdata v1.9+ there is also access list support, like this: + +``` +[web] + bind to = * + allow connections from = localhost 10.* 192.168.* +``` + +#### Fine-grained access control + +The access list support allows filtering of all incoming connections, by specific IP addresses, ranges +or validated DNS lookups. Only connections that match an entry on the list will be allowed: + +``` +[web] + allow connections from = localhost 192.168.* 1.2.3.4 homeip.net +``` + +Connections from the IP addresses are allowed if the connection IP matches one of the patterns given. +The alias localhost is always checked against 127.0.0.1, any other symbolic names need to resolve in +both directions using DNS. In the above example the IP address of `homeip.net` must reverse DNS resolve +to the incoming IP address and a DNS lookup on `homeip.net` must return the incoming IP address as +one of the resolved addresses. + +More specific control of what each incoming connection can do can be specified through the access control +list settings: + +``` +[web] + allow connections from = 160.1.* + allow badges from = 160.1.1.2 + allow streaming from = 160.1.2.* + allow management from = control.subnet.ip + allow netdata.conf from = updates.subnet.ip + allow dashboard from = frontend.subnet.ip +``` + +In this example only connections from `160.1.x.x` are allowed, only the specific IP address `160.1.1.2` +can access badges, only IP addresses in the smaller range `160.1.2.x` can stream data. The three +hostnames shown can access specific features, this assumes that DNS is setup to resolve these names +to IP addresses within the `160.1.x.x` range and that reverse DNS is setup for these hosts. + + +#### Use an authenticating web server in proxy mode + +Use one web server to provide authentication in front of **all your Netdata servers**. So, you will be accessing all your Netdata with URLs like `http://{HOST}/netdata/{NETDATA_HOSTNAME}/` and authentication will be shared among all of them (you will sign-in once for all your servers). Instructions are provided on how to set the proxy configuration to have Netdata run behind [nginx](Running-behind-nginx.md), [Apache](Running-behind-apache.md), [lighttpd](Running-behind-lighttpd.md) and [Caddy](Running-behind-caddy.md). + +To use this method, you should firewall protect all your Netdata servers, so that only the web server IP will be allowed to directly access Netdata. To do this, run this on each of your servers (or use your firewall manager): + +```sh +PROXY_IP="1.2.3.4" +iptables -t filter -I INPUT -p tcp --dport 19999 \! -s ${PROXY_IP} -m conntrack --ctstate NEW -j DROP +``` + +*commands to allow direct access to Netdata from a web server proxy* + +The above will prevent anyone except your web server to access a Netdata dashboard running on the host. + +For Netdata v1.9+ you can also use `netdata.conf`: + +``` +[web] + allow connections from = localhost 1.2.3.4 +``` + +Of course you can add more IPs. + +For Netdata prior to v1.9, if you want to allow multiple IPs, use this: + +```sh +# space separated list of IPs to allow access Netdata +NETDATA_ALLOWED="1.2.3.4 5.6.7.8 9.10.11.12" +NETDATA_PORT=19999 + +# create a new filtering chain || or empty an existing one named netdata +iptables -t filter -N netdata 2>/dev/null || iptables -t filter -F netdata +for x in ${NETDATA_ALLOWED} +do + # allow this IP + iptables -t filter -A netdata -s ${x} -j ACCEPT +done + +# drop all other IPs +iptables -t filter -A netdata -j DROP + +# delete the input chain hook (if it exists) +iptables -t filter -D INPUT -p tcp --dport ${NETDATA_PORT} -m conntrack --ctstate NEW -j netdata 2>/dev/null + +# add the input chain hook (again) +# to send all new Netdata connections to our filtering chain +iptables -t filter -I INPUT -p tcp --dport ${NETDATA_PORT} -m conntrack --ctstate NEW -j netdata +``` + +_script to allow access to Netdata only from a number of hosts_ + +You can run the above any number of times. Each time it runs it refreshes the list of allowed hosts. + +#### Other methods + +Of course, there are many more methods you could use to protect Netdata: + +- bind Netdata to localhost and use `ssh -L 19998:127.0.0.1:19999 remote.netdata.ip` to forward connections of local port 19998 to remote port 19999. This way you can ssh to a Netdata server and then use `http://127.0.0.1:19998/` on your computer to access the remote Netdata dashboard. + +- If you are always under a static IP, you can use the script given above to allow direct access to your Netdata servers without authentication, from all your static IPs. + +- install all your Netdata in **headless data collector** mode, forwarding all metrics in real-time to a parent + Netdata server, which will be protected with authentication using an nginx server running locally at the parent + Netdata server. This requires more resources (you will need a bigger parent Netdata server), but does not require + any firewall changes, since all the child Netdata servers will not be listening for incoming connections. + +## Anonymous Statistics + +### Registry or how to not send any information to a third party server + +The default configuration uses a public registry under registry.my-netdata.io (more information about the registry here: [mynetdata-menu-item](/registry/README.md) ). Please be aware that if you use that public registry, you submit the following information to a third party server: + +- The url where you open the web-ui in the browser (via http request referrer) +- The hostnames of the Netdata servers + +If sending this information to the central Netdata registry violates your security policies, you can configure Netdata to [run your own registry](/registry/README.md#run-your-own-registry). + +### Opt-out of anonymous statistics + +Starting with v1.30, Netdata collects anonymous usage information by default and sends it to a self hosted PostHog instance within the Netdata infrastructure. Read +about the information collected, and learn how to-opt, on our [anonymous statistics](anonymous-statistics.md) page. + +The usage statistics are _vital_ for us, as we use them to discover bugs and prioritize new features. We thank you for +_actively_ contributing to Netdata's future. + +## Netdata directories + +| path|owner|permissions|Netdata|comments| +|:---|:----|:----------|:------|:-------| +| `/etc/netdata`|user `root`
group `netdata`|dirs `0755`
files `0640`|reads|**Netdata config files**
may contain sensitive information, so group `netdata` is allowed to read them.| +| `/usr/libexec/netdata`|user `root`
group `root`|executable by anyone
dirs `0755`
files `0644` or `0755`|executes|**Netdata plugins**
permissions depend on the file - not all of them should have the executable flag.
there are a few plugins that run with escalated privileges (Linux capabilities or `setuid`) - these plugins should be executable only by group `netdata`.| +| `/usr/share/netdata`|user `root`
group `netdata`|readable by anyone
dirs `0755`
files `0644`|reads and sends over the network|**Netdata web static files**
these files are sent over the network to anyone that has access to the Netdata web server. Netdata checks the ownership of these files (using settings at the `[web]` section of `netdata.conf`) and refuses to serve them if they are not properly owned. Symbolic links are not supported. Netdata also refuses to serve URLs with `..` in their name.| +| `/var/cache/netdata`|user `netdata`
group `netdata`|dirs `0750`
files `0660`|reads, writes, creates, deletes|**Netdata ephemeral database files**
Netdata stores its ephemeral real-time database here.| +| `/var/lib/netdata`|user `netdata`
group `netdata`|dirs `0750`
files `0660`|reads, writes, creates, deletes|**Netdata permanent database files**
Netdata stores here the registry data, health alarm log db, etc.| +| `/var/log/netdata`|user `netdata`
group `root`|dirs `0755`
files `0644`|writes, creates|**Netdata log files**
all the Netdata applications, logs their errors or other informational messages to files in this directory. These files should be log rotated.| + + diff --git a/docs/overview/netdata-monitoring-stack.md b/docs/overview/netdata-monitoring-stack.md new file mode 100644 index 0000000..ae92522 --- /dev/null +++ b/docs/overview/netdata-monitoring-stack.md @@ -0,0 +1,62 @@ + + +# Use Netdata standalone or as part of your monitoring stack + +Netdata is an extremely powerful monitoring, visualization, and troubleshooting platform. While you can use it as an +effective standalone tool, we also designed it to be open and interoperable with other tools you might already be using. + +Netdata helps you collect everything and scales to infrastructure of any size, but it doesn't lock-in data or force you +to use specific tools or methodologies. Each feature is extensible and interoperable so they can work in parallel with +other tools. For example, you can use Netdata to collect metrics, visualize metrics with a second open-source program, +and centralize your metrics in a cloud-based time-series database solution for long-term storage or further analysis. + +You can build a new monitoring stack, including Netdata, or integrate Netdata's metrics with your existing monitoring +stack. No matter which route you take, Netdata helps you monitor infrastructure of any size. + +Here are a few ways to enrich your existing monitoring and troubleshooting stack with Netdata: + +## Collect metrics from Prometheus endpoints + +Netdata automatically detects 600 popular endpoints and collects per-second metrics from them via the [generic +Prometheus collector](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/prometheus). This even +includes support for Windows 10 via [`windows_exporter`](https://github.com/prometheus-community/windows_exporter). + +This collector is installed and enabled on all Agent installations by default, so you don't need to waste time +configuring Netdata. Netdata will detect these Prometheus metrics endpoints and collect even more granular metrics than +your existing solutions. You can now use all of Netdata's meaningfully-visualized charts to diagnose issues and +troubleshoot anomalies. + +## Export metrics to external time-series databases + +Netdata can send its per-second metrics to external time-series databases, such as InfluxDB, Prometheus, Graphite, +TimescaleDB, ElasticSearch, AWS Kinesis Data Streams, Google Cloud Pub/Sub Service, and many others. + +To [export metrics to external time-series databases](/docs/export/external-databases.md), you configure an [exporting +_connector_](/docs/export/enable-connector.md). These connectors support filtering and resampling for granular control +over which metrics you export, and at what volume. You can export resampled metrics as collected, as averages, or the +sum of interpolated values based on your needs and other monitoring tools. + +Once you have Netdata's metrics in a secondary time-series database, you can use them however you'd like, such as +additional visualization/dashboarding tools or aggregation of data from multiple sources. + +## Visualize metrics with Grafana + +One popular monitoring stack is Netdata, Graphite, and Grafana. Netdata acts as the stack's metrics collection +powerhouse, Graphite the time-series database, and Grafana the visualization platform. With Netdata at the core, you can +be confident that your monitoring stack is powered by all possible metrics, from all possible sources, from every node +in your infrastructure. + +Of course, just because you export or visualize metrics elsewhere, it doesn't mean Netdata's equivalent features +disappear. You can always build new dashboards in Netdata Cloud, drill down into per-second metrics using Netdata's +charts, or use Netdata's health watchdog to send notifications whenever an anomaly strikes. + +## What's next? + +Whether you're using Netdata standalone or as part of a larger monitoring stack, the next step is the same: [**Get +Netdata**](/docs/get-started.mdx). + + diff --git a/docs/overview/what-is-netdata.md b/docs/overview/what-is-netdata.md new file mode 100644 index 0000000..3df1d94 --- /dev/null +++ b/docs/overview/what-is-netdata.md @@ -0,0 +1,74 @@ + + +# What is Netdata? + +Netdata helps sysadmins, SREs, DevOps engineers, and IT professionals collect all possible metrics from systems and +applications, visualize these metrics in real-time, and troubleshoot complex performance problems. + +Netdata's solution uses two components, the Netdata Agent and Netdata Cloud, to deliver real-time performance and health +monitoring for both single nodes and entire infrastructure. + +## Netdata Agent + +Netdata's distributed monitoring Agent collects thousands of metrics from systems, hardware, and applications with zero +configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT +devices. + +You can [install](/docs/get-started.mdx) Netdata on most Linux distributions (Ubuntu, Debian, CentOS, and more), +container/microservice platforms (Kubernetes clusters, Docker), and many other operating systems (FreeBSD, macOS), with +no `sudo` required. + +![The Netdata +Agent](https://user-images.githubusercontent.com/1153921/94492596-72a86b00-019f-11eb-91ab-224e6ac9ea21.png) + +## Netdata Cloud + +Netdata Cloud is a web application that gives you real-time visibility for your entire infrastructure. With Netdata +Cloud, you can view key metrics, insightful charts, and active alarms from all your nodes in a single web interface. +When an anomaly strikes, seamlessly navigate to any node to troubleshoot and discover the root cause with the familiar +Netdata dashboard. + +**[Netdata Cloud is free](https://www.netdata.cloud/blog/why-netdata-is-free/)**! You can add an entire infrastructure +of nodes, invite all your colleagues, and visualize any number of metrics, charts, and alarms entirely for free. + +While Netdata Cloud offers a centralized method of monitoring your Agents, your metrics data is not stored or +centralized in any way. Metrics data remains with your nodes and is only streamed to your browser, through Cloud, when +you're viewing the Netdata Cloud interface. + +![Netdata Cloud](https://user-images.githubusercontent.com/1153921/94492597-73410180-019f-11eb-9a9e-032420baa489.png) + +## What you can do with Netdata + +Netdata is designed to be both simple to use and flexible for every monitoring, visualization, and troubleshooting use +case: + +- **Collect**: Netdata collects all available metrics from your system and applications with 300+ collectors, + Kubernetes service discovery, and in-depth container monitoring, all while using only 1% CPU and a few MB of RAM. It + even collects metrics from Windows machines. +- **Visualize**: The dashboard meaningfully presents charts to help you understand the relationships between your + hardware, operating system, running apps/services, and the rest of your infrastructure. Add nodes to Netdata Cloud + for a complete view of your infrastructure from a single pane of glass. +- **Monitor**: Netdata's health watchdog uses hundreds of preconfigured alarms to notify you via Slack, email, + PagerDuty and more when an anomaly strikes. Customize with dynamic thresholds, hysteresis, alarm templates, and + role-based notifications. +- **Troubleshoot**: 1s granularity helps you detect and analyze anomalies other monitoring platforms might have + missed. Interactive visualizations reduce your reliance on the console, and historical metrics help you trace issues + back to their root cause. +- **Store**: Netdata's efficient database engine efficiently stores per-second metrics for days, weeks, or even + months. Every distributed node stores metrics locally, simplifying deployment, slashing costs, and enriching + Netdata's interactive dashboards. +- **Export**: Integrate per-second metrics with other time-series databases like Graphite, Prometheus, InfluxDB, + TimescaleDB, and more with Netdata's interoperable and extensible core. +- **Stream**: Aggregate metrics from any number of distributed nodes in one place for in-depth analysis, including + ephemeral nodes in a Kubernetes cluster. + +## What's next? + +Learn more about [why you should use Netdata](/docs/overview/why-netdata.md), or [how Netdata works with your existing +monitoring stack](/docs/overview/netdata-monitoring-stack.md). + + diff --git a/docs/overview/why-netdata.md b/docs/overview/why-netdata.md new file mode 100644 index 0000000..9a308f2 --- /dev/null +++ b/docs/overview/why-netdata.md @@ -0,0 +1,63 @@ + + +# Why use Netdata? + +Netdata takes a different approach to helping people build extraordinary infrastructure. It was built out of frustration +with existing monitoring tools that are too complex, too expensive, and don't help their users actually troubleshoot +complex performance and health issues. + +Netdata is: + +## Simple to deploy + +- **One-line deployment** for Linux distributions, plus support for Kubernetes/Docker infrastructures. +- **Zero configuration and maintenance** required to collect thousands of metrics, every second, from the underlying + OS and running applications. +- **Prebuilt charts and alarms** alert you to common anomalies and performance issues without manual configuration. +- **Distributed storage** to simplify the cost and complexity of storing metrics data from any number of nodes. + +## Powerful and scalable + +- **1% CPU utilization, a few MB of RAM, and minimal disk I/O** to run the monitoring Agent on bare metal, virtual + machines, containers, and even IoT devices. +- **Per-second granularity** for an unlimited number of metrics based on the hardware and applications you're running + on your nodes. +- **Interoperable exporters** let you connect Netdata's per-second metrics with an existing monitoring stack and other + time-series databases. + +## Optimized for troubleshooting + +- **Visual anomaly detection** with a UI/UX that emphasizes the relationships between charts. +- **Customizable dashboards** to pinpoint correlated metrics, respond to incidents, and help you streamline your + workflows. +- **Distributed metrics in a centralized interface** to assist users or teams trace complex issues between distributed + nodes. + +## Comparison with other monitoring solutions + +Netdata offers many benefits over the existing monitoring landscape, whether they're expensive SaaS products or other +open-source tools. + +| Netdata | Others (open-source and commercial) | +| :-------------------------------------------------------------- | :--------------------------------------------------------------- | +| **High resolution metrics** (1s granularity) | Low resolution metrics (10s granularity at best) | +| Collects **thousands of metrics per node** | Collects just a few metrics | +| Fast UI optimized for **anomaly detection** | UI is good for just an abstract view | +| **Long-term, autonomous storage** at one-second granularity | Centralized metrics in an expensive data lake at 10s granularity | +| **Meaningful presentation**, to help you understand the metrics | You have to know the metrics before you start | +| Install and get results **immediately** | Long sales process and complex installation process | +| Use it for **troubleshooting** performance problems | Only gathers _statistics of past performance_ | +| **Kills the console** for tracing performance issues | The console is always required for troubleshooting | +| Requires **zero dedicated resources** | Require large dedicated resources | + +## What's next? + +Whether you already have a monitoring stack you want to integrate Netdata into, or are building something from the +ground-up, you should read more on how Netdata can work either [standalone or as an interoperable part of a monitoring +stack](/docs/overview/netdata-monitoring-stack.md). + + diff --git a/docs/quickstart/infrastructure.md b/docs/quickstart/infrastructure.md new file mode 100644 index 0000000..9db66c0 --- /dev/null +++ b/docs/quickstart/infrastructure.md @@ -0,0 +1,185 @@ + + +# Infrastructure monitoring with Netdata + +[Netdata Cloud](https://app.netdata.cloud) provides scalable infrastructure monitoring for any number of distributed +nodes running the Netdata Agent. A node is any system in your infrastructure that you want to monitor, whether it's a +physical or virtual machine (VM), container, cloud deployment, or edge/IoT device. + +The Netdata Agent uses zero-configuration collectors to gather metrics from every application and container instantly, +and uses Netdata's [distributed data architecture](/docs/store/distributed-data-architecture.md) to store metrics +locally. Without a slow and troublesome centralized data lake for your infrastructure's metrics, you reduce the +resources you need to invest in, and the complexity of, monitoring your infrastructure. + +Netdata Cloud unifies infrastructure monitoring by _centralizing the interface_ you use to query and visualize your +nodes' metrics, not the data. By streaming metrics values to your browser, with Netdata Cloud acting as the secure proxy +between them, you can monitor your infrastructure using customizable, interactive, and real-time visualizations from any +number of distributed nodes. + +In this quickstart guide, you'll learn the basics of using Netdata Cloud to monitor an infrastructure with dashboards, +composite charts, and alarm viewing. You'll then learn about the most critical ways to configure the Agent on each of +your nodes to maximize the value you get from Netdata. + +This quickstart assumes you've installed the Netdata Agent on more than one node in your infrastructure, and connected +those nodes to your Space in Netdata Cloud. If you haven't yet, see the [Netdata +Cloud](https://learn.netdata.cloud/docs/cloud) docs for details on signing up for Netdata Cloud, installation, and +connection process. + +> If you want to monitor a Kubernetes cluster with Netdata, see our [k8s installation +> doc](/packaging/installer/methods/kubernetes.md) for setup details, and then read our guide, [_Monitor a Kubernetes +> cluster with Netdata_](/docs/guides/monitor/kubernetes-k8s-netdata.md). + +## Set up your Netdata Cloud experience + +Start your infrastructure monitoring experience by setting up your Netdata Cloud account. + +### Organize Spaces and War Rooms + +Spaces are high-level containers to help you organize your team members and the nodes they can view in each War Room. +You already have at least one Space in your Netdata Cloud account. + +A single Space puts all your metrics in one easily-accessible place, while multiple Spaces creates logical division +between different users and different pieces of a large infrastructure. For example, a large organization might have one +SRE team for the user-facing SaaS application, and a second IT team for managing employees' hardware. Since these teams +don't monitor the same nodes, they can work in separate Spaces and then further organize their nodes into War Rooms. + +Next, set up War Rooms. Netdata Cloud creates dashboards and visualizations based on the nodes added to a given War +Room. You can [organize War Rooms](https://learn.netdata.cloud/docs/cloud/war-rooms#war-room-organization) in any way +you want, such as by the application type, for end-to-end application monitoring, or as an incident response tool. + +Learn more about [Spaces](https://learn.netdata.cloud/docs/cloud/spaces) and [War +Rooms](https://learn.netdata.cloud/docs/cloud/war-rooms), including how to manage each, in their respective reference +documentation. + +### Invite your team + +Netdata Cloud makes an infrastructure's real-time metrics available and actionable to all organization members. By +inviting others, you can better synchronize with your team or colleagues to understand your infrastructure's heartbeat. +When something goes wrong, you'll be ready to collaboratively troubleshoot complex performance problems from a single +pane of glass. + +To [invite new users](https://learn.netdata.cloud/docs/cloud/manage/invite-your-team), click on **Invite Users** in the +Space management Area. Choose which War Rooms to add this user to, then click **Send**. + +If your team members have trouble signing in, direct them to the [Netdata Cloud sign +in](https://learn.netdata.cloud/docs/cloud/manage/sign-in) doc. + +### See an overview of your infrastructure + +The default way to visualize the health and performance of an infrastructure with Netdata Cloud is the +[**Overview**](/docs/visualize/overview-infrastructure.md), which is the default interface of every War Room. The +Overview features composite charts, which display aggregated metrics from every node in a given War Room. These metrics +are streamed on-demand from individual nodes and composited onto a single, familiar dashboard. + +![The War Room +Overview](https://user-images.githubusercontent.com/1153921/108732681-09791980-74eb-11eb-9ba2-98cb1b6608de.png) + +Read more about the Overview in the [infrastructure overview](/docs/visualize/overview-infrastructure.md) doc. + +Netdata Cloud also features the [**Nodes view**](https://learn.netdata.cloud/docs/cloud/visualize/nodes), which you can +use to configure and see a few key metrics from every node in the War Room, view health status, and more. + +### Drill down to specific nodes + +Both the Overview and Nodes view offer easy access to **single-node dashboards** for targeted analysis. You can use +single-node dashboards in Netdata Cloud to drill down on specific issues, scrub backward in time to investigate +historical data, and see like metrics presented meaningfully to help you troubleshoot performance problems. + +Read about the process in the [infrastructure +overview](/docs/visualize/overview-infrastructure.md#drill-down-with-single-node-dashboards) doc, then learn about [interacting with +dashboards and charts](/docs/visualize/interact-dashboards-charts.md) to get the most from all of Netdata's real-time +metrics. + +### Create new dashboards + +You can use Netdata Cloud to create new dashboards that match your infrastructure's topology or help you diagnose +complex issues by aggregating correlated charts from any number of nodes. For example, you could monitor the system CPU +from every node in your infrastructure on a single dashboard. + +![An example system CPU +dashboard](https://user-images.githubusercontent.com/1153921/108732974-4b09c480-74eb-11eb-87a2-c67e569c08b6.png) + +Read more about [creating new dashboards](/docs/visualize/create-dashboards.md) for more details about the process and +additional tips on best leveraging the feature to help you troubleshoot complex performance problems. + +## Set up your nodes + +You get the most value out of Netdata Cloud's infrastructure monitoring capabilities if each node collects every +possible metric. For example, if a node in your infrastructure is responsible for serving a MySQL database, you should +ensure that the Netdata Agent on that node is properly collecting and streaming all MySQL-related metrics. + +In most cases, collectors autodetect their data source and require no configuration, but you may need to configure +certain behaviors based on your infrastructure. Or, you may want to enable/configure advanced functionality, such as +longer metrics retention or streaming. + +### Configure the Netdata Agent on your nodes + +You can configure any node in your infrastructure if you need to, although most users will find the default settings +work extremely well for monitoring their infrastructures. + +Each node has a configuration file called `netdata.conf`, which is typically at `/etc/netdata/netdata.conf`. The best +way to edit this file is using the `edit-config` script, which ensures updates to the Netdata Agent do not overwrite +your changes. For example: + +```bash +cd /etc/netdata +sudo ./edit-config netdata.conf +``` + +Our [configuration basics doc](/docs/configure/nodes.md) contains more information about `netdata.conf`, `edit-config`, +along with simple examples to get you familiar with editing your node's configuration. + +After you've learned the basics, you should [secure your infrastructure's nodes](/docs/configure/secure-nodes.md) using +one of our recommended methods. These security best practices ensure no untrusted parties gain access to the metrics +collected on any of your nodes. + +### Collect metrics from systems and applications + +Netdata has [300+ pre-installed collectors](/collectors/COLLECTORS.md) that gather thousands of metrics with zero +configuration. Collectors search each of your nodes in default locations and ports to find running applications and +gather as many metrics as they can without you having to configure them individually. + +Most collectors work without configuration, but you should read up on [how collectors +work](/docs/collect/how-collectors-work.md) and [how to enable/configure](/docs/collect/enable-configure.md) them so +that you can see metrics from those applications in Netdata Cloud. + +In addition, find detailed information about which [system](/docs/collect/system-metrics.md), +[container](/docs/collect/container-metrics.md), and [application](/docs/collect/application-metrics.md) metrics you can +collect from across your infrastructure with Netdata. + +## What's next? + +Netdata has many features that help you monitor the health of your nodes and troubleshoot complex performance problems. +Once you have a handle on configuration and are collecting all the right metrics, try out some of Netdata's other +infrastructure-focused features: + +- [See an overview of your infrastructure](/docs/visualize/overview-infrastructure.md) using Netdata Cloud's composite + charts and real-time visualizations. +- [Create new dashboards](/docs/visualize/create-dashboards.md) from any number of nodes and metrics in Netdata Cloud. + +To change how the Netdata Agent runs on each node, dig in to configuration files: + +- [Change how long nodes in your infrastructure retain metrics](/docs/store/change-metrics-storage.md) based on how + many metrics each node collects, your preferred retention period, and the resources you want to dedicate toward + long-term metrics retention. +- [Create new alarms](/docs/monitor/configure-alarms.md), or tweak some of the pre-configured alarms, to stay on top + of anomalies. +- [Enable notifications](/docs/monitor/enable-notifications.md) to Slack, PagerDuty, email, and 30+ other services. +- [Export metrics](/docs/export/external-databases.md) to an external time-series database to use Netdata alongside + other monitoring and troubleshooting tools. + +### Related reference documentation + +- [Netdata Cloud · Spaces](https://learn.netdata.cloud/docs/cloud/spaces) +- [Netdata Cloud · War Rooms](https://learn.netdata.cloud/docs/cloud/war-rooms) +- [Netdata Cloud · Invite your team](https://learn.netdata.cloud/docs/cloud/manage/invite-your-team) +- [Netdata Cloud · Sign in or sign up with email, Google, or + GitHub](https://learn.netdata.cloud/docs/cloud/manage/sign-in) +- [Netdata Cloud · Nodes view](https://learn.netdata.cloud/docs/cloud/visualize/nodes) + + diff --git a/docs/quickstart/single-node.md b/docs/quickstart/single-node.md new file mode 100644 index 0000000..7855a48 --- /dev/null +++ b/docs/quickstart/single-node.md @@ -0,0 +1,92 @@ + + +# Single-node monitoring with Netdata + +Because it's free, open-source, and requires only 1% CPU utilization to collect thousands of metrics every second, +Netdata is a superb single-node monitoring tool. + +In this quickstart guide, you'll learn how to access your single node's metrics through dashboards, configure your node +to your liking, and make sure the Netdata Agent is collecting metrics from the applications or containers you're running +on your node. + +## See your node's metrics + +To see your node's real-time metrics, you need to access its dashboard. You can either view the local dashboard, which +runs on the node itself, or see the dashboard through Netdata Cloud. Both methods feature real-time, interactive, and +synchronized charts, with the same metrics, and use the same UI. + +The primary difference is that Netdata Cloud also has a few extra features, like creating new dashboards using a +drag-and-drop editor, that enhance your monitoring and troubleshooting experience. + +To see your node's local dashboard, open up your web browser of choice and navigate to `http://NODE:19999`, replacing +`NODE` with the IP address or hostname of your Agent. Hit `Enter`. + +![Animated GIF of navigating to the +dashboard](https://user-images.githubusercontent.com/1153921/80825153-abaec600-8b94-11ea-8b17-1b770a2abaa9.gif) + +To see a node's dashboard in Netdata Cloud, [sign in](https://app.netdata.cloud). From the **Nodes** view in your +**General** War Room, click on the hostname of your node to access its dashboard through Netdata Cloud. + +![Screenshot of an embedded node +dashboard](https://user-images.githubusercontent.com/1153921/87457036-9b678e00-c5bc-11ea-977d-ad561a73beef.png) + +Once you've decided which dashboard you prefer, learn about [interacting with dashboards and +charts](/docs/visualize/interact-dashboards-charts.md) to get the most from Netdata's real-time metrics. + +## Configure your node + +The Netdata Agent is highly configurable so that you can match its behavior to your node. You will find most +configuration options in the `netdata.conf` file, which is typically at `/etc/netdata/netdata.conf`. The best way to +edit this file is using the `edit-config` script, which ensures updates to the Netdata Agent do not overwrite your +changes. For example: + +```bash +cd /etc/netdata +sudo ./edit-config netdata.conf +``` + +Our [configuration basics doc](/docs/configure/nodes.md) contains more information about `netdata.conf`, `edit-config`, +along with simple examples to get you familiar with editing your node's configuration. + +After you've learned the basics, you should [secure your node](/docs/configure/secure-nodes.md) using one of our +recommended methods. These security best practices ensure no untrusted parties gain access to your dashboard or its +metrics. + +## Collect metrics from your system and applications + +Netdata has [300+ pre-installed collectors](/collectors/COLLECTORS.md) that gather thousands of metrics with zero +configuration. Collectors search your node in default locations and ports to find running applications and gather as +many metrics as possible without you having to configure them individually. + +These metrics enrich both the local and Netdata Cloud dashboards. + +Most collectors work without configuration, but you should read up on [how collectors +work](/docs/collect/how-collectors-work.md) and [how to enable/configure](/docs/collect/enable-configure.md) them. + +In addition, find detailed information about which [system](/docs/collect/system-metrics.md), +[container](/docs/collect/container-metrics.md), and [application](/docs/collect/application-metrics.md) metrics you can +collect from across your infrastructure with Netdata. + +## What's next? + +Netdata has many features that help you monitor the health of your node and troubleshoot complex performance problems. +Once you understand configuration, and are certain Netdata is collecting all the important metrics from your node, try +out some of Netdata's other visualization and health monitoring features: + +- [Build new dashboards](/docs/visualize/create-dashboards.md) to put disparate but relevant metrics onto a single + interface. +- [Create new alarms](/docs/monitor/configure-alarms.md), or tweak some of the pre-configured alarms, to stay on top + of anomalies. +- [Enable notifications](/docs/monitor/enable-notifications.md) to Slack, PagerDuty, email, and 30+ other services. +- [Change how long your node stores metrics](/docs/store/change-metrics-storage.md) based on how many metrics it + collects, your preferred retention period, and the resources you want to dedicate toward long-term metrics + retention. +- [Export metrics](/docs/export/external-databases.md) to an external time-series database to use Netdata alongside + other monitoring and troubleshooting tools. + + diff --git a/docs/store/change-metrics-storage.md b/docs/store/change-metrics-storage.md new file mode 100644 index 0000000..c4b77d9 --- /dev/null +++ b/docs/store/change-metrics-storage.md @@ -0,0 +1,98 @@ + + +# Change how long Netdata stores metrics + +The Netdata Agent uses a custom made time-series database (TSDB), named the [`dbengine`](/database/engine/README.md), to store metrics. + +The default settings retain approximately two day's worth of metrics on a system collecting 2,000 metrics every second, +but the Netdata Agent is highly configurable if you want your nodes to store days, weeks, or months worth of per-second +data. + +The Netdata Agent uses the following three fundamental settings in `netdata.conf` to change the behavior of the database engine: + +```conf +[global] + dbengine page cache size = 32 + dbengine multihost disk space = 256 + storage tiers = 1 +``` + +`dbengine page cache size` sets the maximum amount of RAM (in MiB) the database engine uses to cache and index recent +metrics. +`dbengine multihost disk space` sets the maximum disk space (again, in MiB) the database engine uses to store +historical, compressed metrics and `storage tiers` specifies the number of storage tiers you want to have in +your `dbengine`. When the size of stored metrics exceeds the allocated disk space, the database engine removes the +oldest metrics on a rolling basis. + +## Calculate the system resources (RAM, disk space) needed to store metrics + +You can store more or less metrics using the database engine by changing the allocated disk space. Use the calculator +below to find the appropriate value for the `dbengine` based on how many metrics your node(s) collect, whether you are +streaming metrics to a parent node, and more. + +You do not need to edit the `dbengine page cache size` setting to store more metrics using the database engine. However, +if you want to store more metrics _specifically in memory_, you can increase the cache size. + +:::tip + +We advise you to visit the [tiering mechanism](/database/engine/README.md#tiering) reference. This will help you +configure the Agent to retain metrics for longer periods. + +::: + +:::caution + +This calculator provides an estimation of disk and RAM usage for **metrics usage**. Real-life usage may vary based on +the accuracy of the values you enter below, changes in the compression ratio, and the types of metrics stored. + +::: + +Visit the [Netdata Storage Calculator](https://netdata-storage-calculator.herokuapp.com/) app to customize +data retention according to your preferences. + +## Edit `netdata.conf` with recommended database engine settings + +Now that you have a recommended setting for your Agent's `dbengine`, open `netdata.conf` with +[`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) and look for the `[db]` +subsection. Change it to the recommended values you calculated from the calculator. For example: + +```conf +[db] + mode = dbengine + storage tiers = 3 + update every = 1 + dbengine multihost disk space MB = 1024 + dbengine page cache size MB = 32 + dbengine tier 1 update every iterations = 60 + dbengine tier 1 multihost disk space MB = 384 + dbengine tier 1 page cache size MB = 32 + dbengine tier 2 update every iterations = 60 + dbengine tier 2 multihost disk space MB = 16 + dbengine tier 2 page cache size MB = 32 +``` + +Save the file and restart the Agent with `sudo systemctl restart netdata`, or +the [appropriate method](/docs/configure/start-stop-restart.md) for your system, to change the database engine's size. + +## What's next? + +If you have multiple nodes with the Netdata Agent installed, you +can [stream metrics](/docs/metrics-storage-management/how-streaming-works.mdx) from any number of _child_ nodes to a _ +parent_ node and store metrics using a centralized time-series database. Streaming allows you to centralize your data, +run Agents as headless collectors, replicate data, and more. + +Storing metrics with the database engine is completely interoperable +with [exporting to other time-series databases](/docs/export/external-databases.md). With exporting, you can use the +node's resources to surface metrics when [viewing dashboards](/docs/visualize/interact-dashboards-charts.md), while also +archiving metrics elsewhere for further analysis, visualization, or correlation with other tools. + +### Related reference documentation + +- [Netdata Agent · Database engine](/database/engine/README.md) +- [Netdata Agent · Database engine configuration option](/daemon/config/README.md#[db]-section-options) + + diff --git a/docs/store/distributed-data-architecture.md b/docs/store/distributed-data-architecture.md new file mode 100644 index 0000000..62933cf --- /dev/null +++ b/docs/store/distributed-data-architecture.md @@ -0,0 +1,88 @@ + + +# Distributed data architecture + +Netdata uses a distributed data architecture to help you collect and store per-second metrics from any number of nodes. +Every node in your infrastructure, whether it's one or a thousand, stores the metrics it collects. + +Netdata Cloud bridges the gap between many distributed databases by _centralizing the interface_ you use to query and +visualize your nodes' metrics. When you [look at charts in Netdata Cloud](/docs/visualize/interact-dashboards-charts.md) +, the metrics values are queried directly from that node's database and securely streamed to Netdata Cloud, which +proxies them to your browser. + +Netdata's distributed data architecture has a number of benefits: + +- **Performance**: Every query to a node's database takes only a few milliseconds to complete for responsiveness when + viewing dashboards or using features + like [Metric Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations). +- **Scalability**: As your infrastructure scales, install the Netdata Agent on every new node to immediately add it to + your monitoring solution without adding cost or complexity. +- **1-second granularity**: Without an expensive centralized data lake, you can store all of your nodes' per-second + metrics, for any period of time, while keeping costs down. +- **No filtering or selecting of metrics**: Because Netdata's distributed data architecture allows you to store all + metrics, you don't have to configure which metrics you retain. Keep everything for full visibility during + troubleshooting and root cause analysis. +- **Easy maintenance**: There is no centralized data lake to purchase, allocate, monitor, and update, removing + complexity from your monitoring infrastructure. + +## Ephemerality of metrics + +The ephemerality of metrics plays an important role in retention. In environments where metrics collection is dynamic and +new metrics are constantly being generated, we are interested about 2 parameters: + +1. The **expected concurrent number of metrics** as an average for the lifetime of the database. This affects mainly the + storage requirements. + +2. The **expected total number of unique metrics** for the lifetime of the database. This affects mainly the memory + requirements for having all these metrics indexed and available to be queried. + +## Granularity of metrics + +The granularity of metrics (the frequency they are collected and stored, i.e. their resolution) is significantly +affecting retention. + +Lowering the granularity from per second to every two seconds, will double their retention and half the CPU requirements +of the Netdata Agent, without affecting disk space or memory requirements. + +## Long-term metrics storage with Netdata + +Any node running the Netdata Agent can store long-term metrics for any retention period, given you allocate the +appropriate amount of RAM and disk space. + +Read our document on changing [how long Netdata stores metrics](/docs/store/change-metrics-storage.md) on your nodes for +details. + +You can also stream between nodes using [streaming](/streaming/README.md), allowing to replicate databases and create +your own centralized data lake of metrics, if you choose to do so. + +While a distributed data architecture is the default when monitoring infrastructure with Netdata, you can also configure +its behavior based on your needs or the type of infrastructure you manage. + +To archive metrics to an external time-series database, such as InfluxDB, Graphite, OpenTSDB, Elasticsearch, +TimescaleDB, and many others, see details on [integrating Netdata via exporting](/docs/export/external-databases.md). + +When you use the database engine to store your metrics, you can always perform a quick backup of a node's +`/var/cache/netdata/dbengine/` folder using the tool of your choice. + +## Does Netdata Cloud store my metrics? + +Netdata Cloud does not store metric values. + +To enable certain features, such as [viewing active alarms](/docs/monitor/view-active-alarms.md) +or [filtering by hostname/service](https://learn.netdata.cloud/docs/cloud/war-rooms#node-filter), Netdata Cloud does +store configured alarms, their status, and a list of active collectors. + +Netdata does not and never will sell your personal data or data about your deployment. + +## What's next? + +You can configure the Netdata Agent to store days, weeks, or months worth of distributed, per-second data by +[configuring the database engine](/docs/store/change-metrics-storage.md). Use our calculator to determine the system +resources required to retain your desired amount of metrics, and expand or contract the database by editing a single +setting. + + diff --git a/docs/visualize/create-dashboards.md b/docs/visualize/create-dashboards.md new file mode 100644 index 0000000..696cd1a --- /dev/null +++ b/docs/visualize/create-dashboards.md @@ -0,0 +1,64 @@ + + +# Create new dashboards + +With Netdata Cloud, you can build new dashboards that put key metrics from any number of distributed systems in one +place for a bird's eye view of your infrastructure. You can create more meaningful visualizations for troubleshooting or +keep a watchful eye on your infrastructure's most meaningful metrics without moving from node to node. + +In the War Room you want to monitor with this dashboard, click on your War Room's dropdown, then click on the green **+ +Add** button next to **Dashboards**. In the panel, give your new dashboard a name, and click **+ Add**. + +Click the **Add Chart** button to add your first chart card. From the dropdown, select the node you want to add the +chart from, then the context. Netdata Cloud shows you a preview of the chart before you finish adding it. + +The **Add Text** button creates a new card with user-defined text, which you can use to describe or document a +particular dashboard's meaning and purpose. Enrich the dashboards you create with documentation or procedures on how to +respond + +![A bird's eye dashboard for a single +node](https://user-images.githubusercontent.com/1153921/102650776-a654ba80-4128-11eb-9a65-4f9801b03d4b.png) + +Charts in dashboards are [fully interactive](/docs/visualize/interact-dashboards-charts.md) and synchronized. You can +pan through time, zoom, highlight specific timeframes, and more. + +Move any card by clicking on their top panel and dragging them to a new location. Other cards re-sort to the grid system +automatically. You can also resize any card by grabbing the bottom-right corner and dragging it to its new size. + +Hit the **Save** button to finalize your dashboard. Any other member of the War Room can now access it and make changes. + +## Jump to single-node Cloud dashboards + +While dashboards help you associate essential charts from distributed nodes on a single pane of glass, you might need +more detail when troubleshooting an issue. Quickly jump to any node's dashboard by clicking the 3-dot icon in the corner +of any card to open a menu. Hit the **Go to Chart** item. + +Netdata Cloud takes you to the same chart on that node's dashboard. You can now navigate all that node's metrics and +[interact with charts](/docs/visualize/interact-dashboards-charts.md) to further investigate anomalies or troubleshoot +complex performance problems. + +When viewing a single-node Cloud dashboard, you can also click on the add to dashboard icon Dashboard +icon to quickly add that chart to a new or existing dashboard. You might find this useful when +investigating an anomaly and want to quickly populate a dashboard with potentially correlated metrics. + +## Pin dashboards and navigate through Netdata Cloud + +Click on the **Pin** button in any dashboard to put those charts into a separate panel at the bottom of the screen. You +can now navigate through Netdata Cloud freely, individual Cloud dashboards, the Nodes view, different War Rooms, or even +different Spaces, and have those valuable metrics follow you. + +Pinning dashboards helps you correlate potentially related charts across your infrastructure and discover root causes +faster. + +## What's next? + +While it's useful to see real-time metrics on flexible dashboards, you need ways to know precisely when an anomaly +strikes. Every Netdata Agent comes with a health watchdog that uses [alarms](/docs/monitor/configure-alarms.md) and +[notifications](/docs/monitor/enable-notifications.md) to notify you of issues seconds after they strike. + + diff --git a/docs/visualize/interact-dashboards-charts.md b/docs/visualize/interact-dashboards-charts.md new file mode 100644 index 0000000..5dbbf19 --- /dev/null +++ b/docs/visualize/interact-dashboards-charts.md @@ -0,0 +1,131 @@ + + +# Interact with dashboards and charts + +> ⚠️ There is a new version of charts that is currently **only** available on [Netdata Cloud](https://learn.netdata.cloud/docs/cloud/visualize/interact-new-charts). We didn't +> want to keep this valuable feature from you, so after we get this into your hands on the Cloud, we will collect and implement your feedback to make sure we are providing the best possible version of the feature on the Netdata Agent dashboard as quickly as possible. + +You can find Netdata's dashboards in two places: locally served at `http://NODE:19999` by the Netdata Agent, and in +Netdata Cloud. While you access these dashboards differently, they have similar interfaces, identical charts and +metrics, and you interact with both of them the same way. + +> If you're not sure which option is best for you, see our [single-node](/docs/quickstart/single-node.md) and +> [infrastructure](/docs/quickstart/infrastructure.md) quickstart guides. + +Netdata dashboards are single, scrollable pages with many charts stacked on top of one another. As you scroll up or +down, charts appearing in your browser's viewport automatically load and update every second. + +The dashboard is broken up into multiple **sections**, such as **System Overview**, **CPU**, **Disk**, which are +automatically generated based on which [collectors](/docs/collect/how-collectors-work.md) begin collecting metrics when +Netdata starts up. Sections also appear in the right-hand **menu**, along with submenus based on the contexts and +families Netdata creates for your node. + +## Choose timeframes to visualize + +Both the local Agent dashboard and Netdata Cloud feature time & date pickers to help you visualize specific points in +time. In Netdata Cloud, the picker appears in the [Overview](/docs/visualize/overview-infrastructure.md), [Nodes +view](https://learn.netdata.cloud/docs/cloud/visualize/nodes), [new +dashboards](https://learn.netdata.cloud/docs/cloud/visualize/dashboards), and any single-node dashboards you visit. + +Local Agent dashboard: + +![Time & date picker on the local Netdata +dashboard](https://user-images.githubusercontent.com/1153921/101512538-5875d080-3938-11eb-8daf-0fbd0948a04b.png) + +Netdata Cloud: + +![Time & date picker on Netdata +Cloud](https://user-images.githubusercontent.com/1153921/101512689-86f3ab80-3938-11eb-8abc-12171a9b8a5e.png) + +Their behavior is identical. Use the Quick Selector to visualize generic timeframes, or use the calendar or inputs to +select days, hours, minutes or seconds. Click **Apply** to re-render all visualizations with new metrics data, or +**Clear** to restore the default timeframe. + +See reference documentation for the [local Agent dashboard](/web/gui/README.md#time--date-picker) and [Netdata +Cloud](https://learn.netdata.cloud/docs/cloud/war-rooms#time--date-picker) for additional context about how the time & +date picker behaves in each environment. + +## Charts, dimensions, families, and contexts + +A **chart** is an interactive visualization of one or more collected/calculated metrics. You can see the name (also +known as its unique ID) of a chart by looking at the top-left corner of a chart and finding the parenthesized text. On a +Linux system, one of the first charts on the dashboard will be the system CPU chart, with the name `system.cpu`. + +A **dimension** is any value that gets shown on a chart. The value can be raw data or calculated values, such as +percentages, aggregates, and more. Most charts will have more than one dimension, in which case it will display each in +a different color. You can disable or enable showing these dimensions by clicking on them. + +A **family** is _one_ instance of a monitored hardware or software resource that needs to be monitored and displayed +separately from similar instances. For example, if your node has multiple partitions, Netdata will create different +families for `/`, `/boot`, `/home`, and so on. Same goes for entire disks, network devices, and more. + +A **context** groups several charts based on the types of metrics being collected and displayed. For example, the +**Disk** section often has many contexts: `disk.io`, `disk.ops`, `disk.backlog`, `disk.util`, and so on. Netdata uses +this context to create individual charts and then groups them by family. You can always see the context of any chart by +looking at its name or hovering over the chart's date. + +See our [dashboard docs](/web/README.md#charts-contexts-families) for more information about the above distinctions +and how they're used across Netdata to meaningfully organize and present metrics. + +## Interact with charts + +Netdata's charts are fully interactive to help you find meaningful information about complex problems. You can pan +through historical metrics, zoom in and out, select specific timeframes for further analysis, resize charts, and more. +Whenever you use a chart in this way, Netdata synchronizes all the other charts to match it. + +| Change | Method #1 | Method #2 | Method #3 | +| ------------------------------------------------- | ----------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------- | +| **Stop** a chart from updating | `click` | | | +| **Reset** charts to default auto-refreshing state | `double click` | `double tap` (touchpad/touchscreen) | | +| **Select** a certain timeframe | `ALT` + `mouse selection` | `⌘` + `mouse selection` (macOS) | | +| **Pan** forward or back in time | `click and drag` | `touch and drag` (touchpad/touchscreen) | | +| **Zoom** to a specific timeframe | `SHIFT` + `mouse selection` | | | +| **Zoom** in/out | `SHIFT`/`ALT` + `mouse scrollwheel` | `SHIFT`/`ALT` + `two-finger pinch` (touchpad/touchscreen) | `SHIFT`/`ALT` + `two-finger scroll` (touchpad/touchscreen) | + +![Animated GIF of interacting with Netdata +charts](https://user-images.githubusercontent.com/1153921/102652236-051b3380-412b-11eb-8f7c-a2372ed92cd0.gif) + +These interactions can also be triggered using the icons on the bottom-right corner of every chart. They are, +respectively, `Pan Left`, `Reset`, `Pan Right`, `Zoom In`, and `Zoom Out`. + +You can show and hide individual dimensions by clicking on their names. Use `SHIFT + click` to hide or show dimensions +one at a time. Hiding dimensions simplifies the chart and can help you better discover exactly which aspect of your +system is behaving strangely. + +You can resize any chart by clicking-and-dragging the icon on the bottom-right corner of any chart. To restore the chart +to its original height, double-click the same icon. + +![Resizing a chart and resetting it to the default +height](https://user-images.githubusercontent.com/1153921/102652691-24b25c00-412b-11eb-9e2c-95325fcedc67.gif) + +### Composite charts in Netdata Cloud + +Netdata Cloud now supports composite charts in the Overview interface. Composite charts come with a few additional UI +elements and varied interactions, such as the location of dimensions and a utility bar for configuring the state of +individual composite charts. All of these details are covered in the [Overview +reference](https://learn.netdata.cloud/docs/cloud/visualize/overview) doc. + +## What's next? + +Netdata Cloud users can [build new dashboards](/docs/visualize/create-dashboards.md) in just a few clicks. By +aggregating relevant metrics from any number of nodes onto a single interface, you can respond faster to anomalies, +perform more targeted troubleshooting, or keep tabs on a bird's eye view of your infrastructure. + +If you're finished with dashboards for now, skip to Netdata's health watchdog for information on [creating or +configuring](/docs/monitor/configure-alarms.md) alarms, and [send notifications](/docs/monitor/enable-notifications.md) +to get informed when something goes wrong in your infrastructure. + +### Related reference documentation + +- [Netdata Agent · Web dashboards overview](/web/README.md) +- [Netdata Cloud · Interact with new charts](https://learn.netdata.cloud/docs/cloud/visualize/interact-new-charts) +- [Netdata Cloud · War Rooms](https://learn.netdata.cloud/docs/cloud/war-rooms) +- [Netdata Cloud · Overview](https://learn.netdata.cloud/docs/cloud/visualize/overview) +- [Netdata Cloud · Nodes](https://learn.netdata.cloud/docs/cloud/visualize/nodes) +- [Netdata Cloud · Build new dashboards](https://learn.netdata.cloud/docs/cloud/visualize/dashboards) + + diff --git a/docs/visualize/overview-infrastructure.md b/docs/visualize/overview-infrastructure.md new file mode 100644 index 0000000..4edbb0f --- /dev/null +++ b/docs/visualize/overview-infrastructure.md @@ -0,0 +1,111 @@ + + +# See an overview of your infrastructure + +In Netdata Cloud, your nodes are organized into War Rooms. One of the two available views for a War Room is the +[**Overview**](https://learn.netdata.cloud/docs/cloud/visualize/overview), which uses composite charts to display +real-time, aggregated metrics from all the nodes (or a filtered selection) in a given War Room. + +With Overview's composite charts, you can see your infrastructure from a single pane of glass, discover trends or +anomalies, then drill down with filtering or single-node dashboards to see more. In the screenshot below, +each chart visualizes average or sum metrics values from across 5 distributed nodes. + +Netdata also supports robust Kubernetes monitoring using the Overview. Read our [deployment +doc](/packaging/installer/methods/kubernetes.md) for details on visualizing Kubernetes metrics in Netdata Cloud. + +![The War Room +Overview](https://user-images.githubusercontent.com/1153921/108732681-09791980-74eb-11eb-9ba2-98cb1b6608de.png) + +## Using the Overview + +The Overview uses roughly the same interface as local Agent dashboards or single-node dashboards in Netdata Cloud. By +showing all available metrics from all your nodes in a single interface, Netdata Cloud helps you visualize the overall +health of your infrastructure. Best of all, you don't have to worry about creating your own dashboards just to get +started with infrastructure monitoring. + +Let's walk through some examples of using the Overview to monitor and troubleshoot your infrastructure. + +### Filter nodes and pick relevant times + +While not exclusive to Overview, you can use two important features, [node +filtering](https://learn.netdata.cloud/docs/cloud/war-rooms#node-filter) and the [time & date +picker](https://learn.netdata.cloud/docs/cloud/war-rooms#time--date-picker), to widen or narrow your infrastructure +monitoring focus. + +By default, the Overview shows composite charts aggregated from every node in the War Room, but you can change that +behavior on an ad-hoc basis. The node filter allows you to create complex queries against your infrastructure based on +the name, OS, or services running on nodes. For example, use `(name contains aws AND os contains ubuntu) OR services == +apache` to show only nodes that have `aws` in the hostname and are Ubuntu-based, or any nodes that have an Apache +webserver running on them. + +The time & date picker helps you visualize both small and large timeframes depending on your goals, whether that's +establishing a baseline of infrastructure performance or targeted root cause analysis of a specific anomaly. + +For example, use the **Quick Selector** options to pick the 12-hour option first thing in the morning to check your +infrastructure for any odd behavior overnight. Use the 7-day option to observe trends between various days of the week. + +See the [War Rooms](https://learn.netdata.cloud/docs/cloud/war-rooms) docs for more details on both features. + +### Configure composite charts to identify problems + +Let's say you notice a sharp decrease in available RAM for applications, as seen in the example screenshot below. In +this situation, you can see when the anomalous behavior began and that it affects the average available and committed +RAM across your infrastructure. However, when _grouped by dimension_, composite charts cannot show whether an anomaly +affects a single node, a subset of nodes, or an entire infrastructure. + +![Composite charts showing available and committed RAM across an +infrastructure](https://user-images.githubusercontent.com/1153921/99314892-0bae4680-281f-11eb-823e-071a1da25dc7.png) + +Use [_group by node_](https://learn.netdata.cloud/docs/cloud/visualize/overview#group-by-dimension-or-node) to visualize +a single metric across all contributing nodes. If the composite chart has 5 contributing nodes, there will be 5 +lines/areas, one for the most relevant dimension from each node. + +![Finding a problematic node with group by +node](https://user-images.githubusercontent.com/1153921/99315558-0e5d6b80-2820-11eb-91e9-9c46bc4c7298.gif) + +After grouping by node, it's clear that the `Composite-Charts-01` node is experiencing anomalous behavior and should be +investigated further by jumping to its [single-node dashboard](#drill-down-with-single-node-dashboards) in Netdata +Cloud. + +### Drill down with single-node dashboards + +Click on **X Charts** of any composite chart's definition bar to display a dropdown of contributing contexts and nodes +contributing. Click on the link icon next to a +given node to quickly _jump to the same chart in that node's single-node dashboard_ in Netdata Cloud. + +You can use single-node dashboards in Netdata Cloud to drill down on specific issues, scrub backward in time to +investigate historical data, and see like metrics presented meaningfully to help you troubleshoot performance problems. +All of the familiar [interactions](/docs/visualize/interact-dashboards-charts.md) are available, as is adding any chart +to a [new dashboard](/docs/visualize/create-dashboards.md). + +## Nodes view + +You can also use the **Nodes view** to monitor the health status and user-configurable key metrics from multiple nodes +in a War Room. Read the [Nodes view doc](https://learn.netdata.cloud/docs/cloud/visualize/nodes) for details. + +![The Nodes view](https://user-images.githubusercontent.com/1153921/108733066-5fe65800-74eb-11eb-98e0-abaccd36deaf.png) + +## What's next? + +To troubleshoot complex performance issues using Netdata, you need to understand how to interact with its meaningful +visualizations. Learn more about [interaction](/docs/visualize/interact-dashboards-charts.md) to see historical metrics, +highlight timeframes for targeted analysis, and more. + +If you're a Kubernetes user, read about Netdata's [Kubernetes +visualizations](https://learn.netdata.cloud/docs/cloud/visualize/kubernetes) for details about the health map and +time-series k8s charts, and our tutorial, [_Kubernetes monitoring with Netdata: Overview and +visualizations_](/docs/guides/monitor/kubernetes-k8s-netdata.md), for a full walkthrough. + +### Related reference documentation + +- [Netdata Cloud · War Rooms](https://learn.netdata.cloud/docs/cloud/war-rooms) +- [Netdata Cloud · Overview](https://learn.netdata.cloud/docs/cloud/visualize/overview) +- [Netdata Cloud · Nodes view](https://learn.netdata.cloud/docs/cloud/visualize/nodes) +- [Netdata Cloud · Kubernetes visualizations](https://learn.netdata.cloud/docs/cloud/visualize/kubernetes) + + diff --git a/docs/why-netdata/1s-granularity.md b/docs/why-netdata/1s-granularity.md new file mode 100644 index 0000000..4fc7fab --- /dev/null +++ b/docs/why-netdata/1s-granularity.md @@ -0,0 +1,59 @@ + + +# 1s granularity + +High resolution metrics are required to effectively monitor and troubleshoot systems and applications. + +## Why? + +- The world is going real-time. Today, customer experience is significantly affected by response time, so SLAs are tighter than ever before. It is just not practical to monitor a 2-second SLA with 10-second metrics. + +- IT goes virtual. Unlike real hardware, virtual environments are not linear, nor predictable. You cannot expect resources to be available when your applications need them. They will eventually be, but not exactly at the time they are needed. The latency of virtual environments is affected by many factors, most of which are outside our control, like: the maintenance policy of the hosting provider, the work load of third party virtual machines running on the same physical servers combined with the resource allocation and throttling policy among virtual machines, the provisioning system of the hosting provider, etc. + +## What do others do? + +So, why don't most monitoring platforms and monitoring SaaS providers offer high resolution metrics? + +They want to, but they can't, at least not massively. + +The reasons lie in their design decisions: + +1. Time-series databases (prometheus, graphite, opentsdb, influxdb, etc) centralize all the metrics. At scale, these databases can easily become the bottleneck of the whole infrastructure. + +2. SaaS providers base their business models on centralizing all the metrics. On top of the time-series database bottleneck they also have increased bandwidth costs. So, massively supporting high resolution metrics, destroys their business model. + +Of course, since a couple of decades the world has fixed this kind of scaling problems: instead of scaling up, scale out, horizontally. That is, instead of investing on bigger and bigger central components, decentralize the application so that it can scale by adding more smaller nodes to it. + +There have been many attempts to fix this problem for monitoring. But so far, all solutions required centralization of metrics, which can only scale up. So, although the problem is somehow managed, it is still the key problem of all monitoring platforms and one of the key reasons for increased monitoring costs. + +Another important factor is how resource efficient data collection can be when running per second. Most solutions fail to do it properly. The data collection agent is consuming significant system resources when running "per second", influencing the monitored systems and applications to a great degree. + +Finally, per second data collection is a lot harder. Busy virtual environments have [a constant latency of about 100ms, spread randomly to all data sources](https://docs.google.com/presentation/d/18C8bCTbtgKDWqPa57GXIjB2PbjjpjsUNkLtZEz6YK8s/edit#slide=id.g422e696d87_0_57). If data collection is not implemented properly, this latency introduces a random error of +/- 10%, which is quite significant for a monitoring system. + +So, the monitoring industry fails to massively provide high resolution metrics, mainly for 3 reasons: + +1. Centralization of metrics makes monitoring cost inefficient at that rate. +2. Data collection needs optimization, otherwise it will significantly affect the monitored systems. +3. Data collection is a lot harder, especially on busy virtual environments. + +## What does Netdata do differently? + +Netdata decentralizes monitoring completely. Each Netdata node is autonomous. It collects metrics locally, it stores them locally, it runs checks against them to trigger alarms locally, and provides an API for the dashboards to visualize them. This allows Netdata to scale to infinity. + +Of course, Netdata can centralize metrics when needed. For example, it is not practical to keep metrics locally on ephemeral nodes. For these cases, Netdata streams the metrics in real-time, from the ephemeral nodes to one or more non-ephemeral nodes nearby. This centralization is again distributed. On a large infrastructure, there may be many centralization points. + +To eliminate the error introduced by data collection latencies on busy virtual environments, Netdata interpolates collected metrics. It does this using microsecond timings, per data source, offering measurements with an error rate of 0.0001%. When running [in debug mode, Netdata calculates this error rate](https://github.com/netdata/netdata/blob/36199f449852f8077ea915a3a14a33fa2aff6d85/database/rrdset.c#L1070-L1099) for every point collected, ensuring that the database works with acceptable accuracy. + +Finally, Netdata is really fast. Optimization is a core product feature. On modern hardware, Netdata can collect metrics with a rate of above 1M metrics per second per core (this includes everything, parsing data sources, interpolating data, storing data in the time series database, etc). So, for a few thousands metrics per second per node, Netdata needs negligible CPU resources (just 1-2% of a single core). + +Netdata has been designed to: + +- Solve the centralization problem of monitoring +- Replace the console for performance troubleshooting. + +So, for Netdata 1s granularity is easy, the natural outcome... + + diff --git a/docs/why-netdata/README.md b/docs/why-netdata/README.md new file mode 100644 index 0000000..c482ee9 --- /dev/null +++ b/docs/why-netdata/README.md @@ -0,0 +1,35 @@ + + +# Why Netdata + +> Any performance monitoring solution that does not go down to per second +> collection and visualization of the data, is useless. +> It will make you happy to have it, but it will not help you more than that. + +Netdata is built around 4 principles: + +1. **[Per second data collection for all metrics.](/docs/why-netdata/1s-granularity.md)** + + _It is impossible to monitor a 2 second SLA, with 10 second metrics._ + +2. **[Collect and visualize all the metrics from all possible sources.](/docs/why-netdata/unlimited-metrics.md)** + + _To troubleshoot slowdowns, we need all the available metrics. The console should not provide more metrics._ + +3. **[Meaningful presentation, optimized for visual anomaly detection.](/docs/why-netdata/meaningful-presentation.md)** + + _Metrics are a lot more than name-value pairs over time. The monitoring tool should know all the metrics. Users should not!_ + +4. **[Immediate results, just install and use.](/docs/why-netdata/immediate-results.md)** + + _Most of our infrastructure is standardized. There is no point to configure everything metric by metric._ + +Unlike other monitoring solutions that focus on metrics visualization, +Netdata's helps us troubleshoot slowdowns without touching the console. + +So, everything is a bit different. + + diff --git a/docs/why-netdata/immediate-results.md b/docs/why-netdata/immediate-results.md new file mode 100644 index 0000000..b35aa53 --- /dev/null +++ b/docs/why-netdata/immediate-results.md @@ -0,0 +1,46 @@ + + +# Immediate results + +Most of our infrastructure is based on standardized systems and applications. + +It is a tremendous waste of time and effort, in a global scale, to require from all users to configure their infrastructure dashboards and alarms metric by metric. + +## Why? + +Most of the existing monitoring solutions, focus on providing a platform "for building your monitoring". So, they provide the tools to collect metrics, store them, visualize them, check them and query them. And we are all expected to go through this process. + +However, most of our infrastructure is standardized. We run well known Linux distributions, the same kernel, the same database, the same web server, etc. + +So, why can't we have a monitoring system that can be installed and instantly provide feature rich dashboards and alarms about everything we use? Is there any reason you would like to monitor your web server differently than me? + +What a waste of time and money! Hundreds of thousands of people doing the same thing over and over again, trying to understand what the metrics are, how to visualize them, how to configure alarms for them and how to query them when issues arise. + +## What do others do? + +Open-source solutions rely almost entirely on configuration. So, you have to go through endless metric-by-metric configuration yourself. The result will reflect your skills, your experience, your understanding. + +Monitoring SaaS providers offer a very basic set of pre-configured metrics, dashboards and alarms. They assume that you will configure the rest you may need. So, once more, the result will reflect your skills, your experience, your understanding. + +## What does Netdata do? + +1. Metrics are auto-detected, so for 99% of the cases data collection works out of the box. +2. Metrics are converted to human readable units, right after data collection, before storing them into the database. +3. Metrics are structured, organized in charts, families and applications, so that they can be browsed. +4. Dashboards are automatically generated, so all metrics are available for exploration immediately after installation. +5. Dashboards are not just visualizing metrics; they are a tool, optimized for visual anomaly detection. +6. Hundreds of pre-configured alarm templates are automatically attached to collected metrics. + +The result is that Netdata can be used immediately after installation! + +Netdata: + +- Helps engineers understand and learn what the metrics are. +- Does not require any configuration. Of course there are thousands of options to tweak, but the defaults are pretty good for most systems. +- Does not introduce any query languages or any other technology to be learned. Of course some familiarity with the tool is required, but nothing too complicated. +- Includes all the community expertise and experience for monitoring systems and applications. + + diff --git a/docs/why-netdata/meaningful-presentation.md b/docs/why-netdata/meaningful-presentation.md new file mode 100644 index 0000000..fc670e3 --- /dev/null +++ b/docs/why-netdata/meaningful-presentation.md @@ -0,0 +1,68 @@ + + +# Meaningful presentation + +Metrics are a lot more than name-value pairs over time. It is just not practical to require from all users to have a deep understanding of all metrics for monitoring their systems and applications. + +## Why? + +There is a plethora of metrics. And each of them has a context, a meaning, a way to be interpreted. + +Traditionally, monitoring solutions instruct engineers to collect only the metrics they understand. This is a good strategy as long as you have a clear understanding of what you need and you have the skills, the expertise and the experience to select them. + +For most people, this is an impossible task. It is just not practical to assume that any engineer will have a deep understanding of how the kernel works, how the networking stack works, how the system manages its memory, how it schedules processes, how web servers work, how databases work, etc. + +The result is that for most of the world, monitoring sucks. It is incomplete, inefficient, and in most of the cases only useful for providing an illusion that the infrastructure is being monitored. It is not! According to the [State of Monitoring 2017](http://start.bigpanda.io/state-of-monitoring-report-2017), only 11% of the companies are satisfied with their existing monitoring infrastructure, and on the average they use 6-7 monitoring tools. + +But even if all the metrics are collected, an even bigger challenge is revealed: What to do with them? How to use them? + +The existing monitoring solutions, assume the engineers will: + +- Design dashboards +- Configure alarms +- Use a query language to investigate issues + +However, all these have to be configured metric by metric. + +The monitoring industry believes there is this "IT Operations Hero", a person combining these abilities: + +1. Has a deep understanding of IT architectures and is a skillful SysAdmin. +2. Is a superb Network Administrator (can even read and understand the Linux kernel networking stack). +3. Is an exceptional database administrator. +4. Is fluent in software engineering, capable of understanding the internal workings of applications. +5. Masters Data Science, statistical algorithms and is fluent in writing advanced mathematical queries to reveal the meaning of metrics. + +Of course this person does not exist! + +## What do others do? + +Most solutions are based on a time-series database. A database that tracks name-value pairs, over time. + +Data collection blindly collects metrics and stores them into the database, dashboard editors query the database to visualize the metrics. They may also provide a query editor, that users can use to query the database by hand. + +Of course, it is just not practical to work that way when the database has 10,000 unique metrics. Most of them will be just noise, not because they are not useful, but because no one understands them! + +So, they collect very limited metrics. Basic dashboards can be created with these metrics, but for any issue that needs to be troubleshooted, the monitoring system is just not adequate. It cannot help. So, engineers are using the console to access the rest of the metrics and find the root cause. + +## What does Netdata do? + +In Netdata, the meaning of metrics is incorporated into the database: + +1. all metrics are converted and stored to human-friendly units. This is a data-collection process, not a visualization process. For example, cpu utilization in Netdata is stored as percentage, not as kernel ticks. + +2. all metrics are organized into human-friendly charts, sharing the same context and units (similar to what other monitoring solutions call `cardinality`). So, when Netdata developer collect metrics, they configure the correlation of the metrics right in data collection, which is stored in the database too. + +3. all charts are then organized in families, and chart families are organized in applications. These structures are responsible for providing the menu at the right side of Netdata dashboards for exploring the whole database. + +The result is a system that can be browsed by humans, even if the database has 100,000 unique metrics. It is pretty natural for everyone to browse them, understand their meaning and their scope. + +Of course, this process makes data collection significantly more time consuming. Netdata developers need to normalize and correlate and categorize every single metric Netdata collects. + +But it simplifies everything else. Data collection, metrics database and visualization are de-coupled, thus the query engine is simpler, and the visualization is straight forward. + +Netdata goes a step further, by enriching the dashboard with information that is useful for most people. So, to improve clarity and help users be more effective, Netdata includes right in the dashboard the community knowledge and expertise about the metrics. So, that Netdata users can focus on solving their infrastructure problem, not on the technicalities of data collection and visualization. + + diff --git a/docs/why-netdata/unlimited-metrics.md b/docs/why-netdata/unlimited-metrics.md new file mode 100644 index 0000000..b79a4ed --- /dev/null +++ b/docs/why-netdata/unlimited-metrics.md @@ -0,0 +1,49 @@ + + +# Unlimited metrics + +All metrics are important and all metrics should be available when you need them. + +## Why? + +Collecting all the metrics breaks the first rule of every monitoring text book: "collect only the metrics you need", "collect only the metrics you understand". + +Unfortunately, this does not work! Filtering out most metrics is like reading a book by skipping most of its pages... + +For many people, monitoring is about: + +- Detecting outages +- Capacity planning + +However, **slowdowns are 10 times more common** compared to outages (check slide 14 of [Online Performance is Business Performance ](https://www.slideshare.net/KenGodskind/alertsitetrac) reported by Trac Research/AlertSite). Designing a monitoring system targeting only outages and capacity planning solves just a tiny part of the operational problems we face. Check also [Downtime vs. Slowtime: Which Hurts More?](https://dzone.com/articles/downtime-vs-slowtime-which-hurts-more). + +To troubleshoot a slowdown, a lot more metrics are needed. Actually all the metrics are needed, since the real cause of a slowdown is most probably quite complex. If we knew the possible reasons, chances are we would have fixed them before they become a problem. + +## What do others do? + +Most monitoring solutions, when they are able to detect something, provide just a hint (e.g. "hey, there is a 20% drop in requests per second over the last minute") and they expect us to use the console for determining the root cause. + +Of course this introduces a lot more problems: how to troubleshoot a slowdown using the console, if the slowdown lifetime is just a few seconds, randomly spread throughout the day? + +You can't! You will spend your entire day on the console, waiting for the problem to happen again while you are logged in. A blame war starts: developers blame the systems, sysadmins blame the hosting provider, someone says it is a DNS problem, another one believes it is network related, etc. We have all experienced this, multiple times... + +So, why do monitoring solutions and SaaS providers filter out metrics? + +They can't do otherwise! + +1. Centralization of metrics depends on metrics filtering, to control monitoring costs. Time-series databases limit the number of metrics collected, because the number of metrics influences their performance significantly. They get congested at scale. +2. It is a lot easier to provide an illusion of monitoring by using a few basic metrics. +3. Troubleshooting slowdowns is the hardest IT problem to solve, so most solutions just avoid it. + +## What does Netdata do? + +Netdata collects, stores and visualizes everything, every single metric exposed by systems and applications. + +Due to Netdata's distributed nature, the number of metrics collected does not have any noticeable effect on the performance or the cost of the monitoring infrastructure. + +Of course, since Netdata is also about [meaningful presentation](meaningful-presentation.md), the number of metrics makes Netdata development slower. We, the Netdata developers, need to have a good understanding of the metrics before adding them into Netdata. We need to organize the metrics, add information related to them, configure alarms for them, so that you, the Netdata users, will have the best out-of-the-box experience and all the information required to kill the console for troubleshooting slowdowns. + + -- cgit v1.2.3