diff options
Diffstat (limited to '')
| -rw-r--r-- | doc/Add-more-charts-to-netdata.md | 429 | ||||
| -rw-r--r-- | doc/Demo-Sites.md | 19 | ||||
| -rw-r--r-- | doc/Donations-netdata-has-received.md | 23 | ||||
| -rw-r--r-- | doc/Netdata-Security-and-Disclosure-Information.md | 37 | ||||
| -rw-r--r-- | doc/Performance.md | 73 | ||||
| -rw-r--r-- | doc/Running-behind-apache.md | 268 | ||||
| -rw-r--r-- | doc/Running-behind-caddy.md | 27 | ||||
| -rw-r--r-- | doc/Running-behind-lighttpd.md | 60 | ||||
| -rw-r--r-- | doc/Running-behind-nginx.md | 202 | ||||
| -rw-r--r-- | doc/Third-Party-Plugins.md | 29 | ||||
| -rw-r--r-- | doc/Why-Netdata.md | 170 | ||||
| -rw-r--r-- | doc/a-github-star-is-important.md | 13 | ||||
| -rw-r--r-- | doc/high-performance-netdata.md | 149 | ||||
| -rw-r--r-- | doc/netdata-for-IoT.md | 199 | ||||
| -rw-r--r-- | doc/netdata-security.md | 179 | ||||
| -rw-r--r-- | docker/Dockerfile | 2 | ||||
| -rw-r--r-- | docker/README.md | 117 | ||||
| -rwxr-xr-x | docker/build.sh | 15 |
18 files changed, 1998 insertions, 13 deletions
diff --git a/doc/Add-more-charts-to-netdata.md b/doc/Add-more-charts-to-netdata.md new file mode 100644 index 000000000..1512a25e7 --- /dev/null +++ b/doc/Add-more-charts-to-netdata.md @@ -0,0 +1,429 @@ +# Add more charts to netdata + +netdata collects system metrics by itself. It has many [internal plugins](../collectors) for collecting most of the metrics presented by default when it starts, collecting data from `/proc`, `/sys` and other Linux kernel sources. + +To collect non-system metrics, netdata supports a plugin architecture. The following are the currently available external plugins: + +- **[Web Servers](#web-servers)**, such as apache, nginx, nginx_plus, tomcat, litespeed +- **[Web Logs](#web-log-parsers)**, such as apache, nginx, lighttpd, gunicorn, squid access logs, apache cache.log +- **[Load Balancers](#load-balancers)**, like haproxy +- **[Message Brokers](#message-brokers)**, like rabbitmq, beanstalkd +- **[Database Servers](#database-servers)**, such as mysql, mariadb, postgres, couchdb, mongodb, rethinkdb +- **[Social Sharing Servers](#social-sharing-servers)**, like retroshare +- **[Proxy Servers](#proxy-servers)**, like squid +- **[HTTP accelerators](#http-accelerators)**, like varnish cache +- **[Search engines](#search-engines)**, like elasticsearch +- **[Name Servers](#name-servers)** (DNS), like bind, nsd, powerdns, dnsdist +- **[DHCP Servers](#dhcp-servers)**, like ISC DHCP +- **[UPS](#ups)**, such as APC UPS, NUT +- **[RAID](#raid)**, such as linux software raid (mdadm), MegaRAID +- **[Mail Servers](#mail-servers)**, like postfix, exim, dovecot +- **[File Servers](#file-servers)**, like samba, NFS, ftp, sftp, WebDAV +- **[System](#system)**, for processes and other system metrics +- **[Sensors](#sensors)**, like temperature, fans speed, voltage, humidity, HDD/SSD S.M.A.R.T attributes +- **[Network](#network)**, such as SNMP devices, `fping`, access points, dns_query_time +- **[Time Servers](#time-servers)**, like chrony +- **[Security](#security)**, like FreeRADIUS, OpenVPN, Fail2ban +- **[Telephony Servers](#telephony-servers)**, like openSIPS +- **[Go applications](#go-applications)** +- **[Household appliances](#household-appliances)**, like SMA WebBox (solar power), Fronius Symo solar power, Stiebel Eltron heating +- **[Java Processes](#java-processes)**, via JMX or Spring Boot Actuator +- **[Provisioning Systems](#provisioning-systems)**, like Puppet +- **[Game Servers](#game-servers)**, like SpigotMC +- **[Distributed Computing Clients](#distributed-computing-clients)**, like BOINC +- **[Skeleton Plugins](#skeleton-plugins)**, for writing your own data collectors + +Check also [Third Party Plugins](Third-Party-Plugins.md) for a list of plugins distributed by third parties. + +## configuring plugins + +netdata comes with **internal** and **external** plugins: + +1. The **internal** ones are written in `C` and run as threads within the netdata daemon. +2. The **external** ones can be written in any computer language. The netdata daemon spawns these as processes (shown with `ps fax`) and reads their metrics using pipes (so the `stdout` of external plugins is connected to netdata for metrics collection and the `stderr` of external plugins is connected to `/var/log/netdata/error.log`). + +To make it easier to develop plugins, and minimize the number of threads and processes running, netdata supports **plugin orchestrators**, each of them supporting one or more data collection **modules**. Currently we ship plugin orchestrators for 4 languages: `C`, `python`, `node.js` and `bash` and 2 more are under development (`go` and `java`). + +#### enabling and disabling plugins + +To control which plugins netdata run, edit `netdata.conf` and check the `[plugins]` section. It looks like this: + +``` +[plugins] + # enable running new plugins = yes + # check for new plugins every = 60 + # proc = yes + # diskspace = yes + # cgroups = yes + # tc = yes + # nfacct = yes + # idlejitter = yes + # freeipmi = yes + # node.d = yes + # python.d = yes + # fping = yes + # charts.d = yes + # apps = yes +``` + +The default for all plugins is the option `enable running new plugins`. So, setting this to `no` will disable all the plugins, except the ones specifically enabled. + +#### enabling and disabling modules + +Each of the **plugins** may support one or more data collection **modules**. To control which of its modules run, you have to consult the configuration of the **plugin** (see table below). + +#### modules configuration + +Most **modules** come with **auto-detection**, configured to work out-of-the-box on popular operating systems with the default settings. + +However, there are cases that auto-detection fails. Usually the reason is that the applications to be monitored do not allow netdata to connect. In most of the cases, allowing the user `netdata` from `localhost` to connect and collect metrics, will automatically enable data collection for the application in question (it will require a netdata restart). + +You can verify netdata **external plugins and their modules** are able to collect metrics, following this procedure: + +```sh +# become user netdata +sudo su -s /bin/bash netdata + +# execute the plugin in debug mode, for a specific module. +# example for the python plugin, mysql module: +/usr/libexec/netdata/plugins.d/python.d.plugin 1 debug trace mysql +``` + +Similarly, you can use `charts.d.plugin` for BASH plugins and `node.d.plugin` for node.js plugins. +Other plugins (like `apps.plugin`, `freeipmi.plugin`, `fping.plugin`) use the native netdata plugin API and can be run directly. + +If you need to configure a netdata plugin or module, all user supplied configuration is kept at `/etc/netdata` while the stock versions of all files is at `/usr/lib/netdata/conf.d`. +To copy a stock file and edit it, run `/etc/netdata/edit-config`. Running this command without an argument, will list the available stock files. + +Each file should provide plenty of examples and documentation about each module and plugin. + +This is a map of the all supported configuration options: + +#### map of configuration files + +plugin | language | plugin<br/>configuration | modules<br/>configuration | +---:|:---:|:---:|:---| +`apps.plugin`<br/>(external plugin for monitoring the process tree on Linux and FreeBSD)|`C`|`netdata.conf` section `[plugin:apps]`|Custom configuration for the processes to be monitored at `apps_groups.conf` +`freebsd.plugin`<br/>(internal plugin for monitoring FreeBSD system resources)|`C`|`netdata.conf` section `[plugin:freebsd]`|one section for each module `[plugin:freebsd:MODULE]`. Each module may provide additional sections in the form of `[plugin:freebsd:MODULE:SUBSECTION]`. +`cgroups.plugin`<br/>(internal plugin for monitoring Linux containers, VMs and systemd services)|`C`|`netdata.conf` section `[plugin:cgroups]`|N/A +`charts.d.plugin`<br/>(external plugin orchestrator for BASH modules)|`BASH`|`charts.d.conf`|a file for each module in `/etc/netdata/charts.d/` +`diskspace.plugin`<br/>(internal plugin for collecting Linux mount points usage)|`C`|`netdata.conf` section `[plugin:diskspace]`|N/A +`fping.plugin`<br/>(external plugin for collecting network latencies)|`C`|`fping.conf`|This plugin is a wrapper for the `fping` command. +`freeipmi.plugin`<br/>(external plugin for collecting IPMI h/w sensors)|`C`|`netdata.conf` section `[plugin:freeipmi]` +`idlejitter.plugin`<br/>(internal plugin for monitoring CPU jitter)|`C`|N/A|N/A +`macos.plugin`<br/>(internal plugin for monitoring MacOS system resources)|`C`|`netdata.conf` section `[plugin:macos]`|one section for each module `[plugin:macos:MODULE]`. Each module may provide additional sections in the form of `[plugin:macos:MODULE:SUBSECTION]`. +`node.d.plugin`<br/>(external plugin orchestrator of node.js modules)|`node.js`|`node.d.conf`|a file for each module in `/etc/netdata/node.d/`. +`proc.plugin`<br/>(internal plugin for monitoring Linux system resources)|`C`|`netdata.conf` section `[plugin:proc]`|one section for each module `[plugin:proc:MODULE]`. Each module may provide additional sections in the form of `[plugin:proc:MODULE:SUBSECTION]`. +`python.d.plugin`<br/>(external plugin orchestrator for running python modules)|`python`<br/>v2 or v3<br/>both are supported|`python.d.conf`|a file for each module in `/etc/netdata/python.d/`. +`statsd.plugin`<br/>(internal plugin for collecting statsd metrics)|`C`|`netdata.conf` section `[statsd]`|Synthetic statsd charts can be configured with files in `/etc/netdata/statsd.d/`. +`tc.plugin`<br/>(internal plugin for collecting Linux traffic QoS)|`C`|`netdata.conf` section `[plugin:tc]`|The plugin runs an external helper called `tc-qos-helper.sh` to interface with the `tc` command. This helper supports a few additional options using `tc-qos-helper.conf`. + + +## writing data collection modules + +You can add custom plugins following the [External Plugins Guide](../collectors/plugins.d/). + +--- + +## available data collection modules + +These are all the data collection plugins currently available. + +### Web Servers + +application|language|notes| +:---------:|:------:|:----| +apache|python<br/>v2 or v3|Connects to multiple apache servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [apache.chart.py](../collectors/python.d.plugin/apache)<br/>configuration file: [python.d/apache.conf](../collectors/python.d.plugin/apache)| +apache|BASH<br/>Shell Script|Connects to an apache server (local or remote) to collect real-time performance metrics.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [apache.chart.sh](../collectors/charts.d.plugin/apache)<br/>configuration file: [charts.d/apache.conf](../collectors/charts.d.plugin/apache)| +ipfs|python<br/>v2 or v3|Connects to multiple ipfs servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [ipfs.chart.py](../collectors/python.d.plugin/ipfs)<br/>configuration file: [python.d/ipfs.conf](../collectors/python.d.plugin/ipfs)| +litespeed|python<br/>v2 or v3|reads the litespeed `rtreport` files to collect metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [litespeed.chart.py](../collectors/python.d.plugin/litespeed)<br/>configuration file: [python.d/litespeed.conf](../collectors/python.d.plugin/litespeed) +nginx|python<br/>v2 or v3|Connects to multiple nginx servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [nginx.chart.py](../collectors/python.d.plugin/nginx)<br/>configuration file: [python.d/nginx.conf](../collectors/python.d.plugin/nginx)| +nginx_plus|python<br/>v2 or v3|Connects to multiple nginx_plus servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [nginx_plus.chart.py](../collectors/python.d.plugin/nginx_plus)<br/>configuration file: [python.d/nginx_plus.conf](../collectors/python.d.plugin/nginx_plus)| +nginx|BASH<br/>Shell Script|Connects to an nginx server (local or remote) to collect real-time performance metrics.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [nginx.chart.sh](../collectors/charts.d.plugin/nginx)<br/>configuration file: [charts.d/nginx.conf](../collectors/charts.d.plugin/nginx)| +phpfpm|python<br/>v2 or v3|Connects to multiple phpfpm servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [phpfpm.chart.py](../collectors/python.d.plugin/phpfpm)<br/>configuration file: [python.d/phpfpm.conf](../collectors/python.d.plugin/phpfpm)| +phpfpm|BASH<br/>Shell Script|Connects to one or more phpfpm servers (local or remote) to collect real-time performance metrics.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [phpfpm.chart.sh](../collectors/charts.d.plugin/phpfpm)<br/>configuration file: [charts.d/phpfpm.conf](../collectors/charts.d.plugin/phpfpm)| +tomcat|python<br/>v2 or v3|Connects to multiple tomcat servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [tomcat.chart.py](../collectors/python.d.plugin/tomcat)<br/>configuration file: [python.d/tomcat.conf](../collectors/python.d.plugin/tomcat)| +tomcat|BASH<br/>Shell Script|Connects to a tomcat server (local or remote) to collect real-time performance metrics.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [tomcat.chart.sh](../collectors/charts.d.plugin/tomcat)<br/>configuration file: [charts.d/tomcat.conf](../collectors/charts.d.plugin/tomcat)| + + +--- + +### Web Log Parsers + +application|language|notes| +:---------:|:------:|:----| +web_log|python<br/>v2 or v3|powerful plugin, capable of incrementally parsing any number of web server log files <br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [web_log.chart.py](../collectors/python.d.plugin/web_log)<br/>configuration file: [python.d/web_log.conf](../collectors/python.d.plugin/web_log)| + + +--- + +### Database Servers + +application|language|notes| +:---------:|:------:|:----| +couchdb|python<br/>v2 or v3|Connects to multiple couchdb servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [couchdb.chart.py](../collectors/python.d.plugin/couchdb)<br/>configuration file: [python.d/couchdb.conf](../collectors/python.d.plugin/couchdb)| +memcached|python<br/>v2 or v3|Connects to multiple memcached servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [memcached.chart.py](../collectors/python.d.plugin/memcached)<br/>configuration file: [python.d/memcached.conf](../collectors/python.d.plugin/memcached)| +mongodb|python<br/>v2 or v3|Connects to multiple `mongodb` servers (local or remote) to collect real-time performance metrics.<br/> <br/>Requires package `python-pymongo`.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [mongodb.chart.py](../collectors/python.d.plugin/mongodb)<br/>configuration file: [python.d/mongodb.conf](../collectors/python.d.plugin/mongodb)| +mysql<br/>mariadb|python<br/>v2 or v3|Connects to multiple mysql or mariadb servers (local or remote) to collect real-time performance metrics.<br/> <br/>Requires package `python-mysqldb` (faster and preferred), or `python-pymysql`. <br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [mysql.chart.py](../collectors/python.d.plugin/mysql)<br/>configuration file: [python.d/mysql.conf](../collectors/python.d.plugin/mysql)| +mysql<br/>mariadb|BASH<br/>Shell Script|Connects to multiple mysql or mariadb servers (local or remote) to collect real-time performance metrics.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [mysql.chart.sh](../collectors/charts.d.plugin/mysql)<br/>configuration file: [charts.d/mysql.conf](../collectors/charts.d.plugin/mysql)| +postgres|python<br/>v2 or v3|Connects to multiple postgres servers (local or remote) to collect real-time performance metrics.<br/> <br/>Requires package `python-psycopg2`.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [postgres.chart.py](../collectors/python.d.plugin/postgres)<br/>configuration file: [python.d/postgres.conf](../collectors/python.d.plugin/postgres)| +redis|python<br/>v2 or v3|Connects to multiple redis servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [redis.chart.py](../collectors/python.d.plugin/redis)<br/>configuration file: [python.d/redis.conf](../collectors/python.d.plugin/redis)| +rethinkdb|python<br/>v2 or v3|Connects to multiple rethinkdb servers (local or remote) to collect real-time metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [rethinkdb.chart.py](../collectors/python.d.plugin/rethinkdbs)<br/>configuration file: [python.d/rethinkdb.conf](../collectors/python.d.plugin/rethinkdbs)| + + +--- + +### Social Sharing Servers + +application|language|notes| +:---------:|:------:|:----| +retroshare|python<br/>v2 or v3|Connects to multiple retroshare servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [retroshare.chart.py](../collectors/python.d.plugin/retroshare)<br/>configuration file: [python.d/retroshare.conf](../collectors/python.d.plugin/retroshare)| + + +--- + +### Proxy Servers + +application|language|notes| +:---------:|:------:|:----| +squid|python<br/>v2 or v3|Connects to multiple squid servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [squid.chart.py](../collectors/python.d.plugin/squid)<br/>configuration file: [python.d/squid.conf](../collectors/python.d.plugin/squid)| +squid|BASH<br/>Shell Script|Connects to a squid server (local or remote) to collect real-time performance metrics.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [squid.chart.sh](../collectors/charts.d.plugin/squid)<br/>configuration file: [charts.d/squid.conf](../collectors/charts.d.plugin/squid)| + + +--- + +### HTTP Accelerators + +application|language|notes| +:---------:|:------:|:----| +varnish|python<br/>v2 or v3|Uses the varnishstat command to provide varnish cache statistics (client metrics, cache perfomance, thread-related metrics, backend health, memory usage etc.).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [varnish.chart.py](../collectors/python.d.plugin/varnish)<br/>configuration file: [python.d/varnish.conf](../collectors/python.d.plugin/varnish)| + + +--- + +### Search Engines + +application|language|notes| +:---------:|:------:|:----| +elasticsearch|python<br/>v2 or v3|Monitor elasticsearch performance and health metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [elasticsearch.chart.py](../collectors/python.d.plugin/elasticsearch)<br/>configuration file: [python.d/elasticsearch.conf](../collectors/python.d.plugin/elasticsearch)| + + +--- + +### Name Servers + +application|language|notes| +:---------:|:------:|:----| +named|node.js|Connects to multiple named (ISC-Bind) servers (local or remote) to collect real-time performance metrics. All versions of bind after 9.9.10 are supported.<br/> <br/>netdata plugin: [node.d.plugin](../collectors/node.d.plugin#nodedplugin)<br/>plugin module: [named.node.js](../collectors/node.d.plugin/named)<br/>configuration file: [node.d/named.conf](../collectors/node.d.plugin/named)| +bind_rndc|python<br/>v2 or v3|Parses named.stats dump file to collect real-time performance metrics. All versions of bind after 9.6 are supported.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [bind_rndc.chart.py](../collectors/python.d.plugin/bind_rndc)<br/>configuration file: [python.d/bind_rndc.conf](../collectors/python.d.plugin/bind_rndc)| +nsd|python<br/>v2 or v3|Charts the nsd received queries and zones.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [nsd.chart.py](../collectors/python.d.plugin/nsd)<br/>configuration file: [python.d/nsd.conf](../collectors/python.d.plugin/nsd) +powerdns|python<br/>v2 or v3|Monitors powerdns performance and health metrics <br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [powerdns.chart.py](../collectors/python.d.plugin/powerdns)<br/>configuration file: [python.d/powerdns.conf](../collectors/python.d.plugin/powerdns)| +dnsdist|python<br/>v2 or v3|Monitors dnsdist performance and health metrics <br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [dnsdist.chart.py](../collectors/python.d.plugin/dnsdist)<br/>configuration file: [python.d/dnsdist.conf](../collectors/python.d.plugin/dnsdist)| +unbound|python<br/>v2 or v3|Monitors Unbound performance and resource usage metrics <br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [unbound.chart.py](../collectors/python.d.plugin/unbound)<br/>configuration file: [python.d/unbound.conf](../collectors/python.d.plugin/unbound)| + + +--- + +### DHCP Servers + +application|language|notes| +:---------:|:------:|:----| +isc dhcp|python<br/>v2 or v3|Monitor lease database to show all active leases.<br/> <br/>Python v2 requires package `python-ipaddress`.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [isc-dhcpd.chart.py](../collectors/python.d.plugin/isc_dhcpd)<br/>configuration file: [python.d/isc-dhcpd.conf](../collectors/python.d.plugin/isc_dhcpd)| + + +--- + +### Load Balancers + +application|language|notes| +:---------:|:------:|:----| +haproxy|python<br/>v2 or v3|Monitor frontend, backend and health metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [haproxy.chart.py](../collectors/python.d.plugin/haproxy)<br/>configuration file: [python.d/haproxy.conf](../collectors/python.d.plugin/haproxy)| +traefik|python<br/>v2 or v3|Connects to multiple traefik instances (local or remote) to collect API metrics (response status code, response time, average response time and server uptime).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [traefik.chart.py](../collectors/python.d.plugin/traefik)<br/>configuration file: [python.d/traefik.conf](../collectors/python.d.plugin/traefik)| + +--- + +### Message Brokers + +application|language|notes| +:---------:|:------:|:----| +rabbitmq|python<br/>v2 or v3|Monitor rabbitmq performance and health metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [rabbitmq.chart.py](../collectors/python.d.plugin/rabbitmq)<br/>configuration file: [python.d/rabbitmq.conf](../collectors/python.d.plugin/rabbitmq)| +beanstalkd|python<br/>v2 or v3|Provides server and tube level statistics.<br/> <br/>Requires beanstalkc python package (`pip install beanstalkc` or install package `python-beanstalkc`, which also installs `python-yaml`).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [beanstalk.chart.py](../collectors/python.d.plugin/beanstalk)<br/>configuration file: [python.d/beanstalk.conf](../collectors/python.d.plugin/beanstalk)| + + +--- + +### UPS + +application|language|notes| +:---------:|:------:|:----| +apcupsd|BASH<br/>Shell Script|Connects to an apcupsd server to collect real-time statistics of an APC UPS.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [apcupsd.chart.sh](../collectors/charts.d.plugin/apcupsd)<br/>configuration file: [charts.d/apcupsd.conf](../collectors/charts.d.plugin/apcupsd)| +nut|BASH<br/>Shell Script|Connects to a nut server (upsd) to collect real-time UPS statistics.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [nut.chart.sh](../collectors/charts.d.plugin/nut)<br/>configuration file: [charts.d/nut.conf](../collectors/charts.d.plugin/nut)| + + +--- + +### RAID + +application|language|notes| +:---------:|:------:|:----| +mdstat|python<br/>v2 or v3|Parses `/proc/mdstat` to get mds health metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [mdstat.chart.py](../collectors/python.d.plugin/mdstat)<br/>configuration file: [python.d/mdstat.conf](../collectors/python.d.plugin/mdstat)| +megacli|python<br/>v2 or v3|Collects adapter, physical drives and battery stats..<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [megacli.chart.py](../collectors/python.d.plugin/megacli)<br/>configuration file: [python.d/megacli.conf](../collectors/python.d.plugin/megacli)| + +--- + +### Mail Servers + +application|language|notes| +:---------:|:------:|:----| +dovecot|python<br/>v2 or v3|Connects to multiple dovecot servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [dovecot.chart.py](../collectors/python.d.plugin/dovecot)<br/>configuration file: [python.d/dovecot.conf](../collectors/python.d.plugin/dovecot)| +exim|python<br/>v2 or v3|Charts the exim queue size.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [exim.chart.py](../collectors/python.d.plugin/exim)<br/>configuration file: [python.d/exim.conf](../collectors/python.d.plugin/exim)| +exim|BASH<br/>Shell Script|Charts the exim queue size.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [exim.chart.sh](../collectors/charts.d.plugin/exim)<br/>configuration file: [charts.d/exim.conf](../collectors/charts.d.plugin/exim)| +postfix|python<br/>v2 or v3|Charts the postfix queue size (supports multiple queues).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [postfix.chart.py](../collectors/python.d.plugin/postfix)<br/>configuration file: [python.d/postfix.conf](../collectors/python.d.plugin/postfix)| +postfix|BASH<br/>Shell Script|Charts the postfix queue size.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [postfix.chart.sh](../collectors/charts.d.plugin/postfix)<br/>configuration file: [charts.d/postfix.conf](../collectors/charts.d.plugin/postfix)| + + +--- + +### File Servers + +application|language|notes| +:---------:|:------:|:----| +NFS Client|`C`|This is handled entirely by the netdata daemon.<br/> <br/>Configuration: `netdata.conf`, section `[plugin:proc:/proc/net/rpc/nfs]`. +NFS Server|`C`|This is handled entirely by the netdata daemon.<br/> <br/>Configuration: `netdata.conf`, section `[plugin:proc:/proc/net/rpc/nfsd]`. +samba|python<br/>v2 or v3|Performance metrics of Samba SMB2 file sharing.<br/> <br/>documentation page: [python.d.plugin module samba](../collectors/python.d.plugin/samba)<br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [samba.chart.py](../collectors/python.d.plugin/samba)<br/>configuration file: [python.d/samba.conf](../collectors/python.d.plugin/samba)| + + +--- + +### System + +application|language|notes| +:---------:|:------:|:----| +apps|C|`apps.plugin` collects resource usage statistics for all processes running in the system. It groups the entire process tree and reports dozens of metrics for CPU utilization, memory footprint, disk I/O, swap memory, network connections, open files and sockets, etc. It reports metrics for application groups, users and user groups.<br/> <br/>[Documentation of `apps.plugin`](../collectors/apps.plugin/).<br/> <br/>netdata plugin: [`apps_plugin.c`](../collectors/apps.plugin)<br/>configuration file: [`apps_groups.conf`](../collectors/apps.plugin)| +cpu_apps|BASH<br/>Shell Script|Collects the CPU utilization of select apps.<br/><br/>DEPRECATED IN FAVOR OF `apps.plugin`. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [cpu_apps.chart.sh](../collectors/charts.d.plugin/cpu_apps)<br/>configuration file: [charts.d/cpu_apps.conf](../collectors/charts.d.plugin/cpu_apps)| +load_average|BASH<br/>Shell Script|Collects the current system load average.<br/><br/>DEPRECATED IN FAVOR OF THE NETDATA INTERNAL ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [load_average.chart.sh](../collectors/charts.d.plugin/load_average)<br/>configuration file: [charts.d/load_average.conf](../collectors/charts.d.plugin/load_average)| +mem_apps|BASH<br/>Shell Script|Collects the memory footprint of select applications.<br/><br/>DEPRECATED IN FAVOR OF `apps.plugin`. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [mem_apps.chart.sh](../collectors/charts.d.plugin/mem_apps)<br/>configuration file: [charts.d/mem_apps.conf](../collectors/charts.d.plugin/mem_apps)| + + +--- + +### Sensors + +application|language|notes| +:---------:|:------:|:----| +cpufreq|python<br/>v2 or v3|Collects the current CPU frequency from `/sys/devices`.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [cpufreq.chart.py](../collectors/python.d.plugin/cpufreq)<br/>configuration file: [python.d/cpufreq.conf](../collectors/python.d.plugin/cpufreq)| +cpufreq|BASH<br/>Shell Script|Collects current CPU frequency from `/sys/devices`.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [cpufreq.chart.sh](../collectors/charts.d.plugin/cpufreq)<br/>configuration file: [charts.d/cpufreq.conf](../collectors/charts.d.plugin/cpufreq)| +IPMI|C|Collects temperatures, voltages, currents, power, fans and `SEL` events from IPMI using `libipmimonitoring`.<br/>Check [Monitoring IPMI](../collectors/freeipmi.plugin/) for more information<br/> <br/>netdata plugin: [freeipmi.plugin](../collectors/freeipmi.plugin)<br/>configuration file: none required - to enable it, compile/install netdata with `--enable-plugin-freeipmi`| +hddtemp|python<br/>v2 or v3|Connects to multiple hddtemp servers (local or remote) to collect real-time performance metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [hddtemp.chart.py](../collectors/python.d.plugin/hddtemp)<br/>configuration file: [python.d/hddtemp.conf](../collectors/python.d.plugin/hddtemp)| +hddtemp|BASH<br/>Shell Script|Connects to a hddtemp server (local or remote) to collect real-time performance metrics.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [hddtemp.chart.sh](../collectors/charts.d.plugin/hddtemp)<br/>configuration file: [charts.d/hddtemp.conf](../collectors/charts.d.plugin/hddtemp)| +sensors|BASH<br/>Shell Script|Collects sensors values from files in `/sys`.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [sensors.chart.sh](../collectors/charts.d.plugin/sensors)<br/>configuration file: [charts.d/sensors.conf](../collectors/charts.d.plugin/sensors)| +sensors|python<br/>v2 or v3|Uses `lm-sensors` to collect sensor data.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [sensors.chart.py](../collectors/python.d.plugin/sensors)<br/>configuration file: [python.d/sensors.conf](../collectors/python.d.plugin/sensors)| +smartd_log|python<br/>v2 or v3|Collects the S.M.A.R.T attributes from `smartd` log files.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [smartd_log.chart.py](../collectors/python.d.plugin/smartd_log)<br/>configuration file: [python.d/smartd_log.conf](../collectors/python.d.plugin/smartd_log)| +w1sensor|python<br/>v2 or v3|Collects data from connected 1-Wire sensors.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [w1sensor.chart.py](../collectors/python.d.plugin/w1sensor)<br/>configuration file: [python.d/w1sensor.conf](../collectors/python.d.plugin/w1sensor)| + + +--- + +### Network + +application|language|notes| +:---------:|:------:|:----| +ap|BASH<br/>Shell Script|Uses the `iw` command to provide statistics of wireless clients connected to a wireless access point running on this host (works well with `hostapd`).<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [ap.chart.sh](../collectors/charts.d.plugin/ap)<br/>configuration file: [charts.d/ap.conf](../collectors/charts.d.plugin/ap)| +fping|C|Charts network latency statistics for any number of nodes, using the `fping` command. A recent (probably unreleased) version of fping is required. The plugin supplied can install it in `/usr/local`.<br/> <br/>netdata plugin: [fping.plugin](../collectors/fping.plugin) (this is a shell wrapper to start fping - once fping is started, netdata and fping communicate directly - it can also install the right version of fping)<br/>configuration file: [fping.conf](../collectors/fping.plugin)| +snmp|node.js|Connects to multiple snmp servers to collect real-time performance metrics.<br/> <br/>netdata plugin: [node.d.plugin](../collectors/node.d.plugin#nodedplugin)<br/>plugin module: [snmp.node.js](../collectors/node.d.plugin/snmp)<br/>configuration file: [node.d/snmp.conf](../collectors/node.d.plugin/snmp)| +dns_query_time|python<br/>v2 or v3|Provides DNS query time statistics.<br/> <br/>Requires package `dnspython` (`pip install dnspython` or install package `python-dnspython`).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [dns_query_time.chart.py](../collectors/python.d.plugin/dns_query_time)<br/>configuration file: [python.d/dns_query_time.conf](../collectors/python.d.plugin/dns_query_time)| +http|python<br />v2 or v3|Monitors a generic web page for status code and returned content in HTML +port|ptyhon<br />v2 or v3|Checks if a generic TCP port for its availability and response time + + +--- + +### Time Servers + +application|language|notes| +:---------:|:------:|:----| +chrony|python<br/>v2 or v3|Uses the chronyc command to provide chrony statistics (Frequency, Last offset, RMS offset, Residual freq, Root delay, Root dispersion, Skew, System time).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [chrony.chart.py](../collectors/python.d.plugin/chrony)<br/>configuration file: [python.d/chrony.conf](../collectors/python.d.plugin/chrony)| +ntpd|python<br/>v2 or v3|Connects to multiple ntpd servers (local or remote) to provide statistics of system variables and optional also peer variables (if enabled in the configuration).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [ntpd.chart.py](../collectors/python.d.plugin/ntpd)<br/>configuration file: [python.d/ntpd.conf](../collectors/python.d.plugin/ntpd)| + + +--- + +### Security + +application|language|notes| +:---------:|:------:|:----| +freeradius|python<br/>v2 or v3|Uses the radclient command to provide freeradius statistics (authentication, accounting, proxy-authentication, proxy-accounting).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [freeradius.chart.py](../collectors/python.d.plugin/freeradius)<br/>configuration file: [python.d/freeradius.conf](../collectors/python.d.plugin/freeradius)| +openvpn|python<br/>v2 or v3|All data from openvpn-status.log in your dashboard! <br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [ovpn_status_log.chart.py](../collectors/python.d.plugin/ovpn_status_log)<br/>configuration file: [python.d/ovpn_status_log.conf](../collectors/python.d.plugin/ovpn_status_log)| +fail2ban|python<br/>v2 or v3|Monitor fail2ban log file to show all bans for all active jails <br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [fail2ban.chart.py](../collectors/python.d.plugin/fail2ban)<br/>configuration file: [python.d/fail2ban.conf](../collectors/python.d.plugin/fail2ban)| + + +--- + +### Telephony Servers + +application|language|notes| +:---------:|:------:|:----| +opensips|BASH<br/>Shell Script|Connects to an opensips server (local only) to collect real-time performance metrics.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [opensips.chart.sh](../collectors/charts.d.plugin/opensips)<br/>configuration file: [charts.d/opensips.conf](../collectors/charts.d.plugin/opensips)| + + +--- + +### Go applications + +application|language|notes| +:---------:|:------:|:----| +go_expvar|python<br/>v2 or v3|Parses metrics exposed by applications written in the Go programming language using the [expvar package](https://golang.org/pkg/expvar/).<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [go_expvar.chart.py](../collectors/python.d.plugin/go_expvar)<br/>configuration file: [python.d/go_expvar.conf](../collectors/python.d.plugin/go_expvar)<br/>documentation: [Monitoring Go Applications](../collectors/python.d.plugin/go_expvar/)| + + +--- + +### Household Appliances + +application|language|notes| +:---------:|:------:|:----| +sma_webbox|node.js|Connects to multiple remote SMA webboxes to collect real-time performance metrics of the photovoltaic (solar) power generation.<br/> <br/>netdata plugin: [node.d.plugin](../collectors/node.d.plugin#nodedplugin)<br/>plugin module: [sma_webbox.node.js](../collectors/node.d.plugin/sma_webbox)<br/>configuration file: [node.d/sma_webbox.conf](../collectors/node.d.plugin/sma_webbox)| +fronius|node.js|Connects to multiple remote Fronius Symo servers to collect real-time performance metrics of the photovoltaic (solar) power generation.<br/> <br/>netdata plugin: [node.d.plugin](../collectors/node.d.plugin#nodedplugin)<br/>plugin module: [fronius.node.js](../collectors/node.d.plugin/fronius)<br/>configuration file: [node.d/fronius.conf](../collectors/node.d.plugin/fronius)| +stiebeleltron|node.js|Collects the temperatures and other metrics from your Stiebel Eltron heating system using their Internet Service Gateway (ISG web).<br/> <br/>netdata plugin: [node.d.plugin](../collectors/node.d.plugin#nodedplugin)<br/>plugin module: [stiebeleltron.node.js](../collectors/node.d.plugin/stiebeleltron)<br/>configuration file: [node.d/stiebeleltron.conf](../collectors/node.d.plugin/stiebeleltron)| + + +--- + +### Java Processes + +application|language|notes| +:---------:|:------:|:----| +Spring Boot Application|java|Monitors running Java [Spring Boot](https://spring.io/) applications that expose their metrics with the use of the **Spring Boot Actuator** included in Spring Boot library.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [springboot](../collectors/python.d.plugin/springboot)<br/>configuration file: [python.d/springboot.conf](../collectors/python.d.plugin/springboot) + + +--- + +### Provisioning Systems + +application|language|notes| +:---------:|:------:|:----| +puppet|python<br/>v2 or v3|Connects to multiple Puppet Server and Puppet DB instances (local or remote) to collect real-time status metrics.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [puppet.chart.py](../collectors/python.d.plugin/puppet)<br/>configuration file: [python.d/puppet.conf](../collectors/python.d.plugin/puppet)| + +--- + +### Game Servers + +application|language|notes| +:---------:|:------:|:----| +SpigotMC|Python<br/>v2 or v3|Monitors Spigot Minecraft server ticks per second and number of online players using the Minecraft remote console.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [spigotmc.chart.py](../collectors/python.d.plugin/spigotmc)<br/>configuration file: [python.d/spigotmc.conf](../collectors/python.d.plugin/spigotmc)| + +--- + +### Distributed Computing Clients + +application|language|notes| +:---------:|:------:|:----| +BOINC|Python<br/>v2 or v3|Monitors task states for local and remote BOINC client software using the remote GUI RPC interface. Also provides alarms for a handful of error conditions. Requires manual configuration<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [boinc.chart.py](../collectors/python.d.plugin/boinc)<br/>configuration file: [python.d/boinc.conf](../collectors/python.d.plugin/boinc)| + +--- + +### Skeleton Plugins + +application|language|notes| +:---------:|:------:|:----| +example|BASH<br/>Shell Script|Skeleton plugin in BASH.<br/><br/>DEPRECATED IN FAVOR OF THE PYTHON ONE. It is still supplied only as an example module to shell scripting plugins.<br/> <br/>netdata plugin: [charts.d.plugin](../collectors/charts.d.plugin#chartsdplugin)<br/>plugin module: [example.chart.sh](../collectors/charts.d.plugin/example)<br/>configuration file: [charts.d/example.conf](../collectors/charts.d.plugin/example)| +example|python<br/>v2 or v3|Skeleton plugin in Python.<br/> <br/>netdata plugin: [python.d.plugin](../collectors/python.d.plugin)<br/>plugin module: [example.chart.py](../collectors/python.d.plugin/example)<br/>configuration file: [python.d/example.conf](../collectors/python.d.plugin/example)| diff --git a/doc/Demo-Sites.md b/doc/Demo-Sites.md new file mode 100644 index 000000000..c9e0594ba --- /dev/null +++ b/doc/Demo-Sites.md @@ -0,0 +1,19 @@ +# Demo Sites + +Live demo installations of netdata are available at **[https://my-netdata.io](https://my-netdata.io)**: + +Location | netdata demo URL | 60 mins reqs | VM Donated by +:-------:|:-----------------:|:----------:|:------------- +London (UK)|**[london.my-netdata.io](https://london.my-netdata.io)**<br/>(this is the global netdata **registry** and has **named** and **mysql** charts)|[](https://london.my-netdata.io)|[DigitalOcean.com](https://m.do.co/c/83dc9f941745) +Atlanta (USA)|**[cdn77.my-netdata.io](https://cdn77.my-netdata.io)**<br/>(with **named** and **mysql** charts)|[](https://cdn77.my-netdata.io)|[CDN77.com](https://www.cdn77.com/) +Israel|**[octopuscs.my-netdata.io](https://octopuscs.my-netdata.io)**|[](https://octopuscs.my-netdata.io)|[OctopusCS.com](https://www.octopuscs.com) +Roubaix (France)|**[ventureer.my-netdata.io](https://ventureer.my-netdata.io)**|[](https://ventureer.my-netdata.io)|[Ventureer.com](https://ventureer.com/) +Madrid (Spain)|**[stackscale.my-netdata.io](https://stackscale.my-netdata.io)**|[](https://stackscale.my-netdata.io)|[StackScale Spain](https://www.stackscale.es/) +Bangalore (India)|**[bangalore.my-netdata.io](https://bangalore.my-netdata.io)**|[](https://bangalore.my-netdata.io)|[DigitalOcean.com](https://m.do.co/c/83dc9f941745) +Frankfurt (Germany)|**[frankfurt.my-netdata.io](https://frankfurt.my-netdata.io)**|[](https://frankfurt.my-netdata.io)|[DigitalOcean.com](https://m.do.co/c/83dc9f941745) +New York (USA)|**[newyork.my-netdata.io](https://newyork.my-netdata.io)**|[](https://newyork.my-netdata.io)|[DigitalOcean.com](https://m.do.co/c/83dc9f941745) +San Francisco (USA)|**[sanfrancisco.my-netdata.io](https://sanfrancisco.my-netdata.io)**|[](https://sanfrancisco.my-netdata.io)|[DigitalOcean.com](https://m.do.co/c/83dc9f941745) +Singapore|**[singapore.my-netdata.io](https://singapore.my-netdata.io)**|[](https://singapore.my-netdata.io)|[DigitalOcean.com](https://m.do.co/c/83dc9f941745) +Toronto (Canada)|**[toronto.my-netdata.io](https://toronto.my-netdata.io)**|[](https://toronto.my-netdata.io)|[DigitalOcean.com](https://m.do.co/c/83dc9f941745) + +*Netdata dashboards are mobile and touch friendly.* diff --git a/doc/Donations-netdata-has-received.md b/doc/Donations-netdata-has-received.md new file mode 100644 index 000000000..53cff3864 --- /dev/null +++ b/doc/Donations-netdata-has-received.md @@ -0,0 +1,23 @@ +# Donations received + +This is a list of the donations we have received for netdata (sorted alphabetically on their name): + +what donated|related links|who donated|description of the donation +----:|:-----:|:---:|:----------- +Packages Distribution|-|**[PackageCloud.io](https://packagecloud.io/)**|**PackageCloud.io** donated to a free open-source subscription to their awesome Package Distribution services. +Cross Browser Testing|-|**[BrowserStack.com](https://www.browserstack.com/)**|**BrowserStack.com** donated a free subscription to their awesome Browser Testing services (all three of them: Live, Screenshots, Responsive). +Cloud VM|[cdn77.my-netdata.io](http://cdn77.my-netdata.io)|**[CDN77.com](https://www.cdn77.com/)**|**CDN77.com** donated a VM with 2 CPU cores, 4GB RAM and 20GB HD, on their excellent CDN network. +Localization Management|[netdata localization project](https://crowdin.com/project/netdata) (check issue [#279](https://github.com/netdata/netdata/issues/279))|**[Crowdin.com](https://crowdin.com/)**|**Crowdin.com** donated an open source license to their Localization Management Platform. +Cloud VMs|[london.my-netdata.io](https://london.my-netdata.io) (Several VMs)|**[DigitalOcean.com](https://www.digitalocean.com/)**|**DigitalOcean.com** donated 1000 USD to be used in their excellent Cloud Computing services. Many thanks to [Justin Paine](https://github.com/xxdesmus) for making this happen. +Development IDE|-|**[JetBrains.com](https://www.jetbrains.com/)**|**JetBrains.com** donated an open source license for 4 developers for 1 year, to their excellent IDEs. +Cloud VM|[octopuscs.my-netdata.io](https://octopuscs.my-netdata.io)|**[OctopusCS.com](https://octopuscs.com/)**|**OctopusCS.com** donated a VM with 4 CPU cores, 16GB RAM and 50GB HD in their excellent Cloud Computing services. +Cloud VM|[ventureer.my-netdata.io](https://ventureer.my-netdata.io)|**[Ventureer.com](https://ventureer.com/)**|**Ventureer.com** donated a VM with 4 CPU cores, 8GB RAM and 50GB HD in their excellent Cloud Computing services. +Cloud VM|[stackscale.my-netdata.io](https://stackscale.my-netdata.io)|**[stackscale.com](https://www.stackscale.com/)**|**StackScale.com** donated a VM with 4 CPU cores, 16GB RAM and 100GB HD in their excellent Cloud Computing services. + +Thank you! + +--- + +**Do you want to donate?** We are thirsty for on-line services that can help us make netdata better. We also try to build a network of demo sites (VMs) that can help us show the full potential of netdata. + +Please contact me at costa@tsaousis.gr. diff --git a/doc/Netdata-Security-and-Disclosure-Information.md b/doc/Netdata-Security-and-Disclosure-Information.md new file mode 100644 index 000000000..86adfeeb9 --- /dev/null +++ b/doc/Netdata-Security-and-Disclosure-Information.md @@ -0,0 +1,37 @@ +# Netdata Security and Disclosure Information + +This page describes netdata security and disclosure information. + +## Security Announcements + +Every time a security issue is fixed in netdata, we immediately release a new version of it. So, to get notified of all security incidents, please subscribe to our releases on github. + +## Report a Vulnerability + +We’re extremely grateful for security researchers and users that report vulnerabilities to Netdata Open Source Community. All reports are thoroughly investigated by a set of community volunteers. + +To make a report, please email the private [security@netdata.cloud](mailto:security@netdata.cloud) list with the security details and the details expected for [all netdata bug reports](../.github/ISSUE_TEMPLATE/bug_report.md). + +## When Should I Report a Vulnerability? + +- You think you discovered a potential security vulnerability in Netdata +- You are unsure how a vulnerability affects Netdata +- You think you discovered a vulnerability in another project that Netdata depends on (e.g. python, node, etc) + +### When Should I NOT Report a Vulnerability? + +- You need help tuning Netdata for security +- You need help applying security related updates +- Your issue is not security related + +## Security Vulnerability Response + +Each report is acknowledged and analyzed by Netdata Team members within 3 working days. This will set off a Security Release Process. + +Any vulnerability information shared with Netdata Team stays within Netdata project and will not be disseminated to other projects unless it is necessary to get the issue fixed. + +As the security issue moves from triage, to identified fix, to release planning we will keep the reporter updated. + +## Public Disclosure Timing + +A public disclosure date is negotiated by the Netdata team and the bug submitter. We prefer to fully disclose the bug as soon as possible once a user mitigation is available. It is reasonable to delay disclosure when the bug or the fix is not yet fully understood, the solution is not well-tested, or for vendor coordination. The timeframe for disclosure is from immediate (especially if it's already publicly known) to a few weeks. As a basic default, we expect report date to disclosure date to be on the order of 7 days. The Netdata team holds the final say when setting a disclosure date. diff --git a/doc/Performance.md b/doc/Performance.md new file mode 100644 index 000000000..ef15a871a --- /dev/null +++ b/doc/Performance.md @@ -0,0 +1,73 @@ +# Netdata Performance + +Netdata performance is affected by: + +**Data collection** +- the number of charts for which data are collected +- the number of plugins running +- the technology of the plugins (i.e. BASH plugins are slower than binary plugins) +- the frequency of data collection + +You can control all the above. + +**Web clients accessing the data** +- the duration of the charts in the dashboard +- the number of charts refreshes requested +- the compression level of the web responses + +--- + +## Netdata Daemon + +For most server systems, with a few hundred charts and a few thousand dimensions, the netdata daemon, without any web clients accessing it, should not use more than 1% of a single core. + +To prove netdata scalability, check issue [#1323](https://github.com/netdata/netdata/issues/1323#issuecomment-265501668) where netdata collects 95.000 metrics per second, with 12% CPU utilization of a single core! + +In embedded systems, if the netdata daemon is using a lot of CPU without any web clients accessing it, you should lower the data collection frequency. To set the data collection frequency, edit `/etc/netdata/netdata.conf` and set `update_every` to a higher number (this is the frequency in seconds data are collected for all charts: higher number of seconds = lower frequency, the default is 1 for per second data collection). You can also set this frequency per module or chart. Check the **[[Configuration]]** section. + +## Plugins + +If a plugin is using a lot of CPU, you should lower its update frequency, or if you wrote it, re-factor it to be more CPU efficient. Check **[[External Plugins]]** for more details on writing plugins. + +## CPU consumption when web clients are accessing dashboards + +Netdata is very efficient when servicing web clients. On most server platforms, netdata should be able to serve **1800 web client requests per second per core** for auto-refreshing charts. + +Normally, each user connected will request less than 10 chart refreshes per second (the page may have hundreds of charts, but only the visible are refreshed). So you can expect 180 users per CPU core accessing dashboards before having any delays. + +Netdata runs with the lowest possible process priority, so even if 1000 users are accessing dashboards, it should not influence your applications. CPU utilization will reach 100%, but your applications should get all the CPU they need. + +To lower the CPU utilization of netdata when clients are accessing the dashboard, set `web compression level = 1`, or disable web compression completely by setting `enable web responses gzip compression = no`. Both settings are in the `[web]` section. + + +## Monitoring a heavy loaded system + +Netdata, while running, does not depend on disk I/O (apart its log files and `access.log` is written with buffering enabled and can be disabled). Some plugins that need disk may stop and show gaps during heavy system load, but the netdata daemon itself should be able to work and collect values from `/proc` and `/sys` and serve web clients accessing it. + +Keep in mind that netdata saves its database when it exits and loads it back when restarted. While it is running though, its DB is only stored in RAM and no I/O takes place for it. + + +## Running netdata in embedded devices + +Embedded devices usually have very limited CPU resources available, and in most cases, just a single core. + +We suggest to do the following: + +#### external plugins + + `charts.d.plugin` and `apps.plugin`, each consumes twice the CPU resources of the netdata daemon. + + If you don't need them, disable them (edit `/etc/netdata/netdata.conf` and search for the plugins section). + + If you need them, increase their `update every` value (again in `/etc/netdata/netdata.conf`), so that they do not run that frequently. + +#### internal plugins + +If netdata is still using a lot of CPU, lower its update frequency. Going from per second updates, to once every 2 seconds updates, will cut the CPU resources of all netdata programs **in half**, and you will still have very frequent updates. + +If the CPU of the embedded device is too weak, try setting even lower update frequency. Experiment with `update every = 5` or `update every = 10` (higher number = lower frequency), until you get acceptable results. + +#### Single threaded web server + +Normally, netdata spawns a thread for each web client. This allows netdata to utilize all the available cores for servicing chart refreshes. You can however disable this feature and serve all charts one after another, using a single thread / core. This will might lower the CPU pressure on the embedded device. To enable the single threaded web server, edit `/etc/netdata/netdata.conf` and set `mode = single-threaded` in the `[web]` section. + diff --git a/doc/Running-behind-apache.md b/doc/Running-behind-apache.md new file mode 100644 index 000000000..02d2be92f --- /dev/null +++ b/doc/Running-behind-apache.md @@ -0,0 +1,268 @@ +# netdata via apache's mod_proxy + +Below you can find instructions for configuring an apache server to: + +1. proxy a single netdata via an HTTP and HTTPS virtual host +2. dynamically proxy any number of netdata +3. add user authentication +4. adjust netdata settings to get optimal results + + +## Requirements + +Make sure your apache has installed `mod_proxy` and `mod_proxy_http`. + +On debian/ubuntu systems, install them with this: + +```sh +sudo apt-get install libapache2-mod-proxy-html +``` + +Also make sure they are enabled: + +``` +sudo a2enmod proxy +sudo a2enmod proxy_http +``` + +Ensure your rewrite module is enabled: + +``` +sudo a2enmod rewrite +``` + +--- + +## netdata on an existing virtual host + +On any **existing** and already **working** apache virtual host, you can redirect requests for URL `/netdata/` to one or more netdata servers. + +### proxy one netdata, running on the same server apache runs + +Add the following on top of any existing virtual host. It will allow you to access netdata as `http://virtual.host/netdata/`. + +``` +<VirtualHost *:80> + + RewriteEngine On + ProxyRequests Off + ProxyPreserveHost On + + <Proxy *> + Require all granted + </Proxy> + + # Local netdata server accessed with '/netdata/', at localhost:19999 + ProxyPass "/netdata/" "http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on + ProxyPassReverse "/netdata/" "http://localhost:19999/" + + # if the user did not give the trailing /, add it + # for HTTP (if the virtualhost is HTTP, use this) + RewriteRule ^/netdata$ http://%{HTTP_HOST}/netdata/ [L,R=301] + # for HTTPS (if the virtualhost is HTTPS, use this) + #RewriteRule ^/netdata$ https://%{HTTP_HOST}/netdata/ [L,R=301] + + # rest of virtual host config here + +</VirtualHost> +``` + +### proxy multiple netdata running on multiple servers + +Add the following on top of any existing virtual host. It will allow you to access multiple netdata as `http://virtual.host/netdata/HOSTNAME/`, where `HOSTNAME` is the hostname of any other netdata server you have (to access the `localhost` netdata, use `http://virtual.host/netdata/localhost/`). + +``` +<VirtualHost *:80> + + RewriteEngine On + ProxyRequests Off + ProxyPreserveHost On + + <Proxy *> + Require all granted + </Proxy> + + # proxy any host, on port 19999 + ProxyPassMatch "^/netdata/([A-Za-z0-9\._-]+)/(.*)" "http://$1:19999/$2" connectiontimeout=5 timeout=30 keepalive=on + + # make sure the user did not forget to add a trailing / + # for HTTP (if the virtualhost is HTTP, use this) + RewriteRule "^/netdata/([A-Za-z0-9\._-]+)$" http://%{HTTP_HOST}/netdata/$1/ [L,R=301] + # for HTTPS (if the virtualhost is HTTPS, use this) + RewriteRule "^/netdata/([A-Za-z0-9\._-]+)$" https://%{HTTP_HOST}/netdata/$1/ [L,R=301] + + # rest of virtual host config here + +</VirtualHost> +``` + +> IMPORTANT<br/> +> The above config allows your apache users to connect to port 19999 on any server on your network. + +If you want to control the servers your users can connect to, replace the `ProxyPassMatch` line with the following. This allows only `server1`, `server2`, `server3` and `server4`. + +``` + ProxyPassMatch "^/netdata/(server1|server2|server3|server4)/(.*)" "http://$1:19999/$2" connectiontimeout=5 timeout=30 keepalive=on +``` + +## netdata on a dedicated virtual host + +You can proxy netdata through apache, using a dedicated apache virtual host. + +Create a new apache site: + +```sh +nano /etc/apache2/sites-available/netdata.conf +``` + +with this content: + +``` +<VirtualHost *:80> + RewriteEngine On + ProxyRequests Off + ProxyPreserveHost On + + ServerName netdata.domain.tld + + <Proxy *> + Require all granted + </Proxy> + + ProxyPass "/" "http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on + ProxyPassReverse "/" "http://localhost:19999/" + + ErrorLog ${APACHE_LOG_DIR}/netdata-error.log + CustomLog ${APACHE_LOG_DIR}/netdata-access.log combined +</VirtualHost> +``` + +Enable the VirtualHost: + +```sh +sudo a2ensite netdata.conf && service apache2 reload +``` + +## Netdata proxy in Plesk +_Assuming the main goal is to make Netdata running in HTTPS._ +1. Make a subdomain for Netdata on which you enable and force HTTPS - You can use a free Let's Encrypt certificate +2. Go to "Apache & nginx Settings", and in the following section, add: +``` +RewriteEngine on +RewriteRule (.*) http://localhost:19999/$1 [P,L] +``` +3. Optional: If your server is remote, then just replace "localhost" with your actual hostname or IP, it just works. + +Repeat the operation for as many servers as you need. + + +## Enable Basic Auth + +If you wish to add an authentication (user/password) to access your netdata, do these: + +Install the package `apache2-utils`. On debian / ubuntu run `sudo apt-get install apache2-utils`. + +Then, generate password for user `netdata`, using `htpasswd -c /etc/apache2/.htpasswd netdata` + +Modify the virtual host with these: + +``` + # replace the <Proxy *> section + <Proxy *> + Order deny,allow + Allow from all + </Proxy> + + # add a <Location /netdata/> section + <Location /netdata/> + AuthType Basic + AuthName "Protected site" + AuthUserFile /etc/apache2/.htpasswd + Require valid-user + Order deny,allow + Allow from all + </Location> +``` + +Specify `Location /` if netdata is running on dedicated virtual host. + +Note: Changes are applied by reloading or restarting Apache. + +# Netdata configuration + +You might edit `/etc/netdata/netdata.conf` to optimize your setup a bit. For applying these changes you need to restart netdata. + +## Response compression + +If you plan to use netdata exclusively via apache, you can gain some performance by preventing double compression of its output (netdata compresses its response, apache re-compresses it) by editing `/etc/netdata/netdata.conf` and setting: + +``` +[web] + enable gzip compression = no +``` + +Once you disable compression at netdata (and restart it), please verify you receive compressed responses from apache (it is important to receive compressed responses - the charts will be more snappy). + +## Limit direct access to netdata + +You would also need to instruct netdata to listen only on `localhost`, `127.0.0.1` or `::1`. + +``` +[web] + bind to = localhost +``` +or +``` +[web] + bind to = 127.0.0.1 +``` +or +``` +[web] + bind to = ::1 +``` + +--- + +You can also use a unix domain socket. This will also provide a faster route between apache and netdata: + +``` +[web] + bind to = unix:/tmp/netdata.sock +``` +_note: netdata v1.8+ support unix domain sockets_ + +At the apache side, prepend the 2nd argument to `ProxyPass` with `unix:/tmp/netdata.sock|`, like this: + +``` +ProxyPass "/netdata/" "unix:/tmp/netdata.sock|http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on +``` + +--- + +If your apache server is not on localhost, you can set: + +``` +[web] + bind to = * + allow connections from = IP_OF_APACHE_SERVER +``` +_note: netdata v1.9+ support `allow connections from`_ + +`allow connections from` accepts [netdata simple patterns](../libnetdata/simple_pattern/) to match against the connection IP address. + +## prevent the double access.log + +apache logs accesses and netdata logs them too. You can prevent netdata from generating its access log, by setting this in `/etc/netdata/netdata.conf`: + +``` +[global] + access log = none +``` + +## Troubleshooting mod_proxy + +Make sure the requests reach netdata, by examing `/var/log/netdata/access.log`. + +1. if the requests do not reach netdata, your apache does not forward them. +2. if the requests reach netdata by the URLs are wrong, you have not re-written them properly. diff --git a/doc/Running-behind-caddy.md b/doc/Running-behind-caddy.md new file mode 100644 index 000000000..2fc3fd634 --- /dev/null +++ b/doc/Running-behind-caddy.md @@ -0,0 +1,27 @@ +# netdata via Caddy + +To run netdata via [Caddy's proxying,](https://caddyserver.com/docs/proxy) set your Caddyfile up like this: + +``` +netdata.domain.tld { + proxy / localhost:19999 +} +``` + +Other directives can be added between the curly brackets as needed. + +To run netdata in a subfolder: + +``` +netdata.domain.tld { + proxy /netdata/ localhost:19999 { + without /netdata + } +} +``` + +## limit direct access to netdata + +You would also need to instruct netdata to listen only to `127.0.0.1` or `::1`. + +To limit access to netdata only from localhost, set `bind socket to IP = 127.0.0.1` or `bind socket to IP = ::1` in `/etc/netdata/netdata.conf`. diff --git a/doc/Running-behind-lighttpd.md b/doc/Running-behind-lighttpd.md new file mode 100644 index 000000000..17fb9c629 --- /dev/null +++ b/doc/Running-behind-lighttpd.md @@ -0,0 +1,60 @@ +# lighttpd v1.4.x + +Here is a config for accessing netdata in a suburl via lighttpd 1.4.46 and newer: + +```txt +$HTTP["url"] =~ "^/netdata/" { + proxy.server = ( "" => ("netdata" => ( "host" => "127.0.0.1", "port" => 19999 ))) + proxy.header = ( "map-urlpath" => ( "/netdata/" => "/") ) +} +``` + +If you have older lighttpd you have to use a chain (such as bellow), as explained [at this stackoverflow answer](http://stackoverflow.com/questions/14536554/lighttpd-configuration-to-proxy-rewrite-from-one-domain-to-another). + +```txt +$HTTP["url"] =~ "^/netdata/" { + proxy.server = ( "" => ("" => ( "host" => "127.0.0.1", "port" => 19998 ))) +} + +$SERVER["socket"] == ":19998" { + url.rewrite-once = ( "^/netdata(.*)$" => "/$1" ) + proxy.server = ( "" => ( "" => ( "host" => "127.0.0.1", "port" => 19999 ))) +} +``` + +--- + +If the only thing the server is exposing via the web is netdata (and thus no suburl rewriting required), +then you can get away with just +``` +proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => 19999 ))) +``` +Though if it's public facing you might then want to put some authentication on it. htdigest support +looks like: +``` +auth.backend = "htdigest" +auth.backend.htdigest.userfile = "/etc/lighttpd/lighttpd.htdigest" +auth.require = ( "" => ( "method" => "digest", + "realm" => "netdata", + "require" => "valid-user" + ) + ) +``` +other auth methods, and more info on htdigest, can be found in lighttpd's [mod_auth docs](http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModAuth). + +--- + +It seems that lighttpd (or some versions of it), fail to proxy compressed web responses. +To solve this issue, disable web response compression in netdata. + +Open /etc/netdata/netdata.conf and set in [global]: + +``` +enable web responses gzip compression = no +``` + +## limit direct access to netdata + +You would also need to instruct netdata to listen only to `127.0.0.1` or `::1`. + +To limit access to netdata only from localhost, set `bind socket to IP = 127.0.0.1` or `bind socket to IP = ::1` in `/etc/netdata/netdata.conf`. diff --git a/doc/Running-behind-nginx.md b/doc/Running-behind-nginx.md new file mode 100644 index 000000000..76062e035 --- /dev/null +++ b/doc/Running-behind-nginx.md @@ -0,0 +1,202 @@ +# netdata via nginx + +To pass netdata via a nginx, use this: + +### As a virtual host + +``` +upstream backend { + # the netdata server + server 127.0.0.1:19999; + keepalive 64; +} + +server { + # nginx listens to this + listen 80; + + # the virtual host name of this + server_name netdata.example.com; + + location / { + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_pass http://backend; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + } +} +``` + +### As a subfolder to an existing virtual host + +``` +upstream netdata { + server 127.0.0.1:19999; + keepalive 64; +} + +server { + listen 80; + + # the virtual host name of this subfolder should be exposed + #server_name netdata.example.com; + + location = /netdata { + return 301 /netdata/; + } + + location ~ /netdata/(?<ndpath>.*) { + proxy_redirect off; + proxy_set_header Host $host; + + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + proxy_pass http://netdata/$ndpath$is_args$args; + + gzip on; + gzip_proxied any; + gzip_types *; + } +} +``` + +### As a subfolder for multiple netdata servers, via one nginx + +``` +upstream backend-server1 { + server 10.1.1.103:19999; + keepalive 64; +} +upstream backend-server2 { + server 10.1.1.104:19999; + keepalive 64; +} + +server { + listen 80; + + # the virtual host name of this subfolder should be exposed + #server_name netdata.example.com; + + location ~ /netdata/(?<behost>.*)/(?<ndpath>.*) { + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + proxy_pass http://backend-$behost/$ndpath$is_args$args; + + gzip on; + gzip_proxied any; + gzip_types *; + } + + # make sure there is a trailing slash at the browser + # or the URLs will be wrong + location ~ /netdata/(?<behost>.*) { + return 301 /netdata/$behost/; + } +} +``` + +Of course you can add as many backend servers as you like. + +Using the above, you access netdata on the backend servers, like this: + +- `http://nginx.server/netdata/server1/` to reach `backend-server1` +- `http://nginx.server/netdata/server2/` to reach `backend-server2` + + +### Enable authentication + +Create an authentication file to enable the nginx basic authentication. +Do not use authentication without SSL/TLS! +If you haven't one you can do the following: + +``` +printf "yourusername:$(openssl passwd -apr1)" > /etc/nginx/passwords +``` + +And enable the authentication inside your server directive: + +``` +server { + # ... + auth_basic "Protected"; + auth_basic_user_file passwords; + # ... +} +``` + +## limit direct access to netdata + +If your nginx is on `localhost`, you can use this to protect your netdata: + +``` +[web] + bind to = 127.0.0.1 ::1 +``` + +--- + +You can also use a unix domain socket. This will also provide a faster route between nginx and netdata: + +``` +[web] + bind to = unix:/tmp/netdata.sock +``` +_note: netdata v1.8+ support unix domain sockets_ + +At the nginx side, use something like this to use the same unix domain socket: + +``` +upstream backend { + server unix:/tmp/netdata.sock; + keepalive 64; +} +``` + +--- + +If your nginx server is not on localhost, you can set: + +``` +[web] + bind to = * + allow connections from = IP_OF_NGINX_SERVER +``` + +_note: netdata v1.9+ support `allow connections from`_ + +`allow connections from` accepts [netdata simple patterns](../libnetdata/simple_pattern/) to match against the connection IP address. + +## prevent the double access.log + +nginx logs accesses and netdata logs them too. You can prevent netdata from generating its access log, by setting this in `/etc/netdata/netdata.conf`: + +``` +[global] + access log = none +``` + +## SELinux + +If you get an 502 Bad Gateway error you might check your nginx error log: + +```sh +# cat /var/log/nginx/error.log: +2016/09/09 12:34:05 [crit] 5731#5731: *1 connect() to 127.0.0.1:19999 failed (13: Permission denied) while connecting to upstream, client: 1.2.3.4, server: netdata.example.com, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:19999/", host: "netdata.example.com" +``` + +If you see something like the above, chances are high that SELinux prevents nginx from connecting to the backend server. To fix that, just use this policy: `setsebool -P httpd_can_network_connect true`. diff --git a/doc/Third-Party-Plugins.md b/doc/Third-Party-Plugins.md new file mode 100644 index 000000000..d50aa417d --- /dev/null +++ b/doc/Third-Party-Plugins.md @@ -0,0 +1,29 @@ +# Third-party Plugins + +The following is a list of Netdata plugins distributed by third parties: + +## Nvidia GPUs + +[netdata nv plugin](https://github.com/coraxx/netdata_nv_plugin) monitors nvidia GPUs. + + + +## teamspeak 3 + +[teamspeak 3 plugin](https://github.com/coraxx/netdata_ts3_plugin) polls active users and bandwidth from TeamSpeak 3 servers. + +## SSH + +[SSH module](https://github.com/Yaser-Amiri/netdata-ssh-module) monitors failed authentication requests of SSH server. + +## interactive users count + +Collect [number of currently logged-on users](https://github.com/veksh/netdata-numsessions) + +## CyberPower UPS + +[cyberups plugin](https://github.com/HawtDogFlvrWtr/netdata_cyberpwrups_plugin) polls the USB connected CyberPower UPS for stats. + +## Nim + +There is an unofficial [nim plugin helper](https://github.com/FedericoCeratto/nim-netdata-plugin) diff --git a/doc/Why-Netdata.md b/doc/Why-Netdata.md new file mode 100644 index 000000000..57ff722ec --- /dev/null +++ b/doc/Why-Netdata.md @@ -0,0 +1,170 @@ +# Why Netdata + + + +## Netdata is unique! + +The following is an animated GIF showing **netdata**'s ability to monitor QoS. The timings of this animation have not been altered, this is the real thing: + + + +Check the details on this animation: + +1. At the beginning the charts auto-refresh, in real-time +2. Charts can be dragged and zoomed (either mouse or touch) +3. You pan or zoom one, the others follow +4. Mouse over on one, selects the same timestamp on all +5. Dimensions can be enabled or disabled +6. All refreshes are instant (an 8 year old core-2 duo computer was used to record this) + +There are a lot of excellent open source tools for collecting and visualizing performance metrics. Check for example [collectd](https://collectd.org/), [OpenTSDB](http://opentsdb.net/), [influxdb](https://influxdata.com/), [Grafana](http://grafana.org/), etc. + +So, why **netdata**? + +Well, **netdata** has a quite different approach. + +## Simplicity + +> Most monitoring solutions require endless configuration of whatever imaginable. Well, this is a linux box. Why do we need to configure every single metric we need to monitor. Of course it has a CPU and RAM and a few disks, and ethernet ports, it might run a firewall, a web server, or a database server and so on. Why do we need to configure all these metrics? + +**Netdata** has been designed to auto-detect everything. Of course you can enable, tweak or disable things. But by default, if **netdata** can retrieve `/server-status` from an web server you run on your linux box, it will automatically collect all performance metrics. This happens for apache, squid, nginx, mysql, opensips, etc. It will also automatically collect all available system values for CPU, memory, disks, network interfaces, QoS (with labels if you also use [FireQOS](http://firehol.org)), etc. Even for applications that do not offer performance metrics, it will automatically group the whole process tree and provide metrics like CPU usage, memory allocated, opened files, sockets, disk activity, swap activity, etc per application group. + +Netdata supports plenty of [configuration](../daemon/config/). However, we have done everything we can to allow netdata to auto-detect as much as possible. + +Even netdata plugins are designed to support configuration-less operation. So, you just install and run netdata. You will need to configure something only if it cannot be auto-detected. + +> Take any performance monitoring solution and try to troubleshoot a performance problem. At the end of the day you will have to ssh to the server to understand what exactly is happening. You will have to use `iostat`, `iotop`, `vmstat`, `top`, `iperf`, `ethtool` and probably a few dozen more console tools to figure it out. + +With **netdata**, this need is eliminated significantly. Of course you will ssh. Just not for monitoring performance. + +If you install **netdata** you will prefer it over the console tools. **Netdata** visualizes the data, while the console tools just show their values. The detail is the same - I have spent pretty much time reading the source code of the console tools, to figure out what needs to do done in netdata, so that the data, the values, will be the same. Actually, **netdata** is more precise than most console tools, it will interpolate all collected values to second boundary, so that even if something took a few microseconds more to be collected, netdata will correctly estimate the per second value. + +**Netdata** visualizes data in ways you cannot even imagine on a console. It allows you to see the present in real-time, much like the console tools, but also the recent past, compare different metrics with each other, zoom in to see the recent past in detail, or zoom out to have a helicopter view of what is happening in longer durations, build custom dashboards with just the charts you need for a specific purpose. + +Most engineers that install netdata, ssh to the server to tweak system or application settings and at the same time they monitor the result of the new settings in **netdata** on their browser. + +## Per second data collection and visualization + +**Per second data collection and visualization** is usually only available in dedicated console tools, like `top`, `vmstat`, `iostat`, etc. Netdata brings per second data collection and visualization to all applications, accessible through the web. + +*You are not convinced per second data collection is important?* +**Click** this image for a demo: + +[](http://netdata.firehol.org/demo2.html) + +## Realtime monitoring + +> Any performance monitoring solution that does not go down to per second collection and visualization of the data, is useless. It will make you happy to have it, but it will not help you more than that. + +Visualizing the present in **real-time and in great detail**, is the most important value a performance monitoring solution should provide. The next most important is the last hour, again per second. The next is the last 8 hours and so on, up to a week, or at most a month. In my 20+ years in IT, I needed just once or twice to look a year back. And this was mainly out of curiosity. + +Of course real-time monitoring requires resources. **netdata** is designed to be very efficient: + +1. collecting performance data is a repeating process - you do the same thing again and again. **Netdata** has been designed to learn from each iteration, so that the next one will be faster. It learns the sizes of files (it even keeps them open when it can), the number of lines and words per line they contain, the sizes of the buffers it needs to process them, etc. It adapts, so that everything will be as ready as possible for the next iteration. +2. internally, it uses hashes and indexes (b-trees), to speed up lookups of metrics, charts, dimensions, settings. +3. it has an in-memory round robin database based on a custom floating point number that allows it to pack values and flags together, in 32 bits, to lower its memory footprint. +4. its internal web server is capable of generating JSON responses from live performance data with speeds comparable to static content delivery (it does not use `printf`, it is actually 11 times faster than in generating JSON compared to `printf`). + +**Netdata** will use some CPU and memory, but it **will not produce any disk I/O at all**, apart its logs (which you can disable if you like). + +Most servers should have plenty of CPU resources (I consider a hardware upgrade or application split when a server averages around 40% CPU utilization at the peak hour). Even if a server has limited CPU resources available, you can just lower the data collection frequency of **netdata**. Going from per second to every 2 seconds data collection, will cut the **netdata** CPU requirements in half and you will still get charts that are just 2 seconds behind. + +The same goes for memory. If you just keep an hour of data (which is perfect for performance troubleshooting), you will most probably need 15-20MB. You can also enable the kernel de-duper (Kernel Same-Page Merging) and **netdata** will offer to it all its round robin database. KSM can free 20-60% of the memory used by **netdata** (guess why: there are a lot of metrics that are always zero or just constant). + +When netdata runs on modern computers (even on CELERON processors), most chart queries are replied in less than 3 milliseconds! **Not seconds, MILLISECONDS!** Less than 3 milliseconds for calculating the chart, generating JSON text, compressing it and sending it to your web browser. Timings are logged in netdata's `access.log` for you to examine. + +Netdata is written in plain `C` and the key system plugins are written in `C` too. Its speed can only be compared to the native console system administration tools. + +You can also stress test your netdata installation by running the script `tests/stress.sh` found in the distribution. Most modern server hardware can serve more than 300 chart refreshes per second per core. A raspberry pi 2, can serve 300+ chart refreshes per second utilizing all of its 4 cores. + + +## No disk I/O at all + +Netdata does not use any disk I/O, apart from its logs and even these can be disabled. + +Netdata will use some memory (you size it, check [[Memory Requirements]] and CPU (below 2% of a single core for the daemon, plugins may require more, check [[Performance]]), but normally your systems should have plenty of these resources available and spare. + +The design goal of **NO DISK I/O AT ALL** effectively means netdata will not disrupt your applications. + +## No root access + +You don't need to run netdata as root. If started as root, netdata will switch to the `netdata` user (or any other user given in its configuration or command line argument). + +There are a few plugins that in order to collect values need root access. These (and only these) are setuid to root. + +## Visualizes QoS + +Netdata visualizes `tc` QoS classes automatically. If you also use FireQOS, it will also collect interface and class names. + +Check this animated GIF (generated with [ScreenToGif](https://github.com/NickeManarin/ScreenToGif)): + + + +## Embedded web server + +> Most solutions require dedicated servers to actually use the monitoring console. To my perspective, this is totally unneeded for performance monitoring. All of us have a spectacular tool on our desktops, that allows us to connect in real time to any server in the world: **the web browser**. It shouldn't be so hard to use the same tool to connect in real-time to all our servers. + +With **netdata**, there is no need to centralize anything for performance monitoring. You view everything directly from their source. No need to run something else to access netdata. Of course you can use a firewall, or a reverse proxy, to limit access to it. But for most systems, inside your DMZ, just running it will be enough. + +Still, with **netdata** you can build dashboards with charts from any number of servers. And these charts will be connected to each other much like the ones that come from the same server. You will hover on one and all of them will show the relative value for the selected timestamp. You will zoom or pan one and all of them will follow. **Netdata** achieves that because the logic that connects the charts together is at the browser, not the server, so that all charts presented on the same page are connected, no matter where they come from. + +## Performance monitoring, scaled properly + +"Properly"? What is "properly"? + +We know software solutions can **scale up** (i.e. you replace its resources with bigger ones), or **scale out** (i.e. you add more smaller resources to it). In both cases, to get more of it, you need to supply **more resources**. + +So, what is "scaled properly"? + +Traditionally, monitoring solutions centralize all metric data to provide unified dashboards across all servers. So, you install agents on all your servers to collect system and application metrics which are then sent to a central place for storage and processing. Depending on the solution you use, the central place can either **scale up** or **scale out** (or a mix of the two). + +"Scaled properly" is something completely different. "Scaled properly" minimizes the need for a "central place", so that **there is nothing to be scaled**! + +Wait a moment! You cannot take out the "central place" of a monitoring solution! + +Yes, we can! well... most of it, but before explaining how, let's see what happens today: + +Monitoring solutions are a key component for any online service. These solutions usually consume considerable amount of resources. This is true for both "scale-up" and "scale-out" solutions. These resources require maintenance and administration too. To balance the resources required, these monitoring solutions follow a few simple rules: + +1. The number of metrics collected per server is limited. They collect CPU, RAM, DISK, NETWORK metrics and a few application metrics. + +2. The data collection frequency of each metric is also very low, at best it is once every 10 or 15 seconds, at worst every 5 or 10 mins. + +Due to all the above, most centralized monitoring solutions are usually good for alarms and **statistics of past performance**. The alarms usually trigger every 1 to 5 minutes and you get a few low-resolution charts about the past performance of your servers. + +Well... there is something wrong in this approach! Can you see it? + +Let's see the netdata approach: + +1. Data collection happens **per second**. This allows true real-time performance monitoring. + +2. **Thousands of metrics** per server and application are collected, **every single second**. The number of metrics collected is not a problem. + +3. Data do not leave the server they are collected. Data are not centralized, so the need for a huge central place that will process and store gazillions of data is not needed. + + > Ok, I hear a few of you complaining already - you will find out... patience... + +4. netdata does not use any DISK I/O while running (apart its log files - and even these can be disabled) and netdata runs with the lowest possible process priority, so that **your applications will never be affected by it**. + +5. Each netdata is standalone. Your web browser connects directly to each server to present real-time dashboards. The charts are so snappy, so real-time, so fast that we can call netdata, **a console killer for performance monitoring**. + +The charting libraries **netdata** uses, are the fastest possible ([Dygraphs](http://dygraphs.com/) do make the difference!) and **netdata** respects browser resources. Data are just rendered on a canvas. No processing in javascript at all. + +6. netdata is very efficient: just 2% of a single core is required and some RAM, and you can actually control how much of both you want to allocate to it. + + +Server side, chart data generation scales pretty well. You can expect 400+ chart refreshes per second per core on modern hardware. For a page with 10 charts visible (the page may have hundreds, but only the visible are refreshed), just a tiny fraction of a single CPU core will be used for servicing them. Even these refreshes stop when you switch tabs on your browser, you focus on another window, scroll to a part of the page without charts, zoom or pan a chart. And of course the **netdata** server runs with the lowest possible process priority, so that your production environment, your applications, will not be slowed down by the netdata server. + +7. netdata dashboards can be multi-server (check: [http://my-netdata.io](http://my-netdata.io)) - your browser connects to each netdata server directly. + +So, using netdata, your monitoring infrastructure is embedded on each server, limiting significantly the need of additional resources. netdata is very resource efficient and utilizes server resources that already exist and are spare (on each server). + +Of course, there are a few issues that need to be addressed with this approach: + +1. We need an index of all netdata installations we have +2. We need a place to handle notifications and alarms +3. We need a place to save statistics of past performance + +Our approach uses the netdata [registry](../registry/). The registry solves the problem of maintaining a list of all the netdata installations we have. It does this transparently, without any configuration. It tracks the netdata servers your web browser has visited and bookmarks them at the `my-netdata` menu. + +Every netdata can be a registry. You can use the global one we provided for free, or pick one of your netdata servers and turn it to a registry for your network. diff --git a/doc/a-github-star-is-important.md b/doc/a-github-star-is-important.md new file mode 100644 index 000000000..c00fba300 --- /dev/null +++ b/doc/a-github-star-is-important.md @@ -0,0 +1,13 @@ +# A GitHub start is important + +**GitHub stars** allow netdata to expand its reach, its community, especially attract people with skills willing to contribute to it. + +Compared to its first release, netdata is now **twice as fast**, has all its bugs settled and a lot more functionality. This happened because a lot of people find it useful, use it daily at home and work, **rely on it** and **contribute to it**. + +**GitHub stars** also **motivate** us. They state that you find our work **useful**. They give us strength to continue, to work **harder** to make it even **better**. + +So, give netdata a **GitHub star**, at the top right of this page. + +Thank you! + +Costa Tsaousis diff --git a/doc/high-performance-netdata.md b/doc/high-performance-netdata.md new file mode 100644 index 000000000..1671acab8 --- /dev/null +++ b/doc/high-performance-netdata.md @@ -0,0 +1,149 @@ +# High performance netdata + +If you plan to run a netdata public on the internet, you will get the most performance out of it by following these rules: + +## 1. run behind nginx + +The internal web server is optimized to provide the best experience with few clients connected to it. Normally a web browser will make 4-6 concurrent connections to a web server, so that it can send requests in parallel. To best serve a single client, netdata spawns a thread for each connection it receives (so 4-6 threads per connected web browser). + +If you plan to have your netdata public on the internet, this strategy wastes resources. It provides a lock-free environment so each thread is autonomous to serve the browser, but it does not scale well. Running netdata behind nginx, idle connections to netdata can be reused, thus improving significantly the performance of netdata. + +In the following nginx configuration we do the following: + +- allow nginx to maintain up to 1024 idle connections to netdata (so netdata will have up to 1024 threads waiting for requests) + +- allow nginx to compress the responses of netdata (later we will disable gzip compression at netdata) + +- we disable wordpress pingback attacks and allow only GET, HEAD and OPTIONS requests. + +``` +upstream backend { + server 127.0.0.1:19999; + keepalive 1024; +} + +server { + listen *:80; + server_name my.web.server.name; + + location / { + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Server $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_pass http://backend; + proxy_http_version 1.1; + proxy_pass_request_headers on; + proxy_set_header Connection "keep-alive"; + proxy_store off; + gzip on; + gzip_proxied any; + gzip_types *; + + # Block any HTTP requests other than GET, HEAD, and OPTIONS + limit_except GET HEAD OPTIONS { + deny all; + } + } + + # WordPress Pingback Request Denial + if ($http_user_agent ~* "WordPress") { + return 403; + } + +} +``` + +Then edit `/etc/netdata/netdata.conf` and set these config options: + +``` +[global] + bind socket to IP = 127.0.0.1 + access log = none + disconnect idle web clients after seconds = 3600 + enable web responses gzip compression = no +``` + +These options: + +- `[global].bind socket to IP = 127.0.0.1` makes netdata listen only for requests from localhost (nginx). +- `[global].access log = none` disables the access.log of netdata. It is not needed since netdata only listens for requests on 127.0.0.1 and thus only nginx can access it. nginx has its own access.log for your record. +- `[global].disconnect idle web clients after seconds = 3600` will kill inactive web threads after an hour of inactivity. +- `[global].enable web responses gzip compression = no` disables gzip compression at netdata (nginx will compress the responses). + +## 2. increase open files limit (non-systemd) + +By default Linux limits open file descriptors per process to 1024. This means that less than half of this number of client connections can be accepted by both nginx and netdata. To increase them, create 2 new files: + +1. `/etc/security/limits.d/nginx.conf`, with these contents: + + ``` +nginx soft nofile 10000 +nginx hard nofile 30000 +``` + +2. `/etc/security/limits.d/netdata.conf`, with these contents: + + ``` +netdata soft nofile 10000 +netdata hard nofile 30000 +``` + +and to activate them, run: + +```sh +sysctl -p +``` + +## 2b. increase open files limit (systemd) + +Thanks to [@leleobhz](https://github.com/netdata/netdata/issues/655#issue-163932584), this is what you need to raise the limits using systemd: + +This is based on https://ma.ttias.be/increase-open-files-limit-in-mariadb-on-centos-7-with-systemd/ and here worked as following: + +1. Create the folders in /etc: + + ``` +mkdir -p /etc/systemd/system/netdata.service.d +mkdir -p /etc/systemd/system/nginx.service.d +``` + +2. Create limits.conf in each folder as following: + + ``` +[Service] +LimitNOFILE=30000 +``` + +3. Reload systemd daemon list and restart services: + + ```sh +systemctl daemon-reload +systemctl restart netdata.service +systemctl restart nginx.service +``` + +You can check limits with following commands: + +```sh +cat /proc/$(ps aux | grep "nginx: master process" | grep -v grep | awk '{print $2}')/limits | grep "Max open files" +cat /proc/$(ps aux | grep "netdata" | head -n1 | grep -v grep | awk '{print $2}')/limits | grep "Max open files" +``` + +View of the files: + +```sh +# tree /etc/systemd/system/*service.d/etc/systemd/system/netdata.service.d +/etc/systemd/system/netdata.service.d +└── limits.conf +/etc/systemd/system/nginx.service.d +└── limits.conf + +0 directories, 2 files + +# cat /proc/$(ps aux | grep "nginx: master process" | grep -v grep | awk '{print $2}')/limits | grep "Max open files" +Max open files 30000 30000 files + +# cat /proc/$(ps aux | grep "netdata" | head -n1 | grep -v grep | awk '{print $2}')/limits | grep "Max open files" +Max open files 30000 30000 files + +``` diff --git a/doc/netdata-for-IoT.md b/doc/netdata-for-IoT.md new file mode 100644 index 000000000..ea7798722 --- /dev/null +++ b/doc/netdata-for-IoT.md @@ -0,0 +1,199 @@ +# Netdata for IoT + + + +> New to netdata? Check its demo: **[https://my-netdata.io/](https://my-netdata.io/)** +> +> [](https://registry.my-netdata.io/#netdata_registry) [](https://registry.my-netdata.io/#netdata_registry) [](https://registry.my-netdata.io/#netdata_registry) +> +> [](https://registry.my-netdata.io/#netdata_registry) [](https://registry.my-netdata.io/#netdata_registry) [](https://registry.my-netdata.io/#netdata_registry) + +--- + +netdata is a **very efficient** server performance monitoring solution. When running in server hardware, it can collect thousands of system and application metrics **per second** with just 1% CPU utilization of a single core. Its web server responds to most data requests in about **half a millisecond** making its web dashboards spontaneous, amazingly fast! + +netdata can also be a very efficient real-time monitoring solution for **IoT devices** (RPIs, routers, media players, wifi access points, industrial controllers and sensors of all kinds). Netdata will generally run everywhere a Linux kernel runs (and it is glibc and [musl-libc](https://www.musl-libc.org/) friendly). + +You can use it as both a data collection agent (where you pull data using its API), for embedding its charts on other web pages / consoles, but also for accessing it directly with your browser to view its dashboard. + +The netdata web API already provides **reduce** functions allowing it to report **average** and **max** for any timeframe. It can also respond in many formats including JSON, JSONP, CSV, HTML. Its API is also a **google charts** provider so it can directly be used by google sheets, google charts, google widgets. + + + +Although netdata has been significantly optimized to lower the CPU and RAM resources it consumes, the plethora of data collection plugins may be inappropriate for weak IoT devices. + +> keep in mind that netdata on RPi 2 and 3 does not require any tuning. The default settings will be good. The following tunables apply only when running netdata on RPi 1 or other very weak IoT devices. + +Here are a few tricks to control the resources consumed by netdata: + +## 1. Disable External plugins + +External plugins can consume more system resources than the netdata server. Disable the ones you don't need. + +Edit `/etc/netdata/netdata.conf`, find the `[plugins]` section: + +``` +[plugins] + proc = yes + + tc = no + idlejitter = no + cgroups = no + checks = no + apps = no + charts.d = no + node.d = no + + plugins directory = /usr/libexec/netdata/plugins.d + enable running new plugins = no + check for new plugins every = 60 +``` + +In detail: + +plugin|description +:---:|:--------- +`proc`|the internal plugin used to monitor the system. Normally, you don't want to disable this. You can disable individual functions of it at the next section. +`tc`|monitoring network interfaces QoS (tc classes) +`idlejitter`|internal plugin (written in C) that attempts show if the systems starved for CPU. Disabling it will eliminate a thread. +`cgroups`|monitoring linux containers. Most probably you are not going to need it. This will also eliminate another thread. +`checks`|a debugging plugin, which is disabled by default. +`apps`|a plugin that monitors system processes. It is very complex and heavy (heavier than the netdata daemon), so if you don't need to monitor the process tree, you can disable it. +`charts.d`|BASH plugins (squid, nginx, mysql, etc). This is again a heavy plugin. +`node.d`|node.js plugin, currently used for SNMP data collection and monitoring named (the name server). + +For most IoT devices, you can disable all plugins except `proc`. For `proc` there is another section that controls which functions of it you need. Check the next section. + +--- + +## 2. Disable internal plugins + +In this section you can select which modules of the `proc` plugin you need. All these are run in a single thread, one after another. Still, each one needs some RAM and consumes some CPU cycles. + +``` +[plugin:proc] + # /proc/net/dev = yes # network interfaces + # /proc/diskstats = yes # disks + # /proc/net/snmp = yes # generic IPv4 + # /proc/net/snmp6 = yes # generic IPv6 + # /proc/net/netstat = yes # TCP and UDP + # /proc/net/stat/conntrack = yes # firewall + # /proc/net/ip_vs/stats = yes # IP load balancer + # /proc/net/stat/synproxy = yes # Anti-DDoS + # /proc/stat = yes # CPU, context switches + # /proc/meminfo = yes # Memory + # /proc/vmstat = yes # Memory operations + # /proc/net/rpc/nfsd = yes # NFS Server + # /proc/sys/kernel/random/entropy_avail = yes # Cryptography + # /proc/interrupts = yes # Interrupts + # /proc/softirqs = yes # SoftIRQs + # /proc/loadavg = yes # Load Average + # /sys/kernel/mm/ksm = yes # Memory deduper + # netdata server resources = yes # netdata charts +``` + +--- + +## 3. Disable logs + +Normally, you will not need them. To disable them, set: + +``` +[global] + debug log = none + error log = none + access log = none +``` + +--- + +## 4. Set memory mode to RAM + +Setting the memory mode to `ram` will disable loading and saving the round robin database. This will not affect anything while running netdata, but it might be required if you have very limited storage available. + +``` +[global] + memory mode = ram +``` + +--- + +## 5. CPU utilization + +If after disabling the plugins you don't need, netdata still uses a lot of CPU without any clients accessing the dashboard, try lowering its data collection frequency. Going from "once per second" to "once every two seconds" will not have a significant difference on the user experience, but it will cut the CPU resources required **in half**. + +To set the update frequency, edit `/etc/netdata/netdata.conf` and set: + +``` +[global] + update every = 2 +``` + +You may have to increase this to 5 or 10 if the CPU of the device is weak. + +Keep in mind this will also force dashboard chart refreshes to happen at the same rate. So increasing this number actually lowers data collection frequency but also lowers dashboard chart refreshes frequency. + +This is a dashboard on a device with `[global].update every = 5` (this device is a media player and is now playing a movie): + + + +--- + +## 6. Lower memory requirements + +You can set the default size of the round robin database for all charts, using: + +``` +[global] + history = 600 +``` + +The units for history is `[global].update every` seconds. So if `[global].update every = 6` and `[global].history = 600`, you will have an hour of data ( 6 x 600 = 3.600 ), which will store 600 points per dimension, one every 6 seconds. + +Check also [[Memory Requirements]] for directions on calculating the size of the round robin database. + +--- + +## 7. Disable gzip compression of responses + +Gzip compression of the web responses is using more CPU that the rest of netdata. You can lower the compression level or disable gzip compression completely. You can disable it, like this: + +``` +[web] + enable gzip compression = no +``` + +To lower the compression level, do this: + +``` +[web] + enable gzip compression = yes + gzip compression level = 1 +``` + +--- + +Finally, if no web server is installed on your device, you can use port tcp/80 for netdata: + +``` +[global] + port = 80 +``` + +--- + +## 8. Monitoring RPi temperature + +The python version of the sensors plugin uses `lm-sensors`. Unfortunately the temperature reading of RPi are not supported by `lm-sensors`. + +netdata also has a bash version of the sensors plugin that can read RPi temperatures. It is disabled by default to avoid the conflicts with the python version. + +To enable it, edit `/etc/netdata/charts.d.conf` and uncomment this line: + +```sh +sensors=force +``` + +Then restart netdata. You will get this: + + diff --git a/doc/netdata-security.md b/doc/netdata-security.md new file mode 100644 index 000000000..79858656b --- /dev/null +++ b/doc/netdata-security.md @@ -0,0 +1,179 @@ +# Netdata Security + +We have given special attention to all aspects of netdata, ensuring that everything throughout its operation is as secure as possible. netdata has been designed with security in mind. + +**Table of Contents** + +1. [your data are safe with netdata](#your-data-are-safe-with-netdata) +2. [your systems are safe with netdata](#your-systems-are-safe-with-netdata) +3. [netdata is read-only](#netdata-is-read-only) +4. [netdata viewers authentication](#netdata-viewers-authentication) + - [why netdata should be protected](#why-netdata-should-be-protected) + - [protect netdata from the internet](#protect-netdata-from-the-internet) + - [expose netdata only in a private LAN](#expose-netdata-only-in-a-private-lan) + - [use an authenticating web server in proxy mode](#use-an-authenticating-web-server-in-proxy-mode) + - [other methods](#other-methods) +5. [registry or how to not send any information to a third party server](#registry-or-how-to-not-send-any-information-to-a-third-party-server) + +## your data are safe with netdata + +netdata collects raw data from many sources. For each source, netdata uses a plugin that connects to the source (or reads the relative files produced by the source), receives raw data and processes them to calculate the metrics shown on netdata dashboards. + +Even if netdata plugins connect to your database server, or read your application log file to collect raw data, the product of this data collection process is always a number of **chart metadata and metric values** (summarized data for dashboard visualization). All netdata plugins (internal to the netdata daemon, and external ones written in any computer language), convert raw data collected into metrics, and only these metrics are stored in netdata databases, sent to upstream netdata servers, or archived to backend time-series databases. + +> The **raw data** collected by netdata, do not leave the host they are collected. **The only data netdata exposes are chart metadata and metric values.** + +This means that netdata can safely be used in environments that require the highest level of data isolation (like PCI Level 1). + +## your systems are safe with netdata + +We are very proud that **the netdata daemon runs as a normal system user, without any special privileges**. This is quite an achievement for a monitoring system that collects all kinds of system and application metrics. + +There are a few cases however that raw source data are only exposed to processes with escalated privileges. To support these cases, netdata attempts to minimize and completely isolate the code that runs with escalated privileges. + +So, netdata **plugins**, even those running with escalated capabilities or privileges, perform a **hard coded data collection job**. They do not accept commands from netdata. The communication is strictly **unidirectional**: from the plugin towards the netdata daemon. The original application data collected by each plugin do not leave the process they are collected, are not saved and are not transferred to the netdata daemon. The communication from the plugins to the netdata daemon includes only chart metadata and processed metric values. + +netdata slaves streaming metrics to upstream netdata servers, use exactly the same protocol local plugins use. The raw data collected by the plugins of slave netdata servers are **never leaving the host they are collected**. The only data appearing on the wire are chart metadata and metric values. This communication is also **unidirectional**: slave netdata servers never accept commands from master netdata servers. + +## netdata is read-only + +netdata **dashboards are read-only**. Dashboard users can view and examine metrics collected by netdata, but cannot instruct netdata to do something other than present the already collected metrics. + +netdata dashboards do not expose sensitive information. Business data of any kind, the kernel version, O/S version, application versions, host IPs, etc are not stored and are not exposed by netdata on its dashboards. + +## netdata viewers authentication + +netdata is a monitoring system. It should be protected, the same way you protect all your admin apps. We assume netdata will be installed privately, for your eyes only. + +### why netdata should be protected + +Viewers will be able to get some information about the system netdata is running. This information is everything the dashboard provides. The dashboard includes a list of the services each system runs (the legends of the charts under the `Systemd Services` section), the applications running (the legends of the charts under the `Applications` section), the disks of the system and their names, the user accounts of the system that are running processes (the `Users` and `User Groups` section of the dashboard), the network interfaces and their names (not the IPs) and detailed information about the performance of the system and its applications. + +This information is not sensitive (meaning that it is not your business data), but **it is important for possible attackers**. It will give them clues on what to check, what to try and in the case of DDoS against your applications, they will know if they are doing it right or not. + +Also, viewers could use netdata itself to stress your servers. Although the netdata daemon runs unprivileged, with the minimum process priority (scheduling priority `idle` - lower than nice 19) and adjusts its OutOfMemory (OOM) score to 1000 (so that it will be first to be killed by the kernel if the system starves for memory), some pressure can be applied on your systems if someone attempts a DDoS against netdata. + +### protect netdata from the internet + +netdata is a distributed application. Most likely you will have many installations of it. Since it is distributed and you are expected to jump from server to server, there is very little usability to add authentication local on each netdata. + +Until we add a distributed authentication method to netdata, you have the following options: + +#### expose netdata only in a private LAN + +If your organisation has a private administration and management LAN, you can bind netdata on this network interface on all your servers. This is done in `netdata.conf` with these settings: + +``` +[web] + bind to = 10.1.1.1:19999 localhost:19999 +``` + +You can bind netdata to multiple IPs and ports. If you use hostnames, netdata will resolve them and use all the IPs (in the above example `localhost` usually resolves to both `127.0.0.1` and `::1`). + +**This is the best and the suggested way to protect netdata**. Your systems **should** have a private administration and management LAN, so that all management tasks are performed without any possibility of them being exposed on the internet. + +For cloud based installations, if your cloud provider does not provide such a private LAN (or if you use multiple providers), you can create a virtual management and administration LAN with tools like `tincd` or `gvpe`. These tools create a mesh VPN allowing all servers to communicate securely and privately. Your administration stations join this mesh VPN to get access to management and administration tasks on all your cloud servers. + +For `gvpe` we have developed a [simple provisioning tool](https://github.com/netdata/netdata-demo-site/tree/master/gvpe) you may find handy (it includes statically compiled `gvpe` binaries for Linux and FreeBSD, and also a script to compile `gvpe` on your Mac). We use this to create a management and administration LAN for all netdata demo sites (spread all over the internet using multiple hosting providers). + +--- + +In netdata v1.9+ there is also access list support, like this: + +``` +[web] + bind to = * + allow connections from = localhost 10.* 192.168.* +``` + + +#### use an authenticating web server in proxy mode + +Use **one nginx** (or one apache) server to provide authentication in front of **all your netdata servers**. So, you will be accessing all your netdata with URLs like `http://nginx.host/netdata/{NETDATA_HOSTNAME}/` and authentication will be shared among all of them (you will sign-in once for all your servers). Check [this wiki page for more information on configuring nginx for such a setup](Running-behind-nginx.md#netdata-via-nginx). + +To use this method, you should firewall protect all your netdata servers, so that only the nginx IP will allowed to directly access netdata. To do this, run this on each of your servers (or use your firewall manager): + +```sh +NGINX_IP="1.2.3.4" +iptables -t filter -I INPUT -p tcp --dport 19999 \! -s ${NGINX_IP} -m conntrack --ctstate NEW -j DROP +``` +_commands to allow direct access to netdata from an nginx proxy_ + +The above will prevent anyone except your nginx server to access a netdata dashboard running on the host. + +For netdata v1.9+ you can also use `netdata.conf`: + +``` +[web] + allow connections from = localhost 1.2.3.4 +``` + +Of course you can add more IPs. + +For netdata prior to v1.9, if you want to allow multiple IPs, use this: + +```sh +# space separated list of IPs to allow access netdata +NETDATA_ALLOWED="1.2.3.4 5.6.7.8 9.10.11.12" +NETDATA_PORT=19999 + +# create a new filtering chain || or empty an existing one named netdata +iptables -t filter -N netdata 2>/dev/null || iptables -t filter -F netdata +for x in ${NETDATA_ALLOWED} +do + # allow this IP + iptables -t filter -A netdata -s ${x} -j ACCEPT +done + +# drop all other IPs +iptables -t filter -A netdata -j DROP + +# delete the input chain hook (if it exists) +iptables -t filter -D INPUT -p tcp --dport ${NETDATA_PORT} -m conntrack --ctstate NEW -j netdata 2>/dev/null + +# add the input chain hook (again) +# to send all new netdata connections to our filtering chain +iptables -t filter -I INPUT -p tcp --dport ${NETDATA_PORT} -m conntrack --ctstate NEW -j netdata +``` +_script to allow access to netdata only from a number of hosts_ + +You can run the above any number of times. Each time it runs it refreshes the list of allowed hosts. + +#### other methods + +Of course, there are many more methods you could use to protect netdata: + +- bind netdata to localhost and use `ssh -L 19998:127.0.0.1:19999 remote.netdata.ip` to forward connections of local port 19998 to remote port 19999. This way you can ssh to a netdata server and then use `http://127.0.0.1:19998/` on your computer to access the remote netdata dashboard. + +- If you are always under a static IP, you can use the script given above to allow direct access to your netdata servers without authentication, from all your static IPs. + +- install all your netdata in **headless data collector** mode, forwarding all metrics in real-time to a master netdata server, which will be protected with authentication using an nginx server running locally at the master netdata server. This requires more resources (you will need a bigger master netdata server), but does not require any firewall changes, since all the slave netdata servers will not be listening for incoming connections. + +## registry or how to not send any information to a third party server + +The default configuration uses a public registry under registry.my-netdata.io (more information about the registry here: [mynetdata-menu-item](../registry/) ). Please be aware that if you use that public registry, you submit at least the following information to a third party server, which might violate your security policies: +- Your public ip where the browser runs +- The url where you open the web-ui in the browser (via http request referer) +- The hostnames of the netdata servers + +You are able to run your own registry, which is pretty simple to do: +- If you have just one netdata web-ui, turn on registry and set the url of that web-ui as "registry to announce" +``` +[registry] +enabled = yes +registry to announce = URL_OF_THE_NETDATA_WEB-UI +``` +- If you run multiple netdata servers with web-ui, you need to define one as registry. On that node activate the registry and setting its url as "registry to announce". On all other nodes do not enable the registry but define the same url. + +restart netdata and check with developer tools of your browser which registry is called. + +## netdata directories + +path|owner|permissions| netdata |comments| +:---|:----|:----------|:--------|:-------| +`/etc/netdata`|user `root`<br/>group `netdata`|dirs `0755`<br/>files `0640`|reads|**netdata config files**<br/>may contain sensitive information, so group `netdata` is allowed to read them. +`/usr/libexec/netdata`|user `root`<br/>group `root`|executable by anyone<br/>dirs `0755`<br/>files `0644` or `0755`|executes|**netdata plugins**<br/>permissions depend on the file - not all of them should have the executable flag.<br/>there are a few plugins that run with escalated privileges (Linux capabilities or `setuid`) - these plugins should be executable only by group `netdata`. +`/usr/share/netdata`|user `root`<br/>group `netdata`|readable by anyone<br/>dirs `0755`<br/>files `0644`|reads and sends over the network|**netdata web static files**<br/>these files are sent over the network to anyone that has access to the netdata web server. netdata checks the ownership of these files (using settings at the `[web]` section of `netdata.conf`) and refuses to serve them if they are not properly owned. Symbolic links are not supported. netdata also refuses to serve URLs with `..` in their name. +`/var/cache/netdata`|user `netdata`<br/>group `netdata`|dirs `0750`<br/>files `0660`|reads, writes, creates, deletes|**netdata ephemeral database files**<br/>netdata stores its ephemeral real-time database here. +`/var/lib/netdata`|user `netdata`<br/>group `netdata`|dirs `0750`<br/>files `0660`|reads, writes, creates, deletes|**netdata permanent database files**<br/>netdata stores here the registry data, health alarm log db, etc. +`/var/log/netdata`|user `netdata`<br/>group `root`|dirs `0755`<br/>files `0644`|writes, creates|**netdata log files**<br/>all the netdata applications, logs their errors or other informational messages to files in this directory. These files should be log rotated. diff --git a/docker/Dockerfile b/docker/Dockerfile index 4362a273e..a852f3044 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -86,7 +86,7 @@ RUN \ addgroup -g ${NETDATA_GID} -S netdata && \ adduser -S -H -s /usr/sbin/nologin -u ${NETDATA_GID} -h /etc/netdata -G netdata netdata && \ # Apply the permissions as described in - # https://github.com/netdata/netdata/wiki/netdata-security#netdata-directories + # https://github.com/netdata/netdata/tree/master/doc/netdata-security.md#netdata-directories chown -R root:netdata /etc/netdata && \ chown -R netdata:netdata /var/cache/netdata /var/lib/netdata /usr/share/netdata && \ chown -R root:netdata /usr/lib/netdata && \ diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 000000000..d624855fb --- /dev/null +++ b/docker/README.md @@ -0,0 +1,117 @@ +# Install netdata with Docker + +> :warning: As of Sep 9th, 2018 we ship [new docker builds](https://github.com/netdata/netdata/pull/3995), running netdata in docker with an ENTRYPOINT directive, not a COMMAND directive. Please adapt your execution scripts accordingly. +> More information about ENTRYPOINT vs COMMAND is presented by goinbigdata [here](http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/) and by docker docs [here](https://docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact). +> +> Also, the `latest` is now based on alpine, so **`alpine` is not updated any more** and `armv7hf` is now replaced with `armhf` (to comply with https://github.com/multiarch naming), so **`armv7hf` is not updated** either. + +## Limitations + +Running netdata in a container for monitoring the whole host, can limit its capabilities. Some data is not accessible or not as detailed as when running netdata on the host. + +## Run netdata with docker command + +Quickly start netdata with the docker command line. +Netdata is then available at http://host:19999 + +This is good for an internal network or to quickly analyse a host. + +For a permanent installation on a public server, you should [[secure the netdata instance|netdata-security]]. See below for an example of how to install netdata with an SSL reverse proxy and basic authentication. + +```bash +docker run -d --name=netdata \ + -p 19999:19999 \ + -v /proc:/host/proc:ro \ + -v /sys:/host/sys:ro \ + -v /var/run/docker.sock:/var/run/docker.sock:ro \ + --cap-add SYS_PTRACE \ + --security-opt apparmor=unconfined \ + netdata/netdata +``` + +above can be converted to docker-compose file for ease of management: + +```yaml +version: '3' +services: + netdata: + image: netdata/netdata + hostname: example.com # set to fqdn of host + ports: + - 19999:19999 + cap_add: + - SYS_PTRACE + security_opt: + - apparmor:unconfined + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /var/run/docker.sock:/var/run/docker.sock:ro +``` + +### Docker container names resolution + +If you want to have your container names resolved by netdata it needs to have access to docker group. To achive that just add environment variable `PGID=999` to netdata container, where `999` is a docker group id from your host. This number can be found by running: +```bash +grep docker /etc/group | cut -d ':' -f 3 +``` + +## Install Netdata using Docker Compose with SSL/TLS enabled http proxy + +You can use use the following docker-compose.yml and Caddyfile files to run netdata with docker. +Replace the Domains and email address for Letsencrypt before starting. + +### Prerequisites +* [Docker](https://docs.docker.com/install/#server) +* [Docker Compose](https://docs.docker.com/compose/install/) +* Domain configured in DNS pointing to host. + +### Caddyfile + +This file needs to be placed in /opt with nams Caddyfile. Here you customize your domain and you need to provide your email address to obtain Letsencrypt certificate. +Certificate renewal will happen automatically and will be executed internally by caddy server. + +``` +netdata.example.org { + proxy / netdata:19999 + tls admin@example.org +} +``` + +### docker-compose.yml + +After setting Caddyfile run this with `docker-compose up -d` to have fully functioning netdata setup behind HTTP reverse proxy. + +```yaml +version: '3' +volumes: + caddy: + +services: + caddy: + image: abiosoft/caddy + ports: + - 80:80 + - 443:443 + volumes: + - /opt/Caddyfile:/etc/Caddyfile + - caddy:/root/.caddy + environment: + ACME_AGREE: 'true' + netdata: + restart: always + hostname: netdata.example.org + image: netdata/netdata + cap_add: + - SYS_PTRACE + security_opt: + - apparmor:unconfined + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /var/run/docker.sock:/var/run/docker.sock:ro +``` + +### Restrict access with basic auth + +You can restrict access by following [official caddy guide](https://caddyserver.com/docs/basicauth) and adding lines to Caddyfile. diff --git a/docker/build.sh b/docker/build.sh index 908468d39..faaa2db79 100755 --- a/docker/build.sh +++ b/docker/build.sh @@ -30,7 +30,9 @@ fi # Build images using multi-arch Dockerfile. for ARCH in i386 armhf aarch64 amd64; do - docker build --build-arg ARCH="${ARCH}-v3.8" --tag "${REPOSITORY}:${VERSION}-${ARCH}" --file docker/Dockerfile ./ & + docker build --build-arg ARCH="${ARCH}-v3.8" \ + --tag "${REPOSITORY}:${VERSION}-${ARCH}" \ + --file docker/Dockerfile ./ & done wait @@ -70,14 +72,3 @@ docker --config /tmp/docker manifest push -p "${REPOSITORY}:${VERSION}" # Show current manifest (debugging purpose only) docker --config /tmp/docker manifest inspect "${REPOSITORY}:${VERSION}" -# Push netdata images to firehol organization -# TODO: Remove it after we decide to deprecate firehol/netdata docker repo -if [ "$REPOSITORY" != "netdata" ]; then - echo "$OLD_DOCKER_PASSWORD" | docker login -u "$OLD_DOCKER_USERNAME" --password-stdin - for ARCH in amd64 i386 armhf aarch64; do - docker tag "${REPOSITORY}:${VERSION}-${ARCH}" "firehol/netdata:${ARCH}" - docker push "firehol/netdata:${ARCH}" - done - docker tag "${REPOSITORY}:${VERSION}-amd64" "firehol/netdata:${VERSION}" - docker push "firehol/netdata:${VERSION}" -fi |