diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/.templates/.page-level/_collector-page-template.mdx | 68 | ||||
| -rw-r--r-- | docs/.templates/.page-level/_concept-page-template.md | 30 | ||||
| -rw-r--r-- | docs/.templates/.page-level/_task-page-template.md | 41 | ||||
| -rw-r--r-- | docs/.templates/.page-level/_tutorial-page-template.mdx | 54 | ||||
| -rw-r--r-- | docs/collect/application-metrics.md | 2 | ||||
| -rw-r--r-- | docs/collect/how-collectors-work.md | 2 | ||||
| -rw-r--r-- | docs/configure/common-changes.md | 4 | ||||
| -rw-r--r-- | docs/contributing/style-guide.md | 4 | ||||
| -rw-r--r-- | docs/dashboard/customize.mdx | 8 | ||||
| -rw-r--r-- | docs/dashboard/import-export-print-snapshot.mdx | 4 | ||||
| -rw-r--r-- | docs/dashboard/reference-web-server.mdx | 278 | ||||
| -rw-r--r-- | docs/guides/collect-apache-nginx-web-logs.md | 43 | ||||
| -rw-r--r-- | docs/guides/configure/performance.md | 16 | ||||
| -rw-r--r-- | docs/guides/step-by-step/step-06.md | 6 | ||||
| -rw-r--r-- | docs/metrics-storage-management/reference-streaming.mdx | 2 | ||||
| -rw-r--r-- | docs/monitor/enable-notifications.md | 2 |
16 files changed, 497 insertions, 67 deletions
diff --git a/docs/.templates/.page-level/_collector-page-template.mdx b/docs/.templates/.page-level/_collector-page-template.mdx new file mode 100644 index 000000000..fcbe7cec7 --- /dev/null +++ b/docs/.templates/.page-level/_collector-page-template.mdx @@ -0,0 +1,68 @@ +<!-- +title: $COLLECTOR_NAME monitoring with Netdata +description: Short summary (will be displayed in search engines) +custom_edit_url: Edit URL of the source file +keywords: [keywords, describing, the main topics] +--> + +# Title + +import { + EnableCollector, + CollectorDebug, +} from '@site/src/components/Collectors/_collector-components.jsx'; + +Short description of what the collector does on a high level. +Why should I use this collector? + +## Configuring $COLLECTOR_NAME + +#### Prerequisites + +List all needed prerequisites: + +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + +<CollectorConfiguration configURL="" moduleName="PLUGIN/COLLECTOR.conf" /> + +### Example + +TODO: Check if we can automatically fetch the [JOBS] section of netdata.conf + +## Other configuration information + +Explain other configuration options, as needed. + +#### Prerequisites + +<!-- If there is only one requirement, use a paragraph instead of a single bullet item. Bullets are social animals and only appear in groups of 2 or more :) --> + +Optional. List all needed prerequisites: + +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## Debugging $COLLECTOR_NAME (optional) + +<CollectorDebug pluginName="" collectorName="" /> + +## Metrics and Alerts produced by this collector + +| Chart | Metrics | Alert | +| ---------- | ----------- | ------------------------ | +| Chart Name | Metric name | [Alert 1](Link to alert) | +| Chart Name | Metric name | [Alert 2](Link to alert) | +| Chart Name | Metric name | [Alert 3](Link to alert) | diff --git a/docs/.templates/.page-level/_concept-page-template.md b/docs/.templates/.page-level/_concept-page-template.md new file mode 100644 index 000000000..685dd2ff3 --- /dev/null +++ b/docs/.templates/.page-level/_concept-page-template.md @@ -0,0 +1,30 @@ +<!-- +title: Noun that describes the concept +description: Short summary (will be displayed in search engines) +custom_edit_url: Edit URL of the source file +keywords: [keywords, describing, the main topics] +--> + +# Title + +Why should the reader care: “What’s in it for me?” + +## Subheading + +Ideally, try to explain one core idea per section. Questions that you could keep in mind while writing: + +- How does it work? +- What are the outcomes? +- What are the positive and negative effects of it? +- Are there alternatives that provide a similar result? + +## Subheading + +Add more subheadings and anything else that needs to be explained... +Remember, if you start to describe about another concept, stop yourself. +Each concept should be about one concept only. + +<!-- Optional --> +### Related links +<!-- Here, you could include links to task topic that describe how to implement the thing you discussed in this concept. --> +- Visit the [related thing documentation](www.related-thing.com) to learn more about related thing. diff --git a/docs/.templates/.page-level/_task-page-template.md b/docs/.templates/.page-level/_task-page-template.md new file mode 100644 index 000000000..0f49201e8 --- /dev/null +++ b/docs/.templates/.page-level/_task-page-template.md @@ -0,0 +1,41 @@ +<!-- +title: Starts with an active verb, like "Create a widget" or "Delete a widget" +description: Short summary (will be displayed in search engines) +custom_edit_url: Edit URL of the source file +keywords: [keywords, describing, the main topics] +--> +# Title + +Short description of why or when the procedure makes sense. + +## Subheading that describes the task +#### Prerequisites +<!-- If there is only one requirement, use a paragraph instead of a single bullet item. Bullets are social animals and only appear in groups of 2 or more :) --> +Optional. List all needed prerequisites: +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## If needed, another task section + +See lines 11-24 + +<!-- Optional --> +## What's next? + +Optional section that explains the next logical steps. + +<!-- Optional --> +## Related links + +- Visit the [related thing documentation](www.related-thing.com) to learn more about related thing. diff --git a/docs/.templates/.page-level/_tutorial-page-template.mdx b/docs/.templates/.page-level/_tutorial-page-template.mdx new file mode 100644 index 000000000..9f64b5ece --- /dev/null +++ b/docs/.templates/.page-level/_tutorial-page-template.mdx @@ -0,0 +1,54 @@ +<!-- +title: Starts with an active verb, like "Create a widget" or "Delete a widget" +description: Short summary (will be displayed in search engines) +custom_edit_url: Edit URL of the source file +author: "Your Name" +author_title: "Your title at Netdata" +author_img: "/img/authors/YourFace.jpg" +keywords: [keywords, describing, the main topics] +--> + +A paragraph that explains what the tutorial does, why it matters, and the expected outcome. + +To achieve goal: + +1. [Do the first task](#first-task) +2. [Do the second task](#second-task) + +## Prerequisites + +<!-- If there is only one requirement, use a paragraph instead of a single bullet item. Bullets are social animals and only appear in groups of 2 or more :) --> + +Optional. List all needed prerequisites: + +- Prerequisite 1 +- Prerequisite 2 +- Prerequisite 3 + +## First task + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## Second task + +To do x: + +1. Step 1 written in active voice + ```bash + Code sample for step + ``` +2. Step 2 + Result of step 2, for example a system reaction; written in passive voice +3. Step 3 + +## What's next? + +Optional section that explains the next logical steps. diff --git a/docs/collect/application-metrics.md b/docs/collect/application-metrics.md index 5af58b105..ffd6b0ac0 100644 --- a/docs/collect/application-metrics.md +++ b/docs/collect/application-metrics.md @@ -36,7 +36,7 @@ Our most popular application collectors: - [Nginx](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/nginx/): Monitor web server status information by gathering metrics via `ngx_http_stub_status_module`. - [Postgres](/collectors/python.d.plugin/postgres/README.md): Collect database health and performance metrics. -- [ElasticSearch](/collectors/python.d.plugin/elasticsearch/README.md): Collect search engine performance and health +- [ElasticSearch](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/elasticsearch): Collect search engine performance and health statistics. Optionally collects per-index metrics. - [PHP-FPM](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/phpfpm/): Collect application summary and processes health metrics by scraping the status page (`/status?full`). diff --git a/docs/collect/how-collectors-work.md b/docs/collect/how-collectors-work.md index 983de35a8..07e34858f 100644 --- a/docs/collect/how-collectors-work.md +++ b/docs/collect/how-collectors-work.md @@ -62,8 +62,6 @@ terms related to collecting metrics. `python` v2/v3. - [charts.d.plugin](/collectors/charts.d.plugin/README.md): An orchestrator for data collection modules written in `bash` v4+. - - [node.d.plugin](/collectors/node.d.plugin/README.md): An orchestrator for data collection modules written in - `node.js`. - **External plugins** gather metrics from external processes, such as a webserver or database, and run as independent processes that communicate with the Netdata daemon via pipes. - **Internal plugins** gather metrics from `/proc`, `/sys`, and other Linux kernel sources. They are written in `C`, diff --git a/docs/configure/common-changes.md b/docs/configure/common-changes.md index cf2e5d78a..93b12d226 100644 --- a/docs/configure/common-changes.md +++ b/docs/configure/common-changes.md @@ -46,7 +46,7 @@ of 5 seconds. ``` Every collector and plugin has its own `update every` setting, which you can also change in the `go.d.conf`, -`python.d.conf`, `node.d.conf`, or `charts.d.conf` files, or in individual collector configuration files. If the `update +`python.d.conf` or `charts.d.conf` files, or in individual collector configuration files. If the `update every` for an individual collector is less than the global, the Netdata Agent uses the global setting. See the [enable or configure a collector](/docs/collect/enable-configure.md) doc for details. @@ -55,7 +55,7 @@ or configure a collector](/docs/collect/enable-configure.md) doc for details. Turn off entire plugins in the [`[plugins]` section](/daemon/config/README.md#plugins-section-options) of `netdata.conf`. -To disable specific collectors, open `go.d.conf`, `python.d.conf`, `node.d.conf`, or `charts.d.conf` and find the line +To disable specific collectors, open `go.d.conf`, `python.d.conf` or `charts.d.conf` and find the line for that specific module. Uncomment the line and change its value to `no`. ## Modify alarms and notifications diff --git a/docs/contributing/style-guide.md b/docs/contributing/style-guide.md index 2af58d54a..5ff61164d 100644 --- a/docs/contributing/style-guide.md +++ b/docs/contributing/style-guide.md @@ -443,7 +443,7 @@ code, for example: inline char *health_stock_config_dir(void) { char buffer[FILENAME_MAX + 1]; snprintfz(buffer, FILENAME_MAX, "%s/health.d", netdata_configured_stock_config_dir); - return config_get(CONFIG_SECTION_HEALTH, "stock health configuration directory", buffer); + return config_get(CONFIG_SECTION_DIRECTORIES, "stock health config", buffer); } ``` ```` @@ -454,7 +454,7 @@ And the prettified result: inline char *health_stock_config_dir(void) { char buffer[FILENAME_MAX + 1]; snprintfz(buffer, FILENAME_MAX, "%s/health.d", netdata_configured_stock_config_dir); - return config_get(CONFIG_SECTION_HEALTH, "stock health configuration directory", buffer); + return config_get(CONFIG_SECTION_DIRECTORIES, "stock health config", buffer); } ``` diff --git a/docs/dashboard/customize.mdx b/docs/dashboard/customize.mdx index d20337911..8f0b222f2 100644 --- a/docs/dashboard/customize.mdx +++ b/docs/dashboard/customize.mdx @@ -24,15 +24,15 @@ Here are a few popular settings: ### Change chart legend position -Find this setting under the **Visual** tab. By default, Netdata places the [legend of -dimensions](/docs/dashboard/dimensions-contexts-families.mdx#dimension) _below_ charts. Click this toggle to -move the legend to the _right_ of charts. +Find this setting under the **Visual** tab. By default, Netdata places the [legend of dimensions](/docs/dashboard/dimensions-contexts-families.mdx#dimension) _below_ charts. +Click this toggle to move the legend to the _right_ of charts. + ### Change theme Find this setting under the **Visual** tab. Choose between Dark (the default) and White. -## Customize the standard dashboard +## Customize the standard dashboard info Netdata stores information about individual charts in the `dashboard_info.js` file. This file includes section and subsection headings, descriptions, colors, titles, tooltips, and other information for Netdata to render on the diff --git a/docs/dashboard/import-export-print-snapshot.mdx b/docs/dashboard/import-export-print-snapshot.mdx index 1333efc2b..df43feb0f 100644 --- a/docs/dashboard/import-export-print-snapshot.mdx +++ b/docs/dashboard/import-export-print-snapshot.mdx @@ -11,9 +11,9 @@ Netdata can export snapshots of the contents of your dashboard at a given time, node running Netdata. Or, you can create a print-ready version of your dashboard to save to PDF or actually print to paper. -Snapshots can be incredibly useful for diagnosing anomalies after they've already happened. Let's say Netdata triggered -a warning alarm while you were asleep. In the morning, you can [pick the +Snapshots can be incredibly useful for diagnosing anomalies after they've already happened. Let's say Netdata triggered a warning alarm while you were asleep. In the morning, you can [select the timeframe](/docs/dashboard/visualization-date-and-time-controls.mdx) when the alarm triggered, export a snapshot, and send it to a + colleague for further analysis. Or, send the Netdata team a snapshot of your dashboard when [filing a bug diff --git a/docs/dashboard/reference-web-server.mdx b/docs/dashboard/reference-web-server.mdx new file mode 100644 index 000000000..55e761a23 --- /dev/null +++ b/docs/dashboard/reference-web-server.mdx @@ -0,0 +1,278 @@ +--- +title: "Web server reference" +description: "The Netdata Agent's local static-threaded web server serves dashboards and real-time visualizations with security and DDoS protection." +type: reference +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/dashboard/reference-web-server.mdx +--- + +# Web server reference + +The Netdata web server is `static-threaded`, with a fixed, configurable number of threads. + +All the threads are concurrently listening for web requests on the same sockets, and the kernel distributes the incoming +requests to them. Each thread uses non-blocking I/O so it can serve any number of web requests in parallel. + +This web server respects the `keep-alive` HTTP header to serve multiple HTTP requests via the same connection. + +## Configuration + +From within your Netdata config directory (typically `/etc/netdata`), [use `edit-config`](/docs/configure/nodes.md) to +open `netdata.conf`. + +``` +sudo ./edit-config netdata.conf +``` + +Scroll down to the `[web]` section to find the following settings. + +## Settings + +| Setting | Default | Description | +|:-------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `ssl key` | `/etc/netdata/ssl/key.pem` | Declare the location of an SSL key to [enable HTTPS](#enable-httpstls-support). | +| `ssl certificate` | `/etc/netdata/ssl/cert.pem` | Declare the location of an SSL certificate to [enable HTTPS](#enable-httpstls-support). | +| `tls version` | `1.3` | Choose which TLS version to use. While all versions are allowed (`1` or `1.0`, `1.1`, `1.2` and `1.3`), we recommend `1.3` for the most secure encryption. If left blank, Netdata uses the highest available protocol version on your system. | +| `tls ciphers` | `none` | Choose which TLS cipher to use. Options include `TLS_AES_256_GCM_SHA384`, `TLS_CHACHA20_POLY1305_SHA256`, and `TLS_AES_128_GCM_SHA256`. If left blank, Netdata uses the default cipher list for that protocol provided by your TLS implementation. | +| `ses max window` | `15` | See [single exponential smoothing](/web/api/queries/ses/README.md). | +| `des max window` | `15` | See [double exponential smoothing](/web/api/queries/des/README.md). | +| `mode` | `static-threaded` | Turns on (`static-threaded` or off (`none`) the static-threaded web server. See the [example](#disable-the-web-server) to turn off the web server and disable the dashboard. | +| `listen backlog` | `4096` | The port backlog. Check `man 2 listen`. | +| `default port` | `19999` | The listen port for the static web server. | +| `web files owner` | `netdata` | The user that owns the web static files. Netdata will refuse to serve a file that is not owned by this user, even if it has read access to that file. If the user given is not found, Netdata will only serve files owned by user given in `run as user`. | +| `web files group` | `netdata` | If this is set, Netdata will check if the file is owned by this group and refuse to serve the file if it's not. | +| `disconnect idle clients after seconds` | `60` | The time in seconds to disconnect web clients after being totally idle. | +| `timeout for first request` | `60` | How long to wait for a client to send a request before closing the socket. Prevents slow request attacks. | +| `accept a streaming request every seconds` | `0` | Can be used to set a limit on how often a parent node will accept streaming requests from child nodes in a [streaming and replication setup](/streaming/README.md). | +| `respect do not track policy` | `no` | If set to `yes`, Netdata will respect the user's browser preferences for [Do Not Track](https://www.eff.org/issues/do-not-track) (DNT) and storing cookies. If DNT is _enabled_ in the browser, and this option is set to `yes`, users will not be able to sign in to Netdata Cloud via their local Agent dashboard, and their node will not connect to any [registry](/registry/README.md). For certain browsers, users must disable DNT and change this option to `yes` for full functionality. | +| `x-frame-options response header` | ` ` | Avoid [clickjacking attacks](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options), by ensuring that the content is not embedded into other sites. | +| `allow connections from` | `localhost *` | Declare which IP addresses or full-qualified domain names (FQDNs) are allowed to connect to the web server, including the [dashboard](/docs/dashboard/interact-charts.mdx) or [HTTP API](/web/api/README.md). This is a global setting with higher priority to any of the ones below. | +| `allow connections by dns` | `heuristic` | See the [access list examples](#access-lists) for details on using `allow` settings. | +| `allow dashboard from` | `localhost *` | | +| `allow dashboard by dns` | `heuristic` | | +| `allow badges from` | `*` | | +| `allow badges by dns` | `heuristic` | | +| `allow streaming from` | `*` | | +| `allow streaming by dns` | `heuristic` | | +| `allow netdata.conf` | `localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.* UNKNOWN` | | +| `allow netdata.conf by dns` | `no` | | +| `allow management from` | `localhost` | | +| `allow management by dns` | `heuristic` | | +| `enable gzip compression` | `yes` | When set to `yes`, Netdata web responses will be GZIP compressed, if the web client accepts such responses. | +| `gzip compression strategy` | `default` | Valid settings are `default`, `filtered`, `huffman only`, `rle` and `fixed`. | +| `gzip compression level` | `3` | Valid settings are 1 (fastest) to 9 (best ratio). | +| `web server threads` | ` ` | How many processor threads the web server is allowed. The default is system-specific, the minimum of `6` or the number of CPU cores. | +| `web server max sockets` | ` ` | Available sockets. The default is system-specific, automatically adjusted to 50% of the max number of open files Netdata is allowed to use (via `/etc/security/limits.conf` or systemd), to allow enough file descriptors to be available for data collection. | +| `custom dashboard_info.js` | ` ` | Specifies the location of a custom `dashboard.js` file. See [customizing the standard dashboard](/docs/dashboard/customize.mdx#customize-the-standard-dashboard) for details. | + +## Examples + +### Disable the web server + +Disable the web server by editing `netdata.conf` and setting: + +``` +[web] + mode = none +``` + +### Change the number of threads + +Control the number of threads and sockets with the following settings: + +``` +[web] + web server threads = 4 + web server max sockets = 512 +``` + +### Binding Netdata to multiple ports + +Netdata can bind to multiple IPs and ports, offering access to different services on each. Up to 100 sockets can be used (increase it at compile time with `CFLAGS="-DMAX_LISTEN_FDS=200" ./netdata-installer.sh ...`). + +The ports to bind are controlled via `[web].bind to`, like this: + +``` +[web] + default port = 19999 + bind to = 127.0.0.1=dashboard^SSL=optional 10.1.1.1:19998=management|netdata.conf hostname:19997=badges [::]:19996=streaming^SSL=force localhost:19995=registry *:http=dashboard unix:/run/netdata/netdata.sock +``` + +Using the above, Netdata will bind to: + +- IPv4 127.0.0.1 at port 19999 (port was used from `default port`). Only the UI (dashboard) and the read API will be accessible on this port. Both HTTP and HTTPS requests will be accepted. +- IPv4 10.1.1.1 at port 19998. The management API and `netdata.conf` will be accessible on this port. +- All the IPs `hostname` resolves to (both IPv4 and IPv6 depending on the resolved IPs) at port 19997. Only badges will be accessible on this port. +- All IPv6 IPs at port 19996. Only metric streaming requests from other Netdata agents will be accepted on this port. Only encrypted streams will be allowed (i.e. child nodes also need to be [configured for TLS](/streaming/README.md). +- All the IPs `localhost` resolves to (both IPv4 and IPv6 depending the resolved IPs) at port 19996. This port will only accept registry API requests. +- All IPv4 and IPv6 IPs at port `http` as set in `/etc/services`. Only the UI (dashboard) and the read API will be accessible on this port. +- Unix domain socket `/run/netdata/netdata.sock`. All requests are serviceable on this socket. Note that in some OSs like Fedora, every service sees a different `/tmp`, so don't create a Unix socket under `/tmp`. `/run` or `/var/run` is suggested. + +The option `[web].default port` is used when an entries in `[web].bind to` do not specify a port. + +Note that the access permissions specified with the `=request type|request type|...` format are available from version 1.12 onwards. +As shown in the example above, these permissions are optional, with the default being to permit all request types on the specified port. +The request types are strings identical to the `allow X from` directives of the access lists, i.e. `dashboard`, `streaming`, `registry`, `netdata.conf`, `badges` and `management`. +The access lists themselves and the general setting `allow connections from` in the next section are applied regardless of the ports that are configured to provide these services. +The API requests are serviced as follows: + +- `dashboard` gives access to the UI, the read API and badges API calls. +- `badges` gives access only to the badges API calls. +- `management` gives access only to the management API calls. + +### Enable HTTPS/TLS support + +Since v1.16.0, Netdata supports encrypted HTTP connections to the web server, plus encryption of streaming data to a +parent from its child nodes, via the TLS protocol. + +Inbound unix socket connections are unaffected, regardless of the TLS settings. + +> While Netdata uses Transport Layer Security (TLS) 1.2 to encrypt communications rather than the obsolete SSL protocol, +> it's still common practice to refer to encrypted web connections as `SSL`. Many vendors, like Nginx and even Netdata +> itself, use `SSL` in configuration files, whereas documentation will always refer to encrypted communications as `TLS` +> or `TLS/SSL`. + +To enable TLS, provide the path to your certificate and private key in the `[web]` section of `netdata.conf`: + +```conf +[web] + ssl key = /etc/netdata/ssl/key.pem + ssl certificate = /etc/netdata/ssl/cert.pem +``` + +Both files must be readable by the `netdata` user. If either of these files do not exist or are unreadable, Netdata will fall back to HTTP. For a parent-child connection, only the parent needs these settings. + +For test purposes, generate self-signed certificates with the following command: + +```bash +openssl req -newkey rsa:2048 -nodes -sha512 -x509 -days 365 -keyout key.pem -out cert.pem +``` + +> If you use 4096 bits for your key and the certificate, Netdata will need more CPU to process the communication. +> `rsa4096` can be up to 4 times slower than `rsa2048`, so we recommend using 2048 bits. Verify the difference +> by running: +> +> ```sh +> openssl speed rsa2048 rsa4096 +> ``` + +### Select TLS version + +Beginning with version `v1.21.0`, specify the TLS version and the ciphers that you want to use: + +```conf +[web] + tls version = 1.3 + tls ciphers = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256 +``` + +If you do not specify these options, Netdata will use the highest available protocol version on your system and the default cipher list for that protocol provided by your TLS implementation. + +#### TLS/SSL enforcement + +When the certificates are defined and unless any other options are provided, a Netdata server will: + +- Redirect all incoming HTTP web server requests to HTTPS. Applies to the dashboard, the API, `netdata.conf` and badges. +- Allow incoming child connections to use both unencrypted and encrypted communications for streaming. + +To change this behavior, you need to modify the `bind to` setting in the `[web]` section of `netdata.conf`. At the end of each port definition, append `^SSL=force` or `^SSL=optional`. What happens with these settings differs, depending on whether the port is used for HTTP/S requests, or for streaming. + +| SSL setting | HTTP requests|HTTPS requests|Unencrypted Streams|Encrypted Streams| +|:---------:|:-----------:|:------------:|:-----------------:|:----------------| +| none | Redirected to HTTPS|Accepted|Accepted|Accepted| +| `force`| Redirected to HTTPS|Accepted|Denied|Accepted| +| `optional`| Accepted|Accepted|Accepted|Accepted| + +Example: + +``` +[web] + bind to = *=dashboard|registry|badges|management|streaming|netdata.conf^SSL=force +``` + +For information how to configure the child to use TLS, check [securing the communication](/streaming/README.md#securing-streaming-communications) in the streaming documentation. There you will find additional details on the expected behavior for client and server nodes, when their respective TLS options are enabled. + +When we define the use of SSL in a Netdata agent for different ports, Netdata will apply the behavior specified on each port. For example, using the configuration line below: + +``` +[web] + bind to = *=dashboard|registry|badges|management|streaming|netdata.conf^SSL=force *:20000=netdata.conf^SSL=optional *:20001=dashboard|registry +``` + +Netdata will: + +- Force all HTTP requests to the default port to be redirected to HTTPS (same port). +- Refuse unencrypted streaming connections from child nodes on the default port. +- Allow both HTTP and HTTPS requests to port 20000 for `netdata.conf` +- Force HTTP requests to port 20001 to be redirected to HTTPS (same port). Only allow requests for the dashboard, the read API and the registry on port 20001. + +#### TLS/SSL errors + +When you start using Netdata with TLS, you may find errors in the Netdata log, which is stored at `/var/log/netdata/error.log` by default. + +Most of the time, these errors are due to incompatibilities between your browser's options related to TLS/SSL protocols and Netdata's internal configuration. The most common error is `error:00000006:lib(0):func(0):EVP lib`. + +In the near future, Netdata will allow our users to change the internal configuration to avoid similar errors. Until then, we're recommending only the most common and safe encryption protocols listed above. + +### Access lists + +Netdata supports access lists in `netdata.conf`: + +``` +[web] + allow connections from = localhost * + allow dashboard from = localhost * + allow badges from = * + allow streaming from = * + allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.* + allow management from = localhost +``` + +`*` does string matches on the IPs or FQDNs of the clients. + +- `allow connections from` matches anyone that connects on the Netdata port(s). + So, if someone is not allowed, it will be connected and disconnected immediately, without reading even + a single byte from its connection. This is a global setting with higher priority to any of the ones below. + +- `allow dashboard from` receives the request and examines if it is a static dashboard file or an API call the + dashboards do. + +- `allow badges from` checks if the API request is for a badge. Badges are not matched by `allow dashboard from`. + +- `allow streaming from` checks if the child willing to stream metrics to this Netdata is allowed. + This can be controlled per API KEY and MACHINE GUID in `stream.conf`. + The setting in `netdata.conf` is checked before the ones in `stream.conf`. + +- `allow netdata.conf from` checks the IP to allow `http://netdata.host:19999/netdata.conf`. + The IPs listed are all the private IPv4 addresses, including link local IPv6 addresses. Keep in mind that connections to Netdata API ports are filtered by `allow connections from`. So, IPs allowed by `allow netdata.conf from` should also be allowed by `allow connections from`. + +- `allow management from` checks the IPs to allow API management calls. Management via the API is currently supported for [health](/web/api/health/README.md#health-management-api) + +In order to check the FQDN of the connection without opening the Netdata agent to DNS-spoofing, a reverse-dns record +must be setup for the connecting host. At connection time the reverse-dns of the peer IP address is resolved, and +a forward DNS resolution is made to validate the IP address against the name-pattern. + +Please note that this process can be expensive on a machine that is serving many connections. Each access list has an +associated configuration option to turn off DNS-based patterns completely to avoid incurring this cost at run-time: + +``` + allow connections by dns = heuristic + allow dashboard by dns = heuristic + allow badges by dns = heuristic + allow streaming by dns = heuristic + allow netdata.conf by dns = no + allow management by dns = heuristic +``` + +The three possible values for each of these options are `yes`, `no` and `heuristic`. The `heuristic` option disables +the check when the pattern only contains IPv4/IPv6 addresses or `localhost`, and enables it when wildcards are +present that may match DNS FQDNs. + +## DDoS protection + +If you publish your Netdata web server to the internet, you may want to apply some protection against DDoS: + +1. Use the `static-threaded` web server (it is the default) +2. Use reasonable `[web].web server max sockets` (the default is) +3. Don't use all your CPU cores for Netdata (lower `[web].web server threads`) +4. Run the `netdata` process with a low process scheduling priority (the default is the lowest) +5. If possible, proxy Netdata via a full featured web server (Nginx, Apache, etc) diff --git a/docs/guides/collect-apache-nginx-web-logs.md b/docs/guides/collect-apache-nginx-web-logs.md index 0298e1d43..a75a4b1cd 100644 --- a/docs/guides/collect-apache-nginx-web-logs.md +++ b/docs/guides/collect-apache-nginx-web-logs.md @@ -12,15 +12,11 @@ By parsing web server log files with Netdata, and seeing the volume of redirects you can better understand what's happening on your infrastructure. Too many bad requests? Maybe a recent deploy missed a few small SVG icons. Too many requests? Time to batten down the hatches—it's a DDoS. -Netdata has been capable of monitoring web log files for quite some time, thanks for the [weblog python.d -module](/collectors/python.d.plugin/web_log/README.md), but we recently refactored this module in Go, and that effort -comes with a ton of improvements. - -You can now use the [LTSV log format](http://ltsv.org/), track TLS and cipher usage, and the whole parser is faster than +You can use the [LTSV log format](http://ltsv.org/), track TLS and cipher usage, and the whole parser is faster than ever. In one test on a system with SSD storage, the collector consistently parsed the logs for 200,000 requests in -200ms, using ~30% of a single core. To learn more about these improvements, see our [v1.19 release post](https://blog.netdata.cloud/posts/release-1.19/). +200ms, using ~30% of a single core. -The [go.d plugin](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog/) is currently compatible +The [web_log](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog/) collector is currently compatible with [Nginx](https://nginx.org/en/) and [Apache](https://httpd.apache.org/). This guide will walk you through using the new Go-based web log collector to turn the logs these web servers @@ -34,33 +30,6 @@ installation procedures. Almost all web server installations will need _no_ configuration to start collecting metrics. As long as your web server has readable access log file, you can configure the web log plugin to access and parse it. -## Configure the web log collector - -To use the Go version of this plugin, you need to explicitly enable it, and disable the deprecated Python version. -First, open `python.d.conf`: - -```bash -cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ -./edit-config python.d.conf -``` - -Find the `web_log` line, uncomment it, and set it to `web_log: no`. Next, open the `go.d.conf` file for editing. - -```bash -./edit-config go.d.conf -``` - -Find the `web_log` line again, uncomment it, and set it to `web_log: yes`. - -Finally, restart Netdata with `sudo systemctl restart netdata`, or the [appropriate -method](/docs/configure/start-stop-restart.md) for your system. You should see metrics in your Netdata dashboard! - - - -If you don't see web log charts, or **web log nginx**/**web log apache** menus on the right-hand side of your dashboard, -continue reading for other configuration options. - ## Custom configuration of the web log collector The web log collector's default configuration comes with a few example jobs that should cover most Linux distributions @@ -152,11 +121,7 @@ documentation](/health/README.md). ## What's next? -Now that you have web log collection up and running, we recommend you take a look at the documentation for our -[python.d](/collectors/python.d.plugin/web_log/README.md) for some ideas of how you can turn these rather "boring" logs -into powerful real-time tools for keeping your servers happy. +Now that you have web log collection up and running, we recommend you take a look at the collector's [documentation](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/weblog/) for some ideas of how you can turn these rather "boring" logs into powerful real-time tools for keeping your servers happy. Don't forget to give GitHub user [Wing924](https://github.com/Wing924) a big 👍 for his hard work in starting up the Go refactoring effort. - - diff --git a/docs/guides/configure/performance.md b/docs/guides/configure/performance.md index 8e0108979..f83634168 100644 --- a/docs/guides/configure/performance.md +++ b/docs/guides/configure/performance.md @@ -72,7 +72,7 @@ seconds, respectively. ### Specific plugin or collector Every collector and plugin has its own `update every` setting, which you can also change in the `go.d.conf`, -`python.d.conf`, `node.d.conf`, or `charts.d.conf` files, or in individual collector configuration files. If the `update +`python.d.conf`, or `charts.d.conf` files, or in individual collector configuration files. If the `update every` for an individual collector is less than the global, the Netdata Agent uses the global setting. See the [enable or configure a collector](/docs/collect/enable-configure.md) doc for details. @@ -103,16 +103,14 @@ Keep in mind that if a plugin/collector has nothing to do, it simply shuts down You will only improve the Agent's performance by disabling plugins/collectors that are actively collecting metrics. Open `netdata.conf` and scroll down to the `[plugins]` section. To disable any plugin, uncomment it and set the value to -`no`. For example, to explicitly keep the `proc` and `go.d` plugins enabled while disabling `python.d`, `charts.d`, and -`node.d`. +`no`. For example, to explicitly keep the `proc` and `go.d` plugins enabled while disabling `python.d` and `charts.d`. ```conf [plugins] proc = yes - python.d = no - charts.d = no - node.d = no - go.d = yes + python.d = no + charts.d = no + go.d = yes ``` Disable specific collectors by opening their respective plugin configuration files, uncommenting the line for the @@ -121,7 +119,6 @@ collector, and setting its value to `no`. ```bash sudo ./edit-config go.d.conf sudo ./edit-config python.d.conf -sudo ./edit-config node.d.conf sudo ./edit-config charts.d.conf ``` @@ -186,7 +183,6 @@ Finally, edit `netdata.conf` with the following settings: ```conf [global] bind socket to IP = 127.0.0.1 - access log = none disconnect idle web clients after seconds = 3600 enable web responses gzip compression = no ``` @@ -218,7 +214,7 @@ If you installation is working correctly, and you're not actively auditing Netda `netdata.conf`. ```conf -[global] +[logs] debug log = none error log = none access log = none diff --git a/docs/guides/step-by-step/step-06.md b/docs/guides/step-by-step/step-06.md index 89a8cb732..f04098fc1 100644 --- a/docs/guides/step-by-step/step-06.md +++ b/docs/guides/step-by-step/step-06.md @@ -39,7 +39,7 @@ they were built in. These modules are primarily written in [Go](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/) (`go.d`) and [Python](/collectors/python.d.plugin/README.md), although some use [Bash](/collectors/charts.d.plugin/README.md) -(`charts.d`) or [Node.js](/collectors/node.d.plugin/README.md) (`node.d`). +(`charts.d`). ## Enable and disable plugins @@ -58,14 +58,14 @@ Enabled: ```conf [plugins] - # node.d = yes + # python.d = yes ``` Disabled: ```conf [plugins] - node.d = no + python.d = no ``` When you explicitly disable a plugin this way, it won't auto-collect metrics using its collectors. diff --git a/docs/metrics-storage-management/reference-streaming.mdx b/docs/metrics-storage-management/reference-streaming.mdx index 1be597dd8..c77ceb37c 100644 --- a/docs/metrics-storage-management/reference-streaming.mdx +++ b/docs/metrics-storage-management/reference-streaming.mdx @@ -54,7 +54,7 @@ node**. This file is automatically generated by Netdata the first time it is sta | `timeout seconds` | `60` | The timeout to connect and send metrics to a parent. | | `default port` | `19999` | The port to use if `destination` does not specify one. | | [`send charts matching`](#send-charts-matching) | `*` | A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to filter which charts are streamed. [Read more →](#send-charts-matching) | -| `buffer size bytes` | `1048576` | The size of the buffer to use when sending metrics. The default `1048576` equals a buffer of 1MB, which is good for 10-20 seconds of data. Increase this if you expect latencies higher than that. The buffer is flushed on reconnect. | +| `buffer size bytes` | `10485760` | The size of the buffer to use when sending metrics. The default `10485760` equals a buffer of 10MB, which is good for 60 seconds of data. Increase this if you expect latencies higher than that. The buffer is flushed on reconnect. | | `reconnect delay seconds` | `5` | How long to wait until retrying to connect to the parent node. | | `initial clock resync iterations` | `60` | Sync the clock of charts for how many seconds when starting. | diff --git a/docs/monitor/enable-notifications.md b/docs/monitor/enable-notifications.md index 8cf697972..438eef391 100644 --- a/docs/monitor/enable-notifications.md +++ b/docs/monitor/enable-notifications.md @@ -67,6 +67,7 @@ notification platform. - [**Email**](/health/notifications/email/README.md) - [**Flock**](/health/notifications/flock/README.md) - [**Google Hangouts**](/health/notifications/hangouts/README.md) +- [**Gotify**](/health/notifications/gotify/README.md) - [**IRC**](/health/notifications/irc/README.md) - [**Kavenegar**](/health/notifications/kavenegar/README.md) - [**Matrix**](/health/notifications/matrix/README.md) @@ -144,4 +145,3 @@ architecture](/docs/store/distributed-data-architecture.md) for the best-in-clas - [Netdata Cloud · Alarm notifications](https://learn.netdata.cloud/docs/cloud/alerts-notifications/notifications) - [Netdata Agent · Notifications](/health/notifications/README.md) - |