diff options
Diffstat (limited to 'docs/metrics-storage-management/reference-streaming.mdx')
| -rw-r--r-- | docs/metrics-storage-management/reference-streaming.mdx | 486 |
1 files changed, 486 insertions, 0 deletions
diff --git a/docs/metrics-storage-management/reference-streaming.mdx b/docs/metrics-storage-management/reference-streaming.mdx new file mode 100644 index 000000000..1be597dd8 --- /dev/null +++ b/docs/metrics-storage-management/reference-streaming.mdx @@ -0,0 +1,486 @@ +--- +title: "Streaming reference" +description: "Each node running Netdata can stream the metrics it collects, in real time, to another node. See all of the available settings in this reference document." +type: reference +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/reference-streaming.mdx +--- + +# Streaming reference + +Each node running Netdata can stream the metrics it collects, in real time, to another node. To learn more, read about +[how streaming works](/docs/metrics-storage-management/how-streaming-works.mdx). + +For a quickstart guide for enabling a simple `parent-child` streaming relationship, see our [stream metrics between +nodes](/docs/metrics-storage-management/enable-streaming.mdx) doc. All other configuration options and scenarios are +covered in the sections below. + +## Configuration + +There are two files responsible for configuring Netdata's streaming capabilities: `stream.conf` and `netdata.conf`. + +From within your Netdata config directory (typically `/etc/netdata`), [use `edit-config`](/docs/configure/nodes.md) to +open either `stream.conf` or `netdata.conf`. + +``` +sudo ./edit-config stream.conf +sudo ./edit-config netdata.conf +``` + +## Settings + +As mentioned above, both `stream.conf` and `netdata.conf` contain settings relevant to streaming. + +### `stream.conf` + +The `stream.conf` file contains three sections. The `[stream]` section is for configuring child nodes. + +The `[API_KEY]` and `[MACHINE_GUID]` sections are both for configuring parent nodes, and share the same settings. +`[API_KEY]` settings affect every child node using that key, whereas `[MACHINE_GUID]` settings affect only the child +node with a matching GUID. + +The file `/var/lib/netdata/registry/netdata.public.unique.id` contains a random GUID that **uniquely identifies each +node**. This file is automatically generated by Netdata the first time it is started and remains unaltered forever. + +#### `[stream]` section + +| Setting | Default | Description | +| :---------------------------------------------- | :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enabled` | `no` | Whether this node streams metrics to any parent. Change to `yes` to enable streaming. | +| [`destination`](#destination) | ` ` | A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`. [Read more →](#destination) | +| `ssl skip certificate verification` | `yes` | If you want to accept self-signed or expired certificates, set to `yes` and uncomment. | +| `CApath` | `/etc/ssl/certs/` | The directory where known certificates are found. Defaults to OpenSSL's default path. | +| `CAfile` | `/etc/ssl/certs/cert.pem` | Add a parent node certificate to the list of known certificates in `CAPath`. | +| `api key` | ` ` | The `API_KEY` to use as the child node. | +| `timeout seconds` | `60` | The timeout to connect and send metrics to a parent. | +| `default port` | `19999` | The port to use if `destination` does not specify one. | +| [`send charts matching`](#send-charts-matching) | `*` | A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to filter which charts are streamed. [Read more →](#send-charts-matching) | +| `buffer size bytes` | `1048576` | The size of the buffer to use when sending metrics. The default `1048576` equals a buffer of 1MB, which is good for 10-20 seconds of data. Increase this if you expect latencies higher than that. The buffer is flushed on reconnect. | +| `reconnect delay seconds` | `5` | How long to wait until retrying to connect to the parent node. | +| `initial clock resync iterations` | `60` | Sync the clock of charts for how many seconds when starting. | + +### `[API_KEY]` and `[MACHINE_GUID]` sections + +| Setting | Default | Description | +| :---------------------------------------------- | :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enabled` | `no` | Whether this API KEY enabled or disabled. | +| [`allow from`](#allow-from) | `*` | A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) matching the IPs of nodes that will stream metrics using this API key. [Read more →](#allow-from) | +| `default history` | `3600` | The default amount of child metrics history to retain when using the `save`, `map`, or `ram` memory modes. | +| [`default memory mode`](#default-memory-mode) | `ram` | The [database](/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `map`, `save`, `ram`, or `none`. [Read more →](#default-memory-mode) | +| `health enabled by default` | `auto` | Whether alarms and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alarms when the child is connected. `yes` enables alarms always, and `no` disables alarms. | +| `default postpone alarms on connect seconds` | `60` | Postpone alarms and notifications for a period of time after the child connects. | +| `default proxy enabled` | ` ` | Route metrics through a proxy. | +| `default proxy destination` | ` ` | Space-separated list of `IP:PORT` for proxies. | +| `default proxy api key` | ` ` | The `API_KEY` of the proxy. | +| `default send charts matching` | `*` | See [`send charts matching`](#send-charts-matching). | + +#### `destination` + +A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using +the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`. + +- `PROTOCOL`: `tcp`, `udp`, or `unix`. (only tcp and unix are supported by parent nodes) +- `HOST`: A IPv4, IPv6 IP, or a hostname, or a unix domain socket path. IPv6 IPs should be given with brackets + `[ip:address]`. +- `INTERFACE` (IPv6 only): The network interface to use. +- `PORT`: The port number or service name (`/etc/services`) to use. +- `SSL`: To enable TLS/SSL encryption of the streaming connection. + +To enable TCP streaming to a parent node at `203.0.113.0` on port `20000` and with TLS/SSL encryption: + +```conf +[stream] + destination = tcp:203.0.113.0:20000:SSL +``` + +#### `send charts matching` + +A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to filter which charts are streamed. + +The default is a single wildcard `*`, which streams all charts. + +To send only a few charts, list them explicitly, or list a group using a wildcard. To send _only_ the `apps.cpu` chart +and charts with contexts beginning with `system.`: + +```conf +[stream] + send charts matching = apps.cpu system.* +``` + +To send all but a few charts, use `!` to create a negative match. To send _all_ charts _but_ `apps.cpu`: + +```conf +[stream] + send charts matching = !apps.cpu * +``` + +#### `allow from` + +A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) matching the IPs of nodes that +will stream metrics using this API key. The order is important, left to right, as the first positive or negative match is used. + +The default is `*`, which accepts all requests including the `API_KEY`. + +To allow from only a specific IP address: + +```conf +[API_KEY] + allow from = 203.0.113.10 +``` + +To allow all IPs starting with `10.*`, except `10.1.2.3`: + +```conf +[API_KEY] + allow from = !10.1.2.3 10.* +``` + +> If you set specific IP addresses here, and also use the `allow connections` setting in the `[web]` section of +> `netdata.conf`, be sure to add the IP address there so that it can access the API port. + +#### `default memory mode` + +The [database](/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `ram`, +`save`, `map`, or `none`. + +- `dbengine`: The default, recommended time-series database (TSDB) for Netdata. Stores recent metrics in memory, then + efficiently spills them to disk for long-term storage. +- `ram`: Stores metrics _only_ in memory, which means metrics are lost when Netdata stops or restarts. Ideal for + streaming configurations that use ephemeral nodes. +- `save`: Stores metrics in memory, but saves metrics to disk when Netdata stops or restarts, and loads historical + metrics on start. +- `map`: Stores metrics in memory-mapped files, like swap, with constant disk write. +- `none`: No database. + +When using `default memory mode = dbengine`, the parent node creates a separate instance of the TSDB to store metrics +from child nodes. The [size of _each_ instance is configurable](/docs/store/change-metrics-storage.md) with the `page +cache size` and `dbengine multihost disk space` settings in the `[global]` section in `netdata.conf`. + +### `netdata.conf` + +| Setting | Default | Description | +| :----------------------------------------- | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **`[global]` section** | | | +| `memory mode` | `dbengine` | Determines the [database type](/database/README.md) to be used on that node. Other options settings include `none`, `ram`, `save`, and `map`. `none` disables the database at this host. This also disables alarms and notifications, as those can't run without a database. | +| **`[web]` section** | | | +| `mode` | `static-threaded` | Determines the [web server](/web/server/README.md) type. The other option is `none`, which disables the dashboard, API, and registry. | +| `accept a streaming request every seconds` | `0` | Set a limit on how often a parent node accepts streaming requests from child nodes. `0` equals no limit. If this is set, you may see `... too busy to accept new streaming request. Will be allowed in X secs` in Netdata's `error.log`. | + +## Examples + +### Per-child settings + +While the `[API_KEY]` section applies settings for any child node using that key, you can also use per-child settings +with the `[MACHINE_GUID]` section. + +For example, the metrics streamed from only the child node with `MACHINE_GUID` are saved in memory, not using the +default `dbengine` as specified by the `API_KEY`, and alarms are disabled. + +```conf +[API_KEY] + enabled = yes + default memory mode = dbengine + health enabled by default = auto + allow from = * + +[MACHINE_GUID] + enabled = yes + memory mode = save + health enabled = no +``` + +### Securing streaming with TLS/SSL + +Netdata does not activate TLS encryption by default. To encrypt streaming connections, you first need to [enable TLS +support](/web/server/README.md#enabling-tls-support) on the parent. With encryption enabled on the receiving side, you +need to instruct the child to use TLS/SSL as well. On the child's `stream.conf`, configure the destination as follows: + +``` +[stream] + destination = host:port:SSL +``` + +The word `SSL` appended to the end of the destination tells the child that connections must be encrypted. + +> While Netdata uses Transport Layer Security (TLS) 1.2 to encrypt communications rather than the obsolete SSL protocol, +> it's still common practice to refer to encrypted web connections as `SSL`. Many vendors, like Nginx and even Netdata +> itself, use `SSL` in configuration files, whereas documentation will always refer to encrypted communications as `TLS` +> or `TLS/SSL`. + +#### Certificate verification + +When TLS/SSL is enabled on the child, the default behavior will be to not connect with the parent unless the server's +certificate can be verified via the default chain. In case you want to avoid this check, add the following to the +child's `stream.conf` file: + +``` +[stream] + ssl skip certificate verification = yes +``` + +#### Trusted certificate + +If you've enabled [certificate verification](#certificate-verification), you might see errors from the OpenSSL library +when there's a problem with checking the certificate chain (`X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY`). More +importantly, OpenSSL will reject self-signed certificates. + +Given these known issues, you have two options. If you trust your certificate, you can set the options `CApath` and +`CAfile` to inform Netdata where your certificates, and the certificate trusted file, are stored. + +For more details about these options, you can read about [verify +locations](https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_load_verify_locations.html). + +Before you changed your streaming configuration, you need to copy your trusted certificate to your child system and add +the certificate to OpenSSL's list. + +On most Linux distributions, the `update-ca-certificates` command searches inside the `/usr/share/ca-certificates` +directory for certificates. You should double-check by reading the `update-ca-certificate` manual (`man +update-ca-certificate`), and then change the directory in the below commands if needed. + +If you have `sudo` configured on your child system, you can use that to run the following commands. If not, you'll have +to log in as `root` to complete them. + +``` +# mkdir /usr/share/ca-certificates/netdata +# cp parent_cert.pem /usr/share/ca-certificates/netdata/parent_cert.crt +# chown -R netdata.netdata /usr/share/ca-certificates/netdata/ +``` + +First, you create a new directory to store your certificates for Netdata. Next, you need to change the extension on your +certificate from `.pem` to `.crt` so it's compatible with `update-ca-certificate`. Finally, you need to change +permissions so the user that runs Netdata can access the directory where you copied in your certificate. + +Next, edit the file `/etc/ca-certificates.conf` and add the following line: + +``` +netdata/parent_cert.crt +``` + +Now you update the list of certificates running the following, again either as `sudo` or `root`: + +``` +# update-ca-certificates +``` + +> Some Linux distributions have different methods of updating the certificate list. For more details, please read this +> guide on [adding trusted root certificates](https://github.com/Busindre/How-to-Add-trusted-root-certificates). + +Once you update your certificate list, you can set the stream parameters for Netdata to trust the parent certificate. +Open `stream.conf` for editing and change the following lines: + +``` +[stream] + CApath = /etc/ssl/certs/ + CAfile = /etc/ssl/certs/parent_cert.pem +``` + +With this configuration, the `CApath` option tells Netdata to search for trusted certificates inside `/etc/ssl/certs`. +The `CAfile` option specifies the Netdata parent certificate is located at `/etc/ssl/certs/parent_cert.pem`. With this +configuration, you can skip using the system's entire list of certificates and use Netdata's parent certificate instead. + +#### Expected behaviors + +With the introduction of TLS/SSL, the parent-child communication behaves as shown in the table below, depending on the +following configurations: + +- **Parent TLS (Yes/No)**: Whether the `[web]` section in `netdata.conf` has `ssl key` and `ssl certificate`. +- **Parent port TLS (-/force/optional)**: Depends on whether the `[web]` section `bind to` contains a `^SSL=force` or + `^SSL=optional` directive on the port(s) used for streaming. +- **Child TLS (Yes/No)**: Whether the destination in the child's `stream.conf` has `:SSL` at the end. +- **Child TLS Verification (yes/no)**: Value of the child's `stream.conf` `ssl skip certificate verification` + parameter (default is no). + +| Parent TLS enabled | Parent port SSL | Child TLS | Child SSL Ver. | Behavior | +| :----------------- | :--------------- | :-------- | :------------- | :--------------------------------------------------------------------------------------------------------------------------------------- | +| No | - | No | no | Legacy behavior. The parent-child stream is unencrypted. | +| Yes | force | No | no | The parent rejects the child connection. | +| Yes | -/optional | No | no | The parent-child stream is unencrypted (expected situation for legacy child nodes and newer parent nodes) | +| Yes | -/force/optional | Yes | no | The parent-child stream is encrypted, provided that the parent has a valid TLS/SSL certificate. Otherwise, the child refuses to connect. | +| Yes | -/force/optional | Yes | yes | The parent-child stream is encrypted. | + +### Proxy + +A proxy is a node that receives metrics from a child, then streams them onward to a parent. To configure a proxy, +configure it as a receiving and a sending Netdata at the same time. + +Netdata proxies may or may not maintain a database for the metrics passing through them. When they maintain a database, +they can also run health checks (alarms and notifications) for the remote host that is streaming the metrics. + +In the following example, the proxy receives metrics from a child node using the `API_KEY` of +`66666666-7777-8888-9999-000000000000`, then stores metrics using `dbengine`. It then uses the `API_KEY` of +`11111111-2222-3333-4444-555555555555` to proxy those same metrics on to a parent node at `203.0.113.0`. + +```conf +[stream] + enabled = yes + destination = 203.0.113.0 + api key = 11111111-2222-3333-4444-555555555555 + +[66666666-7777-8888-9999-000000000000] + enabled = yes + default memory mode = dbengine +``` + +### Ephemeral nodes + +Netdata can help you monitor ephemeral nodes, such as containers in an auto-scaling infrastructure, by always streaming +metrics to any number of permanently-running parent nodes. + +On the parent, set the following in `stream.conf`: + +```conf +[11111111-2222-3333-4444-555555555555] + # enable/disable this API key + enabled = yes + + # one hour of data for each of the child nodes + default history = 3600 + + # do not save child metrics on disk + default memory = ram + + # alarms checks, only while the child is connected + health enabled by default = auto +``` + +On the child nodes, set the following in `stream.conf`: + +```bash +[stream] + # stream metrics to another Netdata + enabled = yes + + # the IP and PORT of the parent + destination = 10.11.12.13:19999 + + # the API key to use + api key = 11111111-2222-3333-4444-555555555555 +``` + +In addition, edit `netdata.conf` on each child node to disable the database and alarms. + +```bash +[global] + # disable the local database + memory mode = none + +[health] + # disable health checks + enabled = no +``` + +## Troubleshooting + +Both parent and child nodes log information at `/var/log/netdata/error.log`. + +If the child manages to connect to the parent you will see something like (on the parent): + +``` +2017-03-09 09:38:52: netdata: INFO : STREAM [receive from [10.11.12.86]:38564]: new client connection. +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [10.11.12.86]:38564: receive thread created (task id 27721) +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: client willing to stream metrics for host 'xxx' with machine_guid '1234567-1976-11e6-ae19-7cdd9077342a': update every = 1, history = 3600, memory mode = ram, health auto +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: initializing communication... +2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: receiving metrics... +``` + +and something like this on the child: + +``` +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: connecting... +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: initializing communication... +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: waiting response from remote netdata... +2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: established communication - sending metrics... +``` + +The following sections describe the most common issues you might encounter when connecting parent and child nodes. + +### Slow connections between parent and child + +When you have a slow connection between parent and child, Netdata raises a few different errors. Most of the +errors will appear in the child's `error.log`. + +```bash +netdata ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM CHILD HOSTNAME [send to PARENT IP:PARENT PORT]: too many data pending - buffer is X bytes long, +Y unsent - we have sent Z bytes in total, W on this connection. Closing connection to flush the data. +``` + +On the parent side, you may see various error messages, most commonly the following: + +``` +netdata ERROR : STREAM_PARENT[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : read failed: end of file +``` + +Another common problem in slow connections is the child sending a partial message to the parent. In this case, the +parent will write the following to its `error.log`: + +``` +ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : sent command 'B' which is not known by netdata, for host 'HOSTNAME'. Disabling it. +``` + +In this example, `B` was part of a `BEGIN` message that was cut due to connection problems. + +Slow connections can also cause problems when the parent misses a message and then receives a command related to the +missed message. For example, a parent might miss a message containing the child's charts, and then doesn't know +what to do with the `SET` message that follows. When that happens, the parent will show a message like this: + +``` +ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : requested a SET on chart 'CHART NAME' of host 'HOSTNAME', without a dimension. Disabling it. +``` + +### Child cannot connect to parent + +When the child can't connect to a parent for any reason (misconfiguration, networking, firewalls, parent +down), you will see the following in the child's `error.log`. + +``` +ERROR : STREAM_SENDER[HOSTNAME] : Failed to connect to 'PARENT IP', port 'PARENT PORT' (errno 113, No route to host) +``` + +### 'Is this a Netdata?' + +This question can appear when Netdata starts the stream and receives an unexpected response. This error can appear when +the parent is using SSL and the child tries to connect using plain text. You will also see this message when +Netdata connects to another server that isn't Netdata. The complete error message will look like this: + +``` +ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM child HOSTNAME [send to PARENT HOSTNAME:PARENT PORT]: server is not replying properly (is it a netdata?). +``` + +### Stream charts wrong + +Chart data needs to be consistent between child and parent nodes. If there are differences between chart data on +a parent and a child, such as gaps in metrics collection, it most often means your child's `memory mode` +does not match the parent's. To learn more about the different ways Netdata can store metrics, and thus keep chart +data consistent, read our [memory mode documentation](/database/README.md). + +### Forbidding access + +You may see errors about "forbidding access" for a number of reasons. It could be because of a slow connection between +the parent and child nodes, but it could also be due to other failures. Look in your parent's `error.log` for errors +that look like this: + +``` +STREAM [receive from [child HOSTNAME]:child IP]: `MESSAGE`. Forbidding access." +``` + +`MESSAGE` will have one of the following patterns: + +- `request without KEY` : The message received is incomplete and the KEY value can be API, hostname, machine GUID. +- `API key 'VALUE' is not valid GUID`: The UUID received from child does not have the format defined in [RFC + 4122](https://tools.ietf.org/html/rfc4122) +- `machine GUID 'VALUE' is not GUID.`: This error with machine GUID is like the previous one. +- `API key 'VALUE' is not allowed`: This stream has a wrong API key. +- `API key 'VALUE' is not permitted from this IP`: The IP is not allowed to use STREAM with this parent. +- `machine GUID 'VALUE' is not allowed.`: The GUID that is trying to send stream is not allowed. +- `Machine GUID 'VALUE' is not permitted from this IP. `: The IP does not match the pattern or IP allowed to connect to + use stream. + +### Netdata could not create a stream + +The connection between parent and child is a stream. When the parent can't convert the initial connection into +a stream, it will write the following message inside `error.log`: + +``` +file descriptor given is not a valid stream +``` + +After logging this error, Netdata will close the stream. |