diff options
Diffstat (limited to 'streaming/README.md')
| -rw-r--r-- | streaming/README.md | 51 |
1 files changed, 28 insertions, 23 deletions
diff --git a/streaming/README.md b/streaming/README.md index 71c1dc28..57c392f4 100644 --- a/streaming/README.md +++ b/streaming/README.md @@ -67,18 +67,18 @@ Alerts for the child can be triggered by any of the involved hosts that maintain You can daisy-chain any number of Netdata, each with or without a database and with or without alerts for the child metrics. -### Mix and match with backends +### Mix and match with exporting engine -All nodes that maintain a database can also send their data to a backend database. +All nodes that maintain a database can also send their data to an external database. This allows quite complex setups. Example: 1. Netdata nodes `A` and `B` do not maintain a database and stream metrics to Netdata node `C`(live streaming functionality). -2. Netdata node `C` maintains a database for `A`, `B`, `C` and archives all metrics to `graphite` with 10 second detail (backends functionality). +2. Netdata node `C` maintains a database for `A`, `B`, `C` and archives all metrics to `graphite` with 10 second detail (exporting functionality). 3. Netdata node `C` also streams data for `A`, `B`, `C` to Netdata `D`, which also collects data from `E`, `F` and `G` from another DMZ (live streaming functionality). 4. Netdata node `D` is just a proxy, without a database, that streams all data to a remote site at Netdata `H`. -5. Netdata node `H` maintains a database for `A`, `B`, `C`, `D`, `E`, `F`, `G`, `H` and sends all data to `opentsdb` with 5 seconds detail (backends functionality) +5. Netdata node `H` maintains a database for `A`, `B`, `C`, `D`, `E`, `F`, `G`, `H` and sends all data to `opentsdb` with 5 seconds detail (exporting functionality) 6. Alerts are triggered by `H` for all hosts. 7. Users can use all Netdata nodes that maintain a database to view metrics (i.e. at `H` all hosts can be viewed). @@ -107,15 +107,7 @@ This also disables the registry (there cannot be a registry without an API). requests from its child nodes. 0 sets no limit, 1 means maximum once every second. If this is set, you may see error log entries "... too busy to accept new streaming request. Will be allowed in X secs". -``` -[backend] - enabled = yes | no - type = graphite | opentsdb - destination = IP:PORT ... - update every = 10 -``` - -`[backend]` configures data archiving to a backend (it archives all databases maintained on +You can [use](/exporting/README.md#configuration) the exporting engine to configure data archiving to an external database (it archives all databases maintained on this host). ### Streaming configuration @@ -156,7 +148,7 @@ a proxy). ``` This is an overview of how these options can be combined: -| target|memory<br/>mode|web<br/>mode|stream<br/>enabled|backend|alarms|dashboard| +| target|memory<br/>mode|web<br/>mode|stream<br/>enabled|exporting|alarms|dashboard| |------|:-------------:|:----------:|:----------------:|:-----:|:----:|:-------:| | headless collector|`none`|`none`|`yes`|only for `data source = as collected`|not possible|no| | headless proxy|`none`|not `none`|`yes`|only for `data source = as collected`|not possible|no| @@ -211,7 +203,6 @@ that use `*` as wildcard (any number of times) and a `!` prefix for a negative m So: `allow from = !10.1.2.3 10.*` will allow all IPs in `10.*` except `10.1.2.3`. The order is important: left to right, the first positive or negative match is used. -`allow from` is available in Netdata v1.9+ ##### Tracing @@ -366,16 +357,30 @@ Note: The `stream-compression` status can be `"enabled" | "disabled" | "N/A"`. A compressed data packet is determined and decompressed on the fly. #### Limitations - This limitation will be withdrawn asap and is work-in-progress. - -The current implementation of streaming data compression can support only a few number of dimensions in a chart with names that cannot exceed the size of 16384 bytes. In case you experience stream connection problems or gaps in the charts please disable stream compresssion in the `stream.conf` file. This limitation can be seen in the error.log file with the sequence of the following messages: -``` -Compression error - data discarded -Message size above limit: +This limitation will be withdrawn asap and is work-in-progress. + +The current implementation of streaming data compression can support only a few number of dimensions in a chart with names that cannot exceed the size of 16384 bytes. In case your instance hit this limitation, the agent will deactivate compression during runtime to avoid stream corruption. This limitation can be seen in the error.log file with the sequence of the following messages: +``` +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: connecting... +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: initializing communication... +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: waiting response from remote netdata... +netdata INFO : STREAM_SENDER[child01] : STREAM_COMPRESSION: Compressor Reset +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: established communication with a parent using protocol version 5 - ready to send metrics... +... +netdata ERROR : PLUGINSD[go.d] : STREAM_COMPRESSION: Compression Failed - Message size 27847 above compression buffer limit: 16384 (errno 9, Bad file descriptor) +netdata ERROR : PLUGINSD[go.d] : STREAM_COMPRESSION: Deactivating compression to avoid stream corruption +netdata ERROR : PLUGINSD[go.d] : STREAM_COMPRESSION child01 [send to my.parent.IP]: Restarting connection without compression +... +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: connecting... +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: initializing communication... +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: waiting response from remote netdata... +netdata INFO : STREAM_SENDER[child01] : Stream is uncompressed! One of the agents (my.parent.IP <-> child01) does not support compression OR compression is disabled. +netdata INFO : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: established communication with a parent using protocol version 4 - ready to send metrics... +netdata INFO : WEB_SERVER[static4] : STREAM child01 [send]: sending metrics... ``` #### How to enable stream compression -Netdata Agents are shipped with data compression disabled by default. You can also configure which streams will use compression. +Netdata Agents are shipped with data compression enabled by default. You can also configure which streams will use compression. With enabled stream compression, a Netdata Agent can negotiate streaming compression with other Netdata Agents. During the negotiation of streaming compression both Netdata Agents should support and enable compression in order to communicate over a compressed stream. The negotiation will result into an uncompressed stream, if one of the Netdata Agents doesn't support **or** has compression disabled. @@ -727,4 +732,4 @@ file descriptor given is not a valid stream After logging this error, Netdata will close the stream. -[](<>) + |