diff options
Diffstat (limited to 'docs/metrics-storage-management/how-streaming-works.mdx')
| -rw-r--r-- | docs/metrics-storage-management/how-streaming-works.mdx | 92 |
1 files changed, 92 insertions, 0 deletions
diff --git a/docs/metrics-storage-management/how-streaming-works.mdx b/docs/metrics-storage-management/how-streaming-works.mdx new file mode 100644 index 000000000..ecbce39bc --- /dev/null +++ b/docs/metrics-storage-management/how-streaming-works.mdx @@ -0,0 +1,92 @@ +--- +title: "How metrics streaming works" +description: "Netdata's real-time streaming allows you to replicate metrics data across multiple nodes, or centralize all your metrics data into a single time-series database (TSDB)." +type: explanation +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/how-streaming-works.mdx +--- + +# How metrics streaming works + +Each node running Netdata can stream the metrics it collects, in real time, to another node. Streaming allows you to +replicate metrics data across multiple nodes, or centralize all your metrics data into a single time-series database +(TSDB). + +When one node streams metrics to another, the node receiving metrics can visualize them on the +[dashboard](/docs/visualize/interact-dashboards-charts.md), run health checks to [trigger +alarms](/docs/monitor/view-active-alarms.md) and [send notifications](/docs/monitor/enable-notifications.md), and +[export](/docs/export/external-databases.md) all metrics to an external TSDB. When Netdata streams metrics to another +Netdata, the receiving one is able to perform everything a Netdata instance is capable of. + +Streaming lets you decide exactly how you want to store and maintain metrics data. While we believe Netdata's +[distributed architecture](/docs/store/distributed-data-architecture.md) is ideal for speed and scale, streaming +provides centralization options for those who want to maintain only a single TSDB instance. + +## Streaming basics + +There are three types of nodes in Netdata's streaming ecosystem. + +- **Parent**: A node, running Netdata, that receives streamed metric data. +- **Child**: A node, running Netdata, that streams metric data to one or more parent. +- **Proxy**: A node, running Netdata, that receives metric data from a child and "forwards" them on to a + separate parent node. + +Netdata uses API keys, which are just random GUIDs, to authorize the communication between child and parent nodes. We +recommend using `uuidgen` for generating API keys, which can then be used across any number of streaming connections. +Or, you can generate unique API keys for each parent-child relationship. + +Once the parent node authorizes the child's API key, the child can start streaming metrics. + +It's important to note that the streaming connection uses TCP, UDP, or Unix sockets, _not HTTP_. To proxy streaming +metrics, you need to use a proxy that tunnels [OSI layer 4-7 +traffic](https://en.wikipedia.org/wiki/OSI_model#Layer_4:_Transport_Layer) without interfering with it, such as +[SOCKS](https://en.wikipedia.org/wiki/SOCKS) or Nginx's [TCP/UDP load +balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/). + +## Supported streaming configurations + +Netdata supports any combination of parent, child, and proxy nodes that you can imagine. Any node can act as both a +parent, child, or proxy at the same time, sending or receiving streaming metrics from any number of other nodes. + +Here are a few example streaming configurations: + +- **Headless collector**: + - Child `A`, _without_ a database or web dashboard, streams metrics to parent `B`. + - `A` metrics are only available via the local Agent dashboard for `B`. + - `B` generates alarms for `A`. +- **Replication**: + - Child `A`, _with_ a database and web dashboard, streams metrics to parent `B`. + - `A` metrics are available on both local Agent dashboards, and can be stored with the same or different metrics + retention policies. + - Both `A` and `B` generate alarms. +- **Proxy**: + - Child `A`, _with or without_ a database, sends metrics to proxy `C`, also _with or without_ a database. `C` sends + metrics to parent `B`. + - Any node with a database can generate alarms. + +## Viewing streamed metrics + +Parent nodes feature a **Replicated Nodes** section in the left-hand panel, which opens with the hamburger icon + +in the top navigation. The parent node, plus any child nodes, appear here. Click on any of the hostnames to switch +between parent and child dashboards, all served by the parent's [web server](/web/server/README.md). + + + +Each child dashboard is also available directly at the following URL pattern: +`http://PARENT-NODE:19999/host/CHILD-HOSTNAME`. + +## What's next? + +Now that you understand the fundamentals of streaming metrics between nodes, go ahead and [enable +streaming](/docs/metrics-storage-management/enable-streaming.mdx) using a simple `parent-child` relationship. For all +the details, see the [streaming reference](/docs/metrics-storage-management/reference-streaming.mdx) doc. + +Take your streaming setup even further by [exporting metrics](/docs/export/external-databases.md) to an external TSDB. + +### Related information & further reading + +- Streaming + - **[How Netdata streams metrics](/docs/metrics-storage-management/how-streaming-works.mdx)** + - [Enable streaming between nodes](/docs/metrics-storage-management/enable-streaming.mdx) + - [Streaming reference](/docs/metrics-storage-management/reference-streaming.mdx)
\ No newline at end of file |