summaryrefslogtreecommitdiffstats
path: root/docs/collect/how-collectors-work.md
blob: 382d4ccc6fe5c8048306cc78a7e198e67cdf3d9c (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
<!--
title: "How Netdata's metrics collectors work"
description: "When Netdata starts, and with zero configuration, it auto-detects thousands of data sources and immediately collects per-second metrics."
custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/collect/how-collectors-work.md"
sidebar_label: "How Netdata's metrics collectors work"
learn_status: "Published"
learn_topic_type: "Concepts"
learn_rel_path: "Concepts"
-->

# How Netdata's metrics collectors work

When Netdata starts, and with zero configuration, it auto-detects thousands of data sources and immediately collects
per-second metrics.

Netdata can immediately collect metrics from these endpoints thanks to 300+ **collectors**, which all come pre-installed
when you [install Netdata](https://github.com/netdata/netdata/blob/master/docs/get-started.mdx).

Every collector has two primary jobs:

-   Look for exposed metrics at a pre- or user-defined endpoint.
-   Gather exposed metrics and use additional logic to build meaningful, interactive visualizations.

If the collector finds compatible metrics exposed on the configured endpoint, it begins a per-second collection job. The
Netdata Agent gathers these metrics, sends them to the [database engine for
storage](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md), and immediately [visualizes them
meaningfully](https://github.com/netdata/netdata/blob/master/docs/visualize/interact-dashboards-charts.md) on dashboards.

Each collector comes with a pre-defined configuration that matches the default setup for that application. This endpoint
can be a URL and port, a socket, a file, a web page, and more.

For example, the [Nginx collector](https://github.com/netdata/go.d.plugin/blob/master/modules/nginx/README.md) searches
at `http://127.0.0.1/stub_status`, which is the default endpoint for exposing Nginx metrics. The [web log collector for
Nginx or Apache](https://github.com/netdata/go.d.plugin/blob/master/README.mdmodules/weblog) searches at
`/var/log/nginx/access.log` and `/var/log/apache2/access.log`, respectively, both of which are standard locations for
access log files on Linux systems.

The endpoint is user-configurable, as are many other specifics of what a given collector does.

## What can Netdata collect?

To quickly find your answer, see our [list of supported collectors](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md).

Generally, Netdata's collectors can be grouped into three types:

-   [Systems](https://github.com/netdata/netdata/blob/master/docs/collect/system-metrics.md): Monitor CPU, memory, disk, networking, systemd, eBPF, and much more.
    Every metric exposed by `/proc`, `/sys`, and other Linux kernel sources.
-   [Containers](https://github.com/netdata/netdata/blob/master/docs/collect/container-metrics.md): Gather metrics from container agents, like `dockerd` or `kubectl`,
    along with the resource usage of containers and the applications they run.
-   [Applications](https://github.com/netdata/netdata/blob/master/docs/collect/application-metrics.md): Collect per-second metrics from web servers, databases, logs,
    message brokers, APM tools, email servers, and much more.

## Collector architecture and terminology

**Collector** is a catch-all term for any Netdata process that gathers metrics from an endpoint. 

While we use _collector_ most often in documentation, release notes, and educational content, you may encounter other
terms related to collecting metrics.

-   **Modules** are a type of collector.
-   **Orchestrators** are external plugins that run and manage one or more modules. They run as independent processes.
    The Go orchestrator is in active development.
    -   [go.d.plugin](https://github.com/netdata/go.d.plugin/blob/master/README.md): An orchestrator for data
        collection modules written in `go`.
    -   [python.d.plugin](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/README.md): An orchestrator for data collection modules written in
        `python` v2/v3.
    -   [charts.d.plugin](https://github.com/netdata/netdata/blob/master/collectors/charts.d.plugin/README.md): An orchestrator for data collection modules written in
        `bash` v4+.
-   **External plugins** gather metrics from external processes, such as a webserver or database, and run as independent
    processes that communicate with the Netdata daemon via pipes.
-   **Internal plugins** gather metrics from `/proc`, `/sys`, and other Linux kernel sources. They are written in `C`,
    and run as threads within the Netdata daemon.

## What's next?

[Enable or configure a collector](https://github.com/netdata/netdata/blob/master/docs/collect/enable-configure.md) if the default settings are not compatible with
your infrastructure.

See our [collectors reference](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md) for detailed information on Netdata's collector architecture,
troubleshooting a collector, developing a custom collector, and more.