summaryrefslogtreecommitdiffstats
path: root/collectors/REFERENCE.md
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2021-02-07 11:49:00 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2021-02-07 12:42:05 +0000
commit2e85f9325a797977eea9dfea0a925775ddd211d9 (patch)
tree452c7f30d62fca5755f659b99e4e53c7b03afc21 /collectors/REFERENCE.md
parentReleasing debian version 1.19.0-4. (diff)
downloadnetdata-2e85f9325a797977eea9dfea0a925775ddd211d9.tar.xz
netdata-2e85f9325a797977eea9dfea0a925775ddd211d9.zip
Merging upstream version 1.29.0.
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'collectors/REFERENCE.md')
-rw-r--r--collectors/REFERENCE.md186
1 files changed, 186 insertions, 0 deletions
diff --git a/collectors/REFERENCE.md b/collectors/REFERENCE.md
new file mode 100644
index 000000000..9c6f0a61e
--- /dev/null
+++ b/collectors/REFERENCE.md
@@ -0,0 +1,186 @@
+<!--
+title: "Collectors configuration reference"
+custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/REFERENCE.md
+-->
+
+# Collectors configuration reference
+
+Welcome to the collector configuration reference guide.
+
+This guide contains detailed information about enabling/disabling plugins or modules, in addition a quick reference to
+the internal plugins API.
+
+To learn the basics of collecting metrics from other applications and services, see the [collector
+quickstart](QUICKSTART.md).
+
+## Netdata's collector architecture
+
+Netdata has an intricate system for organizing and managing its collectors. **Collectors** are the processes/programs
+that actually gather metrics from various sources. Collectors are organized by **plugins**, which help manage all the
+independent processes in a variety of programming languages based on their purpose and performance requirements.
+**Modules** are a type of collector, used primarily to connect to external applications, such as an Nginx web server or
+MySQL database, among many others.
+
+For most users, enabling individual collectors for the application/service you're interested in is far more important
+than knowing which plugin it uses. See our [collectors list](/collectors/COLLECTORS.md) to see whether your favorite app/service has
+a collector, and then read the [collectors quickstart](/collectors/QUICKSTART.md) and the documentation for that specific collector
+to figure out how to enable it.
+
+There are three types of plugins:
+
+- **Internal** plugins organize collectors that gather metrics from `/proc`, `/sys` and other Linux kernel sources.
+ They are written in `C`, and run as threads within the Netdata daemon.
+- **External** plugins organize collectors that gather metrics from external processes, such as a MySQL database or
+ Nginx web server. They can be written in any language, and the `netdata` daemon spawns them as long-running
+ independent processes. They communicate with the daemon via pipes.
+- **Plugin orchestrators**, which are external plugins that instead support a number of **modules**. Modules are a
+ type of collector. We have a few plugin orchestrators available for those who want to develop their own collectors,
+ but focus most of our efforts on the [Go plugin](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/).
+
+## Enable, configure, and disable modules
+
+Most collector modules come with **auto-detection**, configured to work out-of-the-box on popular operating systems with
+the default settings.
+
+However, there are cases that auto-detection fails. Usually, the reason is that the applications to be monitored do not
+allow Netdata to connect. In most of the cases, allowing the user `netdata` from `localhost` to connect and collect
+metrics, will automatically enable data collection for the application in question (it will require a Netdata restart).
+
+View our [collectors quickstart](/collectors/QUICKSTART.md) for explicit details on enabling and configuring collector modules.
+
+## Troubleshoot a collector
+
+First, navigate to your plugins directory, which is usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case
+on your system, open `netdata.conf` and look for the setting `plugins directory`. Once you're in the plugins directory,
+switch to the `netdata` user.
+
+```bash
+cd /usr/libexec/netdata/plugins.d/
+sudo su -s /bin/bash netdata
+```
+
+The next step is based on the collector's orchestrator. You can figure out which orchestrator the collector uses by
+
+uses either
+by viewing the [collectors list](COLLECTORS.md) and referencing the _configuration file_ field. For example, if that
+field contains `go.d`, that collector uses the Go orchestrator.
+
+```bash
+# Go orchestrator (go.d.plugin)
+./go.d.plugin -d -m <MODULE_NAME>
+
+# Python orchestrator (python.d.plugin)
+./python.d.plugin <MODULE_NAME> debug trace
+
+# Node orchestrator (node.d.plugin)
+./node.d.plugin debug 1 <MODULE_NAME>
+
+# Bash orchestrator (bash.d.plugin)
+./charts.d.plugin debug 1 <MODULE_NAME>
+```
+
+The output from the relevant command will provide valuable troubleshooting information. If you can't figure out how to
+enable the collector using the details from this output, feel free to [create an issue on our
+GitHub](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage&template=bug_report.md) to get some
+help from our collectors experts.
+
+## Enable and disable plugins
+
+You can enable or disable individual plugins by opening `netdata.conf` and scrolling down to the `[plugins]` section.
+This section features a list of Netdata's plugins, with a boolean setting to enable or disable them. The exception is
+`statsd.plugin`, which has its own `[statsd]` section. Your `[plugins]` section should look similar to this:
+
+```conf
+[plugins]
+ # PATH environment variable = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/snapd/snap/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
+ # PYTHONPATH environment variable =
+ # proc = yes
+ # diskspace = yes
+ # cgroups = yes
+ # tc = yes
+ # idlejitter = yes
+ # enable running new plugins = yes
+ # check for new plugins every = 60
+ # slabinfo = no
+ # fping = yes
+ # ioping = yes
+ # node.d = yes
+ # python.d = yes
+ # go.d = yes
+ # apps = yes
+ # perf = yes
+ # charts.d = yes
+```
+
+By default, most plugins are enabled, so you don't need to enable them explicitly to use their collectors. To enable or
+disable any specific plugin, remove the comment (`#`) and change the boolean setting to `yes` or `no`.
+
+All **external plugins** are managed by [plugins.d](plugins.d/), which provides additional management options.
+
+## Internal plugins
+
+Each of the internal plugins runs as a thread inside the `netdata` daemon. Once this thread has started, the plugin may
+spawn additional threads according to its design.
+
+### Internal plugins API
+
+The internal data collection API consists of the following calls:
+
+```c
+collect_data() {
+ // collect data here (one iteration)
+
+ collected_number collected_value = collect_a_value();
+
+ // give the metrics to Netdata
+
+ static RRDSET *st = NULL; // the chart
+ static RRDDIM *rd = NULL; // a dimension attached to this chart
+
+ if(unlikely(!st)) {
+ // we haven't created this chart before
+ // create it now
+ st = rrdset_create_localhost(
+ "type"
+ , "id"
+ , "name"
+ , "family"
+ , "context"
+ , "Chart Title"
+ , "units"
+ , "plugin-name"
+ , "module-name"
+ , priority
+ , update_every
+ , chart_type
+ );
+
+ // attach a metric to it
+ rd = rrddim_add(st, "id", "name", multiplier, divider, algorithm);
+ }
+ else {
+ // this chart is already created
+ // let Netdata know we start a new iteration on it
+ rrdset_next(st);
+ }
+
+ // give the collected value(s) to the chart
+ rrddim_set_by_pointer(st, rd, collected_value);
+
+ // signal Netdata we are done with this iteration
+ rrdset_done(st);
+}
+```
+
+Of course, Netdata has a lot of libraries to help you also in collecting the metrics. The best way to find your way
+through this, is to examine what other similar plugins do.
+
+## External Plugins
+
+**External plugins** use the API and are managed by [plugins.d](plugins.d/).
+
+## Write a custom collector
+
+You can add custom collectors by following the [external plugins documentation](../collectors/plugins.d/).
+
+[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fcollectors%2REFERENCE&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)]()