diff options
Diffstat (limited to 'backends')
| -rw-r--r-- | backends/WALKTHROUGH.md | 12 | ||||
| -rw-r--r-- | backends/prometheus/README.md | 72 |
2 files changed, 42 insertions, 42 deletions
diff --git a/backends/WALKTHROUGH.md b/backends/WALKTHROUGH.md index 76dd62f8..12eea2ee 100644 --- a/backends/WALKTHROUGH.md +++ b/backends/WALKTHROUGH.md @@ -176,14 +176,14 @@ Prometheus’s homepage and begin to type ‘netdata\_’ Prometheus should auto  -Let’s now start exploring how we can graph some metrics. Back in our NetData container lets get the CPU spinning with a +Let’s now start exploring how we can graph some metrics. Back in our Netdata container lets get the CPU spinning with a pointless busy loop. On the shell do the following: ```sh [root@netdata /]# while true; do echo "HOT HOT HOT CPU"; done ``` -Our NetData cpu graph should be showing some activity. Let’s represent this in Prometheus. In order to do this let’s +Our Netdata cpu graph should be showing some activity. Let’s represent this in Prometheus. In order to do this let’s keep our metrics page open for reference: <http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes> We are setting out to graph the data in the CPU chart so let’s search for “system.cpu”in the metrics page above. We come across a section of metrics with the first comments `# COMMENT homogeneous chart "system.cpu", context "system.cpu", family @@ -209,18 +209,18 @@ query the dimension also. Place this into our query text box.  -Awesome, this is exactly what we wanted. If you haven’t caught on yet we can emulate entire charts from NetData by using +Awesome, this is exactly what we wanted. If you haven’t caught on yet we can emulate entire charts from Netdata by using the `chart` dimension. If you’d like you can combine the ‘chart’ and ‘instance’ dimension to create per-instance charts. Let’s give this a try: `netdata_system_cpu_percentage_average{chart="system.cpu", instance="netdata:19999"}` -This is the basics of using Prometheus to query NetData. I’d advise everyone at this point to read [this -page](../backends/prometheus/#using-netdata-with-prometheus). The key point here is that NetData can export metrics from +This is the basics of using Prometheus to query Netdata. I’d advise everyone at this point to read [this +page](../backends/prometheus/#using-netdata-with-prometheus). The key point here is that Netdata can export metrics from its internal DB or can send metrics “as-collected” by specifying the ‘source=as-collected’ url parameter like so. <http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes&types=yes&source=as-collected> If you choose to use this method you will need to use Prometheus's set of functions here: <https://prometheus.io/docs/querying/functions/> to obtain useful metrics as you are now dealing with raw counters from the system. For example you will have to use the `irate()` function over a counter to get that metric's rate per second. If your graphing needs are met by using the -metrics returned by NetData's internal database (not specifying any source= url parameter) then use that. If you find +metrics returned by Netdata's internal database (not specifying any source= url parameter) then use that. If you find limitations then consider re-writing your queries using the raw data and using Prometheus functions to get the desired chart. diff --git a/backends/prometheus/README.md b/backends/prometheus/README.md index 10275fa2..a0460d1d 100644 --- a/backends/prometheus/README.md +++ b/backends/prometheus/README.md @@ -110,7 +110,7 @@ scrape_configs: # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP format: [prometheus] # - # sources: as-collected | raw | average | sum | volume + # source: as-collected | raw | average | sum | volume # default is: average #source: [as-collected] # @@ -125,46 +125,46 @@ scrape_configs: #### Install nodes.yml -The following is completely optional, it will enable Prometheus to generate alerts from some NetData sources. Tweak the +The following is completely optional, it will enable Prometheus to generate alerts from some Netdata sources. Tweak the values to your own needs. We will use the following `nodes.yml` file below. Save it at `/opt/prometheus/nodes.yml`, and add a _- "nodes.yml"_ entry under the _rule_files:_ section in the example prometheus.yml file above. ```yaml groups: -- name: nodes - - rules: - - alert: node_high_cpu_usage_70 - expr: avg(rate(netdata_cpu_cpu_percentage_average{dimension="idle"}[1m])) by (job) > 70 - for: 1m - annotations: - description: '{{ $labels.job }} on ''{{ $labels.job }}'' CPU usage is at {{ humanize $value }}%.' - summary: CPU alert for container node '{{ $labels.job }}' - - - alert: node_high_memory_usage_70 - expr: 100 / sum(netdata_system_ram_MB_average) by (job) - * sum(netdata_system_ram_MB_average{dimension=~"free|cached"}) by (job) < 30 - for: 1m - annotations: - description: '{{ $labels.job }} memory usage is {{ humanize $value}}%.' - summary: Memory alert for container node '{{ $labels.job }}' - - - alert: node_low_root_filesystem_space_20 - expr: 100 / sum(netdata_disk_space_GB_average{family="/"}) by (job) - * sum(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}) by (job) < 20 - for: 1m - annotations: - description: '{{ $labels.job }} root filesystem space is {{ humanize $value}}%.' - summary: Root filesystem alert for container node '{{ $labels.job }}' - - - alert: node_root_filesystem_fill_rate_6h - expr: predict_linear(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}[1h], 6 * 3600) < 0 - for: 1h - labels: - severity: critical - annotations: - description: Container node {{ $labels.job }} root filesystem is going to fill up in 6h. - summary: Disk fill alert for Swarm node '{{ $labels.job }}' + - name: nodes + + rules: + - alert: node_high_cpu_usage_70 + expr: sum(sum_over_time(netdata_system_cpu_percentage_average{dimension=~"(user|system|softirq|irq|guest)"}[10m])) by (job) / sum(count_over_time(netdata_system_cpu_percentage_average{dimension="idle"}[10m])) by (job) > 70 + for: 1m + annotations: + description: '{{ $labels.job }} on ''{{ $labels.job }}'' CPU usage is at {{ humanize $value }}%.' + summary: CPU alert for container node '{{ $labels.job }}' + + - alert: node_high_memory_usage_70 + expr: 100 / sum(netdata_system_ram_MB_average) by (job) + * sum(netdata_system_ram_MB_average{dimension=~"free|cached"}) by (job) < 30 + for: 1m + annotations: + description: '{{ $labels.job }} memory usage is {{ humanize $value}}%.' + summary: Memory alert for container node '{{ $labels.job }}' + + - alert: node_low_root_filesystem_space_20 + expr: 100 / sum(netdata_disk_space_GB_average{family="/"}) by (job) + * sum(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}) by (job) < 20 + for: 1m + annotations: + description: '{{ $labels.job }} root filesystem space is {{ humanize $value}}%.' + summary: Root filesystem alert for container node '{{ $labels.job }}' + + - alert: node_root_filesystem_fill_rate_6h + expr: predict_linear(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}[1h], 6 * 3600) < 0 + for: 1h + labels: + severity: critical + annotations: + description: Container node {{ $labels.job }} root filesystem is going to fill up in 6h. + summary: Disk fill alert for Swarm node '{{ $labels.job }}' ``` #### Install prometheus.service |