1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
<!--
title: "Use dimension templates to create dynamic alarms"
custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/guides/monitor/dimension-templates.md
-->
# Use dimension templates to create dynamic alarms
Your ability to monitor the health of your systems and applications relies on your ability to create and maintain
the best set of alarms for your particular needs.
In v1.18 of Netdata, we introduced **dimension templates** for alarms, which simplifies the process of
writing [alarm entities](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#health-entity-reference) for
charts with many dimensions.
Dimension templates can condense many individual entities into one—no more copy-pasting one entity and changing the
`alarm`/`template` and `lookup` lines for each dimension you'd like to monitor.
They are, however, an advanced health monitoring feature. For more basic instructions on creating your first alarm,
check out our [health monitoring documentation](https://github.com/netdata/netdata/blob/master/health/README.md), which also includes
[examples](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#example-alarms).
## The fundamentals of `foreach`
Our dimension templates update creates a new `foreach` parameter to the
existing [`lookup` line](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#alarm-line-lookup). This
is where the magic happens.
You use the `foreach` parameter to specify which dimensions you want to monitor with this single alarm. You can separate
them with a comma (`,`) or a pipe (`|`). You can also use
a [Netdata simple pattern](https://github.com/netdata/netdata/blob/master/libnetdata/simple_pattern/README.md) to create
many alarms with a regex-like syntax.
The `foreach` parameter _has_ to be the last parameter in your `lookup` line, and if you have both `of` and `foreach` in
the same `lookup` line, Netdata will ignore the `of` parameter and use `foreach` instead.
Let's get into some examples so you can see how the new parameter works.
> ⚠️ The following entities are examples to showcase the functionality and syntax of dimension templates. They are not
> meant to be run as-is on production systems.
## Condensing entities with `foreach`
Let's say you want to monitor the `system`, `user`, and `nice` dimensions in your system's overall CPU utilization.
Before dimension templates, you would need the following three entities:
```yaml
alarm: cpu_system
on: system.cpu
lookup: average -10m percentage of system
every: 1m
warn: $this > 50
crit: $this > 80
alarm: cpu_user
on: system.cpu
lookup: average -10m percentage of user
every: 1m
warn: $this > 50
crit: $this > 80
alarm: cpu_nice
on: system.cpu
lookup: average -10m percentage of nice
every: 1m
warn: $this > 50
crit: $this > 80
```
With dimension templates, you can condense these into a single alarm. Take note of the `alarm` and `lookup` lines.
```yaml
alarm: cpu_template
on: system.cpu
lookup: average -10m percentage foreach system,user,nice
every: 1m
warn: $this > 50
crit: $this > 80
```
The `alarm` line specifies the naming scheme Netdata will use. You can use whatever naming scheme you'd like, with `.`
and `_` being the only allowed symbols.
The `lookup` line has changed from `of` to `foreach`, and we're now passing three dimensions.
In this example, Netdata will create three alarms with the names `cpu_template_system`, `cpu_template_user`, and
`cpu_template_nice`. Every minute, each alarm will use the same database query to calculate the average CPU usage for
the `system`, `user`, and `nice` dimensions over the last 10 minutes and send out alarms if necessary.
You can find these three alarms active by clicking on the **Alarms** button in the top navigation, and then clicking on
the **All** tab and scrolling to the **system - cpu** collapsible section.
Let's look at some other examples of how `foreach` works so you can best apply it in your configurations.
### Using a Netdata simple pattern in `foreach`
In the last example, we used `foreach system,user,nice` to create three distinct alarms using dimension templates. But
what if you want to quickly create alarms for _all_ the dimensions of a given chart?
Use a [simple pattern](https://github.com/netdata/netdata/blob/master/libnetdata/simple_pattern/README.md)! One example of a simple pattern is a single wildcard
(`*`).
Instead of monitoring system CPU usage, let's monitor per-application CPU usage using the `apps.cpu` chart. Passing a
wildcard as the simple pattern tells Netdata to create a separate alarm for _every_ process on your system:
```yaml
alarm: app_cpu
on: apps.cpu
lookup: average -10m percentage foreach *
every: 1m
warn: $this > 50
crit: $this > 80
```
This entity will now create alarms for every dimension in the `apps.cpu` chart. Given that most `apps.cpu` charts have
10 or more dimensions, using the wildcard ensures you catch every CPU-hogging process.
To learn more about how to use simple patterns with dimension templates, see
our [simple patterns documentation](https://github.com/netdata/netdata/blob/master/libnetdata/simple_pattern/README.md).
## Using `foreach` with alarm templates
Dimension templates also work
with [alarm templates](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#alarm-line-alarm-or-template).
Alarm templates help you create alarms for all the charts with a given context—for example, all the cores of your
system's CPU.
By combining the two, you can create dozens of individual alarms with a single template entity. Here's how you would
create alarms for the `system`, `user`, and `nice` dimensions for every chart in the `cpu.cpu` context—or, in other
words, every CPU core.
```yaml
template: cpu_template
on: cpu.cpu
lookup: average -10m percentage foreach system,user,nice
every: 1m
warn: $this > 50
crit: $this > 80
```
On a system with a 6-core, 12-thread Ryzen 5 1600 CPU, this one entity creates alarms on the following charts and
dimensions:
- `cpu.cpu0`
- `cpu_template_user`
- `cpu_template_system`
- `cpu_template_nice`
- `cpu.cpu1`
- `cpu_template_user`
- `cpu_template_system`
- `cpu_template_nice`
- `cpu.cpu2`
- `cpu_template_user`
- `cpu_template_system`
- `cpu_template_nice`
- ...
- `cpu.cpu11`
- `cpu_template_user`
- `cpu_template_system`
- `cpu_template_nice`
And how just a few of those dimension template-generated alarms look like in the Netdata dashboard.
All in all, this single entity creates 36 individual alarms. Much easier than writing 36 separate entities in your
health configuration files!
## What's next?
We hope you're excited about the possibilities of using dimension templates! Maybe they'll inspire you to build new
alarms that will help you better monitor the health of your systems.
Or, at the very least, simplify your configuration files.
For information about other advanced features in Netdata's health monitoring toolkit, check out
our [health documentation](https://github.com/netdata/netdata/blob/master/health/README.md). And if you have some cool
alarms you built using dimension templates,
|
