1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
|
plugin_name: python.d.plugin
modules:
- meta:
plugin_name: python.d.plugin
module_name: changefinder
monitored_instance:
name: python.d changefinder
link: ""
categories:
- data-collection.other
icon_filename: ""
related_resources:
integrations:
list: []
info_provided_to_referring_integrations:
description: ""
keywords:
- change detection
- anomaly detection
- machine learning
- ml
most_popular: false
overview:
data_collection:
metrics_description: |
This collector uses the Python [changefinder](https://github.com/shunsukeaihara/changefinder) library to
perform [online](https://en.wikipedia.org/wiki/Online_machine_learning) [changepoint detection](https://en.wikipedia.org/wiki/Change_detection)
on your Netdata charts and/or dimensions.
method_description: >
Instead of this collector just _collecting_ data, it also does some computation on the data it collects to return a
changepoint score for each chart or dimension you configure it to work on. This is
an [online](https://en.wikipedia.org/wiki/Online_machine_learning) machine learning algorithm so there is no batch step
to train the model, instead it evolves over time as more data arrives. That makes this particular algorithm quite cheap
to compute at each step of data collection (see the notes section below for more details) and it should scale fairly
well to work on lots of charts or hosts (if running on a parent node for example).
### Notes
- It may take an hour or two (depending on your choice of `n_score_samples`) for the collector to 'settle' into it's
typical behaviour in terms of the trained models and scores you will see in the normal running of your node. Mainly
this is because it can take a while to build up a proper distribution of previous scores in over to convert the raw
score returned by the ChangeFinder algorithm into a percentile based on the most recent `n_score_samples` that have
already been produced. So when you first turn the collector on, it will have a lot of flags in the beginning and then
should 'settle down' once it has built up enough history. This is a typical characteristic of online machine learning
approaches which need some initial window of time before they can be useful.
- As this collector does most of the work in Python itself, you may want to try it out first on a test or development
system to get a sense of its performance characteristics on a node similar to where you would like to use it.
- On a development n1-standard-2 (2 vCPUs, 7.5 GB memory) vm running Ubuntu 18.04 LTS and not doing any work some of the
typical performance characteristics we saw from running this collector (with defaults) were:
- A runtime (`netdata.runtime_changefinder`) of ~30ms.
- Typically ~1% additional cpu usage.
- About ~85mb of ram (`apps.mem`) being continually used by the `python.d.plugin` under default configuration.
supported_platforms:
include: []
exclude: []
multi_instance: true
additional_permissions:
description: ""
default_behavior:
auto_detection:
description: "By default this collector will work over all `system.*` charts."
limits:
description: ""
performance_impact:
description: ""
setup:
prerequisites:
list:
- title: Python Requirements
description: |
This collector will only work with Python 3 and requires the packages below be installed.
```bash
# become netdata user
sudo su -s /bin/bash netdata
# install required packages for the netdata user
pip3 install --user numpy==1.19.5 changefinder==0.03 scipy==1.5.4
```
**Note**: if you need to tell Netdata to use Python 3 then you can pass the below command in the python plugin section
of your `netdata.conf` file.
```yaml
[ plugin:python.d ]
# update every = 1
command options = -ppython3
```
configuration:
file:
name: python.d/changefinder.conf
description: ""
options:
description: |
There are 2 sections:
* Global variables
* One or more JOBS that can define multiple different instances to monitor.
The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
folding:
title: "Config options"
enabled: true
list:
- name: charts_regex
description: what charts to pull data for - A regex like `system\..*|` or `system\..*|apps.cpu|apps.mem` etc.
default_value: "system\\..*"
required: true
- name: charts_to_exclude
description: |
charts to exclude, useful if you would like to exclude some specific charts.
note: should be a ',' separated string like 'chart.name,chart.name'.
default_value: ""
required: false
- name: mode
description: get ChangeFinder scores 'per_dim' or 'per_chart'.
default_value: "per_chart"
required: true
- name: cf_r
description: default parameters that can be passed to the changefinder library.
default_value: 0.5
required: false
- name: cf_order
description: default parameters that can be passed to the changefinder library.
default_value: 1
required: false
- name: cf_smooth
description: default parameters that can be passed to the changefinder library.
default_value: 15
required: false
- name: cf_threshold
description: the percentile above which scores will be flagged.
default_value: 99
required: false
- name: n_score_samples
description: the number of recent scores to use when calculating the percentile of the changefinder score.
default_value: 14400
required: false
- name: show_scores
description: |
set to true if you also want to chart the percentile scores in addition to the flags. (mainly useful for debugging or if you want to dive deeper on how the scores are evolving over time)
default_value: false
required: false
examples:
folding:
enabled: true
title: "Config"
list:
- name: Default
description: Default configuration.
folding:
enabled: false
config: |
local:
name: 'local'
host: '127.0.0.1:19999'
charts_regex: 'system\..*'
charts_to_exclude: ''
mode: 'per_chart'
cf_r: 0.5
cf_order: 1
cf_smooth: 15
cf_threshold: 99
n_score_samples: 14400
show_scores: false
troubleshooting:
problems:
list:
- name: "Debug Mode"
description: |
If you would like to log in as `netdata` user and run the collector in debug mode to see more detail.
```bash
# become netdata user
sudo su -s /bin/bash netdata
# run collector in debug using `nolock` option if netdata is already running the collector itself.
/usr/libexec/netdata/plugins.d/python.d.plugin changefinder debug trace nolock
```
- name: "Log Messages"
description: |
To see any relevant log messages you can use a command like below.
```bash
grep 'changefinder' /var/log/netdata/error.log
grep 'changefinder' /var/log/netdata/collector.log
```
alerts: []
metrics:
folding:
title: Metrics
enabled: false
description: ""
availability: []
scopes:
- name: global
description: ""
labels: []
metrics:
- name: changefinder.scores
description: ChangeFinder
unit: "score"
chart_type: line
dimensions:
- name: a dimension per chart
- name: changefinder.flags
description: ChangeFinder
unit: "flag"
chart_type: stacked
dimensions:
- name: a dimension per chart
|
