# python.d zscores
Plugin: python.d.plugin
Module: zscores
## Overview
By using smoothed, rolling [Z-Scores](https://en.wikipedia.org/wiki/Standard_score) for selected metrics or charts you can narrow down your focus and shorten root cause analysis.
This collector uses the [Netdata rest api](https://github.com/netdata/netdata/blob/master/web/api/README.md) to get the `mean` and `stddev`
for each dimension on specified charts over a time range (defined by `train_secs` and `offset_secs`).
For each dimension it will calculate a Z-Score as `z = (x - mean) / stddev` (clipped at `z_clip`). Scores are then smoothed over
time (`z_smooth_n`) and, if `mode: 'per_chart'`, aggregated across dimensions to a smoothed, rolling chart level Z-Score at each time step.
This collector is supported on all platforms.
This collector supports collecting metrics from multiple instances of this integration, including remote instances.
### Default Behavior
#### Auto-Detection
This integration doesn't support auto-detection.
#### Limits
The default configuration for this integration does not impose any limits on data collection.
#### Performance Impact
The default configuration for this integration is not expected to impose a significant performance impact on the system.
## Metrics
Metrics grouped by *scope*.
The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.
### Per python.d zscores instance
These metrics refer to the entire monitored application.
This scope has no labels.
Metrics:
| Metric | Dimensions | Unit |
|:------|:----------|:----|
| zscores.z | a dimension per chart or dimension | z |
| zscores.3stddev | a dimension per chart or dimension | count |
## Alerts
There are no alerts configured by default for this integration.
## Setup
### Prerequisites
#### Python Requirements
This collector will only work with Python 3 and requires the below packages be installed.
```bash
# become netdata user
sudo su -s /bin/bash netdata
# install required packages
pip3 install numpy pandas requests netdata-pandas==0.0.38
```
### Configuration
#### File
The configuration file name for this integration is `python.d/zscores.conf`.
You can edit the configuration file using the `edit-config` script from the
Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory).
```bash
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config python.d/zscores.conf
```
#### Options
There are 2 sections:
* Global variables
* One or more JOBS that can define multiple different instances to monitor.
The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
Config options
| Name | Description | Default | Required |
|:----|:-----------|:-------|:--------:|
| charts_regex | what charts to pull data for - A regex like `system\..*/` or `system\..*/apps.cpu/apps.mem` etc. | system\..* | True |
| train_secs | length of time (in seconds) to base calculations off for mean and stddev. | 14400 | True |
| offset_secs | offset (in seconds) preceding latest data to ignore when calculating mean and stddev. | 300 | True |
| train_every_n | recalculate the mean and stddev every n steps of the collector. | 900 | True |
| z_smooth_n | smooth the z score (to reduce sensitivity to spikes) by averaging it over last n values. | 15 | True |
| z_clip | cap absolute value of zscore (before smoothing) for better stability. | 10 | True |
| z_abs | set z_abs: 'true' to make all zscores be absolute values only. | true | True |
| burn_in | burn in period in which to initially calculate mean and stddev on every step. | 2 | True |
| mode | mode can be to get a zscore 'per_dim' or 'per_chart'. | per_chart | True |
| per_chart_agg | per_chart_agg is how you aggregate from dimension to chart when mode='per_chart'. | mean | True |
| update_every | Sets the default data collection frequency. | 5 | False |
| priority | Controls the order of charts at the netdata dashboard. | 60000 | False |
| autodetection_retry | Sets the job re-check interval in seconds. | 0 | False |
| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | False |