diff options
Diffstat (limited to 'src/collectors/python.d.plugin')
78 files changed, 39 insertions, 14752 deletions
diff --git a/src/collectors/python.d.plugin/README.md b/src/collectors/python.d.plugin/README.md index 299cebc0..f8d4184e 100644 --- a/src/collectors/python.d.plugin/README.md +++ b/src/collectors/python.d.plugin/README.md @@ -1,22 +1,13 @@ -<!-- -title: "python.d.plugin" -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/README.md" -sidebar_label: "python.d.plugin" -learn_status: "Published" -learn_topic_type: "Tasks" -learn_rel_path: "Developers/External plugins/python.d.plugin" ---> - # python.d.plugin `python.d.plugin` is a Netdata external plugin. It is an **orchestrator** for data collection modules written in `python`. -1. It runs as an independent process `ps fax` shows it -2. It is started and stopped automatically by Netdata -3. It communicates with Netdata via a unidirectional pipe (sending data to the `netdata` daemon) -4. Supports any number of data collection **modules** -5. Allows each **module** to have one or more data collection **jobs** -6. Each **job** is collecting one or more metrics from a single data source +1. It runs as an independent process `ps fax` shows it +2. It is started and stopped automatically by Netdata +3. It communicates with Netdata via a unidirectional pipe (sending data to the `netdata` daemon) +4. Supports any number of data collection **modules** +5. Allows each **module** to have one or more data collection **jobs** +6. Each **job** is collecting one or more metrics from a single data source ## Disclaimer @@ -25,7 +16,7 @@ Module configurations are written in YAML and **pyYAML is required**. Every configuration file must have one of two formats: -- Configuration for only one job: +- Configuration for only one job: ```yaml update_every : 2 # update frequency @@ -35,7 +26,7 @@ other_var1 : bla # variables passed to module other_var2 : alb ``` -- Configuration for many jobs (ex. mysql): +- Configuration for many jobs (ex. mysql): ```yaml # module defaults: @@ -55,23 +46,19 @@ other_job: ## How to debug a python module -``` +```bash # become user netdata sudo su -s /bin/bash netdata ``` Depending on where Netdata was installed, execute one of the following commands to trace the execution of a python module: -``` +```bash # execute the plugin in debug mode, for a specific module /opt/netdata/usr/libexec/netdata/plugins.d/python.d.plugin <module> debug trace /usr/libexec/netdata/plugins.d/python.d.plugin <module> debug trace ``` -Where `[module]` is the directory name under <https://github.com/netdata/netdata/tree/master/src/collectors/python.d.plugin> +Where `[module]` is the directory name under <https://github.com/netdata/netdata/tree/master/src/collectors/python.d.plugin> **Note**: If you would like execute a collector in debug mode while it is still running by Netdata, you can pass the `nolock` CLI option to the above commands. - -## How to write a new module - -See [develop a custom collector in Python](https://github.com/netdata/netdata/edit/master/docs/developer-and-contributor-corner/python-collector.md). diff --git a/src/collectors/python.d.plugin/am2320/integrations/am2320.md b/src/collectors/python.d.plugin/am2320/integrations/am2320.md index ea0e505c..9aaa1153 100644 --- a/src/collectors/python.d.plugin/am2320/integrations/am2320.md +++ b/src/collectors/python.d.plugin/am2320/integrations/am2320.md @@ -106,8 +106,8 @@ Install the Adafruit Circuit Python AM2320 library: The configuration file name for this integration is `python.d/am2320.conf`. -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script from the +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). ```bash cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata diff --git a/src/collectors/python.d.plugin/anomalies/README.md b/src/collectors/python.d.plugin/anomalies/README.md deleted file mode 100644 index 1d7f8ba1..00000000 --- a/src/collectors/python.d.plugin/anomalies/README.md +++ /dev/null @@ -1,248 +0,0 @@ -<!-- -title: "Anomaly detection with Netdata" -description: "Use ML-driven anomaly detection to narrow your focus to only affected metrics and services/processes on your node to shorten root cause analysis." -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/anomalies/README.md" -sidebar_url: "Anomalies" -sidebar_label: "anomalies" -learn_status: "Published" -learn_rel_path: "Integrations/Monitor/Anything" ---> - -# Anomaly detection with Netdata - -**Note**: Check out the [Netdata Anomaly Advisor](/docs/dashboards-and-charts/anomaly-advisor-tab.md) for a more native anomaly detection experience within Netdata. - -This collector uses the Python [PyOD](https://pyod.readthedocs.io/en/latest/index.html) library to perform unsupervised [anomaly detection](https://en.wikipedia.org/wiki/Anomaly_detection) on your Netdata charts and/or dimensions. - -Instead of this collector just _collecting_ data, it also does some computation on the data it collects to return an anomaly probability and anomaly flag for each chart or custom model you define. This computation consists of a **train** function that runs every `train_n_secs` to train the ML models to learn what 'normal' typically looks like on your node. At each iteration there is also a **predict** function that uses the latest trained models and most recent metrics to produce an anomaly probability and anomaly flag for each chart or custom model you define. - -> As this is a somewhat unique collector and involves often subjective concepts like anomalies and anomaly probabilities, we would love to hear any feedback on it from the community. Please let us know on the [community forum](https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767) or drop us a note at [analytics-ml-team@netdata.cloud](mailto:analytics-ml-team@netdata.cloud) for any and all feedback, both positive and negative. This sort of feedback is priceless to help us make complex features more useful. - -## Charts - -Two charts are produced: - -- **Anomaly Probability** (`anomalies.probability`): This chart shows the probability that the latest observed data is anomalous based on the trained model for that chart (using the [`predict_proba()`](https://pyod.readthedocs.io/en/latest/api_cc.html#pyod.models.base.BaseDetector.predict_proba) method of the trained PyOD model). -- **Anomaly** (`anomalies.anomaly`): This chart shows `1` or `0` predictions of if the latest observed data is considered anomalous or not based on the trained model (using the [`predict()`](https://pyod.readthedocs.io/en/latest/api_cc.html#pyod.models.base.BaseDetector.predict) method of the trained PyOD model). - -Below is an example of the charts produced by this collector and how they might look when things are 'normal' on the node. The anomaly probabilities tend to bounce randomly around a typically low probability range, one or two might randomly jump or drift outside of this range every now and then and show up as anomalies on the anomaly chart. - - - -If we then go onto the system and run a command like `stress-ng --all 2` to create some [stress](https://wiki.ubuntu.com/Kernel/Reference/stress-ng), we see some charts begin to have anomaly probabilities that jump outside the typical range. When the anomaly probabilities change enough, we will start seeing anomalies being flagged on the `anomalies.anomaly` chart. The idea is that these charts are the most anomalous right now so could be a good place to start your troubleshooting. - - - -Then, as the issue passes, the anomaly probabilities should settle back down into their 'normal' range again. - - - -## Requirements - -- This collector will only work with Python 3 and requires the packages below be installed. -- Typically you will not need to do this, but, if needed, to ensure Python 3 is used you can add the below line to the `[plugin:python.d]` section of `netdata.conf` - -```conf -[plugin:python.d] - # update every = 1 - command options = -ppython3 -``` - -Install the required python libraries. - -```bash -# become netdata user -sudo su -s /bin/bash netdata -# install required packages for the netdata user -pip3 install --user netdata-pandas==0.0.38 numba==0.50.1 scikit-learn==0.23.2 pyod==0.8.3 -``` - -## Configuration - -Install the Python requirements above, enable the collector and restart Netdata. - -```bash -cd /etc/netdata/ -sudo ./edit-config python.d.conf -# Set `anomalies: no` to `anomalies: yes` -sudo systemctl restart netdata -``` - -The configuration for the anomalies collector defines how it will behave on your system and might take some experimentation with over time to set it optimally for your node. Out of the box, the config comes with some [sane defaults](https://www.netdata.cloud/blog/redefining-monitoring-with-netdata/) to get you started that try to balance the flexibility and power of the ML models with the goal of being as cheap as possible in term of cost on the node resources. - -_**Note**: If you are unsure about any of the below configuration options then it's best to just ignore all this and leave the `anomalies.conf` file alone to begin with. Then you can return to it later if you would like to tune things a bit more once the collector is running for a while and you have a feeling for its performance on your node._ - -Edit the `python.d/anomalies.conf` configuration file using `edit-config` from the your agent's [config -directory](/docs/netdata-agent/configuration/README.md), which is usually at `/etc/netdata`. - -```bash -cd /etc/netdata # Replace this path with your Netdata config directory, if different -sudo ./edit-config python.d/anomalies.conf -``` - -The default configuration should look something like this. Here you can see each parameter (with sane defaults) and some information about each one and what it does. - -```conf -# - -# JOBS (data collection sources) - -# Pull data from local Netdata node. -anomalies: - name: 'Anomalies' - - # Host to pull data from. - host: '127.0.0.1:19999' - - # Username and Password for Netdata if using basic auth. - # username: '???' - # password: '???' - - # Use http or https to pull data - protocol: 'http' - - # SSL verify parameter for requests.get() calls - tls_verify: true - - # What charts to pull data for - A regex like 'system\..*|' or 'system\..*|apps.cpu|apps.mem' etc. - charts_regex: 'system\..*' - - # Charts to exclude, useful if you would like to exclude some specific charts. - # Note: should be a ',' separated string like 'chart.name,chart.name'. - charts_to_exclude: 'system.uptime,system.entropy' - - # What model to use - can be one of 'pca', 'hbos', 'iforest', 'cblof', 'loda', 'copod' or 'feature_bagging'. - # More details here: https://pyod.readthedocs.io/en/latest/pyod.models.html. - model: 'pca' - - # Max number of observations to train on, to help cap compute cost of training model if you set a very large train_n_secs. - train_max_n: 100000 - - # How often to re-train the model (assuming update_every=1 then train_every_n=1800 represents (re)training every 30 minutes). - # Note: If you want to turn off re-training set train_every_n=0 and after initial training the models will not be retrained. - train_every_n: 1800 - - # The length of the window of data to train on (14400 = last 4 hours). - train_n_secs: 14400 - - # How many prediction steps after a train event to just use previous prediction value for. - # Used to reduce possibility of the training step itself appearing as an anomaly on the charts. - train_no_prediction_n: 10 - - # If you would like to train the model for the first time on a specific window then you can define it using the below two variables. - # Start of training data for initial model. - # initial_train_data_after: 1604578857 - - # End of training data for initial model. - # initial_train_data_before: 1604593257 - - # If you would like to ignore recent data in training then you can offset it by offset_n_secs. - offset_n_secs: 0 - - # How many lagged values of each dimension to include in the 'feature vector' each model is trained on. - lags_n: 5 - - # How much smoothing to apply to each dimension in the 'feature vector' each model is trained on. - smooth_n: 3 - - # How many differences to take in preprocessing your data. - # More info on differencing here: https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average#Differencing - # diffs_n=0 would mean training models on the raw values of each dimension. - # diffs_n=1 means everything is done in terms of differences. - diffs_n: 1 - - # What is the typical proportion of anomalies in your data on average? - # This parameter can control the sensitivity of your models to anomalies. - # Some discussion here: https://github.com/yzhao062/pyod/issues/144 - contamination: 0.001 - - # Set to true to include an "average_prob" dimension on anomalies probability chart which is - # just the average of all anomaly probabilities at each time step - include_average_prob: true - - # Define any custom models you would like to create anomaly probabilities for, some examples below to show how. - # For example below example creates two custom models, one to run anomaly detection user and system cpu for our demo servers - # and one on the cpu and mem apps metrics for the python.d.plugin. - # custom_models: - # - name: 'demos_cpu' - # dimensions: 'london.my-netdata.io::system.cpu|user,london.my-netdata.io::system.cpu|system,newyork.my-netdata.io::system.cpu|user,newyork.my-netdata.io::system.cpu|system' - # - name: 'apps_python_d_plugin' - # dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin' - - # Set to true to normalize, using min-max standardization, features used for the custom models. - # Useful if your custom models contain dimensions on very different scales an model you use does - # not internally do its own normalization. Usually best to leave as false. - # custom_models_normalize: false -``` - -## Custom models - -In the `anomalies.conf` file you can also define some "custom models" which you can use to group one or more metrics into a single model much like is done by default for the charts you specify. This is useful if you have a handful of metrics that exist in different charts but perhaps are related to the same underlying thing you would like to perform anomaly detection on, for example a specific app or user. - -To define a custom model you would include configuration like below in `anomalies.conf`. By default there should already be some commented out examples in there. - -`name` is a name you give your custom model, this is what will appear alongside any other specified charts in the `anomalies.probability` and `anomalies.anomaly` charts. `dimensions` is a string of metrics you want to include in your custom model. By default the [netdata-pandas](https://github.com/netdata/netdata-pandas) library used to pull the data from Netdata uses a "chart.a|dim.1" type of naming convention in the pandas columns it returns, hence the `dimensions` string should look like "chart.name|dimension.name,chart.name|dimension.name". The examples below hopefully make this clear. - -```yaml -custom_models: - # a model for anomaly detection on the netdata user in terms of cpu, mem, threads, processes and sockets. - - name: 'user_netdata' - dimensions: 'users.cpu|netdata,users.mem|netdata,users.threads|netdata,users.processes|netdata,users.sockets|netdata' - # a model for anomaly detection on the netdata python.d.plugin app in terms of cpu, mem, threads, processes and sockets. - - name: 'apps_python_d_plugin' - dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin,apps.threads|python.d.plugin,apps.processes|python.d.plugin,apps.sockets|python.d.plugin' - -custom_models_normalize: false -``` - -## Troubleshooting - -To see any relevant log messages you can use a command like below. - -```bash -`grep 'anomalies' /var/log/netdata/error.log` -``` - -If you would like to log in as `netdata` user and run the collector in debug mode to see more detail. - -```bash -# become netdata user -sudo su -s /bin/bash netdata -# run collector in debug using `nolock` option if netdata is already running the collector itself. -/usr/libexec/netdata/plugins.d/python.d.plugin anomalies debug trace nolock -``` - -## Deepdive tutorial - -If you would like to go deeper on what exactly the anomalies collector is doing under the hood then check out this [deepdive tutorial](https://github.com/netdata/community/blob/main/netdata-agent-api/netdata-pandas/anomalies_collector_deepdive.ipynb) in our community repo where you can play around with some data from our demo servers (or your own if its accessible to you) and work through the calculations step by step. - -(Note: as its a Jupyter Notebook it might render a little prettier on [nbviewer](https://nbviewer.jupyter.org/github/netdata/community/blob/main/netdata-agent-api/netdata-pandas/anomalies_collector_deepdive.ipynb)) - -## Notes - -- Python 3 is required as the [`netdata-pandas`](https://github.com/netdata/netdata-pandas) package uses Python async libraries ([asks](https://pypi.org/project/asks/) and [trio](https://pypi.org/project/trio/)) to make asynchronous calls to the [Netdata REST API](/src/web/api/README.md) to get the required data for each chart. -- Python 3 is also required for the underlying ML libraries of [numba](https://pypi.org/project/numba/), [scikit-learn](https://pypi.org/project/scikit-learn/), and [PyOD](https://pypi.org/project/pyod/). -- It may take a few hours or so (depending on your choice of `train_secs_n`) for the collector to 'settle' into it's typical behaviour in terms of the trained models and probabilities you will see in the normal running of your node. -- As this collector does most of the work in Python itself, with [PyOD](https://pyod.readthedocs.io/en/latest/) leveraging [numba](https://numba.pydata.org/) under the hood, you may want to try it out first on a test or development system to get a sense of its performance characteristics on a node similar to where you would like to use it. -- `lags_n`, `smooth_n`, and `diffs_n` together define the preprocessing done to the raw data before models are trained and before each prediction. This essentially creates a [feature vector](https://en.wikipedia.org/wiki/Feature_(machine_learning)#:~:text=In%20pattern%20recognition%20and%20machine,features%20that%20represent%20some%20object.&text=Feature%20vectors%20are%20often%20combined,score%20for%20making%20a%20prediction.) for each chart model (or each custom model). The default settings for these parameters aim to create a rolling matrix of recent smoothed [differenced](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average#Differencing) values for each chart. The aim of the model then is to score how unusual this 'matrix' of features is for each chart based on what it has learned as 'normal' from the training data. So as opposed to just looking at the single most recent value of a dimension and considering how strange it is, this approach looks at a recent smoothed window of all dimensions for a chart (or dimensions in a custom model) and asks how unusual the data as a whole looks. This should be more flexible in capturing a wider range of [anomaly types](https://andrewm4894.com/2020/10/19/different-types-of-time-series-anomalies/) and be somewhat more robust to temporary 'spikes' in the data that tend to always be happening somewhere in your metrics but often are not the most important type of anomaly (this is all covered in a lot more detail in the [deepdive tutorial](https://nbviewer.jupyter.org/github/netdata/community/blob/main/netdata-agent-api/netdata-pandas/anomalies_collector_deepdive.ipynb)). -- You can see how long model training is taking by looking in the logs for the collector `grep 'anomalies' /var/log/netdata/error.log | grep 'training'` and you should see lines like `2020-12-01 22:02:14: python.d INFO: anomalies[local] : training complete in 2.81 seconds (runs_counter=2700, model=pca, train_n_secs=14400, models=26, n_fit_success=26, n_fit_fails=0, after=1606845731, before=1606860131).`. - - This also gives counts of the number of models, if any, that failed to fit and so had to default back to the DefaultModel (which is currently [HBOS](https://pyod.readthedocs.io/en/latest/_modules/pyod/models/hbos.html)). - - `after` and `before` here refer to the start and end of the training data used to train the models. -- On a development n1-standard-2 (2 vCPUs, 7.5 GB memory) vm running Ubuntu 18.04 LTS and not doing any work some of the typical performance characteristics we saw from running this collector (with defaults) were: - - A runtime (`netdata.runtime_anomalies`) of ~80ms when doing scoring and ~3 seconds when training or retraining the models. - - Typically ~3%-3.5% additional cpu usage from scoring, jumping to ~60% for a couple of seconds during model training. - - About ~150mb of ram (`apps.mem`) being continually used by the `python.d.plugin`. -- If you activate this collector on a fresh node, it might take a little while to build up enough data to calculate a realistic and useful model. -- Some models like `iforest` can be comparatively expensive (on same n1-standard-2 system above ~2s runtime during predict, ~40s training time, ~50% cpu on both train and predict) so if you would like to use it you might be advised to set a relatively high `update_every` maybe 10, 15 or 30 in `anomalies.conf`. -- Setting a higher `train_every_n` and `update_every` is an easy way to devote less resources on the node to anomaly detection. Specifying less charts and a lower `train_n_secs` will also help reduce resources at the expense of covering less charts and maybe a more noisy model if you set `train_n_secs` to be too small for how your node tends to behave. -- If you would like to enable this on a Raspberry Pi, then check out [this guide](/docs/developer-and-contributor-corner/raspberry-pi-anomaly-detection.md) which will guide you through first installing LLVM. - -## Useful links and further reading - -- [PyOD documentation](https://pyod.readthedocs.io/en/latest/), [PyOD Github](https://github.com/yzhao062/pyod). -- [Anomaly Detection](https://en.wikipedia.org/wiki/Anomaly_detection) wikipedia page. -- [Anomaly Detection YouTube playlist](https://www.youtube.com/playlist?list=PL6Zhl9mK2r0KxA6rB87oi4kWzoqGd5vp0) maintained by [andrewm4894](https://github.com/andrewm4894/) from Netdata. -- [awesome-TS-anomaly-detection](https://github.com/rob-med/awesome-TS-anomaly-detection) Github list of useful tools, libraries and resources. -- [Mendeley public group](https://www.mendeley.com/community/interesting-anomaly-detection-papers/) with some interesting anomaly detection papers we have been reading. -- Good [blog post](https://www.anodot.com/blog/what-is-anomaly-detection/) from Anodot on time series anomaly detection. Anodot also have some great whitepapers in this space too that some may find useful. -- Novelty and outlier detection in the [scikit-learn documentation](https://scikit-learn.org/stable/modules/outlier_detection.html). - diff --git a/src/collectors/python.d.plugin/anomalies/anomalies.chart.py b/src/collectors/python.d.plugin/anomalies/anomalies.chart.py deleted file mode 100644 index 24e84cc1..00000000 --- a/src/collectors/python.d.plugin/anomalies/anomalies.chart.py +++ /dev/null @@ -1,425 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: anomalies netdata python.d module -# Author: andrewm4894 -# SPDX-License-Identifier: GPL-3.0-or-later - -import sys -import time -from datetime import datetime -import re -import warnings - -import requests -import numpy as np -import pandas as pd -from netdata_pandas.data import get_data, get_allmetrics_async -from pyod.models.hbos import HBOS -from pyod.models.pca import PCA -from pyod.models.loda import LODA -from pyod.models.iforest import IForest -from pyod.models.cblof import CBLOF -from pyod.models.feature_bagging import FeatureBagging -from pyod.models.copod import COPOD -from sklearn.preprocessing import MinMaxScaler - -from bases.FrameworkServices.SimpleService import SimpleService - -# ignore some sklearn/numpy warnings that are ok -warnings.filterwarnings('ignore', r'All-NaN slice encountered') -warnings.filterwarnings('ignore', r'invalid value encountered in true_divide') -warnings.filterwarnings('ignore', r'divide by zero encountered in true_divide') -warnings.filterwarnings('ignore', r'invalid value encountered in subtract') - -disabled_by_default = True - -ORDER = ['probability', 'anomaly'] - -CHARTS = { - 'probability': { - 'options': ['probability', 'Anomaly Probability', 'probability', 'anomalies', 'anomalies.probability', 'line'], - 'lines': [] - }, - 'anomaly': { - 'options': ['anomaly', 'Anomaly', 'count', 'anomalies', 'anomalies.anomaly', 'stacked'], - 'lines': [] - }, -} - - -class Service(SimpleService): - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.basic_init() - self.charts_init() - self.custom_models_init() - self.data_init() - self.model_params_init() - self.models_init() - self.collected_dims = {'probability': set(), 'anomaly': set()} - - def check(self): - if not (sys.version_info[0] >= 3 and sys.version_info[1] >= 6): - self.error("anomalies collector only works with Python>=3.6") - if len(self.host_charts_dict[self.host]) > 0: - _ = get_allmetrics_async(host_charts_dict=self.host_charts_dict, protocol=self.protocol, user=self.username, pwd=self.password) - return True - - def basic_init(self): - """Perform some basic initialization. - """ - self.order = ORDER - self.definitions = CHARTS - self.protocol = self.configuration.get('protocol', 'http') - self.host = self.configuration.get('host', '127.0.0.1:19999') - self.username = self.configuration.get('username', None) - self.password = self.configuration.get('password', None) - self.tls_verify = self.configuration.get('tls_verify', True) - self.fitted_at = {} - self.df_allmetrics = pd.DataFrame() - self.last_train_at = 0 - self.include_average_prob = bool(self.configuration.get('include_average_prob', True)) - self.reinitialize_at_every_step = bool(self.configuration.get('reinitialize_at_every_step', False)) - - def charts_init(self): - """Do some initialisation of charts in scope related variables. - """ - self.charts_regex = re.compile(self.configuration.get('charts_regex','None')) - self.charts_available = [c for c in list(requests.get(f'{self.protocol}://{self.host}/api/v1/charts', verify=self.tls_verify).json().get('charts', {}).keys())] - self.charts_in_scope = list(filter(self.charts_regex.match, self.charts_available)) - self.charts_to_exclude = self.configuration.get('charts_to_exclude', '').split(',') - if len(self.charts_to_exclude) > 0: - self.charts_in_scope = [c for c in self.charts_in_scope if c not in self.charts_to_exclude] - - def custom_models_init(self): - """Perform initialization steps related to custom models. - """ - self.custom_models = self.configuration.get('custom_models', None) - self.custom_models_normalize = bool(self.configuration.get('custom_models_normalize', False)) - if self.custom_models: - self.custom_models_names = [model['name'] for model in self.custom_models] - self.custom_models_dims = [i for s in [model['dimensions'].split(',') for model in self.custom_models] for i in s] - self.custom_models_dims = [dim if '::' in dim else f'{self.host}::{dim}' for dim in self.custom_models_dims] - self.custom_models_charts = list(set([dim.split('|')[0].split('::')[1] for dim in self.custom_models_dims])) - self.custom_models_hosts = list(set([dim.split('::')[0] for dim in self.custom_models_dims])) - self.custom_models_host_charts_dict = {} - for host in self.custom_models_hosts: - self.custom_models_host_charts_dict[host] = list(set([dim.split('::')[1].split('|')[0] for dim in self.custom_models_dims if dim.startswith(host)])) - self.custom_models_dims_renamed = [f"{model['name']}|{dim}" for model in self.custom_models for dim in model['dimensions'].split(',')] - self.models_in_scope = list(set([f'{self.host}::{c}' for c in self.charts_in_scope] + self.custom_models_names)) - self.charts_in_scope = list(set(self.charts_in_scope + self.custom_models_charts)) - self.host_charts_dict = {self.host: self.charts_in_scope} - for host in self.custom_models_host_charts_dict: - if host not in self.host_charts_dict: - self.host_charts_dict[host] = self.custom_models_host_charts_dict[host] - else: - for chart in self.custom_models_host_charts_dict[host]: - if chart not in self.host_charts_dict[host]: - self.host_charts_dict[host].extend(chart) - else: - self.models_in_scope = [f'{self.host}::{c}' for c in self.charts_in_scope] - self.host_charts_dict = {self.host: self.charts_in_scope} - self.model_display_names = {model: model.split('::')[1] if '::' in model else model for model in self.models_in_scope} - #self.info(f'self.host_charts_dict (len={len(self.host_charts_dict[self.host])}): {self.host_charts_dict}') - - def data_init(self): - """Initialize some empty data objects. - """ - self.data_probability_latest = {f'{m}_prob': 0 for m in self.charts_in_scope} - self.data_anomaly_latest = {f'{m}_anomaly': 0 for m in self.charts_in_scope} - self.data_latest = {**self.data_probability_latest, **self.data_anomaly_latest} - - def model_params_init(self): - """Model parameters initialisation. - """ - self.train_max_n = self.configuration.get('train_max_n', 100000) - self.train_n_secs = self.configuration.get('train_n_secs', 14400) - self.offset_n_secs = self.configuration.get('offset_n_secs', 0) - self.train_every_n = self.configuration.get('train_every_n', 1800) - self.train_no_prediction_n = self.configuration.get('train_no_prediction_n', 10) - self.initial_train_data_after = self.configuration.get('initial_train_data_after', 0) - self.initial_train_data_before = self.configuration.get('initial_train_data_before', 0) - self.contamination = self.configuration.get('contamination', 0.001) - self.lags_n = {model: self.configuration.get('lags_n', 5) for model in self.models_in_scope} - self.smooth_n = {model: self.configuration.get('smooth_n', 5) for model in self.models_in_scope} - self.diffs_n = {model: self.configuration.get('diffs_n', 5) for model in self.models_in_scope} - - def models_init(self): - """Models initialisation. - """ - self.model = self.configuration.get('model', 'pca') - if self.model == 'pca': - self.models = {model: PCA(contamination=self.contamination) for model in self.models_in_scope} - elif self.model == 'loda': - self.models = {model: LODA(contamination=self.contamination) for model in self.models_in_scope} - elif self.model == 'iforest': - self.models = {model: IForest(n_estimators=50, bootstrap=True, behaviour='new', contamination=self.contamination) for model in self.models_in_scope} - elif self.model == 'cblof': - self.models = {model: CBLOF(n_clusters=3, contamination=self.contamination) for model in self.models_in_scope} - elif self.model == 'feature_bagging': - self.models = {model: FeatureBagging(base_estimator=PCA(contamination=self.contamination), contamination=self.contamination) for model in self.models_in_scope} - elif self.model == 'copod': - self.models = {model: COPOD(contamination=self.contamination) for model in self.models_in_scope} - elif self.model == 'hbos': - self.models = {model: HBOS(contamination=self.contamination) for model in self.models_in_scope} - else: - self.models = {model: HBOS(contamination=self.contamination) for model in self.models_in_scope} - self.custom_model_scalers = {model: MinMaxScaler() for model in self.models_in_scope} - - def model_init(self, model): - """Model initialisation of a single model. - """ - if self.model == 'pca': - self.models[model] = PCA(contamination=self.contamination) - elif self.model == 'loda': - self.models[model] = LODA(contamination=self.contamination) - elif self.model == 'iforest': - self.models[model] = IForest(n_estimators=50, bootstrap=True, behaviour='new', contamination=self.contamination) - elif self.model == 'cblof': - self.models[model] = CBLOF(n_clusters=3, contamination=self.contamination) - elif self.model == 'feature_bagging': - self.models[model] = FeatureBagging(base_estimator=PCA(contamination=self.contamination), contamination=self.contamination) - elif self.model == 'copod': - self.models[model] = COPOD(contamination=self.contamination) - elif self.model == 'hbos': - self.models[model] = HBOS(contamination=self.contamination) - else: - self.models[model] = HBOS(contamination=self.contamination) - self.custom_model_scalers[model] = MinMaxScaler() - - def reinitialize(self): - """Reinitialize charts, models and data to a beginning state. - """ - self.charts_init() - self.custom_models_init() - self.data_init() - self.model_params_init() - self.models_init() - - def save_data_latest(self, data, data_probability, data_anomaly): - """Save the most recent data objects to be used if needed in the future. - """ - self.data_latest = data - self.data_probability_latest = data_probability - self.data_anomaly_latest = data_anomaly - - def validate_charts(self, chart, data, algorithm='absolute', multiplier=1, divisor=1): - """If dimension not in chart then add it. - """ - for dim in data: - if dim not in self.collected_dims[chart]: - self.collected_dims[chart].add(dim) - self.charts[chart].add_dimension([dim, dim, algorithm, multiplier, divisor]) - - for dim in list(self.collected_dims[chart]): - if dim not in data: - self.collected_dims[chart].remove(dim) - self.charts[chart].del_dimension(dim, hide=False) - - def add_custom_models_dims(self, df): - """Given a df, select columns used by custom models, add custom model name as prefix, and append to df. - - :param df <pd.DataFrame>: dataframe to append new renamed columns to. - :return: <pd.DataFrame> dataframe with additional columns added relating to the specified custom models. - """ - df_custom = df[self.custom_models_dims].copy() - df_custom.columns = self.custom_models_dims_renamed - df = df.join(df_custom) - - return df - - def make_features(self, arr, train=False, model=None): - """Take in numpy array and preprocess accordingly by taking diffs, smoothing and adding lags. - - :param arr <np.ndarray>: numpy array we want to make features from. - :param train <bool>: True if making features for training, in which case need to fit_transform scaler and maybe sample train_max_n. - :param model <str>: model to make features for. - :return: <np.ndarray> transformed numpy array. - """ - - def lag(arr, n): - res = np.empty_like(arr) - res[:n] = np.nan - res[n:] = arr[:-n] - - return res - - arr = np.nan_to_num(arr) - - diffs_n = self.diffs_n[model] - smooth_n = self.smooth_n[model] - lags_n = self.lags_n[model] - - if self.custom_models_normalize and model in self.custom_models_names: - if train: - arr = self.custom_model_scalers[model].fit_transform(arr) - else: - arr = self.custom_model_scalers[model].transform(arr) - - if diffs_n > 0: - arr = np.diff(arr, diffs_n, axis=0) - arr = arr[~np.isnan(arr).any(axis=1)] - - if smooth_n > 1: - arr = np.cumsum(arr, axis=0, dtype=float) - arr[smooth_n:] = arr[smooth_n:] - arr[:-smooth_n] - arr = arr[smooth_n - 1:] / smooth_n - arr = arr[~np.isnan(arr).any(axis=1)] - - if lags_n > 0: - arr_orig = np.copy(arr) - for lag_n in range(1, lags_n + 1): - arr = np.concatenate((arr, lag(arr_orig, lag_n)), axis=1) - arr = arr[~np.isnan(arr).any(axis=1)] - - if train: - if len(arr) > self.train_max_n: - arr = arr[np.random.randint(arr.shape[0], size=self.train_max_n), :] - - arr = np.nan_to_num(arr) - - return arr - - def train(self, models_to_train=None, train_data_after=0, train_data_before=0): - """Pull required training data and train a model for each specified model. - - :param models_to_train <list>: list of models to train on. - :param train_data_after <int>: integer timestamp for start of train data. - :param train_data_before <int>: integer timestamp for end of train data. - """ - now = datetime.now().timestamp() - if train_data_after > 0 and train_data_before > 0: - before = train_data_before - after = train_data_after - else: - before = int(now) - self.offset_n_secs - after = before - self.train_n_secs - - # get training data - df_train = get_data( - host_charts_dict=self.host_charts_dict, host_prefix=True, host_sep='::', after=after, before=before, - sort_cols=True, numeric_only=True, protocol=self.protocol, float_size='float32', user=self.username, pwd=self.password, - verify=self.tls_verify - ).ffill() - if self.custom_models: - df_train = self.add_custom_models_dims(df_train) - - # train model - self.try_fit(df_train, models_to_train=models_to_train) - self.info(f'training complete in {round(time.time() - now, 2)} seconds (runs_counter={self.runs_counter}, model={self.model}, train_n_secs={self.train_n_secs}, models={len(self.fitted_at)}, n_fit_success={self.n_fit_success}, n_fit_fails={self.n_fit_fail}, after={after}, before={before}).') - self.last_train_at = self.runs_counter - - def try_fit(self, df_train, models_to_train=None): - """Try fit each model and try to fallback to a default model if fit fails for any reason. - - :param df_train <pd.DataFrame>: data to train on. - :param models_to_train <list>: list of models to train. - """ - if models_to_train is None: - models_to_train = list(self.models.keys()) - self.n_fit_fail, self.n_fit_success = 0, 0 - for model in models_to_train: - if model not in self.models: - self.model_init(model) - X_train = self.make_features( - df_train[df_train.columns[df_train.columns.str.startswith(f'{model}|')]].values, - train=True, model=model) - try: - self.models[model].fit(X_train) - self.n_fit_success += 1 - except Exception as e: - self.n_fit_fail += 1 - self.info(e) - self.info(f'training failed for {model} at run_counter {self.runs_counter}, defaulting to hbos model.') - self.models[model] = HBOS(contamination=self.contamination) - self.models[model].fit(X_train) - self.fitted_at[model] = self.runs_counter - - def predict(self): - """Get latest data, make it into a feature vector, and get predictions for each available model. - - :return: (<dict>,<dict>) tuple of dictionaries, one for probability scores and the other for anomaly predictions. - """ - # get recent data to predict on - df_allmetrics = get_allmetrics_async( - host_charts_dict=self.host_charts_dict, host_prefix=True, host_sep='::', wide=True, sort_cols=True, - protocol=self.protocol, numeric_only=True, float_size='float32', user=self.username, pwd=self.password - ) - if self.custom_models: - df_allmetrics = self.add_custom_models_dims(df_allmetrics) - self.df_allmetrics = self.df_allmetrics.append(df_allmetrics).ffill().tail((max(self.lags_n.values()) + max(self.smooth_n.values()) + max(self.diffs_n.values())) * 2) - - # get predictions - data_probability, data_anomaly = self.try_predict() - - return data_probability, data_anomaly - - def try_predict(self): - """Try make prediction and fall back to last known prediction if fails. - - :return: (<dict>,<dict>) tuple of dictionaries, one for probability scores and the other for anomaly predictions. - """ - data_probability, data_anomaly = {}, {} - for model in self.fitted_at.keys(): - model_display_name = self.model_display_names[model] - try: - X_model = np.nan_to_num( - self.make_features( - self.df_allmetrics[self.df_allmetrics.columns[self.df_allmetrics.columns.str.startswith(f'{model}|')]].values, - model=model - )[-1,:].reshape(1, -1) - ) - data_probability[model_display_name + '_prob'] = np.nan_to_num(self.models[model].predict_proba(X_model)[-1][1]) * 10000 - data_anomaly[model_display_name + '_anomaly'] = self.models[model].predict(X_model)[-1] - except Exception as _: - #self.info(e) - if model_display_name + '_prob' in self.data_latest: - #self.info(f'prediction failed for {model} at run_counter {self.runs_counter}, using last prediction instead.') - data_probability[model_display_name + '_prob'] = self.data_latest[model_display_name + '_prob'] - data_anomaly[model_display_name + '_anomaly'] = self.data_latest[model_display_name + '_anomaly'] - else: - #self.info(f'prediction failed for {model} at run_counter {self.runs_counter}, skipping as no previous prediction.') - continue - - return data_probability, data_anomaly - - def get_data(self): - - # initialize to what's available right now - if self.reinitialize_at_every_step or len(self.host_charts_dict[self.host]) == 0: - self.charts_init() - self.custom_models_init() - self.model_params_init() - - # if not all models have been trained then train those we need to - if len(self.fitted_at) < len(self.models_in_scope): - self.train( - models_to_train=[m for m in self.models_in_scope if m not in self.fitted_at], - train_data_after=self.initial_train_data_after, - train_data_before=self.initial_train_data_before - ) - # retrain all models as per schedule from config - elif self.train_every_n > 0 and self.runs_counter % self.train_every_n == 0: - self.reinitialize() - self.train() - - # roll forward previous predictions around a training step to avoid the possibility of having the training itself trigger an anomaly - if (self.runs_counter - self.last_train_at) <= self.train_no_prediction_n: - data_probability = self.data_probability_latest - data_anomaly = self.data_anomaly_latest - else: - data_probability, data_anomaly = self.predict() - if self.include_average_prob: - average_prob = np.mean(list(data_probability.values())) - data_probability['average_prob'] = 0 if np.isnan(average_prob) else average_prob - - data = {**data_probability, **data_anomaly} - - self.validate_charts('probability', data_probability, divisor=100) - self.validate_charts('anomaly', data_anomaly) - - self.save_data_latest(data, data_probability, data_anomaly) - - #self.info(f'len(data)={len(data)}') - #self.info(f'data') - - return data diff --git a/src/collectors/python.d.plugin/anomalies/anomalies.conf b/src/collectors/python.d.plugin/anomalies/anomalies.conf deleted file mode 100644 index ef867709..00000000 --- a/src/collectors/python.d.plugin/anomalies/anomalies.conf +++ /dev/null @@ -1,184 +0,0 @@ -# netdata python.d.plugin configuration for anomalies -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# update_every: 2 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) - -# Pull data from local Netdata node. -anomalies: - name: 'Anomalies' - - # Host to pull data from. - host: '127.0.0.1:19999' - - # Username and Password for Netdata if using basic auth. - # username: '???' - # password: '???' - - # Use http or https to pull data - protocol: 'http' - - # SSL verify parameter for requests.get() calls - tls_verify: true - - # What charts to pull data for - A regex like 'system\..*|' or 'system\..*|apps.cpu|apps.mem' etc. - charts_regex: 'system\..*' - - # Charts to exclude, useful if you would like to exclude some specific charts. - # Note: should be a ',' separated string like 'chart.name,chart.name'. - charts_to_exclude: 'system.uptime,system.entropy' - - # What model to use - can be one of 'pca', 'hbos', 'iforest', 'cblof', 'loda', 'copod' or 'feature_bagging'. - # More details here: https://pyod.readthedocs.io/en/latest/pyod.models.html. - model: 'pca' - - # Max number of observations to train on, to help cap compute cost of training model if you set a very large train_n_secs. - train_max_n: 100000 - - # How often to re-train the model (assuming update_every=1 then train_every_n=1800 represents (re)training every 30 minutes). - # Note: If you want to turn off re-training set train_every_n=0 and after initial training the models will not be retrained. - train_every_n: 1800 - - # The length of the window of data to train on (14400 = last 4 hours). - train_n_secs: 14400 - - # How many prediction steps after a train event to just use previous prediction value for. - # Used to reduce possibility of the training step itself appearing as an anomaly on the charts. - train_no_prediction_n: 10 - - # If you would like to train the model for the first time on a specific window then you can define it using the below two variables. - # Start of training data for initial model. - # initial_train_data_after: 1604578857 - - # End of training data for initial model. - # initial_train_data_before: 1604593257 - - # If you would like to ignore recent data in training then you can offset it by offset_n_secs. - offset_n_secs: 0 - - # How many lagged values of each dimension to include in the 'feature vector' each model is trained on. - lags_n: 5 - - # How much smoothing to apply to each dimension in the 'feature vector' each model is trained on. - smooth_n: 3 - - # How many differences to take in preprocessing your data. - # More info on differencing here: https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average#Differencing - # diffs_n=0 would mean training models on the raw values of each dimension. - # diffs_n=1 means everything is done in terms of differences. - diffs_n: 1 - - # What is the typical proportion of anomalies in your data on average? - # This parameter can control the sensitivity of your models to anomalies. - # Some discussion here: https://github.com/yzhao062/pyod/issues/144 - contamination: 0.001 - - # Set to true to include an "average_prob" dimension on anomalies probability chart which is - # just the average of all anomaly probabilities at each time step - include_average_prob: true - - # Define any custom models you would like to create anomaly probabilities for, some examples below to show how. - # For example below example creates two custom models, one to run anomaly detection user and system cpu for our demo servers - # and one on the cpu and mem apps metrics for the python.d.plugin. - # custom_models: - # - name: 'demos_cpu' - # dimensions: 'london.my-netdata.io::system.cpu|user,london.my-netdata.io::system.cpu|system,newyork.my-netdata.io::system.cpu|user,newyork.my-netdata.io::system.cpu|system' - # - name: 'apps_python_d_plugin' - # dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin' - - # Set to true to normalize, using min-max standardization, features used for the custom models. - # Useful if your custom models contain dimensions on very different scales an model you use does - # not internally do its own normalization. Usually best to leave as false. - # custom_models_normalize: false - -# Standalone Custom models example as an additional collector job. -# custom: -# name: 'custom' -# host: '127.0.0.1:19999' -# protocol: 'http' -# charts_regex: 'None' -# charts_to_exclude: 'None' -# model: 'pca' -# train_max_n: 100000 -# train_every_n: 1800 -# train_n_secs: 14400 -# offset_n_secs: 0 -# lags_n: 5 -# smooth_n: 3 -# diffs_n: 1 -# contamination: 0.001 -# custom_models: -# - name: 'user_netdata' -# dimensions: 'users.cpu|netdata,users.mem|netdata,users.threads|netdata,users.processes|netdata,users.sockets|netdata' -# - name: 'apps_python_d_plugin' -# dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin,apps.threads|python.d.plugin,apps.processes|python.d.plugin,apps.sockets|python.d.plugin' - -# Pull data from some demo nodes for cross node custom models. -# demos: -# name: 'demos' -# host: '127.0.0.1:19999' -# protocol: 'http' -# charts_regex: 'None' -# charts_to_exclude: 'None' -# model: 'pca' -# train_max_n: 100000 -# train_every_n: 1800 -# train_n_secs: 14400 -# offset_n_secs: 0 -# lags_n: 5 -# smooth_n: 3 -# diffs_n: 1 -# contamination: 0.001 -# custom_models: -# - name: 'system.cpu' -# dimensions: 'london.my-netdata.io::system.cpu|user,london.my-netdata.io::system.cpu|system,newyork.my-netdata.io::system.cpu|user,newyork.my-netdata.io::system.cpu|system' -# - name: 'system.ip' -# dimensions: 'london.my-netdata.io::system.ip|received,london.my-netdata.io::system.ip|sent,newyork.my-netdata.io::system.ip|received,newyork.my-netdata.io::system.ip|sent' -# - name: 'system.net' -# dimensions: 'london.my-netdata.io::system.net|received,london.my-netdata.io::system.net|sent,newyork.my-netdata.io::system.net|received,newyork.my-netdata.io::system.net|sent' -# - name: 'system.io' -# dimensions: 'london.my-netdata.io::system.io|in,london.my-netdata.io::system.io|out,newyork.my-netdata.io::system.io|in,newyork.my-netdata.io::system.io|out' - -# Example additional job if you want to also pull data from a child streaming to your -# local parent or even a remote node so long as the Netdata REST API is accessible. -# mychildnode1: -# name: 'mychildnode1' -# host: '127.0.0.1:19999/host/mychildnode1' -# protocol: 'http' -# charts_regex: 'system\..*' -# charts_to_exclude: 'None' -# model: 'pca' -# train_max_n: 100000 -# train_every_n: 1800 -# train_n_secs: 14400 -# offset_n_secs: 0 -# lags_n: 5 -# smooth_n: 3 -# diffs_n: 1 -# contamination: 0.001 diff --git a/src/collectors/python.d.plugin/anomalies/metadata.yaml b/src/collectors/python.d.plugin/anomalies/metadata.yaml deleted file mode 100644 index c14e47bf..00000000 --- a/src/collectors/python.d.plugin/anomalies/metadata.yaml +++ /dev/null @@ -1,87 +0,0 @@ -# NOTE: this file is commented out as users are reccomended to use the -# native anomaly detection capabilities on the agent instead. -# meta: -# plugin_name: python.d.plugin -# module_name: anomalies -# monitored_instance: -# name: python.d anomalies -# link: "" -# categories: [] -# icon_filename: "" -# related_resources: -# integrations: -# list: [] -# info_provided_to_referring_integrations: -# description: "" -# keywords: [] -# most_popular: false -# overview: -# data_collection: -# metrics_description: "" -# method_description: "" -# supported_platforms: -# include: [] -# exclude: [] -# multi_instance: true -# additional_permissions: -# description: "" -# default_behavior: -# auto_detection: -# description: "" -# limits: -# description: "" -# performance_impact: -# description: "" -# setup: -# prerequisites: -# list: [] -# configuration: -# file: -# name: "" -# description: "" -# options: -# description: "" -# folding: -# title: "" -# enabled: true -# list: [] -# examples: -# folding: -# enabled: true -# title: "" -# list: [] -# troubleshooting: -# problems: -# list: [] -# alerts: -# - name: anomalies_anomaly_probabilities -# link: https://github.com/netdata/netdata/blob/master/src/health/health.d/anomalies.conf -# metric: anomalies.probability -# info: average anomaly probability over the last 2 minutes -# - name: anomalies_anomaly_flags -# link: https://github.com/netdata/netdata/blob/master/src/health/health.d/anomalies.conf -# metric: anomalies.anomaly -# info: number of anomalies in the last 2 minutes -# metrics: -# folding: -# title: Metrics -# enabled: false -# description: "" -# availability: [] -# scopes: -# - name: global -# description: "" -# labels: [] -# metrics: -# - name: anomalies.probability -# description: Anomaly Probability -# unit: "probability" -# chart_type: line -# dimensions: -# - name: a dimension per probability -# - name: anomalies.anomaly -# description: Anomaly -# unit: "count" -# chart_type: stacked -# dimensions: -# - name: a dimension per anomaly diff --git a/src/collectors/python.d.plugin/boinc/README.md b/src/collectors/python.d.plugin/boinc/README.md deleted file mode 120000 index 22c10ca1..00000000 --- a/src/collectors/python.d.plugin/boinc/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/boinc.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/boinc/boinc.chart.py b/src/collectors/python.d.plugin/boinc/boinc.chart.py deleted file mode 100644 index a31eda1c..00000000 --- a/src/collectors/python.d.plugin/boinc/boinc.chart.py +++ /dev/null @@ -1,168 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: BOINC netdata python.d module -# Author: Austin S. Hemmelgarn (Ferroin) -# SPDX-License-Identifier: GPL-3.0-or-later - -import socket - -from bases.FrameworkServices.SimpleService import SimpleService -from third_party import boinc_client - -ORDER = [ - 'tasks', - 'states', - 'sched_states', - 'process_states', -] - -CHARTS = { - 'tasks': { - 'options': [None, 'Overall Tasks', 'tasks', 'boinc', 'boinc.tasks', 'line'], - 'lines': [ - ['total', 'Total', 'absolute', 1, 1], - ['active', 'Active', 'absolute', 1, 1] - ] - }, - 'states': { - 'options': [None, 'Tasks per State', 'tasks', 'boinc', 'boinc.states', 'line'], - 'lines': [ - ['new', 'New', 'absolute', 1, 1], - ['downloading', 'Downloading', 'absolute', 1, 1], - ['downloaded', 'Ready to Run', 'absolute', 1, 1], - ['comperror', 'Compute Errors', 'absolute', 1, 1], - ['uploading', 'Uploading', 'absolute', 1, 1], - ['uploaded', 'Uploaded', 'absolute', 1, 1], - ['aborted', 'Aborted', 'absolute', 1, 1], - ['upload_failed', 'Failed Uploads', 'absolute', 1, 1] - ] - }, - 'sched_states': { - 'options': [None, 'Tasks per Scheduler State', 'tasks', 'boinc', 'boinc.sched', 'line'], - 'lines': [ - ['uninit_sched', 'Uninitialized', 'absolute', 1, 1], - ['preempted', 'Preempted', 'absolute', 1, 1], - ['scheduled', 'Scheduled', 'absolute', 1, 1] - ] - }, - 'process_states': { - 'options': [None, 'Tasks per Process State', 'tasks', 'boinc', 'boinc.process', 'line'], - 'lines': [ - ['uninit_proc', 'Uninitialized', 'absolute', 1, 1], - ['executing', 'Executing', 'absolute', 1, 1], - ['suspended', 'Suspended', 'absolute', 1, 1], - ['aborting', 'Aborted', 'absolute', 1, 1], - ['quit', 'Quit', 'absolute', 1, 1], - ['copy_pending', 'Copy Pending', 'absolute', 1, 1] - ] - } -} - -# A simple template used for pre-loading the return dictionary to make -# the _get_data() method simpler. -_DATA_TEMPLATE = { - 'total': 0, - 'active': 0, - 'new': 0, - 'downloading': 0, - 'downloaded': 0, - 'comperror': 0, - 'uploading': 0, - 'uploaded': 0, - 'aborted': 0, - 'upload_failed': 0, - 'uninit_sched': 0, - 'preempted': 0, - 'scheduled': 0, - 'uninit_proc': 0, - 'executing': 0, - 'suspended': 0, - 'aborting': 0, - 'quit': 0, - 'copy_pending': 0 -} - -# Map task states to dimensions -_TASK_MAP = { - boinc_client.ResultState.NEW: 'new', - boinc_client.ResultState.FILES_DOWNLOADING: 'downloading', - boinc_client.ResultState.FILES_DOWNLOADED: 'downloaded', - boinc_client.ResultState.COMPUTE_ERROR: 'comperror', - boinc_client.ResultState.FILES_UPLOADING: 'uploading', - boinc_client.ResultState.FILES_UPLOADED: 'uploaded', - boinc_client.ResultState.ABORTED: 'aborted', - boinc_client.ResultState.UPLOAD_FAILED: 'upload_failed' -} - -# Map scheduler states to dimensions -_SCHED_MAP = { - boinc_client.CpuSched.UNINITIALIZED: 'uninit_sched', - boinc_client.CpuSched.PREEMPTED: 'preempted', - boinc_client.CpuSched.SCHEDULED: 'scheduled', -} - -# Maps process states to dimensions -_PROC_MAP = { - boinc_client.Process.UNINITIALIZED: 'uninit_proc', - boinc_client.Process.EXECUTING: 'executing', - boinc_client.Process.SUSPENDED: 'suspended', - boinc_client.Process.ABORT_PENDING: 'aborted', - boinc_client.Process.QUIT_PENDING: 'quit', - boinc_client.Process.COPY_PENDING: 'copy_pending' -} - - -class Service(SimpleService): - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = CHARTS - self.host = self.configuration.get('host', 'localhost') - self.port = self.configuration.get('port', 0) - self.password = self.configuration.get('password', '') - self.client = boinc_client.BoincClient(host=self.host, port=self.port, passwd=self.password) - self.alive = False - - def check(self): - return self.connect() - - def connect(self): - self.client.connect() - self.alive = self.client.connected and self.client.authorized - return self.alive - - def reconnect(self): - # The client class itself actually disconnects existing - # connections when it is told to connect, so we don't need to - # explicitly disconnect when we're just trying to reconnect. - return self.connect() - - def is_alive(self): - if not self.alive: - return self.reconnect() - return True - - def _get_data(self): - if not self.is_alive(): - return None - - data = dict(_DATA_TEMPLATE) - - try: - results = self.client.get_tasks() - except socket.error: - self.error('Connection is dead') - self.alive = False - return None - - for task in results: - data['total'] += 1 - data[_TASK_MAP[task.state]] += 1 - try: - if task.active_task: - data['active'] += 1 - data[_SCHED_MAP[task.scheduler_state]] += 1 - data[_PROC_MAP[task.active_task_state]] += 1 - except AttributeError: - pass - - return data or None diff --git a/src/collectors/python.d.plugin/boinc/boinc.conf b/src/collectors/python.d.plugin/boinc/boinc.conf deleted file mode 100644 index 16edf55c..00000000 --- a/src/collectors/python.d.plugin/boinc/boinc.conf +++ /dev/null @@ -1,66 +0,0 @@ -# netdata python.d.plugin configuration for boinc -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# update_every: 1 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 1 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# Additionally to the above, boinc also supports the following: -# -# hostname: localhost # The host running the BOINC client -# port: 31416 # The remote GUI RPC port for BOINC -# password: '' # The remote GUI RPC password diff --git a/src/collectors/python.d.plugin/boinc/integrations/boinc.md b/src/collectors/python.d.plugin/boinc/integrations/boinc.md deleted file mode 100644 index d5fcac21..00000000 --- a/src/collectors/python.d.plugin/boinc/integrations/boinc.md +++ /dev/null @@ -1,238 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/boinc/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/boinc/metadata.yaml" -sidebar_label: "BOINC" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Distributed Computing Systems" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# BOINC - - -<img src="https://netdata.cloud/img/bolt.svg" width="150"/> - - -Plugin: python.d.plugin -Module: boinc - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -This collector monitors task counts for the Berkeley Open Infrastructure Networking Computing (BOINC) distributed computing client. - -It uses the same RPC interface that the BOINC monitoring GUI does. - -This collector is supported on all platforms. - -This collector supports collecting metrics from multiple instances of this integration, including remote instances. - - -### Default Behavior - -#### Auto-Detection - -By default, the module will try to auto-detect the password to the RPC interface by looking in `/var/lib/boinc` for this file (this is the location most Linux distributions use for a system-wide BOINC installation), so things may just work without needing configuration for a local system. - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per BOINC instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| boinc.tasks | Total, Active | tasks | -| boinc.states | New, Downloading, Ready to Run, Compute Errors, Uploading, Uploaded, Aborted, Failed Uploads | tasks | -| boinc.sched | Uninitialized, Preempted, Scheduled | tasks | -| boinc.process | Uninitialized, Executing, Suspended, Aborted, Quit, Copy Pending | tasks | - - - -## Alerts - - -The following alerts are available: - -| Alert name | On metric | Description | -|:------------|:----------|:------------| -| [ boinc_total_tasks ](https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf) | boinc.tasks | average number of total tasks over the last 10 minutes | -| [ boinc_active_tasks ](https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf) | boinc.tasks | average number of active tasks over the last 10 minutes | -| [ boinc_compute_errors ](https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf) | boinc.states | average number of compute errors over the last 10 minutes | -| [ boinc_upload_errors ](https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf) | boinc.states | average number of failed uploads over the last 10 minutes | - - -## Setup - -### Prerequisites - -#### Boinc RPC interface - -BOINC requires use of a password to access it's RPC interface. You can find this password in the `gui_rpc_auth.cfg` file in your BOINC directory. - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/boinc.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/boinc.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| update_every | Sets the default data collection frequency. | 5 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | -| name | Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. | | no | -| hostname | Define a hostname where boinc is running. | localhost | no | -| port | The port of boinc RPC interface. | | no | -| password | Provide a password to connect to a boinc RPC interface. | | no | - -</details> - -#### Examples - -##### Configuration of a remote boinc instance - -A basic JOB configuration for a remote boinc instance - -```yaml -remote: - hostname: '1.2.3.4' - port: 1234 - password: 'some-password' - -``` -##### Multi-instance - -> **Note**: When you define multiple jobs, their names must be unique. - -Collecting metrics from local and remote instances. - - -<details open><summary>Config</summary> - -```yaml -localhost: - name: 'local' - host: '127.0.0.1' - port: 1234 - password: 'some-password' - -remote_job: - name: 'remote' - host: '192.0.2.1' - port: 1234 - password: some-other-password - -``` -</details> - - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `boinc` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin boinc debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `boinc` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep boinc -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep boinc /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep boinc -``` - - diff --git a/src/collectors/python.d.plugin/boinc/metadata.yaml b/src/collectors/python.d.plugin/boinc/metadata.yaml deleted file mode 100644 index 9448cbe0..00000000 --- a/src/collectors/python.d.plugin/boinc/metadata.yaml +++ /dev/null @@ -1,198 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: boinc - monitored_instance: - name: BOINC - link: "https://boinc.berkeley.edu/" - categories: - - data-collection.distributed-computing-systems - icon_filename: "bolt.svg" - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: "" - keywords: - - boinc - - distributed - most_popular: false - overview: - data_collection: - metrics_description: "This collector monitors task counts for the Berkeley Open Infrastructure Networking Computing (BOINC) distributed computing client." - method_description: "It uses the same RPC interface that the BOINC monitoring GUI does." - supported_platforms: - include: [] - exclude: [] - multi_instance: true - additional_permissions: - description: "" - default_behavior: - auto_detection: - description: "By default, the module will try to auto-detect the password to the RPC interface by looking in `/var/lib/boinc` for this file (this is the location most Linux distributions use for a system-wide BOINC installation), so things may just work without needing configuration for a local system." - limits: - description: "" - performance_impact: - description: "" - setup: - prerequisites: - list: - - title: "Boinc RPC interface" - description: BOINC requires use of a password to access it's RPC interface. You can find this password in the `gui_rpc_auth.cfg` file in your BOINC directory. - configuration: - file: - name: python.d/boinc.conf - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: update_every - description: Sets the default data collection frequency. - default_value: 5 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - - name: name - description: Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. - default_value: "" - required: false - - name: hostname - description: Define a hostname where boinc is running. - default_value: "localhost" - required: false - - name: port - description: The port of boinc RPC interface. - default_value: "" - required: false - - name: password - description: Provide a password to connect to a boinc RPC interface. - default_value: "" - required: false - examples: - folding: - enabled: true - title: "Config" - list: - - name: Configuration of a remote boinc instance - description: A basic JOB configuration for a remote boinc instance - folding: - enabled: false - config: | - remote: - hostname: '1.2.3.4' - port: 1234 - password: 'some-password' - - name: Multi-instance - description: | - > **Note**: When you define multiple jobs, their names must be unique. - - Collecting metrics from local and remote instances. - config: | - localhost: - name: 'local' - host: '127.0.0.1' - port: 1234 - password: 'some-password' - - remote_job: - name: 'remote' - host: '192.0.2.1' - port: 1234 - password: some-other-password - troubleshooting: - problems: - list: [] - alerts: - - name: boinc_total_tasks - link: https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf - metric: boinc.tasks - info: average number of total tasks over the last 10 minutes - os: "*" - - name: boinc_active_tasks - link: https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf - metric: boinc.tasks - info: average number of active tasks over the last 10 minutes - os: "*" - - name: boinc_compute_errors - link: https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf - metric: boinc.states - info: average number of compute errors over the last 10 minutes - os: "*" - - name: boinc_upload_errors - link: https://github.com/netdata/netdata/blob/master/src/health/health.d/boinc.conf - metric: boinc.states - info: average number of failed uploads over the last 10 minutes - os: "*" - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: boinc.tasks - description: Overall Tasks - unit: "tasks" - chart_type: line - dimensions: - - name: Total - - name: Active - - name: boinc.states - description: Tasks per State - unit: "tasks" - chart_type: line - dimensions: - - name: New - - name: Downloading - - name: Ready to Run - - name: Compute Errors - - name: Uploading - - name: Uploaded - - name: Aborted - - name: Failed Uploads - - name: boinc.sched - description: Tasks per Scheduler State - unit: "tasks" - chart_type: line - dimensions: - - name: Uninitialized - - name: Preempted - - name: Scheduled - - name: boinc.process - description: Tasks per Process State - unit: "tasks" - chart_type: line - dimensions: - - name: Uninitialized - - name: Executing - - name: Suspended - - name: Aborted - - name: Quit - - name: Copy Pending diff --git a/src/collectors/python.d.plugin/ceph/README.md b/src/collectors/python.d.plugin/ceph/README.md deleted file mode 120000 index 654248b7..00000000 --- a/src/collectors/python.d.plugin/ceph/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/ceph.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/ceph/ceph.chart.py b/src/collectors/python.d.plugin/ceph/ceph.chart.py deleted file mode 100644 index 4bcbe197..00000000 --- a/src/collectors/python.d.plugin/ceph/ceph.chart.py +++ /dev/null @@ -1,374 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: ceph netdata python.d module -# Author: Luis Eduardo (lets00) -# SPDX-License-Identifier: GPL-3.0-or-later - -try: - import rados - - CEPH = True -except ImportError: - CEPH = False - -import json -import os - -from bases.FrameworkServices.SimpleService import SimpleService - -# default module values (can be overridden per job in `config`) -update_every = 10 - -ORDER = [ - 'general_usage', - 'general_objects', - 'general_bytes', - 'general_operations', - 'general_latency', - 'pool_usage', - 'pool_objects', - 'pool_read_bytes', - 'pool_write_bytes', - 'pool_read_operations', - 'pool_write_operations', - 'osd_usage', - 'osd_size', - 'osd_apply_latency', - 'osd_commit_latency' -] - -CHARTS = { - 'general_usage': { - 'options': [None, 'Ceph General Space', 'KiB', 'general', 'ceph.general_usage', 'stacked'], - 'lines': [ - ['general_available', 'avail', 'absolute'], - ['general_usage', 'used', 'absolute'] - ] - }, - 'general_objects': { - 'options': [None, 'Ceph General Objects', 'objects', 'general', 'ceph.general_objects', 'area'], - 'lines': [ - ['general_objects', 'cluster', 'absolute'] - ] - }, - 'general_bytes': { - 'options': [None, 'Ceph General Read/Write Data/s', 'KiB/s', 'general', 'ceph.general_bytes', - 'area'], - 'lines': [ - ['general_read_bytes', 'read', 'absolute', 1, 1024], - ['general_write_bytes', 'write', 'absolute', -1, 1024] - ] - }, - 'general_operations': { - 'options': [None, 'Ceph General Read/Write Operations/s', 'operations', 'general', 'ceph.general_operations', - 'area'], - 'lines': [ - ['general_read_operations', 'read', 'absolute', 1], - ['general_write_operations', 'write', 'absolute', -1] - ] - }, - 'general_latency': { - 'options': [None, 'Ceph General Apply/Commit latency', 'milliseconds', 'general', 'ceph.general_latency', - 'area'], - 'lines': [ - ['general_apply_latency', 'apply', 'absolute'], - ['general_commit_latency', 'commit', 'absolute'] - ] - }, - 'pool_usage': { - 'options': [None, 'Ceph Pools', 'KiB', 'pool', 'ceph.pool_usage', 'line'], - 'lines': [] - }, - 'pool_objects': { - 'options': [None, 'Ceph Pools', 'objects', 'pool', 'ceph.pool_objects', 'line'], - 'lines': [] - }, - 'pool_read_bytes': { - 'options': [None, 'Ceph Read Pool Data/s', 'KiB/s', 'pool', 'ceph.pool_read_bytes', 'area'], - 'lines': [] - }, - 'pool_write_bytes': { - 'options': [None, 'Ceph Write Pool Data/s', 'KiB/s', 'pool', 'ceph.pool_write_bytes', 'area'], - 'lines': [] - }, - 'pool_read_operations': { - 'options': [None, 'Ceph Read Pool Operations/s', 'operations', 'pool', 'ceph.pool_read_operations', 'area'], - 'lines': [] - }, - 'pool_write_operations': { - 'options': [None, 'Ceph Write Pool Operations/s', 'operations', 'pool', 'ceph.pool_write_operations', 'area'], - 'lines': [] - }, - 'osd_usage': { - 'options': [None, 'Ceph OSDs', 'KiB', 'osd', 'ceph.osd_usage', 'line'], - 'lines': [] - }, - 'osd_size': { - 'options': [None, 'Ceph OSDs size', 'KiB', 'osd', 'ceph.osd_size', 'line'], - 'lines': [] - }, - 'osd_apply_latency': { - 'options': [None, 'Ceph OSDs apply latency', 'milliseconds', 'osd', 'ceph.apply_latency', 'line'], - 'lines': [] - }, - 'osd_commit_latency': { - 'options': [None, 'Ceph OSDs commit latency', 'milliseconds', 'osd', 'ceph.commit_latency', 'line'], - 'lines': [] - } - -} - - -class Service(SimpleService): - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = CHARTS - self.config_file = self.configuration.get('config_file') - self.keyring_file = self.configuration.get('keyring_file') - self.rados_id = self.configuration.get('rados_id', 'admin') - - def check(self): - """ - Checks module - :return: - """ - if not CEPH: - self.error('rados module is needed to use ceph.chart.py') - return False - if not (self.config_file and self.keyring_file): - self.error('config_file and/or keyring_file is not defined') - return False - - # Verify files and permissions - if not (os.access(self.config_file, os.F_OK)): - self.error('{0} does not exist'.format(self.config_file)) - return False - if not (os.access(self.keyring_file, os.F_OK)): - self.error('{0} does not exist'.format(self.keyring_file)) - return False - if not (os.access(self.config_file, os.R_OK)): - self.error('Ceph plugin does not read {0}, define read permission.'.format(self.config_file)) - return False - if not (os.access(self.keyring_file, os.R_OK)): - self.error('Ceph plugin does not read {0}, define read permission.'.format(self.keyring_file)) - return False - try: - self.cluster = rados.Rados(conffile=self.config_file, - conf=dict(keyring=self.keyring_file), - rados_id=self.rados_id) - self.cluster.connect() - except rados.Error as error: - self.error(error) - return False - self.create_definitions() - return True - - def create_definitions(self): - """ - Create dynamically charts options - :return: None - """ - # Pool lines - for pool in sorted(self._get_df()['pools'], key=lambda x: sorted(x.keys())): - self.definitions['pool_usage']['lines'].append([pool['name'], - pool['name'], - 'absolute']) - self.definitions['pool_objects']['lines'].append(["obj_{0}".format(pool['name']), - pool['name'], - 'absolute']) - self.definitions['pool_read_bytes']['lines'].append(['read_{0}'.format(pool['name']), - pool['name'], - 'absolute', 1, 1024]) - self.definitions['pool_write_bytes']['lines'].append(['write_{0}'.format(pool['name']), - pool['name'], - 'absolute', 1, 1024]) - self.definitions['pool_read_operations']['lines'].append(['read_operations_{0}'.format(pool['name']), - pool['name'], - 'absolute']) - self.definitions['pool_write_operations']['lines'].append(['write_operations_{0}'.format(pool['name']), - pool['name'], - 'absolute']) - - # OSD lines - for osd in sorted(self._get_osd_df()['nodes'], key=lambda x: sorted(x.keys())): - self.definitions['osd_usage']['lines'].append([osd['name'], - osd['name'], - 'absolute']) - self.definitions['osd_size']['lines'].append(['size_{0}'.format(osd['name']), - osd['name'], - 'absolute']) - self.definitions['osd_apply_latency']['lines'].append(['apply_latency_{0}'.format(osd['name']), - osd['name'], - 'absolute']) - self.definitions['osd_commit_latency']['lines'].append(['commit_latency_{0}'.format(osd['name']), - osd['name'], - 'absolute']) - - def get_data(self): - """ - Catch all ceph data - :return: dict - """ - try: - data = {} - df = self._get_df() - osd_df = self._get_osd_df() - osd_perf = self._get_osd_perf() - osd_perf_infos = get_osd_perf_infos(osd_perf) - pool_stats = self._get_osd_pool_stats() - - data.update(self._get_general(osd_perf_infos, pool_stats)) - for pool in df['pools']: - data.update(self._get_pool_usage(pool)) - data.update(self._get_pool_objects(pool)) - for pool_io in pool_stats: - data.update(self._get_pool_rw(pool_io)) - for osd in osd_df['nodes']: - data.update(self._get_osd_usage(osd)) - data.update(self._get_osd_size(osd)) - for osd_apply_commit in osd_perf_infos: - data.update(self._get_osd_latency(osd_apply_commit)) - return data - except (ValueError, AttributeError) as error: - self.error(error) - return None - - def _get_general(self, osd_perf_infos, pool_stats): - """ - Get ceph's general usage - :return: dict - """ - status = self.cluster.get_cluster_stats() - read_bytes_sec = 0 - write_bytes_sec = 0 - read_op_per_sec = 0 - write_op_per_sec = 0 - apply_latency = 0 - commit_latency = 0 - - for pool_rw_io_b in pool_stats: - read_bytes_sec += pool_rw_io_b['client_io_rate'].get('read_bytes_sec', 0) - write_bytes_sec += pool_rw_io_b['client_io_rate'].get('write_bytes_sec', 0) - read_op_per_sec += pool_rw_io_b['client_io_rate'].get('read_op_per_sec', 0) - write_op_per_sec += pool_rw_io_b['client_io_rate'].get('write_op_per_sec', 0) - for perf in osd_perf_infos: - apply_latency += perf['perf_stats']['apply_latency_ms'] - commit_latency += perf['perf_stats']['commit_latency_ms'] - - return { - 'general_usage': int(status['kb_used']), - 'general_available': int(status['kb_avail']), - 'general_objects': int(status['num_objects']), - 'general_read_bytes': read_bytes_sec, - 'general_write_bytes': write_bytes_sec, - 'general_read_operations': read_op_per_sec, - 'general_write_operations': write_op_per_sec, - 'general_apply_latency': apply_latency, - 'general_commit_latency': commit_latency - } - - @staticmethod - def _get_pool_usage(pool): - """ - Process raw data into pool usage dict information - :return: A pool dict with pool name's key and usage bytes' value - """ - return {pool['name']: pool['stats']['kb_used']} - - @staticmethod - def _get_pool_objects(pool): - """ - Process raw data into pool usage dict information - :return: A pool dict with pool name's key and object numbers - """ - return {'obj_{0}'.format(pool['name']): pool['stats']['objects']} - - @staticmethod - def _get_pool_rw(pool): - """ - Get read/write kb and operations in a pool - :return: A pool dict with both read/write bytes and operations. - """ - return { - 'read_{0}'.format(pool['pool_name']): int(pool['client_io_rate'].get('read_bytes_sec', 0)), - 'write_{0}'.format(pool['pool_name']): int(pool['client_io_rate'].get('write_bytes_sec', 0)), - 'read_operations_{0}'.format(pool['pool_name']): int(pool['client_io_rate'].get('read_op_per_sec', 0)), - 'write_operations_{0}'.format(pool['pool_name']): int(pool['client_io_rate'].get('write_op_per_sec', 0)) - } - - @staticmethod - def _get_osd_usage(osd): - """ - Process raw data into osd dict information to get osd usage - :return: A osd dict with osd name's key and usage bytes' value - """ - return {osd['name']: float(osd['kb_used'])} - - @staticmethod - def _get_osd_size(osd): - """ - Process raw data into osd dict information to get osd size (kb) - :return: A osd dict with osd name's key and size bytes' value - """ - return {'size_{0}'.format(osd['name']): float(osd['kb'])} - - @staticmethod - def _get_osd_latency(osd): - """ - Get ceph osd apply and commit latency - :return: A osd dict with osd name's key with both apply and commit latency values - """ - return { - 'apply_latency_osd.{0}'.format(osd['id']): osd['perf_stats']['apply_latency_ms'], - 'commit_latency_osd.{0}'.format(osd['id']): osd['perf_stats']['commit_latency_ms'] - } - - def _get_df(self): - """ - Get ceph df output - :return: ceph df --format json - """ - return json.loads(self.cluster.mon_command(json.dumps({ - 'prefix': 'df', - 'format': 'json' - }), b'')[1].decode('utf-8')) - - def _get_osd_df(self): - """ - Get ceph osd df output - :return: ceph osd df --format json - """ - return json.loads(self.cluster.mon_command(json.dumps({ - 'prefix': 'osd df', - 'format': 'json' - }), b'')[1].decode('utf-8').replace('-nan', '"-nan"')) - - def _get_osd_perf(self): - """ - Get ceph osd performance - :return: ceph osd perf --format json - """ - return json.loads(self.cluster.mon_command(json.dumps({ - 'prefix': 'osd perf', - 'format': 'json' - }), b'')[1].decode('utf-8')) - - def _get_osd_pool_stats(self): - """ - Get ceph osd pool status. - This command is used to get information about both - read/write operation and bytes per second on each pool - :return: ceph osd pool stats --format json - """ - return json.loads(self.cluster.mon_command(json.dumps({ - 'prefix': 'osd pool stats', - 'format': 'json' - }), b'')[1].decode('utf-8')) - - -def get_osd_perf_infos(osd_perf): - # https://github.com/netdata/netdata/issues/8247 - # module uses 'osd_perf_infos' data, its been moved under 'osdstats` since Ceph v14.2 - if 'osd_perf_infos' in osd_perf: - return osd_perf['osd_perf_infos'] - return osd_perf['osdstats']['osd_perf_infos'] diff --git a/src/collectors/python.d.plugin/ceph/ceph.conf b/src/collectors/python.d.plugin/ceph/ceph.conf deleted file mode 100644 index 81788e86..00000000 --- a/src/collectors/python.d.plugin/ceph/ceph.conf +++ /dev/null @@ -1,75 +0,0 @@ -# netdata python.d.plugin configuration for ceph stats -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# update_every: 10 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 10 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# Additionally to the above, ceph plugin also supports the following: -# -# config_file: 'config_file' # Ceph config file. -# keyring_file: 'keyring_file' # Ceph keyring file. netdata user must be added into ceph group -# # and keyring file must be read group permission. -# rados_id: 'rados username' # ID used to connect to ceph cluster. Allows -# # creating a read only key for pulling data v.s. admin -# ---------------------------------------------------------------------- -# AUTO-DETECTION JOBS -# only one of them will run (they have the same name) -# -config_file: '/etc/ceph/ceph.conf' -keyring_file: '/etc/ceph/ceph.client.admin.keyring' -rados_id: 'admin' diff --git a/src/collectors/python.d.plugin/ceph/integrations/ceph.md b/src/collectors/python.d.plugin/ceph/integrations/ceph.md deleted file mode 100644 index d2584a4d..00000000 --- a/src/collectors/python.d.plugin/ceph/integrations/ceph.md +++ /dev/null @@ -1,228 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/ceph/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/ceph/metadata.yaml" -sidebar_label: "Ceph" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Storage, Mount Points and Filesystems" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# Ceph - - -<img src="https://netdata.cloud/img/ceph.svg" width="150"/> - - -Plugin: python.d.plugin -Module: ceph - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -This collector monitors Ceph metrics about Cluster statistics, OSD usage, latency and Pool statistics. - -Uses the `rados` python module to connect to a Ceph cluster. - -This collector is supported on all platforms. - -This collector supports collecting metrics from multiple instances of this integration, including remote instances. - - -### Default Behavior - -#### Auto-Detection - -This integration doesn't support auto-detection. - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per Ceph instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| ceph.general_usage | avail, used | KiB | -| ceph.general_objects | cluster | objects | -| ceph.general_bytes | read, write | KiB/s | -| ceph.general_operations | read, write | operations | -| ceph.general_latency | apply, commit | milliseconds | -| ceph.pool_usage | a dimension per Ceph Pool | KiB | -| ceph.pool_objects | a dimension per Ceph Pool | objects | -| ceph.pool_read_bytes | a dimension per Ceph Pool | KiB/s | -| ceph.pool_write_bytes | a dimension per Ceph Pool | KiB/s | -| ceph.pool_read_operations | a dimension per Ceph Pool | operations | -| ceph.pool_write_operations | a dimension per Ceph Pool | operations | -| ceph.osd_usage | a dimension per Ceph OSD | KiB | -| ceph.osd_size | a dimension per Ceph OSD | KiB | -| ceph.apply_latency | a dimension per Ceph OSD | milliseconds | -| ceph.commit_latency | a dimension per Ceph OSD | milliseconds | - - - -## Alerts - - -The following alerts are available: - -| Alert name | On metric | Description | -|:------------|:----------|:------------| -| [ ceph_cluster_space_usage ](https://github.com/netdata/netdata/blob/master/src/health/health.d/ceph.conf) | ceph.general_usage | cluster disk space utilization | - - -## Setup - -### Prerequisites - -#### `rados` python module - -Make sure the `rados` python module is installed - -#### Granting read permissions to ceph group from keyring file - -Execute: `chmod 640 /etc/ceph/ceph.client.admin.keyring` - -#### Create a specific rados_id - -You can optionally create a rados_id to use instead of admin - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/ceph.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/ceph.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| update_every | Sets the default data collection frequency. | 5 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | -| name | Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. | | no | -| config_file | Ceph config file | | yes | -| keyring_file | Ceph keyring file. netdata user must be added into ceph group and keyring file must be read group permission. | | yes | -| rados_id | A rados user id to use for connecting to the Ceph cluster. | admin | no | - -</details> - -#### Examples - -##### Basic local Ceph cluster - -A basic configuration to connect to a local Ceph cluster. - -```yaml -local: - config_file: '/etc/ceph/ceph.conf' - keyring_file: '/etc/ceph/ceph.client.admin.keyring' - -``` - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `ceph` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin ceph debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `ceph` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep ceph -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep ceph /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep ceph -``` - - diff --git a/src/collectors/python.d.plugin/ceph/metadata.yaml b/src/collectors/python.d.plugin/ceph/metadata.yaml deleted file mode 100644 index 64294113..00000000 --- a/src/collectors/python.d.plugin/ceph/metadata.yaml +++ /dev/null @@ -1,223 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: ceph - monitored_instance: - name: Ceph - link: 'https://ceph.io/' - categories: - - data-collection.storage-mount-points-and-filesystems - icon_filename: 'ceph.svg' - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: '' - keywords: - - ceph - - storage - most_popular: false - overview: - data_collection: - metrics_description: 'This collector monitors Ceph metrics about Cluster statistics, OSD usage, latency and Pool statistics.' - method_description: 'Uses the `rados` python module to connect to a Ceph cluster.' - supported_platforms: - include: [] - exclude: [] - multi_instance: true - additional_permissions: - description: '' - default_behavior: - auto_detection: - description: '' - limits: - description: '' - performance_impact: - description: '' - setup: - prerequisites: - list: - - title: '`rados` python module' - description: 'Make sure the `rados` python module is installed' - - title: 'Granting read permissions to ceph group from keyring file' - description: 'Execute: `chmod 640 /etc/ceph/ceph.client.admin.keyring`' - - title: 'Create a specific rados_id' - description: 'You can optionally create a rados_id to use instead of admin' - configuration: - file: - name: python.d/ceph.conf - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: update_every - description: Sets the default data collection frequency. - default_value: 5 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - - name: name - description: Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. - default_value: '' - required: false - - name: config_file - description: Ceph config file - default_value: '' - required: true - - name: keyring_file - description: Ceph keyring file. netdata user must be added into ceph group and keyring file must be read group permission. - default_value: '' - required: true - - name: rados_id - description: A rados user id to use for connecting to the Ceph cluster. - default_value: 'admin' - required: false - examples: - folding: - enabled: true - title: "Config" - list: - - name: Basic local Ceph cluster - description: A basic configuration to connect to a local Ceph cluster. - folding: - enabled: false - config: | - local: - config_file: '/etc/ceph/ceph.conf' - keyring_file: '/etc/ceph/ceph.client.admin.keyring' - troubleshooting: - problems: - list: [] - alerts: - - name: ceph_cluster_space_usage - link: https://github.com/netdata/netdata/blob/master/src/health/health.d/ceph.conf - metric: ceph.general_usage - info: cluster disk space utilization - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: ceph.general_usage - description: Ceph General Space - unit: "KiB" - chart_type: stacked - dimensions: - - name: avail - - name: used - - name: ceph.general_objects - description: Ceph General Objects - unit: "objects" - chart_type: area - dimensions: - - name: cluster - - name: ceph.general_bytes - description: Ceph General Read/Write Data/s - unit: "KiB/s" - chart_type: area - dimensions: - - name: read - - name: write - - name: ceph.general_operations - description: Ceph General Read/Write Operations/s - unit: "operations" - chart_type: area - dimensions: - - name: read - - name: write - - name: ceph.general_latency - description: Ceph General Apply/Commit latency - unit: "milliseconds" - chart_type: area - dimensions: - - name: apply - - name: commit - - name: ceph.pool_usage - description: Ceph Pools - unit: "KiB" - chart_type: line - dimensions: - - name: a dimension per Ceph Pool - - name: ceph.pool_objects - description: Ceph Pools - unit: "objects" - chart_type: line - dimensions: - - name: a dimension per Ceph Pool - - name: ceph.pool_read_bytes - description: Ceph Read Pool Data/s - unit: "KiB/s" - chart_type: area - dimensions: - - name: a dimension per Ceph Pool - - name: ceph.pool_write_bytes - description: Ceph Write Pool Data/s - unit: "KiB/s" - chart_type: area - dimensions: - - name: a dimension per Ceph Pool - - name: ceph.pool_read_operations - description: Ceph Read Pool Operations/s - unit: "operations" - chart_type: area - dimensions: - - name: a dimension per Ceph Pool - - name: ceph.pool_write_operations - description: Ceph Write Pool Operations/s - unit: "operations" - chart_type: area - dimensions: - - name: a dimension per Ceph Pool - - name: ceph.osd_usage - description: Ceph OSDs - unit: "KiB" - chart_type: line - dimensions: - - name: a dimension per Ceph OSD - - name: ceph.osd_size - description: Ceph OSDs size - unit: "KiB" - chart_type: line - dimensions: - - name: a dimension per Ceph OSD - - name: ceph.apply_latency - description: Ceph OSDs apply latency - unit: "milliseconds" - chart_type: line - dimensions: - - name: a dimension per Ceph OSD - - name: ceph.commit_latency - description: Ceph OSDs commit latency - unit: "milliseconds" - chart_type: line - dimensions: - - name: a dimension per Ceph OSD diff --git a/src/collectors/python.d.plugin/go_expvar/integrations/go_applications_expvar.md b/src/collectors/python.d.plugin/go_expvar/integrations/go_applications_expvar.md index 8f086765..103058db 100644 --- a/src/collectors/python.d.plugin/go_expvar/integrations/go_applications_expvar.md +++ b/src/collectors/python.d.plugin/go_expvar/integrations/go_applications_expvar.md @@ -84,14 +84,14 @@ There are no alerts configured by default for this integration. #### Enable the go_expvar collector -The `go_expvar` collector is disabled by default. To enable it, use `edit-config` from the Netdata [config directory](/docs/netdata-agent/configuration/README.md), which is typically at `/etc/netdata`, to edit the `python.d.conf` file. +The `go_expvar` collector is disabled by default. To enable it, use `edit-config` from the Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md), which is typically at `/etc/netdata`, to edit the `python.d.conf` file. ```bash cd /etc/netdata # Replace this path with your Netdata config directory, if different sudo ./edit-config python.d.conf ``` -Change the value of the `go_expvar` setting to `yes`. Save the file and restart the Netdata Agent with `sudo systemctl restart netdata`, or the [appropriate method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for your system. +Change the value of the `go_expvar` setting to `yes`. Save the file and restart the Netdata Agent with `sudo systemctl restart netdata`, or the [appropriate method](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/start-stop-restart.md) for your system. #### Sample `expvar` usage in a Go application @@ -171,8 +171,8 @@ number of currently running Goroutines and updates these stats every second. The configuration file name for this integration is `python.d/go_expvar.conf`. -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script from the +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). ```bash cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata @@ -231,8 +231,8 @@ See [this issue](https://github.com/netdata/netdata/pull/1902#issuecomment-28449 Please see these two links to the official Netdata documentation for more information about the values: -- [External plugins - charts](/src/collectors/plugins.d/README.md#chart) -- [Chart variables](/src/collectors/python.d.plugin/README.md#global-variables-order-and-chart) +- [External plugins - charts](https://github.com/netdata/netdata/blob/master/src/plugins.d/README.md#chart) +- [Chart variables](https://github.com/netdata/netdata/blob/master/src/collectors/python.d.plugin/README.md#global-variables-order-and-chart) **Line definitions** @@ -255,7 +255,7 @@ hidden: False ``` Please see the following link for more information about the options and their default values: -[External plugins - dimensions](/src/collectors/plugins.d/README.md#dimension) +[External plugins - dimensions](https://github.com/netdata/netdata/blob/master/src/plugins.d/README.md#dimension) Apart from top-level expvars, this plugin can also parse expvars stored in a multi-level map; All dicts in the resulting JSON document are then flattened to one level. diff --git a/src/collectors/python.d.plugin/go_expvar/metadata.yaml b/src/collectors/python.d.plugin/go_expvar/metadata.yaml index aa45968f..b91225e9 100644 --- a/src/collectors/python.d.plugin/go_expvar/metadata.yaml +++ b/src/collectors/python.d.plugin/go_expvar/metadata.yaml @@ -48,7 +48,7 @@ modules: sudo ./edit-config python.d.conf ``` - Change the value of the `go_expvar` setting to `yes`. Save the file and restart the Netdata Agent with `sudo systemctl restart netdata`, or the [appropriate method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for your system. + Change the value of the `go_expvar` setting to `yes`. Save the file and restart the Netdata Agent with `sudo systemctl restart netdata`, or the [appropriate method](/docs/netdata-agent/start-stop-restart.md) for your system. - title: "Sample `expvar` usage in a Go application" description: | The `expvar` package exposes metrics over HTTP and is very easy to use. @@ -200,7 +200,7 @@ modules: Please see these two links to the official Netdata documentation for more information about the values: - - [External plugins - charts](/src/collectors/plugins.d/README.md#chart) + - [External plugins - charts](/src/plugins.d/README.md#chart) - [Chart variables](/src/collectors/python.d.plugin/README.md#global-variables-order-and-chart) **Line definitions** @@ -224,7 +224,7 @@ modules: ``` Please see the following link for more information about the options and their default values: - [External plugins - dimensions](/src/collectors/plugins.d/README.md#dimension) + [External plugins - dimensions](/src/plugins.d/README.md#dimension) Apart from top-level expvars, this plugin can also parse expvars stored in a multi-level map; All dicts in the resulting JSON document are then flattened to one level. diff --git a/src/collectors/python.d.plugin/haproxy/README.md b/src/collectors/python.d.plugin/haproxy/README.md index 8ade512b..bc54d863 100644 --- a/src/collectors/python.d.plugin/haproxy/README.md +++ b/src/collectors/python.d.plugin/haproxy/README.md @@ -1,12 +1,3 @@ -<!-- -title: "HAProxy monitoring with Netdata" -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/haproxy/README.md" -sidebar_label: "haproxy-python.d.plugin" -learn_status: "Published" -learn_topic_type: "References" -learn_rel_path: "Integrations/Monitor/Webapps" ---> - # HAProxy collector Monitors frontend and backend metrics such as bytes in, bytes out, sessions current, sessions in queue current. diff --git a/src/collectors/python.d.plugin/openldap/README.md b/src/collectors/python.d.plugin/openldap/README.md deleted file mode 120000 index 45f36b9b..00000000 --- a/src/collectors/python.d.plugin/openldap/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/openldap.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/openldap/integrations/openldap.md b/src/collectors/python.d.plugin/openldap/integrations/openldap.md deleted file mode 100644 index 3f363343..00000000 --- a/src/collectors/python.d.plugin/openldap/integrations/openldap.md +++ /dev/null @@ -1,249 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/openldap/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/openldap/metadata.yaml" -sidebar_label: "OpenLDAP" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Authentication and Authorization" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# OpenLDAP - - -<img src="https://netdata.cloud/img/statsd.png" width="150"/> - - -Plugin: python.d.plugin -Module: openldap - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -This collector monitors OpenLDAP metrics about connections, operations, referrals and more. - -Statistics are taken from the monitoring interface of a openLDAP (slapd) server - - -This collector is supported on all platforms. - -This collector only supports collecting metrics from a single instance of this integration. - - -### Default Behavior - -#### Auto-Detection - -This collector doesn't work until all the prerequisites are checked. - - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per OpenLDAP instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| openldap.total_connections | connections | connections/s | -| openldap.traffic_stats | sent | KiB/s | -| openldap.operations_status | completed, initiated | ops/s | -| openldap.referrals | sent | referrals/s | -| openldap.entries | sent | entries/s | -| openldap.ldap_operations | bind, search, unbind, add, delete, modify, compare | ops/s | -| openldap.waiters | write, read | waiters/s | - - - -## Alerts - -There are no alerts configured by default for this integration. - - -## Setup - -### Prerequisites - -#### Configure the openLDAP server to expose metrics to monitor it. - -Follow instructions from https://www.openldap.org/doc/admin24/monitoringslapd.html to activate monitoring interface. - - -#### Install python-ldap module - -Install python ldap module - -1. From pip package manager - -```bash -pip install ldap -``` - -2. With apt package manager (in most deb based distros) - - -```bash -apt-get install python-ldap -``` - - -3. With yum package manager (in most rpm based distros) - - -```bash -yum install python-ldap -``` - - -#### Insert credentials for Netdata to access openLDAP server - -Use the `ldappasswd` utility to set a password for the username you will use. - - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/openldap.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/openldap.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| update_every | Sets the default data collection frequency. | 5 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | -| name | Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. | | no | -| username | The bind user with right to access monitor statistics | | yes | -| password | The password for the binded user | | yes | -| server | The listening address of the LDAP server. In case of TLS, use the hostname which the certificate is published for. | | yes | -| port | The listening port of the LDAP server. Change to 636 port in case of TLS connection. | 389 | yes | -| use_tls | Make True if a TLS connection is used over ldaps:// | no | no | -| use_start_tls | Make True if a TLS connection is used over ldap:// | no | no | -| cert_check | False if you want to ignore certificate check | True | yes | -| timeout | Seconds to timeout if no connection exist | | yes | - -</details> - -#### Examples - -##### Basic - -A basic example configuration. - -```yaml -username: "cn=admin" -password: "pass" -server: "localhost" -port: "389" -check_cert: True -timeout: 1 - -``` - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `openldap` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin openldap debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `openldap` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep openldap -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep openldap /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep openldap -``` - - diff --git a/src/collectors/python.d.plugin/openldap/metadata.yaml b/src/collectors/python.d.plugin/openldap/metadata.yaml deleted file mode 100644 index 3826b22c..00000000 --- a/src/collectors/python.d.plugin/openldap/metadata.yaml +++ /dev/null @@ -1,225 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: openldap - monitored_instance: - name: OpenLDAP - link: "https://www.openldap.org/" - categories: - - data-collection.authentication-and-authorization - icon_filename: "statsd.png" - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: "" - keywords: - - openldap - - RBAC - - Directory access - most_popular: false - overview: - data_collection: - metrics_description: "This collector monitors OpenLDAP metrics about connections, operations, referrals and more." - method_description: | - Statistics are taken from the monitoring interface of a openLDAP (slapd) server - supported_platforms: - include: [] - exclude: [] - multi_instance: false - additional_permissions: - description: "" - default_behavior: - auto_detection: - description: | - This collector doesn't work until all the prerequisites are checked. - limits: - description: "" - performance_impact: - description: "" - setup: - prerequisites: - list: - - title: Configure the openLDAP server to expose metrics to monitor it. - description: | - Follow instructions from https://www.openldap.org/doc/admin24/monitoringslapd.html to activate monitoring interface. - - title: Install python-ldap module - description: | - Install python ldap module - - 1. From pip package manager - - ```bash - pip install ldap - ``` - - 2. With apt package manager (in most deb based distros) - - - ```bash - apt-get install python-ldap - ``` - - - 3. With yum package manager (in most rpm based distros) - - - ```bash - yum install python-ldap - ``` - - title: Insert credentials for Netdata to access openLDAP server - description: | - Use the `ldappasswd` utility to set a password for the username you will use. - configuration: - file: - name: "python.d/openldap.conf" - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: update_every - description: Sets the default data collection frequency. - default_value: 5 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - - name: name - description: Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. - default_value: "" - required: false - - name: username - description: The bind user with right to access monitor statistics - default_value: "" - required: true - - name: password - description: The password for the binded user - default_value: "" - required: true - - name: server - description: The listening address of the LDAP server. In case of TLS, use the hostname which the certificate is published for. - default_value: "" - required: true - - name: port - description: The listening port of the LDAP server. Change to 636 port in case of TLS connection. - default_value: "389" - required: true - - name: use_tls - description: Make True if a TLS connection is used over ldaps:// - default_value: False - required: false - - name: use_start_tls - description: Make True if a TLS connection is used over ldap:// - default_value: False - required: false - - name: cert_check - description: False if you want to ignore certificate check - default_value: "True" - required: true - - name: timeout - description: Seconds to timeout if no connection exist - default_value: "" - required: true - examples: - folding: - enabled: true - title: "Config" - list: - - name: Basic - description: A basic example configuration. - folding: - enabled: false - config: | - username: "cn=admin" - password: "pass" - server: "localhost" - port: "389" - check_cert: True - timeout: 1 - troubleshooting: - problems: - list: [] - alerts: [] - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: openldap.total_connections - description: Total Connections - unit: "connections/s" - chart_type: line - dimensions: - - name: connections - - name: openldap.traffic_stats - description: Traffic - unit: "KiB/s" - chart_type: line - dimensions: - - name: sent - - name: openldap.operations_status - description: Operations Status - unit: "ops/s" - chart_type: line - dimensions: - - name: completed - - name: initiated - - name: openldap.referrals - description: Referrals - unit: "referrals/s" - chart_type: line - dimensions: - - name: sent - - name: openldap.entries - description: Entries - unit: "entries/s" - chart_type: line - dimensions: - - name: sent - - name: openldap.ldap_operations - description: Operations - unit: "ops/s" - chart_type: line - dimensions: - - name: bind - - name: search - - name: unbind - - name: add - - name: delete - - name: modify - - name: compare - - name: openldap.waiters - description: Waiters - unit: "waiters/s" - chart_type: line - dimensions: - - name: write - - name: read diff --git a/src/collectors/python.d.plugin/openldap/openldap.chart.py b/src/collectors/python.d.plugin/openldap/openldap.chart.py deleted file mode 100644 index aba14395..00000000 --- a/src/collectors/python.d.plugin/openldap/openldap.chart.py +++ /dev/null @@ -1,216 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: openldap netdata python.d module -# Author: Manolis Kartsonakis (ekartsonakis) -# SPDX-License-Identifier: GPL-3.0+ - -try: - import ldap - - HAS_LDAP = True -except ImportError: - HAS_LDAP = False - -from bases.FrameworkServices.SimpleService import SimpleService - -DEFAULT_SERVER = 'localhost' -DEFAULT_PORT = '389' -DEFAULT_TLS = False -DEFAULT_CERT_CHECK = True -DEFAULT_TIMEOUT = 1 -DEFAULT_START_TLS = False - -ORDER = [ - 'total_connections', - 'bytes_sent', - 'operations', - 'referrals_sent', - 'entries_sent', - 'ldap_operations', - 'waiters' -] - -CHARTS = { - 'total_connections': { - 'options': [None, 'Total Connections', 'connections/s', 'ldap', 'openldap.total_connections', 'line'], - 'lines': [ - ['total_connections', 'connections', 'incremental'] - ] - }, - 'bytes_sent': { - 'options': [None, 'Traffic', 'KiB/s', 'ldap', 'openldap.traffic_stats', 'line'], - 'lines': [ - ['bytes_sent', 'sent', 'incremental', 1, 1024] - ] - }, - 'operations': { - 'options': [None, 'Operations Status', 'ops/s', 'ldap', 'openldap.operations_status', 'line'], - 'lines': [ - ['completed_operations', 'completed', 'incremental'], - ['initiated_operations', 'initiated', 'incremental'] - ] - }, - 'referrals_sent': { - 'options': [None, 'Referrals', 'referrals/s', 'ldap', 'openldap.referrals', 'line'], - 'lines': [ - ['referrals_sent', 'sent', 'incremental'] - ] - }, - 'entries_sent': { - 'options': [None, 'Entries', 'entries/s', 'ldap', 'openldap.entries', 'line'], - 'lines': [ - ['entries_sent', 'sent', 'incremental'] - ] - }, - 'ldap_operations': { - 'options': [None, 'Operations', 'ops/s', 'ldap', 'openldap.ldap_operations', 'line'], - 'lines': [ - ['bind_operations', 'bind', 'incremental'], - ['search_operations', 'search', 'incremental'], - ['unbind_operations', 'unbind', 'incremental'], - ['add_operations', 'add', 'incremental'], - ['delete_operations', 'delete', 'incremental'], - ['modify_operations', 'modify', 'incremental'], - ['compare_operations', 'compare', 'incremental'] - ] - }, - 'waiters': { - 'options': [None, 'Waiters', 'waiters/s', 'ldap', 'openldap.waiters', 'line'], - 'lines': [ - ['write_waiters', 'write', 'incremental'], - ['read_waiters', 'read', 'incremental'] - ] - }, -} - -# Stuff to gather - make tuples of DN dn and attrib to get -SEARCH_LIST = { - 'total_connections': ( - 'cn=Total,cn=Connections,cn=Monitor', 'monitorCounter', - ), - 'bytes_sent': ( - 'cn=Bytes,cn=Statistics,cn=Monitor', 'monitorCounter', - ), - 'completed_operations': ( - 'cn=Operations,cn=Monitor', 'monitorOpCompleted', - ), - 'initiated_operations': ( - 'cn=Operations,cn=Monitor', 'monitorOpInitiated', - ), - 'referrals_sent': ( - 'cn=Referrals,cn=Statistics,cn=Monitor', 'monitorCounter', - ), - 'entries_sent': ( - 'cn=Entries,cn=Statistics,cn=Monitor', 'monitorCounter', - ), - 'bind_operations': ( - 'cn=Bind,cn=Operations,cn=Monitor', 'monitorOpCompleted', - ), - 'unbind_operations': ( - 'cn=Unbind,cn=Operations,cn=Monitor', 'monitorOpCompleted', - ), - 'add_operations': ( - 'cn=Add,cn=Operations,cn=Monitor', 'monitorOpInitiated', - ), - 'delete_operations': ( - 'cn=Delete,cn=Operations,cn=Monitor', 'monitorOpCompleted', - ), - 'modify_operations': ( - 'cn=Modify,cn=Operations,cn=Monitor', 'monitorOpCompleted', - ), - 'compare_operations': ( - 'cn=Compare,cn=Operations,cn=Monitor', 'monitorOpCompleted', - ), - 'search_operations': ( - 'cn=Search,cn=Operations,cn=Monitor', 'monitorOpCompleted', - ), - 'write_waiters': ( - 'cn=Write,cn=Waiters,cn=Monitor', 'monitorCounter', - ), - 'read_waiters': ( - 'cn=Read,cn=Waiters,cn=Monitor', 'monitorCounter', - ), -} - - -class Service(SimpleService): - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = CHARTS - self.server = configuration.get('server', DEFAULT_SERVER) - self.port = configuration.get('port', DEFAULT_PORT) - self.username = configuration.get('username') - self.password = configuration.get('password') - self.timeout = configuration.get('timeout', DEFAULT_TIMEOUT) - self.use_tls = configuration.get('use_tls', DEFAULT_TLS) - self.cert_check = configuration.get('cert_check', DEFAULT_CERT_CHECK) - self.use_start_tls = configuration.get('use_start_tls', DEFAULT_START_TLS) - self.alive = False - self.conn = None - - def disconnect(self): - if self.conn: - self.conn.unbind() - self.conn = None - self.alive = False - - def connect(self): - try: - if self.use_tls: - self.conn = ldap.initialize('ldaps://%s:%s' % (self.server, self.port)) - else: - self.conn = ldap.initialize('ldap://%s:%s' % (self.server, self.port)) - self.conn.set_option(ldap.OPT_NETWORK_TIMEOUT, self.timeout) - if (self.use_tls or self.use_start_tls) and not self.cert_check: - self.conn.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER) - if self.use_start_tls or self.use_tls: - self.conn.set_option(ldap.OPT_X_TLS_NEWCTX, 0) - if self.use_start_tls: - self.conn.protocol_version = ldap.VERSION3 - self.conn.start_tls_s() - if self.username and self.password: - self.conn.simple_bind(self.username, self.password) - except ldap.LDAPError as error: - self.error(error) - return False - - self.alive = True - return True - - def reconnect(self): - self.disconnect() - return self.connect() - - def check(self): - if not HAS_LDAP: - self.error("'python-ldap' package is needed") - return None - - return self.connect() and self.get_data() - - def get_data(self): - if not self.alive and not self.reconnect(): - return None - - data = dict() - for key in SEARCH_LIST: - dn = SEARCH_LIST[key][0] - attr = SEARCH_LIST[key][1] - try: - num = self.conn.search(dn, ldap.SCOPE_BASE, 'objectClass=*', [attr, ]) - result_type, result_data = self.conn.result(num, 1) - except ldap.LDAPError as error: - self.error("Empty result. Check bind username/password. Message: ", error) - self.alive = False - return None - - if result_type != 101: - continue - - try: - data[key] = int(list(result_data[0][1].values())[0][0]) - except (ValueError, IndexError) as error: - self.debug(error) - continue - - return data diff --git a/src/collectors/python.d.plugin/openldap/openldap.conf b/src/collectors/python.d.plugin/openldap/openldap.conf deleted file mode 100644 index 5fd99a52..00000000 --- a/src/collectors/python.d.plugin/openldap/openldap.conf +++ /dev/null @@ -1,75 +0,0 @@ -# netdata python.d.plugin configuration for openldap -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# postfix is slow, so once every 10 seconds -update_every: 10 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 1 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# ---------------------------------------------------------------------- -# OPENLDAP EXTRA PARAMETERS - -# Set here your LDAP connection settings - -#username : "cn=admin,dc=example,dc=com" # The bind user with right to access monitor statistics -#password : "yourpass" # The password for the binded user -#server : 'localhost' # The listening address of the LDAP server. In case of TLS, use the hostname which the certificate is published for. -#port : 389 # The listening port of the LDAP server. Change to 636 port in case of TLS connection -#use_tls : False # Make True if a TLS connection is used over ldaps:// -#use_start_tls: False # Make True if a TLS connection is used over ldap:// -#cert_check : True # False if you want to ignore certificate check -#timeout : 1 # Seconds to timeout if no connection exi diff --git a/src/collectors/python.d.plugin/oracledb/README.md b/src/collectors/python.d.plugin/oracledb/README.md deleted file mode 120000 index a75e3611..00000000 --- a/src/collectors/python.d.plugin/oracledb/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/oracle_db.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/oracledb/integrations/oracle_db.md b/src/collectors/python.d.plugin/oracledb/integrations/oracle_db.md deleted file mode 100644 index 4cf1b54a..00000000 --- a/src/collectors/python.d.plugin/oracledb/integrations/oracle_db.md +++ /dev/null @@ -1,260 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/oracledb/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/oracledb/metadata.yaml" -sidebar_label: "Oracle DB" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Databases" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# Oracle DB - - -<img src="https://netdata.cloud/img/oracle.svg" width="150"/> - - -Plugin: python.d.plugin -Module: oracledb - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -This collector monitors OracleDB database metrics about sessions, tables, memory and more. - -It collects the metrics via the supported database client library - -This collector is supported on all platforms. - -This collector supports collecting metrics from multiple instances of this integration, including remote instances. - -In order for this collector to work, it needs a read-only user `netdata` in the RDBMS. - - -### Default Behavior - -#### Auto-Detection - -When the requirements are met, databases on the local host on port 1521 will be auto-detected - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - -These metrics refer to the entire monitored application. - -### Per Oracle DB instance - - - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| oracledb.session_count | total, active | sessions | -| oracledb.session_limit_usage | usage | % | -| oracledb.logons | logons | events/s | -| oracledb.physical_disk_read_writes | reads, writes | events/s | -| oracledb.sorts_on_disks | sorts | events/s | -| oracledb.full_table_scans | full table scans | events/s | -| oracledb.database_wait_time_ratio | wait time ratio | % | -| oracledb.shared_pool_free_memory | free memory | % | -| oracledb.in_memory_sorts_ratio | in-memory sorts | % | -| oracledb.sql_service_response_time | time | seconds | -| oracledb.user_rollbacks | rollbacks | events/s | -| oracledb.enqueue_timeouts | enqueue timeouts | events/s | -| oracledb.cache_hit_ration | buffer, cursor, library, row | % | -| oracledb.global_cache_blocks | corrupted, lost | events/s | -| oracledb.activity | parse count, execute count, user commits, user rollbacks | events/s | -| oracledb.wait_time | application, configuration, administrative, concurrency, commit, network, user I/O, system I/O, scheduler, other | ms | -| oracledb.tablespace_size | a dimension per active tablespace | KiB | -| oracledb.tablespace_usage | a dimension per active tablespace | KiB | -| oracledb.tablespace_usage_in_percent | a dimension per active tablespace | % | -| oracledb.allocated_size | a dimension per active tablespace | B | -| oracledb.allocated_usage | a dimension per active tablespace | B | -| oracledb.allocated_usage_in_percent | a dimension per active tablespace | % | - - - -## Alerts - -There are no alerts configured by default for this integration. - - -## Setup - -### Prerequisites - -#### Install the python-oracledb package - -You can follow the official guide below to install the required package: - -Source: https://python-oracledb.readthedocs.io/en/latest/user_guide/installation.html - - -#### Create a read only user for netdata - -Follow the official instructions for your oracle RDBMS to create a read-only user for netdata. The operation may follow this approach - -Connect to your Oracle database with an administrative user and execute: - -```bash -CREATE USER netdata IDENTIFIED BY <PASSWORD>; - -GRANT CONNECT TO netdata; -GRANT SELECT_CATALOG_ROLE TO netdata; -``` - - -#### Edit the configuration - -Edit the configuration troubleshooting: - -1. Provide a valid user for the netdata collector to access the database -2. Specify the network target this database is listening. - - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/oracledb.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/oracledb.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| update_every | Sets the default data collection frequency. | 5 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | -| user | The username for the user account. | no | yes | -| password | The password for the user account. | no | yes | -| server | The IP address or hostname (and port) of the Oracle Database Server. | no | yes | -| service | The Oracle Database service name. To view the services available on your server run this query, `select SERVICE_NAME from gv$session where sid in (select sid from V$MYSTAT)`. | no | yes | -| protocol | one of the strings "tcp" or "tcps" indicating whether to use unencrypted network traffic or encrypted network traffic | no | yes | - -</details> - -#### Examples - -##### Basic - -A basic example configuration, two jobs described for two databases. - -```yaml -local: - user: 'netdata' - password: 'secret' - server: 'localhost:1521' - service: 'XE' - protocol: 'tcps' - -remote: - user: 'netdata' - password: 'secret' - server: '10.0.0.1:1521' - service: 'XE' - protocol: 'tcps' - -``` - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `oracledb` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin oracledb debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `oracledb` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep oracledb -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep oracledb /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep oracledb -``` - - diff --git a/src/collectors/python.d.plugin/oracledb/metadata.yaml b/src/collectors/python.d.plugin/oracledb/metadata.yaml deleted file mode 100644 index f2ab8312..00000000 --- a/src/collectors/python.d.plugin/oracledb/metadata.yaml +++ /dev/null @@ -1,309 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: oracledb - monitored_instance: - name: Oracle DB - link: "https://docs.oracle.com/en/database/oracle/oracle-database/" - categories: - - data-collection.database-servers - icon_filename: "oracle.svg" - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: "" - keywords: - - database - - oracle - - data warehouse - - SQL - most_popular: false - overview: - data_collection: - metrics_description: "This collector monitors OracleDB database metrics about sessions, tables, memory and more." - method_description: "It collects the metrics via the supported database client library" - supported_platforms: - include: [] - exclude: [] - multi_instance: true - additional_permissions: - description: | - In order for this collector to work, it needs a read-only user `netdata` in the RDBMS. - default_behavior: - auto_detection: - description: "When the requirements are met, databases on the local host on port 1521 will be auto-detected" - limits: - description: "" - performance_impact: - description: "" - setup: - prerequisites: - list: - - title: Install the python-oracledb package - description: | - You can follow the official guide below to install the required package: - - Source: https://python-oracledb.readthedocs.io/en/latest/user_guide/installation.html - - title: Create a read only user for netdata - description: | - Follow the official instructions for your oracle RDBMS to create a read-only user for netdata. The operation may follow this approach - - Connect to your Oracle database with an administrative user and execute: - - ```bash - CREATE USER netdata IDENTIFIED BY <PASSWORD>; - - GRANT CONNECT TO netdata; - GRANT SELECT_CATALOG_ROLE TO netdata; - ``` - - title: Edit the configuration - description: | - Edit the configuration troubleshooting: - - 1. Provide a valid user for the netdata collector to access the database - 2. Specify the network target this database is listening. - configuration: - file: - name: "python.d/oracledb.conf" - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: update_every - description: Sets the default data collection frequency. - default_value: 5 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - - name: user - description: The username for the user account. - default_value: no - required: true - - name: password - description: The password for the user account. - default_value: no - required: true - - name: server - description: The IP address or hostname (and port) of the Oracle Database Server. - default_value: no - required: true - - name: service - description: The Oracle Database service name. To view the services available on your server run this query, `select SERVICE_NAME from gv$session where sid in (select sid from V$MYSTAT)`. - default_value: no - required: true - - name: protocol - description: one of the strings "tcp" or "tcps" indicating whether to use unencrypted network traffic or encrypted network traffic - default_value: no - required: true - examples: - folding: - enabled: true - title: "Config" - list: - - name: Basic - folding: - enabled: false - description: A basic example configuration, two jobs described for two databases. - config: | - local: - user: 'netdata' - password: 'secret' - server: 'localhost:1521' - service: 'XE' - protocol: 'tcps' - - remote: - user: 'netdata' - password: 'secret' - server: '10.0.0.1:1521' - service: 'XE' - protocol: 'tcps' - troubleshooting: - problems: - list: [] - alerts: [] - metrics: - folding: - title: Metrics - enabled: false - description: "These metrics refer to the entire monitored application." - availability: [] - scopes: - - name: global - description: "" - labels: [] - metrics: - - name: oracledb.session_count - description: Session Count - unit: "sessions" - chart_type: line - dimensions: - - name: total - - name: active - - name: oracledb.session_limit_usage - description: Session Limit Usage - unit: "%" - chart_type: area - dimensions: - - name: usage - - name: oracledb.logons - description: Logons - unit: "events/s" - chart_type: area - dimensions: - - name: logons - - name: oracledb.physical_disk_read_writes - description: Physical Disk Reads/Writes - unit: "events/s" - chart_type: area - dimensions: - - name: reads - - name: writes - - name: oracledb.sorts_on_disks - description: Sorts On Disk - unit: "events/s" - chart_type: line - dimensions: - - name: sorts - - name: oracledb.full_table_scans - description: Full Table Scans - unit: "events/s" - chart_type: line - dimensions: - - name: full table scans - - name: oracledb.database_wait_time_ratio - description: Database Wait Time Ratio - unit: "%" - chart_type: line - dimensions: - - name: wait time ratio - - name: oracledb.shared_pool_free_memory - description: Shared Pool Free Memory - unit: "%" - chart_type: line - dimensions: - - name: free memory - - name: oracledb.in_memory_sorts_ratio - description: In-Memory Sorts Ratio - unit: "%" - chart_type: line - dimensions: - - name: in-memory sorts - - name: oracledb.sql_service_response_time - description: SQL Service Response Time - unit: "seconds" - chart_type: line - dimensions: - - name: time - - name: oracledb.user_rollbacks - description: User Rollbacks - unit: "events/s" - chart_type: line - dimensions: - - name: rollbacks - - name: oracledb.enqueue_timeouts - description: Enqueue Timeouts - unit: "events/s" - chart_type: line - dimensions: - - name: enqueue timeouts - - name: oracledb.cache_hit_ration - description: Cache Hit Ratio - unit: "%" - chart_type: stacked - dimensions: - - name: buffer - - name: cursor - - name: library - - name: row - - name: oracledb.global_cache_blocks - description: Global Cache Blocks Events - unit: "events/s" - chart_type: area - dimensions: - - name: corrupted - - name: lost - - name: oracledb.activity - description: Activities - unit: "events/s" - chart_type: stacked - dimensions: - - name: parse count - - name: execute count - - name: user commits - - name: user rollbacks - - name: oracledb.wait_time - description: Wait Time - unit: "ms" - chart_type: stacked - dimensions: - - name: application - - name: configuration - - name: administrative - - name: concurrency - - name: commit - - name: network - - name: user I/O - - name: system I/O - - name: scheduler - - name: other - - name: oracledb.tablespace_size - description: Size - unit: "KiB" - chart_type: line - dimensions: - - name: a dimension per active tablespace - - name: oracledb.tablespace_usage - description: Usage - unit: "KiB" - chart_type: line - dimensions: - - name: a dimension per active tablespace - - name: oracledb.tablespace_usage_in_percent - description: Usage - unit: "%" - chart_type: line - dimensions: - - name: a dimension per active tablespace - - name: oracledb.allocated_size - description: Size - unit: "B" - chart_type: line - dimensions: - - name: a dimension per active tablespace - - name: oracledb.allocated_usage - description: Usage - unit: "B" - chart_type: line - dimensions: - - name: a dimension per active tablespace - - name: oracledb.allocated_usage_in_percent - description: Usage - unit: "%" - chart_type: line - dimensions: - - name: a dimension per active tablespace diff --git a/src/collectors/python.d.plugin/oracledb/oracledb.chart.py b/src/collectors/python.d.plugin/oracledb/oracledb.chart.py deleted file mode 100644 index 455cf270..00000000 --- a/src/collectors/python.d.plugin/oracledb/oracledb.chart.py +++ /dev/null @@ -1,846 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: oracledb netdata python.d module -# Author: ilyam8 (Ilya Mashchenko) -# SPDX-License-Identifier: GPL-3.0-or-later - -from copy import deepcopy - -from bases.FrameworkServices.SimpleService import SimpleService - -try: - import oracledb as cx_Oracle - - HAS_ORACLE_NEW = True - HAS_ORACLE_OLD = False -except ImportError: - HAS_ORACLE_NEW = False - try: - import cx_Oracle - - HAS_ORACLE_OLD = True - except ImportError: - HAS_ORACLE_OLD = False - -ORDER = [ - 'session_count', - 'session_limit_usage', - 'logons', - 'physical_disk_read_write', - 'sorts_on_disk', - 'full_table_scans', - 'database_wait_time_ratio', - 'shared_pool_free_memory', - 'in_memory_sorts_ratio', - 'sql_service_response_time', - 'user_rollbacks', - 'enqueue_timeouts', - 'cache_hit_ratio', - 'global_cache_blocks', - 'activity', - 'wait_time', - 'tablespace_size', - 'tablespace_usage', - 'tablespace_usage_in_percent', - 'allocated_size', - 'allocated_usage', - 'allocated_usage_in_percent', -] - -CHARTS = { - 'session_count': { - 'options': [None, 'Session Count', 'sessions', 'session activity', 'oracledb.session_count', 'line'], - 'lines': [ - ['session_count', 'total', 'absolute', 1, 1000], - ['average_active_sessions', 'active', 'absolute', 1, 1000], - ] - }, - 'session_limit_usage': { - 'options': [None, 'Session Limit Usage', '%', 'session activity', 'oracledb.session_limit_usage', 'area'], - 'lines': [ - ['session_limit_percent', 'usage', 'absolute', 1, 1000], - ] - }, - 'logons': { - 'options': [None, 'Logons', 'events/s', 'session activity', 'oracledb.logons', 'area'], - 'lines': [ - ['logons_per_sec', 'logons', 'absolute', 1, 1000], - ] - }, - 'physical_disk_read_write': { - 'options': [None, 'Physical Disk Reads/Writes', 'events/s', 'disk activity', - 'oracledb.physical_disk_read_writes', 'area'], - 'lines': [ - ['physical_reads_per_sec', 'reads', 'absolute', 1, 1000], - ['physical_writes_per_sec', 'writes', 'absolute', -1, 1000], - ] - }, - 'sorts_on_disk': { - 'options': [None, 'Sorts On Disk', 'events/s', 'disk activity', 'oracledb.sorts_on_disks', 'line'], - 'lines': [ - ['disk_sort_per_sec', 'sorts', 'absolute', 1, 1000], - ] - }, - 'full_table_scans': { - 'options': [None, 'Full Table Scans', 'events/s', 'disk activity', 'oracledb.full_table_scans', 'line'], - 'lines': [ - ['long_table_scans_per_sec', 'full table scans', 'absolute', 1, 1000], - ] - }, - 'database_wait_time_ratio': { - 'options': [None, 'Database Wait Time Ratio', '%', 'database and buffer activity', - 'oracledb.database_wait_time_ratio', 'line'], - 'lines': [ - ['database_wait_time_ratio', 'wait time ratio', 'absolute', 1, 1000], - ] - }, - 'shared_pool_free_memory': { - 'options': [None, 'Shared Pool Free Memory', '%', 'database and buffer activity', - 'oracledb.shared_pool_free_memory', 'line'], - 'lines': [ - ['shared_pool_free_percent', 'free memory', 'absolute', 1, 1000], - ] - }, - 'in_memory_sorts_ratio': { - 'options': [None, 'In-Memory Sorts Ratio', '%', 'database and buffer activity', - 'oracledb.in_memory_sorts_ratio', 'line'], - 'lines': [ - ['memory_sorts_ratio', 'in-memory sorts', 'absolute', 1, 1000], - ] - }, - 'sql_service_response_time': { - 'options': [None, 'SQL Service Response Time', 'seconds', 'database and buffer activity', - 'oracledb.sql_service_response_time', 'line'], - 'lines': [ - ['sql_service_response_time', 'time', 'absolute', 1, 1000], - ] - }, - 'user_rollbacks': { - 'options': [None, 'User Rollbacks', 'events/s', 'database and buffer activity', - 'oracledb.user_rollbacks', 'line'], - 'lines': [ - ['user_rollbacks_per_sec', 'rollbacks', 'absolute', 1, 1000], - ] - }, - 'enqueue_timeouts': { - 'options': [None, 'Enqueue Timeouts', 'events/s', 'database and buffer activity', - 'oracledb.enqueue_timeouts', 'line'], - 'lines': [ - ['enqueue_timeouts_per_sec', 'enqueue timeouts', 'absolute', 1, 1000], - ] - }, - 'cache_hit_ratio': { - 'options': [None, 'Cache Hit Ratio', '%', 'cache', 'oracledb.cache_hit_ration', 'stacked'], - 'lines': [ - ['buffer_cache_hit_ratio', 'buffer', 'absolute', 1, 1000], - ['cursor_cache_hit_ratio', 'cursor', 'absolute', 1, 1000], - ['library_cache_hit_ratio', 'library', 'absolute', 1, 1000], - ['row_cache_hit_ratio', 'row', 'absolute', 1, 1000], - ] - }, - 'global_cache_blocks': { - 'options': [None, 'Global Cache Blocks Events', 'events/s', 'cache', 'oracledb.global_cache_blocks', 'area'], - 'lines': [ - ['global_cache_blocks_corrupted', 'corrupted', 'incremental', 1, 1000], - ['global_cache_blocks_lost', 'lost', 'incremental', 1, 1000], - ] - }, - 'activity': { - 'options': [None, 'Activities', 'events/s', 'activities', 'oracledb.activity', 'stacked'], - 'lines': [ - ['activity_parse_count_total', 'parse count', 'incremental', 1, 1000], - ['activity_execute_count', 'execute count', 'incremental', 1, 1000], - ['activity_user_commits', 'user commits', 'incremental', 1, 1000], - ['activity_user_rollbacks', 'user rollbacks', 'incremental', 1, 1000], - ] - }, - 'wait_time': { - 'options': [None, 'Wait Time', 'ms', 'wait time', 'oracledb.wait_time', 'stacked'], - 'lines': [ - ['wait_time_application', 'application', 'absolute', 1, 1000], - ['wait_time_configuration', 'configuration', 'absolute', 1, 1000], - ['wait_time_administrative', 'administrative', 'absolute', 1, 1000], - ['wait_time_concurrency', 'concurrency', 'absolute', 1, 1000], - ['wait_time_commit', 'commit', 'absolute', 1, 1000], - ['wait_time_network', 'network', 'absolute', 1, 1000], - ['wait_time_user_io', 'user I/O', 'absolute', 1, 1000], - ['wait_time_system_io', 'system I/O', 'absolute', 1, 1000], - ['wait_time_scheduler', 'scheduler', 'absolute', 1, 1000], - ['wait_time_other', 'other', 'absolute', 1, 1000], - ] - }, - 'tablespace_size': { - 'options': [None, 'Size', 'KiB', 'tablespace', 'oracledb.tablespace_size', 'line'], - 'lines': [], - }, - 'tablespace_usage': { - 'options': [None, 'Usage', 'KiB', 'tablespace', 'oracledb.tablespace_usage', 'line'], - 'lines': [], - }, - 'tablespace_usage_in_percent': { - 'options': [None, 'Usage', '%', 'tablespace', 'oracledb.tablespace_usage_in_percent', 'line'], - 'lines': [], - }, - 'allocated_size': { - 'options': [None, 'Size', 'B', 'tablespace', 'oracledb.allocated_size', 'line'], - 'lines': [], - }, - 'allocated_usage': { - 'options': [None, 'Usage', 'B', 'tablespace', 'oracledb.allocated_usage', 'line'], - 'lines': [], - }, - 'allocated_usage_in_percent': { - 'options': [None, 'Usage', '%', 'tablespace', 'oracledb.allocated_usage_in_percent', 'line'], - 'lines': [], - }, -} - -CX_CONNECT_STRING_OLD = "{0}/{1}@//{2}/{3}" - -QUERY_SYSTEM = ''' -SELECT - metric_name, - value -FROM - gv$sysmetric -ORDER BY - begin_time -''' -QUERY_TABLESPACE = ''' -SELECT - m.tablespace_name, - m.used_space * t.block_size AS used_bytes, - m.tablespace_size * t.block_size AS max_bytes, - m.used_percent -FROM - dba_tablespace_usage_metrics m - JOIN dba_tablespaces t ON m.tablespace_name = t.tablespace_name -''' -QUERY_ALLOCATED = ''' -SELECT - nvl(b.tablespace_name,nvl(a.tablespace_name,'UNKNOWN')) tablespace_name, - bytes_alloc used_bytes, - bytes_alloc-nvl(bytes_free,0) max_bytes, - ((bytes_alloc-nvl(bytes_free,0))/ bytes_alloc)*100 used_percent -FROM - (SELECT - sum(bytes) bytes_free, - tablespace_name - FROM sys.dba_free_space - GROUP BY tablespace_name - ) a, - (SELECT - sum(bytes) bytes_alloc, - tablespace_name - FROM sys.dba_data_files - GROUP BY tablespace_name - ) b -WHERE a.tablespace_name (+) = b.tablespace_name -''' -QUERY_ACTIVITIES_COUNT = ''' -SELECT - name, - value -FROM - v$sysstat -WHERE - name IN ( - 'parse count (total)', - 'execute count', - 'user commits', - 'user rollbacks' - ) -''' -QUERY_WAIT_TIME = ''' -SELECT - n.wait_class, - round(m.time_waited / m.INTSIZE_CSEC, 3) -FROM - v$waitclassmetric m, - v$system_wait_class n -WHERE - m.wait_class_id = n.wait_class_id - AND n.wait_class != 'Idle' -''' -# QUERY_SESSION_COUNT = ''' -# SELECT -# status, -# type -# FROM -# v$session -# GROUP BY -# status, -# type -# ''' -# QUERY_PROCESSES_COUNT = ''' -# SELECT -# COUNT(*) -# FROM -# v$process -# ''' -# QUERY_PROCESS = ''' -# SELECT -# program, -# pga_used_mem, -# pga_alloc_mem, -# pga_freeable_mem, -# pga_max_mem -# FROM -# gv$process -# ''' - -# PROCESS_METRICS = [ -# 'pga_used_memory', -# 'pga_allocated_memory', -# 'pga_freeable_memory', -# 'pga_maximum_memory', -# ] - - -SYS_METRICS = { - 'Average Active Sessions': 'average_active_sessions', - 'Session Count': 'session_count', - 'Session Limit %': 'session_limit_percent', - 'Logons Per Sec': 'logons_per_sec', - 'Physical Reads Per Sec': 'physical_reads_per_sec', - 'Physical Writes Per Sec': 'physical_writes_per_sec', - 'Disk Sort Per Sec': 'disk_sort_per_sec', - 'Long Table Scans Per Sec': 'long_table_scans_per_sec', - 'Database Wait Time Ratio': 'database_wait_time_ratio', - 'Shared Pool Free %': 'shared_pool_free_percent', - 'Memory Sorts Ratio': 'memory_sorts_ratio', - 'SQL Service Response Time': 'sql_service_response_time', - 'User Rollbacks Per Sec': 'user_rollbacks_per_sec', - 'Enqueue Timeouts Per Sec': 'enqueue_timeouts_per_sec', - 'Buffer Cache Hit Ratio': 'buffer_cache_hit_ratio', - 'Cursor Cache Hit Ratio': 'cursor_cache_hit_ratio', - 'Library Cache Hit Ratio': 'library_cache_hit_ratio', - 'Row Cache Hit Ratio': 'row_cache_hit_ratio', - 'Global Cache Blocks Corrupted': 'global_cache_blocks_corrupted', - 'Global Cache Blocks Lost': 'global_cache_blocks_lost', -} - - -class Service(SimpleService): - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = deepcopy(CHARTS) - self.user = configuration.get('user') - self.password = configuration.get('password') - self.server = configuration.get('server') - self.service = configuration.get('service') - self.protocol = configuration.get('protocol', 'tcps') - self.alive = False - self.conn = None - self.active_tablespaces = set() - - def connect(self): - if self.conn: - self.conn.close() - self.conn = None - if HAS_ORACLE_NEW: - try: - self.conn = cx_Oracle.connect( - f'{self.user}/{self.password}@{self.protocol}://{self.server}/{self.service}') - except cx_Oracle.DatabaseError as error: - self.error(error) - return False - else: - try: - self.conn = cx_Oracle.connect( - CX_CONNECT_STRING_OLD.format( - self.user, - self.password, - self.server, - self.service, - )) - except cx_Oracle.DatabaseError as error: - self.error(error) - return False - - self.alive = True - return True - - def reconnect(self): - return self.connect() - - def check(self): - if not HAS_ORACLE_NEW and not HAS_ORACLE_OLD: - self.error("'oracledb' package is needed to use oracledb module") - return False - - if not all([ - self.user, - self.password, - self.server, - self.service - ]): - self.error("one of these parameters is not specified: user, password, server, service") - return False - - if not self.connect(): - return False - - return bool(self.get_data()) - - def get_data(self): - if not self.alive and not self.reconnect(): - return None - - data = dict() - - # SYSTEM - try: - rv = self.gather_system_metrics() - except cx_Oracle.Error as error: - self.error(error) - self.alive = False - return None - else: - for name, value in rv: - if name not in SYS_METRICS: - continue - data[SYS_METRICS[name]] = int(float(value) * 1000) - - # ACTIVITIES COUNT - try: - rv = self.gather_activities_count() - except cx_Oracle.Error as error: - self.error(error) - self.alive = False - return None - else: - for name, amount in rv: - cleaned = name.replace(' ', '_').replace('(', '').replace(')', '') - new_name = 'activity_{0}'.format(cleaned) - data[new_name] = int(float(amount) * 1000) - - # WAIT TIME - try: - rv = self.gather_wait_time_metrics() - except cx_Oracle.Error as error: - self.error(error) - self.alive = False - return None - else: - for name, amount in rv: - cleaned = name.replace(' ', '_').replace('/', '').lower() - new_name = 'wait_time_{0}'.format(cleaned) - data[new_name] = amount - - # TABLESPACE - try: - rv = self.gather_tablespace_metrics() - except cx_Oracle.Error as error: - self.error(error) - self.alive = False - return None - else: - for name, offline, size, used, used_in_percent in rv: - # TODO: skip offline? - if not (not offline and self.charts): - continue - # TODO: remove inactive? - if name not in self.active_tablespaces: - self.active_tablespaces.add(name) - self.add_tablespace_to_charts(name) - data['{0}_tablespace_size'.format(name)] = int(size * 1000) - data['{0}_tablespace_used'.format(name)] = int(used * 1000) - data['{0}_tablespace_used_in_percent'.format(name)] = int(used_in_percent * 1000) - - # ALLOCATED SPACE - try: - rv = self.gather_allocated_metrics() - except cx_Oracle.Error as error: - self.error(error) - self.alive = False - return None - else: - for name, offline, size, used, used_in_percent in rv: - # TODO: skip offline? - if not (not offline and self.charts): - continue - # TODO: remove inactive? - if name not in self.active_tablespaces: - self.active_tablespaces.add(name) - self.add_tablespace_to_charts(name) - data['{0}_allocated_size'.format(name)] = int(size * 1000) - data['{0}_allocated_used'.format(name)] = int(used * 1000) - data['{0}_allocated_used_in_percent'.format(name)] = int(used_in_percent * 1000) - - return data or None - - def gather_system_metrics(self): - - """ - :return: - - [['Buffer Cache Hit Ratio', 100], - ['Memory Sorts Ratio', 100], - ['Redo Allocation Hit Ratio', 100], - ['User Transaction Per Sec', 0], - ['Physical Reads Per Sec', 0], - ['Physical Reads Per Txn', 0], - ['Physical Writes Per Sec', 0], - ['Physical Writes Per Txn', 0], - ['Physical Reads Direct Per Sec', 0], - ['Physical Reads Direct Per Txn', 0], - ['Physical Writes Direct Per Sec', 0], - ['Physical Writes Direct Per Txn', 0], - ['Physical Reads Direct Lobs Per Sec', 0], - ['Physical Reads Direct Lobs Per Txn', 0], - ['Physical Writes Direct Lobs Per Sec', 0], - ['Physical Writes Direct Lobs Per Txn', 0], - ['Redo Generated Per Sec', Decimal('4.66666666666667')], - ['Redo Generated Per Txn', 280], - ['Logons Per Sec', Decimal('0.0166666666666667')], - ['Logons Per Txn', 1], - ['Open Cursors Per Sec', 0.35], - ['Open Cursors Per Txn', 21], - ['User Commits Per Sec', 0], - ['User Commits Percentage', 0], - ['User Rollbacks Per Sec', 0], - ['User Rollbacks Percentage', 0], - ['User Calls Per Sec', Decimal('0.0333333333333333')], - ['User Calls Per Txn', 2], - ['Recursive Calls Per Sec', 14.15], - ['Recursive Calls Per Txn', 849], - ['Logical Reads Per Sec', Decimal('0.683333333333333')], - ['Logical Reads Per Txn', 41], - ['DBWR Checkpoints Per Sec', 0], - ['Background Checkpoints Per Sec', 0], - ['Redo Writes Per Sec', Decimal('0.0333333333333333')], - ['Redo Writes Per Txn', 2], - ['Long Table Scans Per Sec', 0], - ['Long Table Scans Per Txn', 0], - ['Total Table Scans Per Sec', Decimal('0.0166666666666667')], - ['Total Table Scans Per Txn', 1], - ['Full Index Scans Per Sec', 0], - ['Full Index Scans Per Txn', 0], - ['Total Index Scans Per Sec', Decimal('0.216666666666667')], - ['Total Index Scans Per Txn', 13], - ['Total Parse Count Per Sec', 0.35], - ['Total Parse Count Per Txn', 21], - ['Hard Parse Count Per Sec', 0], - ['Hard Parse Count Per Txn', 0], - ['Parse Failure Count Per Sec', 0], - ['Parse Failure Count Per Txn', 0], - ['Cursor Cache Hit Ratio', Decimal('52.3809523809524')], - ['Disk Sort Per Sec', 0], - ['Disk Sort Per Txn', 0], - ['Rows Per Sort', 8.6], - ['Execute Without Parse Ratio', Decimal('27.5862068965517')], - ['Soft Parse Ratio', 100], - ['User Calls Ratio', Decimal('0.235017626321974')], - ['Host CPU Utilization (%)', Decimal('0.124311845142959')], - ['Network Traffic Volume Per Sec', 0], - ['Enqueue Timeouts Per Sec', 0], - ['Enqueue Timeouts Per Txn', 0], - ['Enqueue Waits Per Sec', 0], - ['Enqueue Waits Per Txn', 0], - ['Enqueue Deadlocks Per Sec', 0], - ['Enqueue Deadlocks Per Txn', 0], - ['Enqueue Requests Per Sec', Decimal('216.683333333333')], - ['Enqueue Requests Per Txn', 13001], - ['DB Block Gets Per Sec', 0], - ['DB Block Gets Per Txn', 0], - ['Consistent Read Gets Per Sec', Decimal('0.683333333333333')], - ['Consistent Read Gets Per Txn', 41], - ['DB Block Changes Per Sec', 0], - ['DB Block Changes Per Txn', 0], - ['Consistent Read Changes Per Sec', 0], - ['Consistent Read Changes Per Txn', 0], - ['CPU Usage Per Sec', 0], - ['CPU Usage Per Txn', 0], - ['CR Blocks Created Per Sec', 0], - ['CR Blocks Created Per Txn', 0], - ['CR Undo Records Applied Per Sec', 0], - ['CR Undo Records Applied Per Txn', 0], - ['User Rollback UndoRec Applied Per Sec', 0], - ['User Rollback Undo Records Applied Per Txn', 0], - ['Leaf Node Splits Per Sec', 0], - ['Leaf Node Splits Per Txn', 0], - ['Branch Node Splits Per Sec', 0], - ['Branch Node Splits Per Txn', 0], - ['PX downgraded 1 to 25% Per Sec', 0], - ['PX downgraded 25 to 50% Per Sec', 0], - ['PX downgraded 50 to 75% Per Sec', 0], - ['PX downgraded 75 to 99% Per Sec', 0], - ['PX downgraded to serial Per Sec', 0], - ['Physical Read Total IO Requests Per Sec', Decimal('2.16666666666667')], - ['Physical Read Total Bytes Per Sec', Decimal('35498.6666666667')], - ['GC CR Block Received Per Second', 0], - ['GC CR Block Received Per Txn', 0], - ['GC Current Block Received Per Second', 0], - ['GC Current Block Received Per Txn', 0], - ['Global Cache Average CR Get Time', 0], - ['Global Cache Average Current Get Time', 0], - ['Physical Write Total IO Requests Per Sec', Decimal('0.966666666666667')], - ['Global Cache Blocks Corrupted', 0], - ['Global Cache Blocks Lost', 0], - ['Current Logons Count', 49], - ['Current Open Cursors Count', 64], - ['User Limit %', Decimal('0.00000114087015416959')], - ['SQL Service Response Time', 0], - ['Database Wait Time Ratio', 0], - ['Database CPU Time Ratio', 0], - ['Response Time Per Txn', 0], - ['Row Cache Hit Ratio', 100], - ['Row Cache Miss Ratio', 0], - ['Library Cache Hit Ratio', 100], - ['Library Cache Miss Ratio', 0], - ['Shared Pool Free %', Decimal('7.82380268491548')], - ['PGA Cache Hit %', Decimal('98.0399767109115')], - ['Process Limit %', Decimal('17.6666666666667')], - ['Session Limit %', Decimal('15.2542372881356')], - ['Executions Per Txn', 29], - ['Executions Per Sec', Decimal('0.483333333333333')], - ['Txns Per Logon', 0], - ['Database Time Per Sec', 0], - ['Physical Write Total Bytes Per Sec', 15308.8], - ['Physical Read IO Requests Per Sec', 0], - ['Physical Read Bytes Per Sec', 0], - ['Physical Write IO Requests Per Sec', 0], - ['Physical Write Bytes Per Sec', 0], - ['DB Block Changes Per User Call', 0], - ['DB Block Gets Per User Call', 0], - ['Executions Per User Call', 14.5], - ['Logical Reads Per User Call', 20.5], - ['Total Sorts Per User Call', 2.5], - ['Total Table Scans Per User Call', 0.5], - ['Current OS Load', 0.0390625], - ['Streams Pool Usage Percentage', 0], - ['PQ QC Session Count', 0], - ['PQ Slave Session Count', 0], - ['Queries parallelized Per Sec', 0], - ['DML statements parallelized Per Sec', 0], - ['DDL statements parallelized Per Sec', 0], - ['PX operations not downgraded Per Sec', 0], - ['Session Count', 72], - ['Average Synchronous Single-Block Read Latency', 0], - ['I/O Megabytes per Second', 0.05], - ['I/O Requests per Second', Decimal('3.13333333333333')], - ['Average Active Sessions', 0], - ['Active Serial Sessions', 1], - ['Active Parallel Sessions', 0], - ['Captured user calls', 0], - ['Replayed user calls', 0], - ['Workload Capture and Replay status', 0], - ['Background CPU Usage Per Sec', Decimal('1.22578833333333')], - ['Background Time Per Sec', 0.0147551], - ['Host CPU Usage Per Sec', Decimal('0.116666666666667')], - ['Cell Physical IO Interconnect Bytes', 3048448], - ['Temp Space Used', 0], - ['Total PGA Allocated', 200657920], - ['Total PGA Used by SQL Workareas', 0], - ['Run Queue Per Sec', 0], - ['VM in bytes Per Sec', 0], - ['VM out bytes Per Sec', 0]] - """ - - metrics = list() - with self.conn.cursor() as cursor: - cursor.execute(QUERY_SYSTEM) - for metric_name, value in cursor.fetchall(): - metrics.append([metric_name, value]) - return metrics - - def gather_tablespace_metrics(self): - """ - :return: - - [['SYSTEM', 874250240.0, 3233169408.0, 27.040038107400033, 0], - ['SYSAUX', 498860032.0, 3233169408.0, 15.429443033997678, 0], - ['TEMP', 0.0, 3233177600.0, 0.0, 0], - ['USERS', 1048576.0, 3233169408.0, 0.03243182981397305, 0]] - """ - metrics = list() - with self.conn.cursor() as cursor: - cursor.execute(QUERY_TABLESPACE) - for tablespace_name, used_bytes, max_bytes, used_percent in cursor.fetchall(): - if used_bytes is None: - offline = True - used = 0 - else: - offline = False - used = float(used_bytes) - if max_bytes is None: - size = 0 - else: - size = float(max_bytes) - if used_percent is None: - used_percent = 0 - else: - used_percent = float(used_percent) - metrics.append( - [ - tablespace_name, - offline, - size, - used, - used_percent, - ] - ) - return metrics - - def gather_allocated_metrics(self): - """ - :return: - - [['SYSTEM', 874250240.0, 3233169408.0, 27.040038107400033, 0], - ['SYSAUX', 498860032.0, 3233169408.0, 15.429443033997678, 0], - ['TEMP', 0.0, 3233177600.0, 0.0, 0], - ['USERS', 1048576.0, 3233169408.0, 0.03243182981397305, 0]] - """ - metrics = list() - with self.conn.cursor() as cursor: - cursor.execute(QUERY_ALLOCATED) - for tablespace_name, used_bytes, max_bytes, used_percent in cursor.fetchall(): - if used_bytes is None: - offline = True - used = 0 - else: - offline = False - used = float(used_bytes) - if max_bytes is None: - size = 0 - else: - size = float(max_bytes) - if used_percent is None: - used_percent = 0 - else: - used_percent = float(used_percent) - metrics.append( - [ - tablespace_name, - offline, - size, - used, - used_percent, - ] - ) - return metrics - - def gather_wait_time_metrics(self): - """ - :return: - - [['Other', 0], - ['Application', 0], - ['Configuration', 0], - ['Administrative', 0], - ['Concurrency', 0], - ['Commit', 0], - ['Network', 0], - ['User I/O', 0], - ['System I/O', 0.002], - ['Scheduler', 0]] - """ - metrics = list() - with self.conn.cursor() as cursor: - cursor.execute(QUERY_WAIT_TIME) - for wait_class_name, value in cursor.fetchall(): - metrics.append([wait_class_name, value]) - return metrics - - def gather_activities_count(self): - """ - :return: - - [('user commits', 9104), - ('user rollbacks', 17), - ('parse count (total)', 483695), - ('execute count', 2020356)] - """ - with self.conn.cursor() as cursor: - cursor.execute(QUERY_ACTIVITIES_COUNT) - return cursor.fetchall() - - # def gather_process_metrics(self): - # """ - # :return: - # - # [['PSEUDO', 'pga_used_memory', 0], - # ['PSEUDO', 'pga_allocated_memory', 0], - # ['PSEUDO', 'pga_freeable_memory', 0], - # ['PSEUDO', 'pga_maximum_memory', 0], - # ['oracle@localhost.localdomain (PMON)', 'pga_used_memory', 1793827], - # ['oracle@localhost.localdomain (PMON)', 'pga_allocated_memory', 1888651], - # ['oracle@localhost.localdomain (PMON)', 'pga_freeable_memory', 0], - # ['oracle@localhost.localdomain (PMON)', 'pga_maximum_memory', 1888651], - # ... - # ... - # """ - # - # metrics = list() - # with self.conn.cursor() as cursor: - # cursor.execute(QUERY_PROCESS) - # for row in cursor.fetchall(): - # for i, name in enumerate(PROCESS_METRICS, 1): - # metrics.append([row[0], name, row[i]]) - # return metrics - - # def gather_processes_count(self): - # with self.conn.cursor() as cursor: - # cursor.execute(QUERY_PROCESSES_COUNT) - # return cursor.fetchone()[0] # 53 - - # def gather_sessions_count(self): - # with self.conn.cursor() as cursor: - # cursor.execute(QUERY_SESSION_COUNT) - # total, active, inactive = 0, 0, 0 - # for status, _ in cursor.fetchall(): - # total += 1 - # active += status == 'ACTIVE' - # inactive += status == 'INACTIVE' - # return [total, active, inactive] - - def add_tablespace_to_charts(self, name): - self.charts['tablespace_size'].add_dimension( - [ - '{0}_tablespace_size'.format(name), - name, - 'absolute', - 1, - 1024 * 1000, - ]) - self.charts['tablespace_usage'].add_dimension( - [ - '{0}_tablespace_used'.format(name), - name, - 'absolute', - 1, - 1024 * 1000, - ]) - self.charts['tablespace_usage_in_percent'].add_dimension( - [ - '{0}_tablespace_used_in_percent'.format(name), - name, - 'absolute', - 1, - 1000, - ]) - self.charts['allocated_size'].add_dimension( - [ - '{0}_allocated_size'.format(name), - name, - 'absolute', - 1, - 1000, - ]) - self.charts['allocated_usage'].add_dimension( - [ - '{0}_allocated_used'.format(name), - name, - 'absolute', - 1, - 1000, - ]) - self.charts['allocated_usage_in_percent'].add_dimension( - [ - '{0}_allocated_used_in_percent'.format(name), - name, - 'absolute', - 1, - 1000, - ]) diff --git a/src/collectors/python.d.plugin/oracledb/oracledb.conf b/src/collectors/python.d.plugin/oracledb/oracledb.conf deleted file mode 100644 index 027215da..00000000 --- a/src/collectors/python.d.plugin/oracledb/oracledb.conf +++ /dev/null @@ -1,88 +0,0 @@ -# netdata python.d.plugin configuration for oracledb -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# update_every: 1 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 1 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# Additionally to the above, oracledb also supports the following: -# -# user: username # the username for the user account. Required. -# password: password # the password for the user account. Required. -# server: localhost:1521 # the IP address or hostname (and port) of the Oracle Database Server. Required. -# service: XE # the Oracle Database service name. Required. To view the services available on your server, -# run this query: `select SERVICE_NAME from gv$session where sid in (select sid from V$MYSTAT)`. -# protocol: tcp/tcps # one of the strings "tcp" or "tcps" indicating whether to use unencrypted network traffic -# or encrypted network traffic -# -# ---------------------------------------------------------------------- -# AUTO-DETECTION JOBS -# only one of them will run (they have the same name) - -#local: -# user: 'netdata' -# password: 'secret' -# server: 'localhost:1521' -# service: 'XE' -# protocol: 'tcps' - -#remote: -# user: 'netdata' -# password: 'secret' -# server: '10.0.0.1:1521' -# service: 'XE' -# protocol: 'tcps' diff --git a/src/collectors/python.d.plugin/pandas/integrations/pandas.md b/src/collectors/python.d.plugin/pandas/integrations/pandas.md index e0b5418c..b36bea07 100644 --- a/src/collectors/python.d.plugin/pandas/integrations/pandas.md +++ b/src/collectors/python.d.plugin/pandas/integrations/pandas.md @@ -108,8 +108,8 @@ sudo pip install 'sqlalchemy<2.0' psycopg2-binary The configuration file name for this integration is `python.d/pandas.conf`. -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script from the +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). ```bash cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata @@ -136,8 +136,8 @@ Every configuration JOB starts with a `job_name` value which will appear in the | chart_configs | an array of chart configuration dictionaries | [] | yes | | chart_configs.name | name of the chart to be displayed in the dashboard. | None | yes | | chart_configs.title | title of the chart to be displayed in the dashboard. | None | yes | -| chart_configs.family | [family](/docs/dashboards-and-charts/netdata-charts.md#families) of the chart to be displayed in the dashboard. | None | yes | -| chart_configs.context | [context](/docs/dashboards-and-charts/netdata-charts.md#contexts) of the chart to be displayed in the dashboard. | None | yes | +| chart_configs.family | [family](https://github.com/netdata/netdata/blob/master/docs/dashboards-and-charts/netdata-charts.md#families) of the chart to be displayed in the dashboard. | None | yes | +| chart_configs.context | [context](https://github.com/netdata/netdata/blob/master/docs/dashboards-and-charts/netdata-charts.md#contexts) of the chart to be displayed in the dashboard. | None | yes | | chart_configs.type | the type of the chart to be displayed in the dashboard. | None | yes | | chart_configs.units | the units of the chart to be displayed in the dashboard. | None | yes | | chart_configs.df_steps | a series of pandas operations (one per line) that each returns a dataframe. | None | yes | diff --git a/src/collectors/python.d.plugin/python.d.conf b/src/collectors/python.d.plugin/python.d.conf index 4fcecc75..e2ce1347 100644 --- a/src/collectors/python.d.plugin/python.d.conf +++ b/src/collectors/python.d.plugin/python.d.conf @@ -26,30 +26,23 @@ gc_run: yes gc_interval: 300 # am2320: yes -# anomalies: no -# boinc: yes -# ceph: yes # this is just an example go_expvar: no # haproxy: yes -# openldap: yes -# oracledb: yes # pandas: yes # retroshare: yes -# samba: yes # smartd_log: yes -# spigotmc: yes # traefik: yes # varnish: yes -# w1sensor: yes -# zscores: no ## Disabled for existing installations. adaptec_raid: no # Removed (replaced with go.d/adaptercraid). apache: no # Removed (replaced with go.d/apache). beanstalk: no # Removed (replaced with go.d/beanstalk). +boinc: no # Removed (replaced with go.d/boinc). dovecot: no # Removed (replaced with go.d/dovecot). +ceph: no # Removed (replaced with go.d/ceph). elasticsearch: no # Removed (replaced with go.d/elasticsearch). exim: no # Removed (replaced with go.d/exim). fail2ban: no # Removed (replaced with go.d/fail2ban). @@ -68,15 +61,21 @@ mysql: no # Removed (replaced with go.d/mysql). nginx: no # Removed (replaced with go.d/nginx). nsd: no # Removed (replaced with go.d/nsd). nvidia_smi: no # Removed (replaced with go.d/nvidia_smi). +openldap: no # Removed (replaced with go.d/openldap). +oracledb: no # Removed (replaced with go.d/oracledb). postfix: no # Removed (replaced with go.d/postfix). postgres: no # Removed (replaced with go.d/postgres). proxysql: no # Removed (replaced with go.d/proxysql). redis: no # Removed (replaced with go.d/redis). rethinkdbs: no # Removed (replaced with go.d/rethinkdb). riakkv: no # Removed (replaced with go.d/riak). +samba: no # Removed (replaced with go.d/samba). sensors: no # Removed (replaced with go.d/sensors). squid: no # Removed (replaced with go.d/squid). +spigotmc: no # Removed (replaced with go.d/spigotmc). tomcat: no # Removed (replaced with go.d/tomcat) tor: no # Removed (replaced with go.d/tor). puppet: no # Removed (replaced with go.d/puppet). uwsgi: no # Removed (replaced with go.d/uwsgi). +varnish: no # Removed (replaced with go.d/varnish). +w1sensor: no # Removed (replaced with go.d/w1sensor) diff --git a/src/collectors/python.d.plugin/python.d.plugin.in b/src/collectors/python.d.plugin/python.d.plugin.in index 81e68f94..089fb5a5 100644 --- a/src/collectors/python.d.plugin/python.d.plugin.in +++ b/src/collectors/python.d.plugin/python.d.plugin.in @@ -12,7 +12,8 @@ do done if [ "$pybinary" = "" ] then - echo "ERROR python IS NOT AVAILABLE IN THIS SYSTEM" + echo 1>&2 "python.d ERROR: python is not installed on this system" + echo "EXIT" exit 1 fi exec "$pybinary" "$0" "${filtered[@]}" # ''' diff --git a/src/collectors/python.d.plugin/python_modules/bases/loaders.py b/src/collectors/python.d.plugin/python_modules/bases/loaders.py index 095f3a3b..6ffa2189 100644 --- a/src/collectors/python.d.plugin/python_modules/bases/loaders.py +++ b/src/collectors/python.d.plugin/python_modules/bases/loaders.py @@ -3,27 +3,17 @@ # Author: Ilya Mashchenko (ilyam8) # SPDX-License-Identifier: GPL-3.0-or-later - -from sys import version_info - -PY_VERSION = version_info[:2] - try: - if PY_VERSION > (3, 1): - from pyyaml3 import SafeLoader as YamlSafeLoader - else: - from pyyaml2 import SafeLoader as YamlSafeLoader + from pyyaml3 import SafeLoader as YamlSafeLoader except ImportError: from yaml import SafeLoader as YamlSafeLoader - try: from collections import OrderedDict except ImportError: from third_party.ordereddict import OrderedDict - -DEFAULT_MAPPING_TAG = 'tag:yaml.org,2002:map' if PY_VERSION > (3, 1) else u'tag:yaml.org,2002:map' +DEFAULT_MAPPING_TAG = 'tag:yaml.org,2002:map' def dict_constructor(loader, node): diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/__init__.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/__init__.py deleted file mode 100644 index 4d560e43..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/__init__.py +++ /dev/null @@ -1,316 +0,0 @@ -# SPDX-License-Identifier: MIT - -from error import * - -from tokens import * -from events import * -from nodes import * - -from loader import * -from dumper import * - -__version__ = '3.11' - -try: - from cyaml import * - __with_libyaml__ = True -except ImportError: - __with_libyaml__ = False - -def scan(stream, Loader=Loader): - """ - Scan a YAML stream and produce scanning tokens. - """ - loader = Loader(stream) - try: - while loader.check_token(): - yield loader.get_token() - finally: - loader.dispose() - -def parse(stream, Loader=Loader): - """ - Parse a YAML stream and produce parsing events. - """ - loader = Loader(stream) - try: - while loader.check_event(): - yield loader.get_event() - finally: - loader.dispose() - -def compose(stream, Loader=Loader): - """ - Parse the first YAML document in a stream - and produce the corresponding representation tree. - """ - loader = Loader(stream) - try: - return loader.get_single_node() - finally: - loader.dispose() - -def compose_all(stream, Loader=Loader): - """ - Parse all YAML documents in a stream - and produce corresponding representation trees. - """ - loader = Loader(stream) - try: - while loader.check_node(): - yield loader.get_node() - finally: - loader.dispose() - -def load(stream, Loader=Loader): - """ - Parse the first YAML document in a stream - and produce the corresponding Python object. - """ - loader = Loader(stream) - try: - return loader.get_single_data() - finally: - loader.dispose() - -def load_all(stream, Loader=Loader): - """ - Parse all YAML documents in a stream - and produce corresponding Python objects. - """ - loader = Loader(stream) - try: - while loader.check_data(): - yield loader.get_data() - finally: - loader.dispose() - -def safe_load(stream): - """ - Parse the first YAML document in a stream - and produce the corresponding Python object. - Resolve only basic YAML tags. - """ - return load(stream, SafeLoader) - -def safe_load_all(stream): - """ - Parse all YAML documents in a stream - and produce corresponding Python objects. - Resolve only basic YAML tags. - """ - return load_all(stream, SafeLoader) - -def emit(events, stream=None, Dumper=Dumper, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None): - """ - Emit YAML parsing events into a stream. - If stream is None, return the produced string instead. - """ - getvalue = None - if stream is None: - from StringIO import StringIO - stream = StringIO() - getvalue = stream.getvalue - dumper = Dumper(stream, canonical=canonical, indent=indent, width=width, - allow_unicode=allow_unicode, line_break=line_break) - try: - for event in events: - dumper.emit(event) - finally: - dumper.dispose() - if getvalue: - return getvalue() - -def serialize_all(nodes, stream=None, Dumper=Dumper, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding='utf-8', explicit_start=None, explicit_end=None, - version=None, tags=None): - """ - Serialize a sequence of representation trees into a YAML stream. - If stream is None, return the produced string instead. - """ - getvalue = None - if stream is None: - if encoding is None: - from StringIO import StringIO - else: - from cStringIO import StringIO - stream = StringIO() - getvalue = stream.getvalue - dumper = Dumper(stream, canonical=canonical, indent=indent, width=width, - allow_unicode=allow_unicode, line_break=line_break, - encoding=encoding, version=version, tags=tags, - explicit_start=explicit_start, explicit_end=explicit_end) - try: - dumper.open() - for node in nodes: - dumper.serialize(node) - dumper.close() - finally: - dumper.dispose() - if getvalue: - return getvalue() - -def serialize(node, stream=None, Dumper=Dumper, **kwds): - """ - Serialize a representation tree into a YAML stream. - If stream is None, return the produced string instead. - """ - return serialize_all([node], stream, Dumper=Dumper, **kwds) - -def dump_all(documents, stream=None, Dumper=Dumper, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding='utf-8', explicit_start=None, explicit_end=None, - version=None, tags=None): - """ - Serialize a sequence of Python objects into a YAML stream. - If stream is None, return the produced string instead. - """ - getvalue = None - if stream is None: - if encoding is None: - from StringIO import StringIO - else: - from cStringIO import StringIO - stream = StringIO() - getvalue = stream.getvalue - dumper = Dumper(stream, default_style=default_style, - default_flow_style=default_flow_style, - canonical=canonical, indent=indent, width=width, - allow_unicode=allow_unicode, line_break=line_break, - encoding=encoding, version=version, tags=tags, - explicit_start=explicit_start, explicit_end=explicit_end) - try: - dumper.open() - for data in documents: - dumper.represent(data) - dumper.close() - finally: - dumper.dispose() - if getvalue: - return getvalue() - -def dump(data, stream=None, Dumper=Dumper, **kwds): - """ - Serialize a Python object into a YAML stream. - If stream is None, return the produced string instead. - """ - return dump_all([data], stream, Dumper=Dumper, **kwds) - -def safe_dump_all(documents, stream=None, **kwds): - """ - Serialize a sequence of Python objects into a YAML stream. - Produce only basic YAML tags. - If stream is None, return the produced string instead. - """ - return dump_all(documents, stream, Dumper=SafeDumper, **kwds) - -def safe_dump(data, stream=None, **kwds): - """ - Serialize a Python object into a YAML stream. - Produce only basic YAML tags. - If stream is None, return the produced string instead. - """ - return dump_all([data], stream, Dumper=SafeDumper, **kwds) - -def add_implicit_resolver(tag, regexp, first=None, - Loader=Loader, Dumper=Dumper): - """ - Add an implicit scalar detector. - If an implicit scalar value matches the given regexp, - the corresponding tag is assigned to the scalar. - first is a sequence of possible initial characters or None. - """ - Loader.add_implicit_resolver(tag, regexp, first) - Dumper.add_implicit_resolver(tag, regexp, first) - -def add_path_resolver(tag, path, kind=None, Loader=Loader, Dumper=Dumper): - """ - Add a path based resolver for the given tag. - A path is a list of keys that forms a path - to a node in the representation tree. - Keys can be string values, integers, or None. - """ - Loader.add_path_resolver(tag, path, kind) - Dumper.add_path_resolver(tag, path, kind) - -def add_constructor(tag, constructor, Loader=Loader): - """ - Add a constructor for the given tag. - Constructor is a function that accepts a Loader instance - and a node object and produces the corresponding Python object. - """ - Loader.add_constructor(tag, constructor) - -def add_multi_constructor(tag_prefix, multi_constructor, Loader=Loader): - """ - Add a multi-constructor for the given tag prefix. - Multi-constructor is called for a node if its tag starts with tag_prefix. - Multi-constructor accepts a Loader instance, a tag suffix, - and a node object and produces the corresponding Python object. - """ - Loader.add_multi_constructor(tag_prefix, multi_constructor) - -def add_representer(data_type, representer, Dumper=Dumper): - """ - Add a representer for the given type. - Representer is a function accepting a Dumper instance - and an instance of the given data type - and producing the corresponding representation node. - """ - Dumper.add_representer(data_type, representer) - -def add_multi_representer(data_type, multi_representer, Dumper=Dumper): - """ - Add a representer for the given type. - Multi-representer is a function accepting a Dumper instance - and an instance of the given data type or subtype - and producing the corresponding representation node. - """ - Dumper.add_multi_representer(data_type, multi_representer) - -class YAMLObjectMetaclass(type): - """ - The metaclass for YAMLObject. - """ - def __init__(cls, name, bases, kwds): - super(YAMLObjectMetaclass, cls).__init__(name, bases, kwds) - if 'yaml_tag' in kwds and kwds['yaml_tag'] is not None: - cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml) - cls.yaml_dumper.add_representer(cls, cls.to_yaml) - -class YAMLObject(object): - """ - An object that can dump itself to a YAML stream - and load itself from a YAML stream. - """ - - __metaclass__ = YAMLObjectMetaclass - __slots__ = () # no direct instantiation, so allow immutable subclasses - - yaml_loader = Loader - yaml_dumper = Dumper - - yaml_tag = None - yaml_flow_style = None - - def from_yaml(cls, loader, node): - """ - Convert a representation node to a Python object. - """ - return loader.construct_yaml_object(node, cls) - from_yaml = classmethod(from_yaml) - - def to_yaml(cls, dumper, data): - """ - Convert a Python object to a representation node. - """ - return dumper.represent_yaml_object(cls.yaml_tag, data, cls, - flow_style=cls.yaml_flow_style) - to_yaml = classmethod(to_yaml) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/composer.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/composer.py deleted file mode 100644 index 6b41b806..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/composer.py +++ /dev/null @@ -1,140 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['Composer', 'ComposerError'] - -from error import MarkedYAMLError -from events import * -from nodes import * - -class ComposerError(MarkedYAMLError): - pass - -class Composer(object): - - def __init__(self): - self.anchors = {} - - def check_node(self): - # Drop the STREAM-START event. - if self.check_event(StreamStartEvent): - self.get_event() - - # If there are more documents available? - return not self.check_event(StreamEndEvent) - - def get_node(self): - # Get the root node of the next document. - if not self.check_event(StreamEndEvent): - return self.compose_document() - - def get_single_node(self): - # Drop the STREAM-START event. - self.get_event() - - # Compose a document if the stream is not empty. - document = None - if not self.check_event(StreamEndEvent): - document = self.compose_document() - - # Ensure that the stream contains no more documents. - if not self.check_event(StreamEndEvent): - event = self.get_event() - raise ComposerError("expected a single document in the stream", - document.start_mark, "but found another document", - event.start_mark) - - # Drop the STREAM-END event. - self.get_event() - - return document - - def compose_document(self): - # Drop the DOCUMENT-START event. - self.get_event() - - # Compose the root node. - node = self.compose_node(None, None) - - # Drop the DOCUMENT-END event. - self.get_event() - - self.anchors = {} - return node - - def compose_node(self, parent, index): - if self.check_event(AliasEvent): - event = self.get_event() - anchor = event.anchor - if anchor not in self.anchors: - raise ComposerError(None, None, "found undefined alias %r" - % anchor.encode('utf-8'), event.start_mark) - return self.anchors[anchor] - event = self.peek_event() - anchor = event.anchor - if anchor is not None: - if anchor in self.anchors: - raise ComposerError("found duplicate anchor %r; first occurence" - % anchor.encode('utf-8'), self.anchors[anchor].start_mark, - "second occurence", event.start_mark) - self.descend_resolver(parent, index) - if self.check_event(ScalarEvent): - node = self.compose_scalar_node(anchor) - elif self.check_event(SequenceStartEvent): - node = self.compose_sequence_node(anchor) - elif self.check_event(MappingStartEvent): - node = self.compose_mapping_node(anchor) - self.ascend_resolver() - return node - - def compose_scalar_node(self, anchor): - event = self.get_event() - tag = event.tag - if tag is None or tag == u'!': - tag = self.resolve(ScalarNode, event.value, event.implicit) - node = ScalarNode(tag, event.value, - event.start_mark, event.end_mark, style=event.style) - if anchor is not None: - self.anchors[anchor] = node - return node - - def compose_sequence_node(self, anchor): - start_event = self.get_event() - tag = start_event.tag - if tag is None or tag == u'!': - tag = self.resolve(SequenceNode, None, start_event.implicit) - node = SequenceNode(tag, [], - start_event.start_mark, None, - flow_style=start_event.flow_style) - if anchor is not None: - self.anchors[anchor] = node - index = 0 - while not self.check_event(SequenceEndEvent): - node.value.append(self.compose_node(node, index)) - index += 1 - end_event = self.get_event() - node.end_mark = end_event.end_mark - return node - - def compose_mapping_node(self, anchor): - start_event = self.get_event() - tag = start_event.tag - if tag is None or tag == u'!': - tag = self.resolve(MappingNode, None, start_event.implicit) - node = MappingNode(tag, [], - start_event.start_mark, None, - flow_style=start_event.flow_style) - if anchor is not None: - self.anchors[anchor] = node - while not self.check_event(MappingEndEvent): - #key_event = self.peek_event() - item_key = self.compose_node(node, None) - #if item_key in node.value: - # raise ComposerError("while composing a mapping", start_event.start_mark, - # "found duplicate key", key_event.start_mark) - item_value = self.compose_node(node, item_key) - #node.value[item_key] = item_value - node.value.append((item_key, item_value)) - end_event = self.get_event() - node.end_mark = end_event.end_mark - return node - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/constructor.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/constructor.py deleted file mode 100644 index 8ad1b90a..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/constructor.py +++ /dev/null @@ -1,676 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['BaseConstructor', 'SafeConstructor', 'Constructor', - 'ConstructorError'] - -from error import * -from nodes import * - -import datetime - -import binascii, re, sys, types - -class ConstructorError(MarkedYAMLError): - pass - -class BaseConstructor(object): - - yaml_constructors = {} - yaml_multi_constructors = {} - - def __init__(self): - self.constructed_objects = {} - self.recursive_objects = {} - self.state_generators = [] - self.deep_construct = False - - def check_data(self): - # If there are more documents available? - return self.check_node() - - def get_data(self): - # Construct and return the next document. - if self.check_node(): - return self.construct_document(self.get_node()) - - def get_single_data(self): - # Ensure that the stream contains a single document and construct it. - node = self.get_single_node() - if node is not None: - return self.construct_document(node) - return None - - def construct_document(self, node): - data = self.construct_object(node) - while self.state_generators: - state_generators = self.state_generators - self.state_generators = [] - for generator in state_generators: - for dummy in generator: - pass - self.constructed_objects = {} - self.recursive_objects = {} - self.deep_construct = False - return data - - def construct_object(self, node, deep=False): - if node in self.constructed_objects: - return self.constructed_objects[node] - if deep: - old_deep = self.deep_construct - self.deep_construct = True - if node in self.recursive_objects: - raise ConstructorError(None, None, - "found unconstructable recursive node", node.start_mark) - self.recursive_objects[node] = None - constructor = None - tag_suffix = None - if node.tag in self.yaml_constructors: - constructor = self.yaml_constructors[node.tag] - else: - for tag_prefix in self.yaml_multi_constructors: - if node.tag.startswith(tag_prefix): - tag_suffix = node.tag[len(tag_prefix):] - constructor = self.yaml_multi_constructors[tag_prefix] - break - else: - if None in self.yaml_multi_constructors: - tag_suffix = node.tag - constructor = self.yaml_multi_constructors[None] - elif None in self.yaml_constructors: - constructor = self.yaml_constructors[None] - elif isinstance(node, ScalarNode): - constructor = self.__class__.construct_scalar - elif isinstance(node, SequenceNode): - constructor = self.__class__.construct_sequence - elif isinstance(node, MappingNode): - constructor = self.__class__.construct_mapping - if tag_suffix is None: - data = constructor(self, node) - else: - data = constructor(self, tag_suffix, node) - if isinstance(data, types.GeneratorType): - generator = data - data = generator.next() - if self.deep_construct: - for dummy in generator: - pass - else: - self.state_generators.append(generator) - self.constructed_objects[node] = data - del self.recursive_objects[node] - if deep: - self.deep_construct = old_deep - return data - - def construct_scalar(self, node): - if not isinstance(node, ScalarNode): - raise ConstructorError(None, None, - "expected a scalar node, but found %s" % node.id, - node.start_mark) - return node.value - - def construct_sequence(self, node, deep=False): - if not isinstance(node, SequenceNode): - raise ConstructorError(None, None, - "expected a sequence node, but found %s" % node.id, - node.start_mark) - return [self.construct_object(child, deep=deep) - for child in node.value] - - def construct_mapping(self, node, deep=False): - if not isinstance(node, MappingNode): - raise ConstructorError(None, None, - "expected a mapping node, but found %s" % node.id, - node.start_mark) - mapping = {} - for key_node, value_node in node.value: - key = self.construct_object(key_node, deep=deep) - try: - hash(key) - except TypeError, exc: - raise ConstructorError("while constructing a mapping", node.start_mark, - "found unacceptable key (%s)" % exc, key_node.start_mark) - value = self.construct_object(value_node, deep=deep) - mapping[key] = value - return mapping - - def construct_pairs(self, node, deep=False): - if not isinstance(node, MappingNode): - raise ConstructorError(None, None, - "expected a mapping node, but found %s" % node.id, - node.start_mark) - pairs = [] - for key_node, value_node in node.value: - key = self.construct_object(key_node, deep=deep) - value = self.construct_object(value_node, deep=deep) - pairs.append((key, value)) - return pairs - - def add_constructor(cls, tag, constructor): - if not 'yaml_constructors' in cls.__dict__: - cls.yaml_constructors = cls.yaml_constructors.copy() - cls.yaml_constructors[tag] = constructor - add_constructor = classmethod(add_constructor) - - def add_multi_constructor(cls, tag_prefix, multi_constructor): - if not 'yaml_multi_constructors' in cls.__dict__: - cls.yaml_multi_constructors = cls.yaml_multi_constructors.copy() - cls.yaml_multi_constructors[tag_prefix] = multi_constructor - add_multi_constructor = classmethod(add_multi_constructor) - -class SafeConstructor(BaseConstructor): - - def construct_scalar(self, node): - if isinstance(node, MappingNode): - for key_node, value_node in node.value: - if key_node.tag == u'tag:yaml.org,2002:value': - return self.construct_scalar(value_node) - return BaseConstructor.construct_scalar(self, node) - - def flatten_mapping(self, node): - merge = [] - index = 0 - while index < len(node.value): - key_node, value_node = node.value[index] - if key_node.tag == u'tag:yaml.org,2002:merge': - del node.value[index] - if isinstance(value_node, MappingNode): - self.flatten_mapping(value_node) - merge.extend(value_node.value) - elif isinstance(value_node, SequenceNode): - submerge = [] - for subnode in value_node.value: - if not isinstance(subnode, MappingNode): - raise ConstructorError("while constructing a mapping", - node.start_mark, - "expected a mapping for merging, but found %s" - % subnode.id, subnode.start_mark) - self.flatten_mapping(subnode) - submerge.append(subnode.value) - submerge.reverse() - for value in submerge: - merge.extend(value) - else: - raise ConstructorError("while constructing a mapping", node.start_mark, - "expected a mapping or list of mappings for merging, but found %s" - % value_node.id, value_node.start_mark) - elif key_node.tag == u'tag:yaml.org,2002:value': - key_node.tag = u'tag:yaml.org,2002:str' - index += 1 - else: - index += 1 - if merge: - node.value = merge + node.value - - def construct_mapping(self, node, deep=False): - if isinstance(node, MappingNode): - self.flatten_mapping(node) - return BaseConstructor.construct_mapping(self, node, deep=deep) - - def construct_yaml_null(self, node): - self.construct_scalar(node) - return None - - bool_values = { - u'yes': True, - u'no': False, - u'true': True, - u'false': False, - u'on': True, - u'off': False, - } - - def construct_yaml_bool(self, node): - value = self.construct_scalar(node) - return self.bool_values[value.lower()] - - def construct_yaml_int(self, node): - value = str(self.construct_scalar(node)) - value = value.replace('_', '') - sign = +1 - if value[0] == '-': - sign = -1 - if value[0] in '+-': - value = value[1:] - if value == '0': - return 0 - elif value.startswith('0b'): - return sign*int(value[2:], 2) - elif value.startswith('0x'): - return sign*int(value[2:], 16) - elif value[0] == '0': - return sign*int(value, 8) - elif ':' in value: - digits = [int(part) for part in value.split(':')] - digits.reverse() - base = 1 - value = 0 - for digit in digits: - value += digit*base - base *= 60 - return sign*value - else: - return sign*int(value) - - inf_value = 1e300 - while inf_value != inf_value*inf_value: - inf_value *= inf_value - nan_value = -inf_value/inf_value # Trying to make a quiet NaN (like C99). - - def construct_yaml_float(self, node): - value = str(self.construct_scalar(node)) - value = value.replace('_', '').lower() - sign = +1 - if value[0] == '-': - sign = -1 - if value[0] in '+-': - value = value[1:] - if value == '.inf': - return sign*self.inf_value - elif value == '.nan': - return self.nan_value - elif ':' in value: - digits = [float(part) for part in value.split(':')] - digits.reverse() - base = 1 - value = 0.0 - for digit in digits: - value += digit*base - base *= 60 - return sign*value - else: - return sign*float(value) - - def construct_yaml_binary(self, node): - value = self.construct_scalar(node) - try: - return str(value).decode('base64') - except (binascii.Error, UnicodeEncodeError), exc: - raise ConstructorError(None, None, - "failed to decode base64 data: %s" % exc, node.start_mark) - - timestamp_regexp = re.compile( - ur'''^(?P<year>[0-9][0-9][0-9][0-9]) - -(?P<month>[0-9][0-9]?) - -(?P<day>[0-9][0-9]?) - (?:(?:[Tt]|[ \t]+) - (?P<hour>[0-9][0-9]?) - :(?P<minute>[0-9][0-9]) - :(?P<second>[0-9][0-9]) - (?:\.(?P<fraction>[0-9]*))? - (?:[ \t]*(?P<tz>Z|(?P<tz_sign>[-+])(?P<tz_hour>[0-9][0-9]?) - (?::(?P<tz_minute>[0-9][0-9]))?))?)?$''', re.X) - - def construct_yaml_timestamp(self, node): - value = self.construct_scalar(node) - match = self.timestamp_regexp.match(node.value) - values = match.groupdict() - year = int(values['year']) - month = int(values['month']) - day = int(values['day']) - if not values['hour']: - return datetime.date(year, month, day) - hour = int(values['hour']) - minute = int(values['minute']) - second = int(values['second']) - fraction = 0 - if values['fraction']: - fraction = values['fraction'][:6] - while len(fraction) < 6: - fraction += '0' - fraction = int(fraction) - delta = None - if values['tz_sign']: - tz_hour = int(values['tz_hour']) - tz_minute = int(values['tz_minute'] or 0) - delta = datetime.timedelta(hours=tz_hour, minutes=tz_minute) - if values['tz_sign'] == '-': - delta = -delta - data = datetime.datetime(year, month, day, hour, minute, second, fraction) - if delta: - data -= delta - return data - - def construct_yaml_omap(self, node): - # Note: we do not check for duplicate keys, because it's too - # CPU-expensive. - omap = [] - yield omap - if not isinstance(node, SequenceNode): - raise ConstructorError("while constructing an ordered map", node.start_mark, - "expected a sequence, but found %s" % node.id, node.start_mark) - for subnode in node.value: - if not isinstance(subnode, MappingNode): - raise ConstructorError("while constructing an ordered map", node.start_mark, - "expected a mapping of length 1, but found %s" % subnode.id, - subnode.start_mark) - if len(subnode.value) != 1: - raise ConstructorError("while constructing an ordered map", node.start_mark, - "expected a single mapping item, but found %d items" % len(subnode.value), - subnode.start_mark) - key_node, value_node = subnode.value[0] - key = self.construct_object(key_node) - value = self.construct_object(value_node) - omap.append((key, value)) - - def construct_yaml_pairs(self, node): - # Note: the same code as `construct_yaml_omap`. - pairs = [] - yield pairs - if not isinstance(node, SequenceNode): - raise ConstructorError("while constructing pairs", node.start_mark, - "expected a sequence, but found %s" % node.id, node.start_mark) - for subnode in node.value: - if not isinstance(subnode, MappingNode): - raise ConstructorError("while constructing pairs", node.start_mark, - "expected a mapping of length 1, but found %s" % subnode.id, - subnode.start_mark) - if len(subnode.value) != 1: - raise ConstructorError("while constructing pairs", node.start_mark, - "expected a single mapping item, but found %d items" % len(subnode.value), - subnode.start_mark) - key_node, value_node = subnode.value[0] - key = self.construct_object(key_node) - value = self.construct_object(value_node) - pairs.append((key, value)) - - def construct_yaml_set(self, node): - data = set() - yield data - value = self.construct_mapping(node) - data.update(value) - - def construct_yaml_str(self, node): - value = self.construct_scalar(node) - try: - return value.encode('ascii') - except UnicodeEncodeError: - return value - - def construct_yaml_seq(self, node): - data = [] - yield data - data.extend(self.construct_sequence(node)) - - def construct_yaml_map(self, node): - data = {} - yield data - value = self.construct_mapping(node) - data.update(value) - - def construct_yaml_object(self, node, cls): - data = cls.__new__(cls) - yield data - if hasattr(data, '__setstate__'): - state = self.construct_mapping(node, deep=True) - data.__setstate__(state) - else: - state = self.construct_mapping(node) - data.__dict__.update(state) - - def construct_undefined(self, node): - raise ConstructorError(None, None, - "could not determine a constructor for the tag %r" % node.tag.encode('utf-8'), - node.start_mark) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:null', - SafeConstructor.construct_yaml_null) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:bool', - SafeConstructor.construct_yaml_bool) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:int', - SafeConstructor.construct_yaml_int) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:float', - SafeConstructor.construct_yaml_float) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:binary', - SafeConstructor.construct_yaml_binary) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:timestamp', - SafeConstructor.construct_yaml_timestamp) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:omap', - SafeConstructor.construct_yaml_omap) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:pairs', - SafeConstructor.construct_yaml_pairs) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:set', - SafeConstructor.construct_yaml_set) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:str', - SafeConstructor.construct_yaml_str) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:seq', - SafeConstructor.construct_yaml_seq) - -SafeConstructor.add_constructor( - u'tag:yaml.org,2002:map', - SafeConstructor.construct_yaml_map) - -SafeConstructor.add_constructor(None, - SafeConstructor.construct_undefined) - -class Constructor(SafeConstructor): - - def construct_python_str(self, node): - return self.construct_scalar(node).encode('utf-8') - - def construct_python_unicode(self, node): - return self.construct_scalar(node) - - def construct_python_long(self, node): - return long(self.construct_yaml_int(node)) - - def construct_python_complex(self, node): - return complex(self.construct_scalar(node)) - - def construct_python_tuple(self, node): - return tuple(self.construct_sequence(node)) - - def find_python_module(self, name, mark): - if not name: - raise ConstructorError("while constructing a Python module", mark, - "expected non-empty name appended to the tag", mark) - try: - __import__(name) - except ImportError, exc: - raise ConstructorError("while constructing a Python module", mark, - "cannot find module %r (%s)" % (name.encode('utf-8'), exc), mark) - return sys.modules[name] - - def find_python_name(self, name, mark): - if not name: - raise ConstructorError("while constructing a Python object", mark, - "expected non-empty name appended to the tag", mark) - if u'.' in name: - module_name, object_name = name.rsplit('.', 1) - else: - module_name = '__builtin__' - object_name = name - try: - __import__(module_name) - except ImportError, exc: - raise ConstructorError("while constructing a Python object", mark, - "cannot find module %r (%s)" % (module_name.encode('utf-8'), exc), mark) - module = sys.modules[module_name] - if not hasattr(module, object_name): - raise ConstructorError("while constructing a Python object", mark, - "cannot find %r in the module %r" % (object_name.encode('utf-8'), - module.__name__), mark) - return getattr(module, object_name) - - def construct_python_name(self, suffix, node): - value = self.construct_scalar(node) - if value: - raise ConstructorError("while constructing a Python name", node.start_mark, - "expected the empty value, but found %r" % value.encode('utf-8'), - node.start_mark) - return self.find_python_name(suffix, node.start_mark) - - def construct_python_module(self, suffix, node): - value = self.construct_scalar(node) - if value: - raise ConstructorError("while constructing a Python module", node.start_mark, - "expected the empty value, but found %r" % value.encode('utf-8'), - node.start_mark) - return self.find_python_module(suffix, node.start_mark) - - class classobj: pass - - def make_python_instance(self, suffix, node, - args=None, kwds=None, newobj=False): - if not args: - args = [] - if not kwds: - kwds = {} - cls = self.find_python_name(suffix, node.start_mark) - if newobj and isinstance(cls, type(self.classobj)) \ - and not args and not kwds: - instance = self.classobj() - instance.__class__ = cls - return instance - elif newobj and isinstance(cls, type): - return cls.__new__(cls, *args, **kwds) - else: - return cls(*args, **kwds) - - def set_python_instance_state(self, instance, state): - if hasattr(instance, '__setstate__'): - instance.__setstate__(state) - else: - slotstate = {} - if isinstance(state, tuple) and len(state) == 2: - state, slotstate = state - if hasattr(instance, '__dict__'): - instance.__dict__.update(state) - elif state: - slotstate.update(state) - for key, value in slotstate.items(): - setattr(object, key, value) - - def construct_python_object(self, suffix, node): - # Format: - # !!python/object:module.name { ... state ... } - instance = self.make_python_instance(suffix, node, newobj=True) - yield instance - deep = hasattr(instance, '__setstate__') - state = self.construct_mapping(node, deep=deep) - self.set_python_instance_state(instance, state) - - def construct_python_object_apply(self, suffix, node, newobj=False): - # Format: - # !!python/object/apply # (or !!python/object/new) - # args: [ ... arguments ... ] - # kwds: { ... keywords ... } - # state: ... state ... - # listitems: [ ... listitems ... ] - # dictitems: { ... dictitems ... } - # or short format: - # !!python/object/apply [ ... arguments ... ] - # The difference between !!python/object/apply and !!python/object/new - # is how an object is created, check make_python_instance for details. - if isinstance(node, SequenceNode): - args = self.construct_sequence(node, deep=True) - kwds = {} - state = {} - listitems = [] - dictitems = {} - else: - value = self.construct_mapping(node, deep=True) - args = value.get('args', []) - kwds = value.get('kwds', {}) - state = value.get('state', {}) - listitems = value.get('listitems', []) - dictitems = value.get('dictitems', {}) - instance = self.make_python_instance(suffix, node, args, kwds, newobj) - if state: - self.set_python_instance_state(instance, state) - if listitems: - instance.extend(listitems) - if dictitems: - for key in dictitems: - instance[key] = dictitems[key] - return instance - - def construct_python_object_new(self, suffix, node): - return self.construct_python_object_apply(suffix, node, newobj=True) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/none', - Constructor.construct_yaml_null) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/bool', - Constructor.construct_yaml_bool) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/str', - Constructor.construct_python_str) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/unicode', - Constructor.construct_python_unicode) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/int', - Constructor.construct_yaml_int) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/long', - Constructor.construct_python_long) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/float', - Constructor.construct_yaml_float) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/complex', - Constructor.construct_python_complex) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/list', - Constructor.construct_yaml_seq) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/tuple', - Constructor.construct_python_tuple) - -Constructor.add_constructor( - u'tag:yaml.org,2002:python/dict', - Constructor.construct_yaml_map) - -Constructor.add_multi_constructor( - u'tag:yaml.org,2002:python/name:', - Constructor.construct_python_name) - -Constructor.add_multi_constructor( - u'tag:yaml.org,2002:python/module:', - Constructor.construct_python_module) - -Constructor.add_multi_constructor( - u'tag:yaml.org,2002:python/object:', - Constructor.construct_python_object) - -Constructor.add_multi_constructor( - u'tag:yaml.org,2002:python/object/apply:', - Constructor.construct_python_object_apply) - -Constructor.add_multi_constructor( - u'tag:yaml.org,2002:python/object/new:', - Constructor.construct_python_object_new) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/cyaml.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/cyaml.py deleted file mode 100644 index 2858ab47..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/cyaml.py +++ /dev/null @@ -1,86 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['CBaseLoader', 'CSafeLoader', 'CLoader', - 'CBaseDumper', 'CSafeDumper', 'CDumper'] - -from _yaml import CParser, CEmitter - -from constructor import * - -from serializer import * -from representer import * - -from resolver import * - -class CBaseLoader(CParser, BaseConstructor, BaseResolver): - - def __init__(self, stream): - CParser.__init__(self, stream) - BaseConstructor.__init__(self) - BaseResolver.__init__(self) - -class CSafeLoader(CParser, SafeConstructor, Resolver): - - def __init__(self, stream): - CParser.__init__(self, stream) - SafeConstructor.__init__(self) - Resolver.__init__(self) - -class CLoader(CParser, Constructor, Resolver): - - def __init__(self, stream): - CParser.__init__(self, stream) - Constructor.__init__(self) - Resolver.__init__(self) - -class CBaseDumper(CEmitter, BaseRepresenter, BaseResolver): - - def __init__(self, stream, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding=None, explicit_start=None, explicit_end=None, - version=None, tags=None): - CEmitter.__init__(self, stream, canonical=canonical, - indent=indent, width=width, encoding=encoding, - allow_unicode=allow_unicode, line_break=line_break, - explicit_start=explicit_start, explicit_end=explicit_end, - version=version, tags=tags) - Representer.__init__(self, default_style=default_style, - default_flow_style=default_flow_style) - Resolver.__init__(self) - -class CSafeDumper(CEmitter, SafeRepresenter, Resolver): - - def __init__(self, stream, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding=None, explicit_start=None, explicit_end=None, - version=None, tags=None): - CEmitter.__init__(self, stream, canonical=canonical, - indent=indent, width=width, encoding=encoding, - allow_unicode=allow_unicode, line_break=line_break, - explicit_start=explicit_start, explicit_end=explicit_end, - version=version, tags=tags) - SafeRepresenter.__init__(self, default_style=default_style, - default_flow_style=default_flow_style) - Resolver.__init__(self) - -class CDumper(CEmitter, Serializer, Representer, Resolver): - - def __init__(self, stream, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding=None, explicit_start=None, explicit_end=None, - version=None, tags=None): - CEmitter.__init__(self, stream, canonical=canonical, - indent=indent, width=width, encoding=encoding, - allow_unicode=allow_unicode, line_break=line_break, - explicit_start=explicit_start, explicit_end=explicit_end, - version=version, tags=tags) - Representer.__init__(self, default_style=default_style, - default_flow_style=default_flow_style) - Resolver.__init__(self) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/dumper.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/dumper.py deleted file mode 100644 index 3685cbee..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/dumper.py +++ /dev/null @@ -1,63 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['BaseDumper', 'SafeDumper', 'Dumper'] - -from emitter import * -from serializer import * -from representer import * -from resolver import * - -class BaseDumper(Emitter, Serializer, BaseRepresenter, BaseResolver): - - def __init__(self, stream, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding=None, explicit_start=None, explicit_end=None, - version=None, tags=None): - Emitter.__init__(self, stream, canonical=canonical, - indent=indent, width=width, - allow_unicode=allow_unicode, line_break=line_break) - Serializer.__init__(self, encoding=encoding, - explicit_start=explicit_start, explicit_end=explicit_end, - version=version, tags=tags) - Representer.__init__(self, default_style=default_style, - default_flow_style=default_flow_style) - Resolver.__init__(self) - -class SafeDumper(Emitter, Serializer, SafeRepresenter, Resolver): - - def __init__(self, stream, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding=None, explicit_start=None, explicit_end=None, - version=None, tags=None): - Emitter.__init__(self, stream, canonical=canonical, - indent=indent, width=width, - allow_unicode=allow_unicode, line_break=line_break) - Serializer.__init__(self, encoding=encoding, - explicit_start=explicit_start, explicit_end=explicit_end, - version=version, tags=tags) - SafeRepresenter.__init__(self, default_style=default_style, - default_flow_style=default_flow_style) - Resolver.__init__(self) - -class Dumper(Emitter, Serializer, Representer, Resolver): - - def __init__(self, stream, - default_style=None, default_flow_style=None, - canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None, - encoding=None, explicit_start=None, explicit_end=None, - version=None, tags=None): - Emitter.__init__(self, stream, canonical=canonical, - indent=indent, width=width, - allow_unicode=allow_unicode, line_break=line_break) - Serializer.__init__(self, encoding=encoding, - explicit_start=explicit_start, explicit_end=explicit_end, - version=version, tags=tags) - Representer.__init__(self, default_style=default_style, - default_flow_style=default_flow_style) - Resolver.__init__(self) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/emitter.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/emitter.py deleted file mode 100644 index 9a460a0f..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/emitter.py +++ /dev/null @@ -1,1141 +0,0 @@ -# SPDX-License-Identifier: MIT - -# Emitter expects events obeying the following grammar: -# stream ::= STREAM-START document* STREAM-END -# document ::= DOCUMENT-START node DOCUMENT-END -# node ::= SCALAR | sequence | mapping -# sequence ::= SEQUENCE-START node* SEQUENCE-END -# mapping ::= MAPPING-START (node node)* MAPPING-END - -__all__ = ['Emitter', 'EmitterError'] - -from error import YAMLError -from events import * - -class EmitterError(YAMLError): - pass - -class ScalarAnalysis(object): - def __init__(self, scalar, empty, multiline, - allow_flow_plain, allow_block_plain, - allow_single_quoted, allow_double_quoted, - allow_block): - self.scalar = scalar - self.empty = empty - self.multiline = multiline - self.allow_flow_plain = allow_flow_plain - self.allow_block_plain = allow_block_plain - self.allow_single_quoted = allow_single_quoted - self.allow_double_quoted = allow_double_quoted - self.allow_block = allow_block - -class Emitter(object): - - DEFAULT_TAG_PREFIXES = { - u'!' : u'!', - u'tag:yaml.org,2002:' : u'!!', - } - - def __init__(self, stream, canonical=None, indent=None, width=None, - allow_unicode=None, line_break=None): - - # The stream should have the methods `write` and possibly `flush`. - self.stream = stream - - # Encoding can be overriden by STREAM-START. - self.encoding = None - - # Emitter is a state machine with a stack of states to handle nested - # structures. - self.states = [] - self.state = self.expect_stream_start - - # Current event and the event queue. - self.events = [] - self.event = None - - # The current indentation level and the stack of previous indents. - self.indents = [] - self.indent = None - - # Flow level. - self.flow_level = 0 - - # Contexts. - self.root_context = False - self.sequence_context = False - self.mapping_context = False - self.simple_key_context = False - - # Characteristics of the last emitted character: - # - current position. - # - is it a whitespace? - # - is it an indention character - # (indentation space, '-', '?', or ':')? - self.line = 0 - self.column = 0 - self.whitespace = True - self.indention = True - - # Whether the document requires an explicit document indicator - self.open_ended = False - - # Formatting details. - self.canonical = canonical - self.allow_unicode = allow_unicode - self.best_indent = 2 - if indent and 1 < indent < 10: - self.best_indent = indent - self.best_width = 80 - if width and width > self.best_indent*2: - self.best_width = width - self.best_line_break = u'\n' - if line_break in [u'\r', u'\n', u'\r\n']: - self.best_line_break = line_break - - # Tag prefixes. - self.tag_prefixes = None - - # Prepared anchor and tag. - self.prepared_anchor = None - self.prepared_tag = None - - # Scalar analysis and style. - self.analysis = None - self.style = None - - def dispose(self): - # Reset the state attributes (to clear self-references) - self.states = [] - self.state = None - - def emit(self, event): - self.events.append(event) - while not self.need_more_events(): - self.event = self.events.pop(0) - self.state() - self.event = None - - # In some cases, we wait for a few next events before emitting. - - def need_more_events(self): - if not self.events: - return True - event = self.events[0] - if isinstance(event, DocumentStartEvent): - return self.need_events(1) - elif isinstance(event, SequenceStartEvent): - return self.need_events(2) - elif isinstance(event, MappingStartEvent): - return self.need_events(3) - else: - return False - - def need_events(self, count): - level = 0 - for event in self.events[1:]: - if isinstance(event, (DocumentStartEvent, CollectionStartEvent)): - level += 1 - elif isinstance(event, (DocumentEndEvent, CollectionEndEvent)): - level -= 1 - elif isinstance(event, StreamEndEvent): - level = -1 - if level < 0: - return False - return (len(self.events) < count+1) - - def increase_indent(self, flow=False, indentless=False): - self.indents.append(self.indent) - if self.indent is None: - if flow: - self.indent = self.best_indent - else: - self.indent = 0 - elif not indentless: - self.indent += self.best_indent - - # States. - - # Stream handlers. - - def expect_stream_start(self): - if isinstance(self.event, StreamStartEvent): - if self.event.encoding and not getattr(self.stream, 'encoding', None): - self.encoding = self.event.encoding - self.write_stream_start() - self.state = self.expect_first_document_start - else: - raise EmitterError("expected StreamStartEvent, but got %s" - % self.event) - - def expect_nothing(self): - raise EmitterError("expected nothing, but got %s" % self.event) - - # Document handlers. - - def expect_first_document_start(self): - return self.expect_document_start(first=True) - - def expect_document_start(self, first=False): - if isinstance(self.event, DocumentStartEvent): - if (self.event.version or self.event.tags) and self.open_ended: - self.write_indicator(u'...', True) - self.write_indent() - if self.event.version: - version_text = self.prepare_version(self.event.version) - self.write_version_directive(version_text) - self.tag_prefixes = self.DEFAULT_TAG_PREFIXES.copy() - if self.event.tags: - handles = self.event.tags.keys() - handles.sort() - for handle in handles: - prefix = self.event.tags[handle] - self.tag_prefixes[prefix] = handle - handle_text = self.prepare_tag_handle(handle) - prefix_text = self.prepare_tag_prefix(prefix) - self.write_tag_directive(handle_text, prefix_text) - implicit = (first and not self.event.explicit and not self.canonical - and not self.event.version and not self.event.tags - and not self.check_empty_document()) - if not implicit: - self.write_indent() - self.write_indicator(u'---', True) - if self.canonical: - self.write_indent() - self.state = self.expect_document_root - elif isinstance(self.event, StreamEndEvent): - if self.open_ended: - self.write_indicator(u'...', True) - self.write_indent() - self.write_stream_end() - self.state = self.expect_nothing - else: - raise EmitterError("expected DocumentStartEvent, but got %s" - % self.event) - - def expect_document_end(self): - if isinstance(self.event, DocumentEndEvent): - self.write_indent() - if self.event.explicit: - self.write_indicator(u'...', True) - self.write_indent() - self.flush_stream() - self.state = self.expect_document_start - else: - raise EmitterError("expected DocumentEndEvent, but got %s" - % self.event) - - def expect_document_root(self): - self.states.append(self.expect_document_end) - self.expect_node(root=True) - - # Node handlers. - - def expect_node(self, root=False, sequence=False, mapping=False, - simple_key=False): - self.root_context = root - self.sequence_context = sequence - self.mapping_context = mapping - self.simple_key_context = simple_key - if isinstance(self.event, AliasEvent): - self.expect_alias() - elif isinstance(self.event, (ScalarEvent, CollectionStartEvent)): - self.process_anchor(u'&') - self.process_tag() - if isinstance(self.event, ScalarEvent): - self.expect_scalar() - elif isinstance(self.event, SequenceStartEvent): - if self.flow_level or self.canonical or self.event.flow_style \ - or self.check_empty_sequence(): - self.expect_flow_sequence() - else: - self.expect_block_sequence() - elif isinstance(self.event, MappingStartEvent): - if self.flow_level or self.canonical or self.event.flow_style \ - or self.check_empty_mapping(): - self.expect_flow_mapping() - else: - self.expect_block_mapping() - else: - raise EmitterError("expected NodeEvent, but got %s" % self.event) - - def expect_alias(self): - if self.event.anchor is None: - raise EmitterError("anchor is not specified for alias") - self.process_anchor(u'*') - self.state = self.states.pop() - - def expect_scalar(self): - self.increase_indent(flow=True) - self.process_scalar() - self.indent = self.indents.pop() - self.state = self.states.pop() - - # Flow sequence handlers. - - def expect_flow_sequence(self): - self.write_indicator(u'[', True, whitespace=True) - self.flow_level += 1 - self.increase_indent(flow=True) - self.state = self.expect_first_flow_sequence_item - - def expect_first_flow_sequence_item(self): - if isinstance(self.event, SequenceEndEvent): - self.indent = self.indents.pop() - self.flow_level -= 1 - self.write_indicator(u']', False) - self.state = self.states.pop() - else: - if self.canonical or self.column > self.best_width: - self.write_indent() - self.states.append(self.expect_flow_sequence_item) - self.expect_node(sequence=True) - - def expect_flow_sequence_item(self): - if isinstance(self.event, SequenceEndEvent): - self.indent = self.indents.pop() - self.flow_level -= 1 - if self.canonical: - self.write_indicator(u',', False) - self.write_indent() - self.write_indicator(u']', False) - self.state = self.states.pop() - else: - self.write_indicator(u',', False) - if self.canonical or self.column > self.best_width: - self.write_indent() - self.states.append(self.expect_flow_sequence_item) - self.expect_node(sequence=True) - - # Flow mapping handlers. - - def expect_flow_mapping(self): - self.write_indicator(u'{', True, whitespace=True) - self.flow_level += 1 - self.increase_indent(flow=True) - self.state = self.expect_first_flow_mapping_key - - def expect_first_flow_mapping_key(self): - if isinstance(self.event, MappingEndEvent): - self.indent = self.indents.pop() - self.flow_level -= 1 - self.write_indicator(u'}', False) - self.state = self.states.pop() - else: - if self.canonical or self.column > self.best_width: - self.write_indent() - if not self.canonical and self.check_simple_key(): - self.states.append(self.expect_flow_mapping_simple_value) - self.expect_node(mapping=True, simple_key=True) - else: - self.write_indicator(u'?', True) - self.states.append(self.expect_flow_mapping_value) - self.expect_node(mapping=True) - - def expect_flow_mapping_key(self): - if isinstance(self.event, MappingEndEvent): - self.indent = self.indents.pop() - self.flow_level -= 1 - if self.canonical: - self.write_indicator(u',', False) - self.write_indent() - self.write_indicator(u'}', False) - self.state = self.states.pop() - else: - self.write_indicator(u',', False) - if self.canonical or self.column > self.best_width: - self.write_indent() - if not self.canonical and self.check_simple_key(): - self.states.append(self.expect_flow_mapping_simple_value) - self.expect_node(mapping=True, simple_key=True) - else: - self.write_indicator(u'?', True) - self.states.append(self.expect_flow_mapping_value) - self.expect_node(mapping=True) - - def expect_flow_mapping_simple_value(self): - self.write_indicator(u':', False) - self.states.append(self.expect_flow_mapping_key) - self.expect_node(mapping=True) - - def expect_flow_mapping_value(self): - if self.canonical or self.column > self.best_width: - self.write_indent() - self.write_indicator(u':', True) - self.states.append(self.expect_flow_mapping_key) - self.expect_node(mapping=True) - - # Block sequence handlers. - - def expect_block_sequence(self): - indentless = (self.mapping_context and not self.indention) - self.increase_indent(flow=False, indentless=indentless) - self.state = self.expect_first_block_sequence_item - - def expect_first_block_sequence_item(self): - return self.expect_block_sequence_item(first=True) - - def expect_block_sequence_item(self, first=False): - if not first and isinstance(self.event, SequenceEndEvent): - self.indent = self.indents.pop() - self.state = self.states.pop() - else: - self.write_indent() - self.write_indicator(u'-', True, indention=True) - self.states.append(self.expect_block_sequence_item) - self.expect_node(sequence=True) - - # Block mapping handlers. - - def expect_block_mapping(self): - self.increase_indent(flow=False) - self.state = self.expect_first_block_mapping_key - - def expect_first_block_mapping_key(self): - return self.expect_block_mapping_key(first=True) - - def expect_block_mapping_key(self, first=False): - if not first and isinstance(self.event, MappingEndEvent): - self.indent = self.indents.pop() - self.state = self.states.pop() - else: - self.write_indent() - if self.check_simple_key(): - self.states.append(self.expect_block_mapping_simple_value) - self.expect_node(mapping=True, simple_key=True) - else: - self.write_indicator(u'?', True, indention=True) - self.states.append(self.expect_block_mapping_value) - self.expect_node(mapping=True) - - def expect_block_mapping_simple_value(self): - self.write_indicator(u':', False) - self.states.append(self.expect_block_mapping_key) - self.expect_node(mapping=True) - - def expect_block_mapping_value(self): - self.write_indent() - self.write_indicator(u':', True, indention=True) - self.states.append(self.expect_block_mapping_key) - self.expect_node(mapping=True) - - # Checkers. - - def check_empty_sequence(self): - return (isinstance(self.event, SequenceStartEvent) and self.events - and isinstance(self.events[0], SequenceEndEvent)) - - def check_empty_mapping(self): - return (isinstance(self.event, MappingStartEvent) and self.events - and isinstance(self.events[0], MappingEndEvent)) - - def check_empty_document(self): - if not isinstance(self.event, DocumentStartEvent) or not self.events: - return False - event = self.events[0] - return (isinstance(event, ScalarEvent) and event.anchor is None - and event.tag is None and event.implicit and event.value == u'') - - def check_simple_key(self): - length = 0 - if isinstance(self.event, NodeEvent) and self.event.anchor is not None: - if self.prepared_anchor is None: - self.prepared_anchor = self.prepare_anchor(self.event.anchor) - length += len(self.prepared_anchor) - if isinstance(self.event, (ScalarEvent, CollectionStartEvent)) \ - and self.event.tag is not None: - if self.prepared_tag is None: - self.prepared_tag = self.prepare_tag(self.event.tag) - length += len(self.prepared_tag) - if isinstance(self.event, ScalarEvent): - if self.analysis is None: - self.analysis = self.analyze_scalar(self.event.value) - length += len(self.analysis.scalar) - return (length < 128 and (isinstance(self.event, AliasEvent) - or (isinstance(self.event, ScalarEvent) - and not self.analysis.empty and not self.analysis.multiline) - or self.check_empty_sequence() or self.check_empty_mapping())) - - # Anchor, Tag, and Scalar processors. - - def process_anchor(self, indicator): - if self.event.anchor is None: - self.prepared_anchor = None - return - if self.prepared_anchor is None: - self.prepared_anchor = self.prepare_anchor(self.event.anchor) - if self.prepared_anchor: - self.write_indicator(indicator+self.prepared_anchor, True) - self.prepared_anchor = None - - def process_tag(self): - tag = self.event.tag - if isinstance(self.event, ScalarEvent): - if self.style is None: - self.style = self.choose_scalar_style() - if ((not self.canonical or tag is None) and - ((self.style == '' and self.event.implicit[0]) - or (self.style != '' and self.event.implicit[1]))): - self.prepared_tag = None - return - if self.event.implicit[0] and tag is None: - tag = u'!' - self.prepared_tag = None - else: - if (not self.canonical or tag is None) and self.event.implicit: - self.prepared_tag = None - return - if tag is None: - raise EmitterError("tag is not specified") - if self.prepared_tag is None: - self.prepared_tag = self.prepare_tag(tag) - if self.prepared_tag: - self.write_indicator(self.prepared_tag, True) - self.prepared_tag = None - - def choose_scalar_style(self): - if self.analysis is None: - self.analysis = self.analyze_scalar(self.event.value) - if self.event.style == '"' or self.canonical: - return '"' - if not self.event.style and self.event.implicit[0]: - if (not (self.simple_key_context and - (self.analysis.empty or self.analysis.multiline)) - and (self.flow_level and self.analysis.allow_flow_plain - or (not self.flow_level and self.analysis.allow_block_plain))): - return '' - if self.event.style and self.event.style in '|>': - if (not self.flow_level and not self.simple_key_context - and self.analysis.allow_block): - return self.event.style - if not self.event.style or self.event.style == '\'': - if (self.analysis.allow_single_quoted and - not (self.simple_key_context and self.analysis.multiline)): - return '\'' - return '"' - - def process_scalar(self): - if self.analysis is None: - self.analysis = self.analyze_scalar(self.event.value) - if self.style is None: - self.style = self.choose_scalar_style() - split = (not self.simple_key_context) - #if self.analysis.multiline and split \ - # and (not self.style or self.style in '\'\"'): - # self.write_indent() - if self.style == '"': - self.write_double_quoted(self.analysis.scalar, split) - elif self.style == '\'': - self.write_single_quoted(self.analysis.scalar, split) - elif self.style == '>': - self.write_folded(self.analysis.scalar) - elif self.style == '|': - self.write_literal(self.analysis.scalar) - else: - self.write_plain(self.analysis.scalar, split) - self.analysis = None - self.style = None - - # Analyzers. - - def prepare_version(self, version): - major, minor = version - if major != 1: - raise EmitterError("unsupported YAML version: %d.%d" % (major, minor)) - return u'%d.%d' % (major, minor) - - def prepare_tag_handle(self, handle): - if not handle: - raise EmitterError("tag handle must not be empty") - if handle[0] != u'!' or handle[-1] != u'!': - raise EmitterError("tag handle must start and end with '!': %r" - % (handle.encode('utf-8'))) - for ch in handle[1:-1]: - if not (u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-_'): - raise EmitterError("invalid character %r in the tag handle: %r" - % (ch.encode('utf-8'), handle.encode('utf-8'))) - return handle - - def prepare_tag_prefix(self, prefix): - if not prefix: - raise EmitterError("tag prefix must not be empty") - chunks = [] - start = end = 0 - if prefix[0] == u'!': - end = 1 - while end < len(prefix): - ch = prefix[end] - if u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-;/?!:@&=+$,_.~*\'()[]': - end += 1 - else: - if start < end: - chunks.append(prefix[start:end]) - start = end = end+1 - data = ch.encode('utf-8') - for ch in data: - chunks.append(u'%%%02X' % ord(ch)) - if start < end: - chunks.append(prefix[start:end]) - return u''.join(chunks) - - def prepare_tag(self, tag): - if not tag: - raise EmitterError("tag must not be empty") - if tag == u'!': - return tag - handle = None - suffix = tag - prefixes = self.tag_prefixes.keys() - prefixes.sort() - for prefix in prefixes: - if tag.startswith(prefix) \ - and (prefix == u'!' or len(prefix) < len(tag)): - handle = self.tag_prefixes[prefix] - suffix = tag[len(prefix):] - chunks = [] - start = end = 0 - while end < len(suffix): - ch = suffix[end] - if u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-;/?:@&=+$,_.~*\'()[]' \ - or (ch == u'!' and handle != u'!'): - end += 1 - else: - if start < end: - chunks.append(suffix[start:end]) - start = end = end+1 - data = ch.encode('utf-8') - for ch in data: - chunks.append(u'%%%02X' % ord(ch)) - if start < end: - chunks.append(suffix[start:end]) - suffix_text = u''.join(chunks) - if handle: - return u'%s%s' % (handle, suffix_text) - else: - return u'!<%s>' % suffix_text - - def prepare_anchor(self, anchor): - if not anchor: - raise EmitterError("anchor must not be empty") - for ch in anchor: - if not (u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-_'): - raise EmitterError("invalid character %r in the anchor: %r" - % (ch.encode('utf-8'), anchor.encode('utf-8'))) - return anchor - - def analyze_scalar(self, scalar): - - # Empty scalar is a special case. - if not scalar: - return ScalarAnalysis(scalar=scalar, empty=True, multiline=False, - allow_flow_plain=False, allow_block_plain=True, - allow_single_quoted=True, allow_double_quoted=True, - allow_block=False) - - # Indicators and special characters. - block_indicators = False - flow_indicators = False - line_breaks = False - special_characters = False - - # Important whitespace combinations. - leading_space = False - leading_break = False - trailing_space = False - trailing_break = False - break_space = False - space_break = False - - # Check document indicators. - if scalar.startswith(u'---') or scalar.startswith(u'...'): - block_indicators = True - flow_indicators = True - - # First character or preceded by a whitespace. - preceeded_by_whitespace = True - - # Last character or followed by a whitespace. - followed_by_whitespace = (len(scalar) == 1 or - scalar[1] in u'\0 \t\r\n\x85\u2028\u2029') - - # The previous character is a space. - previous_space = False - - # The previous character is a break. - previous_break = False - - index = 0 - while index < len(scalar): - ch = scalar[index] - - # Check for indicators. - if index == 0: - # Leading indicators are special characters. - if ch in u'#,[]{}&*!|>\'\"%@`': - flow_indicators = True - block_indicators = True - if ch in u'?:': - flow_indicators = True - if followed_by_whitespace: - block_indicators = True - if ch == u'-' and followed_by_whitespace: - flow_indicators = True - block_indicators = True - else: - # Some indicators cannot appear within a scalar as well. - if ch in u',?[]{}': - flow_indicators = True - if ch == u':': - flow_indicators = True - if followed_by_whitespace: - block_indicators = True - if ch == u'#' and preceeded_by_whitespace: - flow_indicators = True - block_indicators = True - - # Check for line breaks, special, and unicode characters. - if ch in u'\n\x85\u2028\u2029': - line_breaks = True - if not (ch == u'\n' or u'\x20' <= ch <= u'\x7E'): - if (ch == u'\x85' or u'\xA0' <= ch <= u'\uD7FF' - or u'\uE000' <= ch <= u'\uFFFD') and ch != u'\uFEFF': - unicode_characters = True - if not self.allow_unicode: - special_characters = True - else: - special_characters = True - - # Detect important whitespace combinations. - if ch == u' ': - if index == 0: - leading_space = True - if index == len(scalar)-1: - trailing_space = True - if previous_break: - break_space = True - previous_space = True - previous_break = False - elif ch in u'\n\x85\u2028\u2029': - if index == 0: - leading_break = True - if index == len(scalar)-1: - trailing_break = True - if previous_space: - space_break = True - previous_space = False - previous_break = True - else: - previous_space = False - previous_break = False - - # Prepare for the next character. - index += 1 - preceeded_by_whitespace = (ch in u'\0 \t\r\n\x85\u2028\u2029') - followed_by_whitespace = (index+1 >= len(scalar) or - scalar[index+1] in u'\0 \t\r\n\x85\u2028\u2029') - - # Let's decide what styles are allowed. - allow_flow_plain = True - allow_block_plain = True - allow_single_quoted = True - allow_double_quoted = True - allow_block = True - - # Leading and trailing whitespaces are bad for plain scalars. - if (leading_space or leading_break - or trailing_space or trailing_break): - allow_flow_plain = allow_block_plain = False - - # We do not permit trailing spaces for block scalars. - if trailing_space: - allow_block = False - - # Spaces at the beginning of a new line are only acceptable for block - # scalars. - if break_space: - allow_flow_plain = allow_block_plain = allow_single_quoted = False - - # Spaces followed by breaks, as well as special character are only - # allowed for double quoted scalars. - if space_break or special_characters: - allow_flow_plain = allow_block_plain = \ - allow_single_quoted = allow_block = False - - # Although the plain scalar writer supports breaks, we never emit - # multiline plain scalars. - if line_breaks: - allow_flow_plain = allow_block_plain = False - - # Flow indicators are forbidden for flow plain scalars. - if flow_indicators: - allow_flow_plain = False - - # Block indicators are forbidden for block plain scalars. - if block_indicators: - allow_block_plain = False - - return ScalarAnalysis(scalar=scalar, - empty=False, multiline=line_breaks, - allow_flow_plain=allow_flow_plain, - allow_block_plain=allow_block_plain, - allow_single_quoted=allow_single_quoted, - allow_double_quoted=allow_double_quoted, - allow_block=allow_block) - - # Writers. - - def flush_stream(self): - if hasattr(self.stream, 'flush'): - self.stream.flush() - - def write_stream_start(self): - # Write BOM if needed. - if self.encoding and self.encoding.startswith('utf-16'): - self.stream.write(u'\uFEFF'.encode(self.encoding)) - - def write_stream_end(self): - self.flush_stream() - - def write_indicator(self, indicator, need_whitespace, - whitespace=False, indention=False): - if self.whitespace or not need_whitespace: - data = indicator - else: - data = u' '+indicator - self.whitespace = whitespace - self.indention = self.indention and indention - self.column += len(data) - self.open_ended = False - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - - def write_indent(self): - indent = self.indent or 0 - if not self.indention or self.column > indent \ - or (self.column == indent and not self.whitespace): - self.write_line_break() - if self.column < indent: - self.whitespace = True - data = u' '*(indent-self.column) - self.column = indent - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - - def write_line_break(self, data=None): - if data is None: - data = self.best_line_break - self.whitespace = True - self.indention = True - self.line += 1 - self.column = 0 - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - - def write_version_directive(self, version_text): - data = u'%%YAML %s' % version_text - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - self.write_line_break() - - def write_tag_directive(self, handle_text, prefix_text): - data = u'%%TAG %s %s' % (handle_text, prefix_text) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - self.write_line_break() - - # Scalar streams. - - def write_single_quoted(self, text, split=True): - self.write_indicator(u'\'', True) - spaces = False - breaks = False - start = end = 0 - while end <= len(text): - ch = None - if end < len(text): - ch = text[end] - if spaces: - if ch is None or ch != u' ': - if start+1 == end and self.column > self.best_width and split \ - and start != 0 and end != len(text): - self.write_indent() - else: - data = text[start:end] - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end - elif breaks: - if ch is None or ch not in u'\n\x85\u2028\u2029': - if text[start] == u'\n': - self.write_line_break() - for br in text[start:end]: - if br == u'\n': - self.write_line_break() - else: - self.write_line_break(br) - self.write_indent() - start = end - else: - if ch is None or ch in u' \n\x85\u2028\u2029' or ch == u'\'': - if start < end: - data = text[start:end] - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end - if ch == u'\'': - data = u'\'\'' - self.column += 2 - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end + 1 - if ch is not None: - spaces = (ch == u' ') - breaks = (ch in u'\n\x85\u2028\u2029') - end += 1 - self.write_indicator(u'\'', False) - - ESCAPE_REPLACEMENTS = { - u'\0': u'0', - u'\x07': u'a', - u'\x08': u'b', - u'\x09': u't', - u'\x0A': u'n', - u'\x0B': u'v', - u'\x0C': u'f', - u'\x0D': u'r', - u'\x1B': u'e', - u'\"': u'\"', - u'\\': u'\\', - u'\x85': u'N', - u'\xA0': u'_', - u'\u2028': u'L', - u'\u2029': u'P', - } - - def write_double_quoted(self, text, split=True): - self.write_indicator(u'"', True) - start = end = 0 - while end <= len(text): - ch = None - if end < len(text): - ch = text[end] - if ch is None or ch in u'"\\\x85\u2028\u2029\uFEFF' \ - or not (u'\x20' <= ch <= u'\x7E' - or (self.allow_unicode - and (u'\xA0' <= ch <= u'\uD7FF' - or u'\uE000' <= ch <= u'\uFFFD'))): - if start < end: - data = text[start:end] - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end - if ch is not None: - if ch in self.ESCAPE_REPLACEMENTS: - data = u'\\'+self.ESCAPE_REPLACEMENTS[ch] - elif ch <= u'\xFF': - data = u'\\x%02X' % ord(ch) - elif ch <= u'\uFFFF': - data = u'\\u%04X' % ord(ch) - else: - data = u'\\U%08X' % ord(ch) - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end+1 - if 0 < end < len(text)-1 and (ch == u' ' or start >= end) \ - and self.column+(end-start) > self.best_width and split: - data = text[start:end]+u'\\' - if start < end: - start = end - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - self.write_indent() - self.whitespace = False - self.indention = False - if text[start] == u' ': - data = u'\\' - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - end += 1 - self.write_indicator(u'"', False) - - def determine_block_hints(self, text): - hints = u'' - if text: - if text[0] in u' \n\x85\u2028\u2029': - hints += unicode(self.best_indent) - if text[-1] not in u'\n\x85\u2028\u2029': - hints += u'-' - elif len(text) == 1 or text[-2] in u'\n\x85\u2028\u2029': - hints += u'+' - return hints - - def write_folded(self, text): - hints = self.determine_block_hints(text) - self.write_indicator(u'>'+hints, True) - if hints[-1:] == u'+': - self.open_ended = True - self.write_line_break() - leading_space = True - spaces = False - breaks = True - start = end = 0 - while end <= len(text): - ch = None - if end < len(text): - ch = text[end] - if breaks: - if ch is None or ch not in u'\n\x85\u2028\u2029': - if not leading_space and ch is not None and ch != u' ' \ - and text[start] == u'\n': - self.write_line_break() - leading_space = (ch == u' ') - for br in text[start:end]: - if br == u'\n': - self.write_line_break() - else: - self.write_line_break(br) - if ch is not None: - self.write_indent() - start = end - elif spaces: - if ch != u' ': - if start+1 == end and self.column > self.best_width: - self.write_indent() - else: - data = text[start:end] - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end - else: - if ch is None or ch in u' \n\x85\u2028\u2029': - data = text[start:end] - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - if ch is None: - self.write_line_break() - start = end - if ch is not None: - breaks = (ch in u'\n\x85\u2028\u2029') - spaces = (ch == u' ') - end += 1 - - def write_literal(self, text): - hints = self.determine_block_hints(text) - self.write_indicator(u'|'+hints, True) - if hints[-1:] == u'+': - self.open_ended = True - self.write_line_break() - breaks = True - start = end = 0 - while end <= len(text): - ch = None - if end < len(text): - ch = text[end] - if breaks: - if ch is None or ch not in u'\n\x85\u2028\u2029': - for br in text[start:end]: - if br == u'\n': - self.write_line_break() - else: - self.write_line_break(br) - if ch is not None: - self.write_indent() - start = end - else: - if ch is None or ch in u'\n\x85\u2028\u2029': - data = text[start:end] - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - if ch is None: - self.write_line_break() - start = end - if ch is not None: - breaks = (ch in u'\n\x85\u2028\u2029') - end += 1 - - def write_plain(self, text, split=True): - if self.root_context: - self.open_ended = True - if not text: - return - if not self.whitespace: - data = u' ' - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - self.whitespace = False - self.indention = False - spaces = False - breaks = False - start = end = 0 - while end <= len(text): - ch = None - if end < len(text): - ch = text[end] - if spaces: - if ch != u' ': - if start+1 == end and self.column > self.best_width and split: - self.write_indent() - self.whitespace = False - self.indention = False - else: - data = text[start:end] - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end - elif breaks: - if ch not in u'\n\x85\u2028\u2029': - if text[start] == u'\n': - self.write_line_break() - for br in text[start:end]: - if br == u'\n': - self.write_line_break() - else: - self.write_line_break(br) - self.write_indent() - self.whitespace = False - self.indention = False - start = end - else: - if ch is None or ch in u' \n\x85\u2028\u2029': - data = text[start:end] - self.column += len(data) - if self.encoding: - data = data.encode(self.encoding) - self.stream.write(data) - start = end - if ch is not None: - spaces = (ch == u' ') - breaks = (ch in u'\n\x85\u2028\u2029') - end += 1 - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/error.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/error.py deleted file mode 100644 index 5466be72..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/error.py +++ /dev/null @@ -1,76 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['Mark', 'YAMLError', 'MarkedYAMLError'] - -class Mark(object): - - def __init__(self, name, index, line, column, buffer, pointer): - self.name = name - self.index = index - self.line = line - self.column = column - self.buffer = buffer - self.pointer = pointer - - def get_snippet(self, indent=4, max_length=75): - if self.buffer is None: - return None - head = '' - start = self.pointer - while start > 0 and self.buffer[start-1] not in u'\0\r\n\x85\u2028\u2029': - start -= 1 - if self.pointer-start > max_length/2-1: - head = ' ... ' - start += 5 - break - tail = '' - end = self.pointer - while end < len(self.buffer) and self.buffer[end] not in u'\0\r\n\x85\u2028\u2029': - end += 1 - if end-self.pointer > max_length/2-1: - tail = ' ... ' - end -= 5 - break - snippet = self.buffer[start:end].encode('utf-8') - return ' '*indent + head + snippet + tail + '\n' \ - + ' '*(indent+self.pointer-start+len(head)) + '^' - - def __str__(self): - snippet = self.get_snippet() - where = " in \"%s\", line %d, column %d" \ - % (self.name, self.line+1, self.column+1) - if snippet is not None: - where += ":\n"+snippet - return where - -class YAMLError(Exception): - pass - -class MarkedYAMLError(YAMLError): - - def __init__(self, context=None, context_mark=None, - problem=None, problem_mark=None, note=None): - self.context = context - self.context_mark = context_mark - self.problem = problem - self.problem_mark = problem_mark - self.note = note - - def __str__(self): - lines = [] - if self.context is not None: - lines.append(self.context) - if self.context_mark is not None \ - and (self.problem is None or self.problem_mark is None - or self.context_mark.name != self.problem_mark.name - or self.context_mark.line != self.problem_mark.line - or self.context_mark.column != self.problem_mark.column): - lines.append(str(self.context_mark)) - if self.problem is not None: - lines.append(self.problem) - if self.problem_mark is not None: - lines.append(str(self.problem_mark)) - if self.note is not None: - lines.append(self.note) - return '\n'.join(lines) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/events.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/events.py deleted file mode 100644 index 283452ad..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/events.py +++ /dev/null @@ -1,87 +0,0 @@ -# SPDX-License-Identifier: MIT - -# Abstract classes. - -class Event(object): - def __init__(self, start_mark=None, end_mark=None): - self.start_mark = start_mark - self.end_mark = end_mark - def __repr__(self): - attributes = [key for key in ['anchor', 'tag', 'implicit', 'value'] - if hasattr(self, key)] - arguments = ', '.join(['%s=%r' % (key, getattr(self, key)) - for key in attributes]) - return '%s(%s)' % (self.__class__.__name__, arguments) - -class NodeEvent(Event): - def __init__(self, anchor, start_mark=None, end_mark=None): - self.anchor = anchor - self.start_mark = start_mark - self.end_mark = end_mark - -class CollectionStartEvent(NodeEvent): - def __init__(self, anchor, tag, implicit, start_mark=None, end_mark=None, - flow_style=None): - self.anchor = anchor - self.tag = tag - self.implicit = implicit - self.start_mark = start_mark - self.end_mark = end_mark - self.flow_style = flow_style - -class CollectionEndEvent(Event): - pass - -# Implementations. - -class StreamStartEvent(Event): - def __init__(self, start_mark=None, end_mark=None, encoding=None): - self.start_mark = start_mark - self.end_mark = end_mark - self.encoding = encoding - -class StreamEndEvent(Event): - pass - -class DocumentStartEvent(Event): - def __init__(self, start_mark=None, end_mark=None, - explicit=None, version=None, tags=None): - self.start_mark = start_mark - self.end_mark = end_mark - self.explicit = explicit - self.version = version - self.tags = tags - -class DocumentEndEvent(Event): - def __init__(self, start_mark=None, end_mark=None, - explicit=None): - self.start_mark = start_mark - self.end_mark = end_mark - self.explicit = explicit - -class AliasEvent(NodeEvent): - pass - -class ScalarEvent(NodeEvent): - def __init__(self, anchor, tag, implicit, value, - start_mark=None, end_mark=None, style=None): - self.anchor = anchor - self.tag = tag - self.implicit = implicit - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - self.style = style - -class SequenceStartEvent(CollectionStartEvent): - pass - -class SequenceEndEvent(CollectionEndEvent): - pass - -class MappingStartEvent(CollectionStartEvent): - pass - -class MappingEndEvent(CollectionEndEvent): - pass - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/loader.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/loader.py deleted file mode 100644 index 1c195531..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/loader.py +++ /dev/null @@ -1,41 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['BaseLoader', 'SafeLoader', 'Loader'] - -from reader import * -from scanner import * -from parser import * -from composer import * -from constructor import * -from resolver import * - -class BaseLoader(Reader, Scanner, Parser, Composer, BaseConstructor, BaseResolver): - - def __init__(self, stream): - Reader.__init__(self, stream) - Scanner.__init__(self) - Parser.__init__(self) - Composer.__init__(self) - BaseConstructor.__init__(self) - BaseResolver.__init__(self) - -class SafeLoader(Reader, Scanner, Parser, Composer, SafeConstructor, Resolver): - - def __init__(self, stream): - Reader.__init__(self, stream) - Scanner.__init__(self) - Parser.__init__(self) - Composer.__init__(self) - SafeConstructor.__init__(self) - Resolver.__init__(self) - -class Loader(Reader, Scanner, Parser, Composer, Constructor, Resolver): - - def __init__(self, stream): - Reader.__init__(self, stream) - Scanner.__init__(self) - Parser.__init__(self) - Composer.__init__(self) - Constructor.__init__(self) - Resolver.__init__(self) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/nodes.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/nodes.py deleted file mode 100644 index ed2a1b43..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/nodes.py +++ /dev/null @@ -1,50 +0,0 @@ -# SPDX-License-Identifier: MIT - -class Node(object): - def __init__(self, tag, value, start_mark, end_mark): - self.tag = tag - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - def __repr__(self): - value = self.value - #if isinstance(value, list): - # if len(value) == 0: - # value = '<empty>' - # elif len(value) == 1: - # value = '<1 item>' - # else: - # value = '<%d items>' % len(value) - #else: - # if len(value) > 75: - # value = repr(value[:70]+u' ... ') - # else: - # value = repr(value) - value = repr(value) - return '%s(tag=%r, value=%s)' % (self.__class__.__name__, self.tag, value) - -class ScalarNode(Node): - id = 'scalar' - def __init__(self, tag, value, - start_mark=None, end_mark=None, style=None): - self.tag = tag - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - self.style = style - -class CollectionNode(Node): - def __init__(self, tag, value, - start_mark=None, end_mark=None, flow_style=None): - self.tag = tag - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - self.flow_style = flow_style - -class SequenceNode(CollectionNode): - id = 'sequence' - -class MappingNode(CollectionNode): - id = 'mapping' - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/parser.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/parser.py deleted file mode 100644 index 97ba0833..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/parser.py +++ /dev/null @@ -1,590 +0,0 @@ -# SPDX-License-Identifier: MIT - -# The following YAML grammar is LL(1) and is parsed by a recursive descent -# parser. -# -# stream ::= STREAM-START implicit_document? explicit_document* STREAM-END -# implicit_document ::= block_node DOCUMENT-END* -# explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* -# block_node_or_indentless_sequence ::= -# ALIAS -# | properties (block_content | indentless_block_sequence)? -# | block_content -# | indentless_block_sequence -# block_node ::= ALIAS -# | properties block_content? -# | block_content -# flow_node ::= ALIAS -# | properties flow_content? -# | flow_content -# properties ::= TAG ANCHOR? | ANCHOR TAG? -# block_content ::= block_collection | flow_collection | SCALAR -# flow_content ::= flow_collection | SCALAR -# block_collection ::= block_sequence | block_mapping -# flow_collection ::= flow_sequence | flow_mapping -# block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END -# indentless_sequence ::= (BLOCK-ENTRY block_node?)+ -# block_mapping ::= BLOCK-MAPPING_START -# ((KEY block_node_or_indentless_sequence?)? -# (VALUE block_node_or_indentless_sequence?)?)* -# BLOCK-END -# flow_sequence ::= FLOW-SEQUENCE-START -# (flow_sequence_entry FLOW-ENTRY)* -# flow_sequence_entry? -# FLOW-SEQUENCE-END -# flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? -# flow_mapping ::= FLOW-MAPPING-START -# (flow_mapping_entry FLOW-ENTRY)* -# flow_mapping_entry? -# FLOW-MAPPING-END -# flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? -# -# FIRST sets: -# -# stream: { STREAM-START } -# explicit_document: { DIRECTIVE DOCUMENT-START } -# implicit_document: FIRST(block_node) -# block_node: { ALIAS TAG ANCHOR SCALAR BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START } -# flow_node: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START } -# block_content: { BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START SCALAR } -# flow_content: { FLOW-SEQUENCE-START FLOW-MAPPING-START SCALAR } -# block_collection: { BLOCK-SEQUENCE-START BLOCK-MAPPING-START } -# flow_collection: { FLOW-SEQUENCE-START FLOW-MAPPING-START } -# block_sequence: { BLOCK-SEQUENCE-START } -# block_mapping: { BLOCK-MAPPING-START } -# block_node_or_indentless_sequence: { ALIAS ANCHOR TAG SCALAR BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START BLOCK-ENTRY } -# indentless_sequence: { ENTRY } -# flow_collection: { FLOW-SEQUENCE-START FLOW-MAPPING-START } -# flow_sequence: { FLOW-SEQUENCE-START } -# flow_mapping: { FLOW-MAPPING-START } -# flow_sequence_entry: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START KEY } -# flow_mapping_entry: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START KEY } - -__all__ = ['Parser', 'ParserError'] - -from error import MarkedYAMLError -from tokens import * -from events import * -from scanner import * - -class ParserError(MarkedYAMLError): - pass - -class Parser(object): - # Since writing a recursive-descendant parser is a straightforward task, we - # do not give many comments here. - - DEFAULT_TAGS = { - u'!': u'!', - u'!!': u'tag:yaml.org,2002:', - } - - def __init__(self): - self.current_event = None - self.yaml_version = None - self.tag_handles = {} - self.states = [] - self.marks = [] - self.state = self.parse_stream_start - - def dispose(self): - # Reset the state attributes (to clear self-references) - self.states = [] - self.state = None - - def check_event(self, *choices): - # Check the type of the next event. - if self.current_event is None: - if self.state: - self.current_event = self.state() - if self.current_event is not None: - if not choices: - return True - for choice in choices: - if isinstance(self.current_event, choice): - return True - return False - - def peek_event(self): - # Get the next event. - if self.current_event is None: - if self.state: - self.current_event = self.state() - return self.current_event - - def get_event(self): - # Get the next event and proceed further. - if self.current_event is None: - if self.state: - self.current_event = self.state() - value = self.current_event - self.current_event = None - return value - - # stream ::= STREAM-START implicit_document? explicit_document* STREAM-END - # implicit_document ::= block_node DOCUMENT-END* - # explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* - - def parse_stream_start(self): - - # Parse the stream start. - token = self.get_token() - event = StreamStartEvent(token.start_mark, token.end_mark, - encoding=token.encoding) - - # Prepare the next state. - self.state = self.parse_implicit_document_start - - return event - - def parse_implicit_document_start(self): - - # Parse an implicit document. - if not self.check_token(DirectiveToken, DocumentStartToken, - StreamEndToken): - self.tag_handles = self.DEFAULT_TAGS - token = self.peek_token() - start_mark = end_mark = token.start_mark - event = DocumentStartEvent(start_mark, end_mark, - explicit=False) - - # Prepare the next state. - self.states.append(self.parse_document_end) - self.state = self.parse_block_node - - return event - - else: - return self.parse_document_start() - - def parse_document_start(self): - - # Parse any extra document end indicators. - while self.check_token(DocumentEndToken): - self.get_token() - - # Parse an explicit document. - if not self.check_token(StreamEndToken): - token = self.peek_token() - start_mark = token.start_mark - version, tags = self.process_directives() - if not self.check_token(DocumentStartToken): - raise ParserError(None, None, - "expected '<document start>', but found %r" - % self.peek_token().id, - self.peek_token().start_mark) - token = self.get_token() - end_mark = token.end_mark - event = DocumentStartEvent(start_mark, end_mark, - explicit=True, version=version, tags=tags) - self.states.append(self.parse_document_end) - self.state = self.parse_document_content - else: - # Parse the end of the stream. - token = self.get_token() - event = StreamEndEvent(token.start_mark, token.end_mark) - assert not self.states - assert not self.marks - self.state = None - return event - - def parse_document_end(self): - - # Parse the document end. - token = self.peek_token() - start_mark = end_mark = token.start_mark - explicit = False - if self.check_token(DocumentEndToken): - token = self.get_token() - end_mark = token.end_mark - explicit = True - event = DocumentEndEvent(start_mark, end_mark, - explicit=explicit) - - # Prepare the next state. - self.state = self.parse_document_start - - return event - - def parse_document_content(self): - if self.check_token(DirectiveToken, - DocumentStartToken, DocumentEndToken, StreamEndToken): - event = self.process_empty_scalar(self.peek_token().start_mark) - self.state = self.states.pop() - return event - else: - return self.parse_block_node() - - def process_directives(self): - self.yaml_version = None - self.tag_handles = {} - while self.check_token(DirectiveToken): - token = self.get_token() - if token.name == u'YAML': - if self.yaml_version is not None: - raise ParserError(None, None, - "found duplicate YAML directive", token.start_mark) - major, minor = token.value - if major != 1: - raise ParserError(None, None, - "found incompatible YAML document (version 1.* is required)", - token.start_mark) - self.yaml_version = token.value - elif token.name == u'TAG': - handle, prefix = token.value - if handle in self.tag_handles: - raise ParserError(None, None, - "duplicate tag handle %r" % handle.encode('utf-8'), - token.start_mark) - self.tag_handles[handle] = prefix - if self.tag_handles: - value = self.yaml_version, self.tag_handles.copy() - else: - value = self.yaml_version, None - for key in self.DEFAULT_TAGS: - if key not in self.tag_handles: - self.tag_handles[key] = self.DEFAULT_TAGS[key] - return value - - # block_node_or_indentless_sequence ::= ALIAS - # | properties (block_content | indentless_block_sequence)? - # | block_content - # | indentless_block_sequence - # block_node ::= ALIAS - # | properties block_content? - # | block_content - # flow_node ::= ALIAS - # | properties flow_content? - # | flow_content - # properties ::= TAG ANCHOR? | ANCHOR TAG? - # block_content ::= block_collection | flow_collection | SCALAR - # flow_content ::= flow_collection | SCALAR - # block_collection ::= block_sequence | block_mapping - # flow_collection ::= flow_sequence | flow_mapping - - def parse_block_node(self): - return self.parse_node(block=True) - - def parse_flow_node(self): - return self.parse_node() - - def parse_block_node_or_indentless_sequence(self): - return self.parse_node(block=True, indentless_sequence=True) - - def parse_node(self, block=False, indentless_sequence=False): - if self.check_token(AliasToken): - token = self.get_token() - event = AliasEvent(token.value, token.start_mark, token.end_mark) - self.state = self.states.pop() - else: - anchor = None - tag = None - start_mark = end_mark = tag_mark = None - if self.check_token(AnchorToken): - token = self.get_token() - start_mark = token.start_mark - end_mark = token.end_mark - anchor = token.value - if self.check_token(TagToken): - token = self.get_token() - tag_mark = token.start_mark - end_mark = token.end_mark - tag = token.value - elif self.check_token(TagToken): - token = self.get_token() - start_mark = tag_mark = token.start_mark - end_mark = token.end_mark - tag = token.value - if self.check_token(AnchorToken): - token = self.get_token() - end_mark = token.end_mark - anchor = token.value - if tag is not None: - handle, suffix = tag - if handle is not None: - if handle not in self.tag_handles: - raise ParserError("while parsing a node", start_mark, - "found undefined tag handle %r" % handle.encode('utf-8'), - tag_mark) - tag = self.tag_handles[handle]+suffix - else: - tag = suffix - #if tag == u'!': - # raise ParserError("while parsing a node", start_mark, - # "found non-specific tag '!'", tag_mark, - # "Please check 'http://pyyaml.org/wiki/YAMLNonSpecificTag' and share your opinion.") - if start_mark is None: - start_mark = end_mark = self.peek_token().start_mark - event = None - implicit = (tag is None or tag == u'!') - if indentless_sequence and self.check_token(BlockEntryToken): - end_mark = self.peek_token().end_mark - event = SequenceStartEvent(anchor, tag, implicit, - start_mark, end_mark) - self.state = self.parse_indentless_sequence_entry - else: - if self.check_token(ScalarToken): - token = self.get_token() - end_mark = token.end_mark - if (token.plain and tag is None) or tag == u'!': - implicit = (True, False) - elif tag is None: - implicit = (False, True) - else: - implicit = (False, False) - event = ScalarEvent(anchor, tag, implicit, token.value, - start_mark, end_mark, style=token.style) - self.state = self.states.pop() - elif self.check_token(FlowSequenceStartToken): - end_mark = self.peek_token().end_mark - event = SequenceStartEvent(anchor, tag, implicit, - start_mark, end_mark, flow_style=True) - self.state = self.parse_flow_sequence_first_entry - elif self.check_token(FlowMappingStartToken): - end_mark = self.peek_token().end_mark - event = MappingStartEvent(anchor, tag, implicit, - start_mark, end_mark, flow_style=True) - self.state = self.parse_flow_mapping_first_key - elif block and self.check_token(BlockSequenceStartToken): - end_mark = self.peek_token().start_mark - event = SequenceStartEvent(anchor, tag, implicit, - start_mark, end_mark, flow_style=False) - self.state = self.parse_block_sequence_first_entry - elif block and self.check_token(BlockMappingStartToken): - end_mark = self.peek_token().start_mark - event = MappingStartEvent(anchor, tag, implicit, - start_mark, end_mark, flow_style=False) - self.state = self.parse_block_mapping_first_key - elif anchor is not None or tag is not None: - # Empty scalars are allowed even if a tag or an anchor is - # specified. - event = ScalarEvent(anchor, tag, (implicit, False), u'', - start_mark, end_mark) - self.state = self.states.pop() - else: - if block: - node = 'block' - else: - node = 'flow' - token = self.peek_token() - raise ParserError("while parsing a %s node" % node, start_mark, - "expected the node content, but found %r" % token.id, - token.start_mark) - return event - - # block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END - - def parse_block_sequence_first_entry(self): - token = self.get_token() - self.marks.append(token.start_mark) - return self.parse_block_sequence_entry() - - def parse_block_sequence_entry(self): - if self.check_token(BlockEntryToken): - token = self.get_token() - if not self.check_token(BlockEntryToken, BlockEndToken): - self.states.append(self.parse_block_sequence_entry) - return self.parse_block_node() - else: - self.state = self.parse_block_sequence_entry - return self.process_empty_scalar(token.end_mark) - if not self.check_token(BlockEndToken): - token = self.peek_token() - raise ParserError("while parsing a block collection", self.marks[-1], - "expected <block end>, but found %r" % token.id, token.start_mark) - token = self.get_token() - event = SequenceEndEvent(token.start_mark, token.end_mark) - self.state = self.states.pop() - self.marks.pop() - return event - - # indentless_sequence ::= (BLOCK-ENTRY block_node?)+ - - def parse_indentless_sequence_entry(self): - if self.check_token(BlockEntryToken): - token = self.get_token() - if not self.check_token(BlockEntryToken, - KeyToken, ValueToken, BlockEndToken): - self.states.append(self.parse_indentless_sequence_entry) - return self.parse_block_node() - else: - self.state = self.parse_indentless_sequence_entry - return self.process_empty_scalar(token.end_mark) - token = self.peek_token() - event = SequenceEndEvent(token.start_mark, token.start_mark) - self.state = self.states.pop() - return event - - # block_mapping ::= BLOCK-MAPPING_START - # ((KEY block_node_or_indentless_sequence?)? - # (VALUE block_node_or_indentless_sequence?)?)* - # BLOCK-END - - def parse_block_mapping_first_key(self): - token = self.get_token() - self.marks.append(token.start_mark) - return self.parse_block_mapping_key() - - def parse_block_mapping_key(self): - if self.check_token(KeyToken): - token = self.get_token() - if not self.check_token(KeyToken, ValueToken, BlockEndToken): - self.states.append(self.parse_block_mapping_value) - return self.parse_block_node_or_indentless_sequence() - else: - self.state = self.parse_block_mapping_value - return self.process_empty_scalar(token.end_mark) - if not self.check_token(BlockEndToken): - token = self.peek_token() - raise ParserError("while parsing a block mapping", self.marks[-1], - "expected <block end>, but found %r" % token.id, token.start_mark) - token = self.get_token() - event = MappingEndEvent(token.start_mark, token.end_mark) - self.state = self.states.pop() - self.marks.pop() - return event - - def parse_block_mapping_value(self): - if self.check_token(ValueToken): - token = self.get_token() - if not self.check_token(KeyToken, ValueToken, BlockEndToken): - self.states.append(self.parse_block_mapping_key) - return self.parse_block_node_or_indentless_sequence() - else: - self.state = self.parse_block_mapping_key - return self.process_empty_scalar(token.end_mark) - else: - self.state = self.parse_block_mapping_key - token = self.peek_token() - return self.process_empty_scalar(token.start_mark) - - # flow_sequence ::= FLOW-SEQUENCE-START - # (flow_sequence_entry FLOW-ENTRY)* - # flow_sequence_entry? - # FLOW-SEQUENCE-END - # flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? - # - # Note that while production rules for both flow_sequence_entry and - # flow_mapping_entry are equal, their interpretations are different. - # For `flow_sequence_entry`, the part `KEY flow_node? (VALUE flow_node?)?` - # generate an inline mapping (set syntax). - - def parse_flow_sequence_first_entry(self): - token = self.get_token() - self.marks.append(token.start_mark) - return self.parse_flow_sequence_entry(first=True) - - def parse_flow_sequence_entry(self, first=False): - if not self.check_token(FlowSequenceEndToken): - if not first: - if self.check_token(FlowEntryToken): - self.get_token() - else: - token = self.peek_token() - raise ParserError("while parsing a flow sequence", self.marks[-1], - "expected ',' or ']', but got %r" % token.id, token.start_mark) - - if self.check_token(KeyToken): - token = self.peek_token() - event = MappingStartEvent(None, None, True, - token.start_mark, token.end_mark, - flow_style=True) - self.state = self.parse_flow_sequence_entry_mapping_key - return event - elif not self.check_token(FlowSequenceEndToken): - self.states.append(self.parse_flow_sequence_entry) - return self.parse_flow_node() - token = self.get_token() - event = SequenceEndEvent(token.start_mark, token.end_mark) - self.state = self.states.pop() - self.marks.pop() - return event - - def parse_flow_sequence_entry_mapping_key(self): - token = self.get_token() - if not self.check_token(ValueToken, - FlowEntryToken, FlowSequenceEndToken): - self.states.append(self.parse_flow_sequence_entry_mapping_value) - return self.parse_flow_node() - else: - self.state = self.parse_flow_sequence_entry_mapping_value - return self.process_empty_scalar(token.end_mark) - - def parse_flow_sequence_entry_mapping_value(self): - if self.check_token(ValueToken): - token = self.get_token() - if not self.check_token(FlowEntryToken, FlowSequenceEndToken): - self.states.append(self.parse_flow_sequence_entry_mapping_end) - return self.parse_flow_node() - else: - self.state = self.parse_flow_sequence_entry_mapping_end - return self.process_empty_scalar(token.end_mark) - else: - self.state = self.parse_flow_sequence_entry_mapping_end - token = self.peek_token() - return self.process_empty_scalar(token.start_mark) - - def parse_flow_sequence_entry_mapping_end(self): - self.state = self.parse_flow_sequence_entry - token = self.peek_token() - return MappingEndEvent(token.start_mark, token.start_mark) - - # flow_mapping ::= FLOW-MAPPING-START - # (flow_mapping_entry FLOW-ENTRY)* - # flow_mapping_entry? - # FLOW-MAPPING-END - # flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? - - def parse_flow_mapping_first_key(self): - token = self.get_token() - self.marks.append(token.start_mark) - return self.parse_flow_mapping_key(first=True) - - def parse_flow_mapping_key(self, first=False): - if not self.check_token(FlowMappingEndToken): - if not first: - if self.check_token(FlowEntryToken): - self.get_token() - else: - token = self.peek_token() - raise ParserError("while parsing a flow mapping", self.marks[-1], - "expected ',' or '}', but got %r" % token.id, token.start_mark) - if self.check_token(KeyToken): - token = self.get_token() - if not self.check_token(ValueToken, - FlowEntryToken, FlowMappingEndToken): - self.states.append(self.parse_flow_mapping_value) - return self.parse_flow_node() - else: - self.state = self.parse_flow_mapping_value - return self.process_empty_scalar(token.end_mark) - elif not self.check_token(FlowMappingEndToken): - self.states.append(self.parse_flow_mapping_empty_value) - return self.parse_flow_node() - token = self.get_token() - event = MappingEndEvent(token.start_mark, token.end_mark) - self.state = self.states.pop() - self.marks.pop() - return event - - def parse_flow_mapping_value(self): - if self.check_token(ValueToken): - token = self.get_token() - if not self.check_token(FlowEntryToken, FlowMappingEndToken): - self.states.append(self.parse_flow_mapping_key) - return self.parse_flow_node() - else: - self.state = self.parse_flow_mapping_key - return self.process_empty_scalar(token.end_mark) - else: - self.state = self.parse_flow_mapping_key - token = self.peek_token() - return self.process_empty_scalar(token.start_mark) - - def parse_flow_mapping_empty_value(self): - self.state = self.parse_flow_mapping_key - return self.process_empty_scalar(self.peek_token().start_mark) - - def process_empty_scalar(self, mark): - return ScalarEvent(None, None, (True, False), u'', mark, mark) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/reader.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/reader.py deleted file mode 100644 index 8d422954..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/reader.py +++ /dev/null @@ -1,191 +0,0 @@ -# SPDX-License-Identifier: MIT -# This module contains abstractions for the input stream. You don't have to -# looks further, there are no pretty code. -# -# We define two classes here. -# -# Mark(source, line, column) -# It's just a record and its only use is producing nice error messages. -# Parser does not use it for any other purposes. -# -# Reader(source, data) -# Reader determines the encoding of `data` and converts it to unicode. -# Reader provides the following methods and attributes: -# reader.peek(length=1) - return the next `length` characters -# reader.forward(length=1) - move the current position to `length` characters. -# reader.index - the number of the current character. -# reader.line, stream.column - the line and the column of the current character. - -__all__ = ['Reader', 'ReaderError'] - -from error import YAMLError, Mark - -import codecs, re - -class ReaderError(YAMLError): - - def __init__(self, name, position, character, encoding, reason): - self.name = name - self.character = character - self.position = position - self.encoding = encoding - self.reason = reason - - def __str__(self): - if isinstance(self.character, str): - return "'%s' codec can't decode byte #x%02x: %s\n" \ - " in \"%s\", position %d" \ - % (self.encoding, ord(self.character), self.reason, - self.name, self.position) - else: - return "unacceptable character #x%04x: %s\n" \ - " in \"%s\", position %d" \ - % (self.character, self.reason, - self.name, self.position) - -class Reader(object): - # Reader: - # - determines the data encoding and converts it to unicode, - # - checks if characters are in allowed range, - # - adds '\0' to the end. - - # Reader accepts - # - a `str` object, - # - a `unicode` object, - # - a file-like object with its `read` method returning `str`, - # - a file-like object with its `read` method returning `unicode`. - - # Yeah, it's ugly and slow. - - def __init__(self, stream): - self.name = None - self.stream = None - self.stream_pointer = 0 - self.eof = True - self.buffer = u'' - self.pointer = 0 - self.raw_buffer = None - self.raw_decode = None - self.encoding = None - self.index = 0 - self.line = 0 - self.column = 0 - if isinstance(stream, unicode): - self.name = "<unicode string>" - self.check_printable(stream) - self.buffer = stream+u'\0' - elif isinstance(stream, str): - self.name = "<string>" - self.raw_buffer = stream - self.determine_encoding() - else: - self.stream = stream - self.name = getattr(stream, 'name', "<file>") - self.eof = False - self.raw_buffer = '' - self.determine_encoding() - - def peek(self, index=0): - try: - return self.buffer[self.pointer+index] - except IndexError: - self.update(index+1) - return self.buffer[self.pointer+index] - - def prefix(self, length=1): - if self.pointer+length >= len(self.buffer): - self.update(length) - return self.buffer[self.pointer:self.pointer+length] - - def forward(self, length=1): - if self.pointer+length+1 >= len(self.buffer): - self.update(length+1) - while length: - ch = self.buffer[self.pointer] - self.pointer += 1 - self.index += 1 - if ch in u'\n\x85\u2028\u2029' \ - or (ch == u'\r' and self.buffer[self.pointer] != u'\n'): - self.line += 1 - self.column = 0 - elif ch != u'\uFEFF': - self.column += 1 - length -= 1 - - def get_mark(self): - if self.stream is None: - return Mark(self.name, self.index, self.line, self.column, - self.buffer, self.pointer) - else: - return Mark(self.name, self.index, self.line, self.column, - None, None) - - def determine_encoding(self): - while not self.eof and len(self.raw_buffer) < 2: - self.update_raw() - if not isinstance(self.raw_buffer, unicode): - if self.raw_buffer.startswith(codecs.BOM_UTF16_LE): - self.raw_decode = codecs.utf_16_le_decode - self.encoding = 'utf-16-le' - elif self.raw_buffer.startswith(codecs.BOM_UTF16_BE): - self.raw_decode = codecs.utf_16_be_decode - self.encoding = 'utf-16-be' - else: - self.raw_decode = codecs.utf_8_decode - self.encoding = 'utf-8' - self.update(1) - - NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]') - def check_printable(self, data): - match = self.NON_PRINTABLE.search(data) - if match: - character = match.group() - position = self.index+(len(self.buffer)-self.pointer)+match.start() - raise ReaderError(self.name, position, ord(character), - 'unicode', "special characters are not allowed") - - def update(self, length): - if self.raw_buffer is None: - return - self.buffer = self.buffer[self.pointer:] - self.pointer = 0 - while len(self.buffer) < length: - if not self.eof: - self.update_raw() - if self.raw_decode is not None: - try: - data, converted = self.raw_decode(self.raw_buffer, - 'strict', self.eof) - except UnicodeDecodeError, exc: - character = exc.object[exc.start] - if self.stream is not None: - position = self.stream_pointer-len(self.raw_buffer)+exc.start - else: - position = exc.start - raise ReaderError(self.name, position, character, - exc.encoding, exc.reason) - else: - data = self.raw_buffer - converted = len(data) - self.check_printable(data) - self.buffer += data - self.raw_buffer = self.raw_buffer[converted:] - if self.eof: - self.buffer += u'\0' - self.raw_buffer = None - break - - def update_raw(self, size=1024): - data = self.stream.read(size) - if data: - self.raw_buffer += data - self.stream_pointer += len(data) - else: - self.eof = True - -#try: -# import psyco -# psyco.bind(Reader) -#except ImportError: -# pass - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/representer.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/representer.py deleted file mode 100644 index 0a1404ec..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/representer.py +++ /dev/null @@ -1,485 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['BaseRepresenter', 'SafeRepresenter', 'Representer', - 'RepresenterError'] - -from error import * -from nodes import * - -import datetime - -import sys, copy_reg, types - -class RepresenterError(YAMLError): - pass - -class BaseRepresenter(object): - - yaml_representers = {} - yaml_multi_representers = {} - - def __init__(self, default_style=None, default_flow_style=None): - self.default_style = default_style - self.default_flow_style = default_flow_style - self.represented_objects = {} - self.object_keeper = [] - self.alias_key = None - - def represent(self, data): - node = self.represent_data(data) - self.serialize(node) - self.represented_objects = {} - self.object_keeper = [] - self.alias_key = None - - def get_classobj_bases(self, cls): - bases = [cls] - for base in cls.__bases__: - bases.extend(self.get_classobj_bases(base)) - return bases - - def represent_data(self, data): - if self.ignore_aliases(data): - self.alias_key = None - else: - self.alias_key = id(data) - if self.alias_key is not None: - if self.alias_key in self.represented_objects: - node = self.represented_objects[self.alias_key] - #if node is None: - # raise RepresenterError("recursive objects are not allowed: %r" % data) - return node - #self.represented_objects[alias_key] = None - self.object_keeper.append(data) - data_types = type(data).__mro__ - if type(data) is types.InstanceType: - data_types = self.get_classobj_bases(data.__class__)+list(data_types) - if data_types[0] in self.yaml_representers: - node = self.yaml_representers[data_types[0]](self, data) - else: - for data_type in data_types: - if data_type in self.yaml_multi_representers: - node = self.yaml_multi_representers[data_type](self, data) - break - else: - if None in self.yaml_multi_representers: - node = self.yaml_multi_representers[None](self, data) - elif None in self.yaml_representers: - node = self.yaml_representers[None](self, data) - else: - node = ScalarNode(None, unicode(data)) - #if alias_key is not None: - # self.represented_objects[alias_key] = node - return node - - def add_representer(cls, data_type, representer): - if not 'yaml_representers' in cls.__dict__: - cls.yaml_representers = cls.yaml_representers.copy() - cls.yaml_representers[data_type] = representer - add_representer = classmethod(add_representer) - - def add_multi_representer(cls, data_type, representer): - if not 'yaml_multi_representers' in cls.__dict__: - cls.yaml_multi_representers = cls.yaml_multi_representers.copy() - cls.yaml_multi_representers[data_type] = representer - add_multi_representer = classmethod(add_multi_representer) - - def represent_scalar(self, tag, value, style=None): - if style is None: - style = self.default_style - node = ScalarNode(tag, value, style=style) - if self.alias_key is not None: - self.represented_objects[self.alias_key] = node - return node - - def represent_sequence(self, tag, sequence, flow_style=None): - value = [] - node = SequenceNode(tag, value, flow_style=flow_style) - if self.alias_key is not None: - self.represented_objects[self.alias_key] = node - best_style = True - for item in sequence: - node_item = self.represent_data(item) - if not (isinstance(node_item, ScalarNode) and not node_item.style): - best_style = False - value.append(node_item) - if flow_style is None: - if self.default_flow_style is not None: - node.flow_style = self.default_flow_style - else: - node.flow_style = best_style - return node - - def represent_mapping(self, tag, mapping, flow_style=None): - value = [] - node = MappingNode(tag, value, flow_style=flow_style) - if self.alias_key is not None: - self.represented_objects[self.alias_key] = node - best_style = True - if hasattr(mapping, 'items'): - mapping = mapping.items() - mapping.sort() - for item_key, item_value in mapping: - node_key = self.represent_data(item_key) - node_value = self.represent_data(item_value) - if not (isinstance(node_key, ScalarNode) and not node_key.style): - best_style = False - if not (isinstance(node_value, ScalarNode) and not node_value.style): - best_style = False - value.append((node_key, node_value)) - if flow_style is None: - if self.default_flow_style is not None: - node.flow_style = self.default_flow_style - else: - node.flow_style = best_style - return node - - def ignore_aliases(self, data): - return False - -class SafeRepresenter(BaseRepresenter): - - def ignore_aliases(self, data): - if data in [None, ()]: - return True - if isinstance(data, (str, unicode, bool, int, float)): - return True - - def represent_none(self, data): - return self.represent_scalar(u'tag:yaml.org,2002:null', - u'null') - - def represent_str(self, data): - tag = None - style = None - try: - data = unicode(data, 'ascii') - tag = u'tag:yaml.org,2002:str' - except UnicodeDecodeError: - try: - data = unicode(data, 'utf-8') - tag = u'tag:yaml.org,2002:str' - except UnicodeDecodeError: - data = data.encode('base64') - tag = u'tag:yaml.org,2002:binary' - style = '|' - return self.represent_scalar(tag, data, style=style) - - def represent_unicode(self, data): - return self.represent_scalar(u'tag:yaml.org,2002:str', data) - - def represent_bool(self, data): - if data: - value = u'true' - else: - value = u'false' - return self.represent_scalar(u'tag:yaml.org,2002:bool', value) - - def represent_int(self, data): - return self.represent_scalar(u'tag:yaml.org,2002:int', unicode(data)) - - def represent_long(self, data): - return self.represent_scalar(u'tag:yaml.org,2002:int', unicode(data)) - - inf_value = 1e300 - while repr(inf_value) != repr(inf_value*inf_value): - inf_value *= inf_value - - def represent_float(self, data): - if data != data or (data == 0.0 and data == 1.0): - value = u'.nan' - elif data == self.inf_value: - value = u'.inf' - elif data == -self.inf_value: - value = u'-.inf' - else: - value = unicode(repr(data)).lower() - # Note that in some cases `repr(data)` represents a float number - # without the decimal parts. For instance: - # >>> repr(1e17) - # '1e17' - # Unfortunately, this is not a valid float representation according - # to the definition of the `!!float` tag. We fix this by adding - # '.0' before the 'e' symbol. - if u'.' not in value and u'e' in value: - value = value.replace(u'e', u'.0e', 1) - return self.represent_scalar(u'tag:yaml.org,2002:float', value) - - def represent_list(self, data): - #pairs = (len(data) > 0 and isinstance(data, list)) - #if pairs: - # for item in data: - # if not isinstance(item, tuple) or len(item) != 2: - # pairs = False - # break - #if not pairs: - return self.represent_sequence(u'tag:yaml.org,2002:seq', data) - #value = [] - #for item_key, item_value in data: - # value.append(self.represent_mapping(u'tag:yaml.org,2002:map', - # [(item_key, item_value)])) - #return SequenceNode(u'tag:yaml.org,2002:pairs', value) - - def represent_dict(self, data): - return self.represent_mapping(u'tag:yaml.org,2002:map', data) - - def represent_set(self, data): - value = {} - for key in data: - value[key] = None - return self.represent_mapping(u'tag:yaml.org,2002:set', value) - - def represent_date(self, data): - value = unicode(data.isoformat()) - return self.represent_scalar(u'tag:yaml.org,2002:timestamp', value) - - def represent_datetime(self, data): - value = unicode(data.isoformat(' ')) - return self.represent_scalar(u'tag:yaml.org,2002:timestamp', value) - - def represent_yaml_object(self, tag, data, cls, flow_style=None): - if hasattr(data, '__getstate__'): - state = data.__getstate__() - else: - state = data.__dict__.copy() - return self.represent_mapping(tag, state, flow_style=flow_style) - - def represent_undefined(self, data): - raise RepresenterError("cannot represent an object: %s" % data) - -SafeRepresenter.add_representer(type(None), - SafeRepresenter.represent_none) - -SafeRepresenter.add_representer(str, - SafeRepresenter.represent_str) - -SafeRepresenter.add_representer(unicode, - SafeRepresenter.represent_unicode) - -SafeRepresenter.add_representer(bool, - SafeRepresenter.represent_bool) - -SafeRepresenter.add_representer(int, - SafeRepresenter.represent_int) - -SafeRepresenter.add_representer(long, - SafeRepresenter.represent_long) - -SafeRepresenter.add_representer(float, - SafeRepresenter.represent_float) - -SafeRepresenter.add_representer(list, - SafeRepresenter.represent_list) - -SafeRepresenter.add_representer(tuple, - SafeRepresenter.represent_list) - -SafeRepresenter.add_representer(dict, - SafeRepresenter.represent_dict) - -SafeRepresenter.add_representer(set, - SafeRepresenter.represent_set) - -SafeRepresenter.add_representer(datetime.date, - SafeRepresenter.represent_date) - -SafeRepresenter.add_representer(datetime.datetime, - SafeRepresenter.represent_datetime) - -SafeRepresenter.add_representer(None, - SafeRepresenter.represent_undefined) - -class Representer(SafeRepresenter): - - def represent_str(self, data): - tag = None - style = None - try: - data = unicode(data, 'ascii') - tag = u'tag:yaml.org,2002:str' - except UnicodeDecodeError: - try: - data = unicode(data, 'utf-8') - tag = u'tag:yaml.org,2002:python/str' - except UnicodeDecodeError: - data = data.encode('base64') - tag = u'tag:yaml.org,2002:binary' - style = '|' - return self.represent_scalar(tag, data, style=style) - - def represent_unicode(self, data): - tag = None - try: - data.encode('ascii') - tag = u'tag:yaml.org,2002:python/unicode' - except UnicodeEncodeError: - tag = u'tag:yaml.org,2002:str' - return self.represent_scalar(tag, data) - - def represent_long(self, data): - tag = u'tag:yaml.org,2002:int' - if int(data) is not data: - tag = u'tag:yaml.org,2002:python/long' - return self.represent_scalar(tag, unicode(data)) - - def represent_complex(self, data): - if data.imag == 0.0: - data = u'%r' % data.real - elif data.real == 0.0: - data = u'%rj' % data.imag - elif data.imag > 0: - data = u'%r+%rj' % (data.real, data.imag) - else: - data = u'%r%rj' % (data.real, data.imag) - return self.represent_scalar(u'tag:yaml.org,2002:python/complex', data) - - def represent_tuple(self, data): - return self.represent_sequence(u'tag:yaml.org,2002:python/tuple', data) - - def represent_name(self, data): - name = u'%s.%s' % (data.__module__, data.__name__) - return self.represent_scalar(u'tag:yaml.org,2002:python/name:'+name, u'') - - def represent_module(self, data): - return self.represent_scalar( - u'tag:yaml.org,2002:python/module:'+data.__name__, u'') - - def represent_instance(self, data): - # For instances of classic classes, we use __getinitargs__ and - # __getstate__ to serialize the data. - - # If data.__getinitargs__ exists, the object must be reconstructed by - # calling cls(**args), where args is a tuple returned by - # __getinitargs__. Otherwise, the cls.__init__ method should never be - # called and the class instance is created by instantiating a trivial - # class and assigning to the instance's __class__ variable. - - # If data.__getstate__ exists, it returns the state of the object. - # Otherwise, the state of the object is data.__dict__. - - # We produce either a !!python/object or !!python/object/new node. - # If data.__getinitargs__ does not exist and state is a dictionary, we - # produce a !!python/object node . Otherwise we produce a - # !!python/object/new node. - - cls = data.__class__ - class_name = u'%s.%s' % (cls.__module__, cls.__name__) - args = None - state = None - if hasattr(data, '__getinitargs__'): - args = list(data.__getinitargs__()) - if hasattr(data, '__getstate__'): - state = data.__getstate__() - else: - state = data.__dict__ - if args is None and isinstance(state, dict): - return self.represent_mapping( - u'tag:yaml.org,2002:python/object:'+class_name, state) - if isinstance(state, dict) and not state: - return self.represent_sequence( - u'tag:yaml.org,2002:python/object/new:'+class_name, args) - value = {} - if args: - value['args'] = args - value['state'] = state - return self.represent_mapping( - u'tag:yaml.org,2002:python/object/new:'+class_name, value) - - def represent_object(self, data): - # We use __reduce__ API to save the data. data.__reduce__ returns - # a tuple of length 2-5: - # (function, args, state, listitems, dictitems) - - # For reconstructing, we calls function(*args), then set its state, - # listitems, and dictitems if they are not None. - - # A special case is when function.__name__ == '__newobj__'. In this - # case we create the object with args[0].__new__(*args). - - # Another special case is when __reduce__ returns a string - we don't - # support it. - - # We produce a !!python/object, !!python/object/new or - # !!python/object/apply node. - - cls = type(data) - if cls in copy_reg.dispatch_table: - reduce = copy_reg.dispatch_table[cls](data) - elif hasattr(data, '__reduce_ex__'): - reduce = data.__reduce_ex__(2) - elif hasattr(data, '__reduce__'): - reduce = data.__reduce__() - else: - raise RepresenterError("cannot represent object: %r" % data) - reduce = (list(reduce)+[None]*5)[:5] - function, args, state, listitems, dictitems = reduce - args = list(args) - if state is None: - state = {} - if listitems is not None: - listitems = list(listitems) - if dictitems is not None: - dictitems = dict(dictitems) - if function.__name__ == '__newobj__': - function = args[0] - args = args[1:] - tag = u'tag:yaml.org,2002:python/object/new:' - newobj = True - else: - tag = u'tag:yaml.org,2002:python/object/apply:' - newobj = False - function_name = u'%s.%s' % (function.__module__, function.__name__) - if not args and not listitems and not dictitems \ - and isinstance(state, dict) and newobj: - return self.represent_mapping( - u'tag:yaml.org,2002:python/object:'+function_name, state) - if not listitems and not dictitems \ - and isinstance(state, dict) and not state: - return self.represent_sequence(tag+function_name, args) - value = {} - if args: - value['args'] = args - if state or not isinstance(state, dict): - value['state'] = state - if listitems: - value['listitems'] = listitems - if dictitems: - value['dictitems'] = dictitems - return self.represent_mapping(tag+function_name, value) - -Representer.add_representer(str, - Representer.represent_str) - -Representer.add_representer(unicode, - Representer.represent_unicode) - -Representer.add_representer(long, - Representer.represent_long) - -Representer.add_representer(complex, - Representer.represent_complex) - -Representer.add_representer(tuple, - Representer.represent_tuple) - -Representer.add_representer(type, - Representer.represent_name) - -Representer.add_representer(types.ClassType, - Representer.represent_name) - -Representer.add_representer(types.FunctionType, - Representer.represent_name) - -Representer.add_representer(types.BuiltinFunctionType, - Representer.represent_name) - -Representer.add_representer(types.ModuleType, - Representer.represent_module) - -Representer.add_multi_representer(types.InstanceType, - Representer.represent_instance) - -Representer.add_multi_representer(object, - Representer.represent_object) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/resolver.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/resolver.py deleted file mode 100644 index 49922deb..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/resolver.py +++ /dev/null @@ -1,225 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['BaseResolver', 'Resolver'] - -from error import * -from nodes import * - -import re - -class ResolverError(YAMLError): - pass - -class BaseResolver(object): - - DEFAULT_SCALAR_TAG = u'tag:yaml.org,2002:str' - DEFAULT_SEQUENCE_TAG = u'tag:yaml.org,2002:seq' - DEFAULT_MAPPING_TAG = u'tag:yaml.org,2002:map' - - yaml_implicit_resolvers = {} - yaml_path_resolvers = {} - - def __init__(self): - self.resolver_exact_paths = [] - self.resolver_prefix_paths = [] - - def add_implicit_resolver(cls, tag, regexp, first): - if not 'yaml_implicit_resolvers' in cls.__dict__: - cls.yaml_implicit_resolvers = cls.yaml_implicit_resolvers.copy() - if first is None: - first = [None] - for ch in first: - cls.yaml_implicit_resolvers.setdefault(ch, []).append((tag, regexp)) - add_implicit_resolver = classmethod(add_implicit_resolver) - - def add_path_resolver(cls, tag, path, kind=None): - # Note: `add_path_resolver` is experimental. The API could be changed. - # `new_path` is a pattern that is matched against the path from the - # root to the node that is being considered. `node_path` elements are - # tuples `(node_check, index_check)`. `node_check` is a node class: - # `ScalarNode`, `SequenceNode`, `MappingNode` or `None`. `None` - # matches any kind of a node. `index_check` could be `None`, a boolean - # value, a string value, or a number. `None` and `False` match against - # any _value_ of sequence and mapping nodes. `True` matches against - # any _key_ of a mapping node. A string `index_check` matches against - # a mapping value that corresponds to a scalar key which content is - # equal to the `index_check` value. An integer `index_check` matches - # against a sequence value with the index equal to `index_check`. - if not 'yaml_path_resolvers' in cls.__dict__: - cls.yaml_path_resolvers = cls.yaml_path_resolvers.copy() - new_path = [] - for element in path: - if isinstance(element, (list, tuple)): - if len(element) == 2: - node_check, index_check = element - elif len(element) == 1: - node_check = element[0] - index_check = True - else: - raise ResolverError("Invalid path element: %s" % element) - else: - node_check = None - index_check = element - if node_check is str: - node_check = ScalarNode - elif node_check is list: - node_check = SequenceNode - elif node_check is dict: - node_check = MappingNode - elif node_check not in [ScalarNode, SequenceNode, MappingNode] \ - and not isinstance(node_check, basestring) \ - and node_check is not None: - raise ResolverError("Invalid node checker: %s" % node_check) - if not isinstance(index_check, (basestring, int)) \ - and index_check is not None: - raise ResolverError("Invalid index checker: %s" % index_check) - new_path.append((node_check, index_check)) - if kind is str: - kind = ScalarNode - elif kind is list: - kind = SequenceNode - elif kind is dict: - kind = MappingNode - elif kind not in [ScalarNode, SequenceNode, MappingNode] \ - and kind is not None: - raise ResolverError("Invalid node kind: %s" % kind) - cls.yaml_path_resolvers[tuple(new_path), kind] = tag - add_path_resolver = classmethod(add_path_resolver) - - def descend_resolver(self, current_node, current_index): - if not self.yaml_path_resolvers: - return - exact_paths = {} - prefix_paths = [] - if current_node: - depth = len(self.resolver_prefix_paths) - for path, kind in self.resolver_prefix_paths[-1]: - if self.check_resolver_prefix(depth, path, kind, - current_node, current_index): - if len(path) > depth: - prefix_paths.append((path, kind)) - else: - exact_paths[kind] = self.yaml_path_resolvers[path, kind] - else: - for path, kind in self.yaml_path_resolvers: - if not path: - exact_paths[kind] = self.yaml_path_resolvers[path, kind] - else: - prefix_paths.append((path, kind)) - self.resolver_exact_paths.append(exact_paths) - self.resolver_prefix_paths.append(prefix_paths) - - def ascend_resolver(self): - if not self.yaml_path_resolvers: - return - self.resolver_exact_paths.pop() - self.resolver_prefix_paths.pop() - - def check_resolver_prefix(self, depth, path, kind, - current_node, current_index): - node_check, index_check = path[depth-1] - if isinstance(node_check, basestring): - if current_node.tag != node_check: - return - elif node_check is not None: - if not isinstance(current_node, node_check): - return - if index_check is True and current_index is not None: - return - if (index_check is False or index_check is None) \ - and current_index is None: - return - if isinstance(index_check, basestring): - if not (isinstance(current_index, ScalarNode) - and index_check == current_index.value): - return - elif isinstance(index_check, int) and not isinstance(index_check, bool): - if index_check != current_index: - return - return True - - def resolve(self, kind, value, implicit): - if kind is ScalarNode and implicit[0]: - if value == u'': - resolvers = self.yaml_implicit_resolvers.get(u'', []) - else: - resolvers = self.yaml_implicit_resolvers.get(value[0], []) - resolvers += self.yaml_implicit_resolvers.get(None, []) - for tag, regexp in resolvers: - if regexp.match(value): - return tag - implicit = implicit[1] - if self.yaml_path_resolvers: - exact_paths = self.resolver_exact_paths[-1] - if kind in exact_paths: - return exact_paths[kind] - if None in exact_paths: - return exact_paths[None] - if kind is ScalarNode: - return self.DEFAULT_SCALAR_TAG - elif kind is SequenceNode: - return self.DEFAULT_SEQUENCE_TAG - elif kind is MappingNode: - return self.DEFAULT_MAPPING_TAG - -class Resolver(BaseResolver): - pass - -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:bool', - re.compile(ur'''^(?:yes|Yes|YES|no|No|NO - |true|True|TRUE|false|False|FALSE - |on|On|ON|off|Off|OFF)$''', re.X), - list(u'yYnNtTfFoO')) - -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:float', - re.compile(ur'''^(?:[-+]?(?:[0-9][0-9_]*)\.[0-9_]*(?:[eE][-+][0-9]+)? - |\.[0-9_]+(?:[eE][-+][0-9]+)? - |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\.[0-9_]* - |[-+]?\.(?:inf|Inf|INF) - |\.(?:nan|NaN|NAN))$''', re.X), - list(u'-+0123456789.')) - -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:int', - re.compile(ur'''^(?:[-+]?0b[0-1_]+ - |[-+]?0[0-7_]+ - |[-+]?(?:0|[1-9][0-9_]*) - |[-+]?0x[0-9a-fA-F_]+ - |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9])+)$''', re.X), - list(u'-+0123456789')) - -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:merge', - re.compile(ur'^(?:<<)$'), - [u'<']) - -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:null', - re.compile(ur'''^(?: ~ - |null|Null|NULL - | )$''', re.X), - [u'~', u'n', u'N', u'']) - -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:timestamp', - re.compile(ur'''^(?:[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] - |[0-9][0-9][0-9][0-9] -[0-9][0-9]? -[0-9][0-9]? - (?:[Tt]|[ \t]+)[0-9][0-9]? - :[0-9][0-9] :[0-9][0-9] (?:\.[0-9]*)? - (?:[ \t]*(?:Z|[-+][0-9][0-9]?(?::[0-9][0-9])?))?)$''', re.X), - list(u'0123456789')) - -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:value', - re.compile(ur'^(?:=)$'), - [u'=']) - -# The following resolver is only for documentation purposes. It cannot work -# because plain scalars cannot start with '!', '&', or '*'. -Resolver.add_implicit_resolver( - u'tag:yaml.org,2002:yaml', - re.compile(ur'^(?:!|&|\*)$'), - list(u'!&*')) - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/scanner.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/scanner.py deleted file mode 100644 index 971da612..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/scanner.py +++ /dev/null @@ -1,1458 +0,0 @@ -# SPDX-License-Identifier: MIT - -# Scanner produces tokens of the following types: -# STREAM-START -# STREAM-END -# DIRECTIVE(name, value) -# DOCUMENT-START -# DOCUMENT-END -# BLOCK-SEQUENCE-START -# BLOCK-MAPPING-START -# BLOCK-END -# FLOW-SEQUENCE-START -# FLOW-MAPPING-START -# FLOW-SEQUENCE-END -# FLOW-MAPPING-END -# BLOCK-ENTRY -# FLOW-ENTRY -# KEY -# VALUE -# ALIAS(value) -# ANCHOR(value) -# TAG(value) -# SCALAR(value, plain, style) -# -# Read comments in the Scanner code for more details. -# - -__all__ = ['Scanner', 'ScannerError'] - -from error import MarkedYAMLError -from tokens import * - -class ScannerError(MarkedYAMLError): - pass - -class SimpleKey(object): - # See below simple keys treatment. - - def __init__(self, token_number, required, index, line, column, mark): - self.token_number = token_number - self.required = required - self.index = index - self.line = line - self.column = column - self.mark = mark - -class Scanner(object): - - def __init__(self): - """Initialize the scanner.""" - # It is assumed that Scanner and Reader will have a common descendant. - # Reader do the dirty work of checking for BOM and converting the - # input data to Unicode. It also adds NUL to the end. - # - # Reader supports the following methods - # self.peek(i=0) # peek the next i-th character - # self.prefix(l=1) # peek the next l characters - # self.forward(l=1) # read the next l characters and move the pointer. - - # Had we reached the end of the stream? - self.done = False - - # The number of unclosed '{' and '['. `flow_level == 0` means block - # context. - self.flow_level = 0 - - # List of processed tokens that are not yet emitted. - self.tokens = [] - - # Add the STREAM-START token. - self.fetch_stream_start() - - # Number of tokens that were emitted through the `get_token` method. - self.tokens_taken = 0 - - # The current indentation level. - self.indent = -1 - - # Past indentation levels. - self.indents = [] - - # Variables related to simple keys treatment. - - # A simple key is a key that is not denoted by the '?' indicator. - # Example of simple keys: - # --- - # block simple key: value - # ? not a simple key: - # : { flow simple key: value } - # We emit the KEY token before all keys, so when we find a potential - # simple key, we try to locate the corresponding ':' indicator. - # Simple keys should be limited to a single line and 1024 characters. - - # Can a simple key start at the current position? A simple key may - # start: - # - at the beginning of the line, not counting indentation spaces - # (in block context), - # - after '{', '[', ',' (in the flow context), - # - after '?', ':', '-' (in the block context). - # In the block context, this flag also signifies if a block collection - # may start at the current position. - self.allow_simple_key = True - - # Keep track of possible simple keys. This is a dictionary. The key - # is `flow_level`; there can be no more that one possible simple key - # for each level. The value is a SimpleKey record: - # (token_number, required, index, line, column, mark) - # A simple key may start with ALIAS, ANCHOR, TAG, SCALAR(flow), - # '[', or '{' tokens. - self.possible_simple_keys = {} - - # Public methods. - - def check_token(self, *choices): - # Check if the next token is one of the given types. - while self.need_more_tokens(): - self.fetch_more_tokens() - if self.tokens: - if not choices: - return True - for choice in choices: - if isinstance(self.tokens[0], choice): - return True - return False - - def peek_token(self): - # Return the next token, but do not delete if from the queue. - while self.need_more_tokens(): - self.fetch_more_tokens() - if self.tokens: - return self.tokens[0] - - def get_token(self): - # Return the next token. - while self.need_more_tokens(): - self.fetch_more_tokens() - if self.tokens: - self.tokens_taken += 1 - return self.tokens.pop(0) - - # Private methods. - - def need_more_tokens(self): - if self.done: - return False - if not self.tokens: - return True - # The current token may be a potential simple key, so we - # need to look further. - self.stale_possible_simple_keys() - if self.next_possible_simple_key() == self.tokens_taken: - return True - - def fetch_more_tokens(self): - - # Eat whitespaces and comments until we reach the next token. - self.scan_to_next_token() - - # Remove obsolete possible simple keys. - self.stale_possible_simple_keys() - - # Compare the current indentation and column. It may add some tokens - # and decrease the current indentation level. - self.unwind_indent(self.column) - - # Peek the next character. - ch = self.peek() - - # Is it the end of stream? - if ch == u'\0': - return self.fetch_stream_end() - - # Is it a directive? - if ch == u'%' and self.check_directive(): - return self.fetch_directive() - - # Is it the document start? - if ch == u'-' and self.check_document_start(): - return self.fetch_document_start() - - # Is it the document end? - if ch == u'.' and self.check_document_end(): - return self.fetch_document_end() - - # TODO: support for BOM within a stream. - #if ch == u'\uFEFF': - # return self.fetch_bom() <-- issue BOMToken - - # Note: the order of the following checks is NOT significant. - - # Is it the flow sequence start indicator? - if ch == u'[': - return self.fetch_flow_sequence_start() - - # Is it the flow mapping start indicator? - if ch == u'{': - return self.fetch_flow_mapping_start() - - # Is it the flow sequence end indicator? - if ch == u']': - return self.fetch_flow_sequence_end() - - # Is it the flow mapping end indicator? - if ch == u'}': - return self.fetch_flow_mapping_end() - - # Is it the flow entry indicator? - if ch == u',': - return self.fetch_flow_entry() - - # Is it the block entry indicator? - if ch == u'-' and self.check_block_entry(): - return self.fetch_block_entry() - - # Is it the key indicator? - if ch == u'?' and self.check_key(): - return self.fetch_key() - - # Is it the value indicator? - if ch == u':' and self.check_value(): - return self.fetch_value() - - # Is it an alias? - if ch == u'*': - return self.fetch_alias() - - # Is it an anchor? - if ch == u'&': - return self.fetch_anchor() - - # Is it a tag? - if ch == u'!': - return self.fetch_tag() - - # Is it a literal scalar? - if ch == u'|' and not self.flow_level: - return self.fetch_literal() - - # Is it a folded scalar? - if ch == u'>' and not self.flow_level: - return self.fetch_folded() - - # Is it a single quoted scalar? - if ch == u'\'': - return self.fetch_single() - - # Is it a double quoted scalar? - if ch == u'\"': - return self.fetch_double() - - # It must be a plain scalar then. - if self.check_plain(): - return self.fetch_plain() - - # No? It's an error. Let's produce a nice error message. - raise ScannerError("while scanning for the next token", None, - "found character %r that cannot start any token" - % ch.encode('utf-8'), self.get_mark()) - - # Simple keys treatment. - - def next_possible_simple_key(self): - # Return the number of the nearest possible simple key. Actually we - # don't need to loop through the whole dictionary. We may replace it - # with the following code: - # if not self.possible_simple_keys: - # return None - # return self.possible_simple_keys[ - # min(self.possible_simple_keys.keys())].token_number - min_token_number = None - for level in self.possible_simple_keys: - key = self.possible_simple_keys[level] - if min_token_number is None or key.token_number < min_token_number: - min_token_number = key.token_number - return min_token_number - - def stale_possible_simple_keys(self): - # Remove entries that are no longer possible simple keys. According to - # the YAML specification, simple keys - # - should be limited to a single line, - # - should be no longer than 1024 characters. - # Disabling this procedure will allow simple keys of any length and - # height (may cause problems if indentation is broken though). - for level in self.possible_simple_keys.keys(): - key = self.possible_simple_keys[level] - if key.line != self.line \ - or self.index-key.index > 1024: - if key.required: - raise ScannerError("while scanning a simple key", key.mark, - "could not found expected ':'", self.get_mark()) - del self.possible_simple_keys[level] - - def save_possible_simple_key(self): - # The next token may start a simple key. We check if it's possible - # and save its position. This function is called for - # ALIAS, ANCHOR, TAG, SCALAR(flow), '[', and '{'. - - # Check if a simple key is required at the current position. - required = not self.flow_level and self.indent == self.column - - # A simple key is required only if it is the first token in the current - # line. Therefore it is always allowed. - assert self.allow_simple_key or not required - - # The next token might be a simple key. Let's save it's number and - # position. - if self.allow_simple_key: - self.remove_possible_simple_key() - token_number = self.tokens_taken+len(self.tokens) - key = SimpleKey(token_number, required, - self.index, self.line, self.column, self.get_mark()) - self.possible_simple_keys[self.flow_level] = key - - def remove_possible_simple_key(self): - # Remove the saved possible key position at the current flow level. - if self.flow_level in self.possible_simple_keys: - key = self.possible_simple_keys[self.flow_level] - - if key.required: - raise ScannerError("while scanning a simple key", key.mark, - "could not found expected ':'", self.get_mark()) - - del self.possible_simple_keys[self.flow_level] - - # Indentation functions. - - def unwind_indent(self, column): - - ## In flow context, tokens should respect indentation. - ## Actually the condition should be `self.indent >= column` according to - ## the spec. But this condition will prohibit intuitively correct - ## constructions such as - ## key : { - ## } - #if self.flow_level and self.indent > column: - # raise ScannerError(None, None, - # "invalid intendation or unclosed '[' or '{'", - # self.get_mark()) - - # In the flow context, indentation is ignored. We make the scanner less - # restrictive then specification requires. - if self.flow_level: - return - - # In block context, we may need to issue the BLOCK-END tokens. - while self.indent > column: - mark = self.get_mark() - self.indent = self.indents.pop() - self.tokens.append(BlockEndToken(mark, mark)) - - def add_indent(self, column): - # Check if we need to increase indentation. - if self.indent < column: - self.indents.append(self.indent) - self.indent = column - return True - return False - - # Fetchers. - - def fetch_stream_start(self): - # We always add STREAM-START as the first token and STREAM-END as the - # last token. - - # Read the token. - mark = self.get_mark() - - # Add STREAM-START. - self.tokens.append(StreamStartToken(mark, mark, - encoding=self.encoding)) - - - def fetch_stream_end(self): - - # Set the current intendation to -1. - self.unwind_indent(-1) - - # Reset simple keys. - self.remove_possible_simple_key() - self.allow_simple_key = False - self.possible_simple_keys = {} - - # Read the token. - mark = self.get_mark() - - # Add STREAM-END. - self.tokens.append(StreamEndToken(mark, mark)) - - # The steam is finished. - self.done = True - - def fetch_directive(self): - - # Set the current intendation to -1. - self.unwind_indent(-1) - - # Reset simple keys. - self.remove_possible_simple_key() - self.allow_simple_key = False - - # Scan and add DIRECTIVE. - self.tokens.append(self.scan_directive()) - - def fetch_document_start(self): - self.fetch_document_indicator(DocumentStartToken) - - def fetch_document_end(self): - self.fetch_document_indicator(DocumentEndToken) - - def fetch_document_indicator(self, TokenClass): - - # Set the current intendation to -1. - self.unwind_indent(-1) - - # Reset simple keys. Note that there could not be a block collection - # after '---'. - self.remove_possible_simple_key() - self.allow_simple_key = False - - # Add DOCUMENT-START or DOCUMENT-END. - start_mark = self.get_mark() - self.forward(3) - end_mark = self.get_mark() - self.tokens.append(TokenClass(start_mark, end_mark)) - - def fetch_flow_sequence_start(self): - self.fetch_flow_collection_start(FlowSequenceStartToken) - - def fetch_flow_mapping_start(self): - self.fetch_flow_collection_start(FlowMappingStartToken) - - def fetch_flow_collection_start(self, TokenClass): - - # '[' and '{' may start a simple key. - self.save_possible_simple_key() - - # Increase the flow level. - self.flow_level += 1 - - # Simple keys are allowed after '[' and '{'. - self.allow_simple_key = True - - # Add FLOW-SEQUENCE-START or FLOW-MAPPING-START. - start_mark = self.get_mark() - self.forward() - end_mark = self.get_mark() - self.tokens.append(TokenClass(start_mark, end_mark)) - - def fetch_flow_sequence_end(self): - self.fetch_flow_collection_end(FlowSequenceEndToken) - - def fetch_flow_mapping_end(self): - self.fetch_flow_collection_end(FlowMappingEndToken) - - def fetch_flow_collection_end(self, TokenClass): - - # Reset possible simple key on the current level. - self.remove_possible_simple_key() - - # Decrease the flow level. - self.flow_level -= 1 - - # No simple keys after ']' or '}'. - self.allow_simple_key = False - - # Add FLOW-SEQUENCE-END or FLOW-MAPPING-END. - start_mark = self.get_mark() - self.forward() - end_mark = self.get_mark() - self.tokens.append(TokenClass(start_mark, end_mark)) - - def fetch_flow_entry(self): - - # Simple keys are allowed after ','. - self.allow_simple_key = True - - # Reset possible simple key on the current level. - self.remove_possible_simple_key() - - # Add FLOW-ENTRY. - start_mark = self.get_mark() - self.forward() - end_mark = self.get_mark() - self.tokens.append(FlowEntryToken(start_mark, end_mark)) - - def fetch_block_entry(self): - - # Block context needs additional checks. - if not self.flow_level: - - # Are we allowed to start a new entry? - if not self.allow_simple_key: - raise ScannerError(None, None, - "sequence entries are not allowed here", - self.get_mark()) - - # We may need to add BLOCK-SEQUENCE-START. - if self.add_indent(self.column): - mark = self.get_mark() - self.tokens.append(BlockSequenceStartToken(mark, mark)) - - # It's an error for the block entry to occur in the flow context, - # but we let the parser detect this. - else: - pass - - # Simple keys are allowed after '-'. - self.allow_simple_key = True - - # Reset possible simple key on the current level. - self.remove_possible_simple_key() - - # Add BLOCK-ENTRY. - start_mark = self.get_mark() - self.forward() - end_mark = self.get_mark() - self.tokens.append(BlockEntryToken(start_mark, end_mark)) - - def fetch_key(self): - - # Block context needs additional checks. - if not self.flow_level: - - # Are we allowed to start a key (not nessesary a simple)? - if not self.allow_simple_key: - raise ScannerError(None, None, - "mapping keys are not allowed here", - self.get_mark()) - - # We may need to add BLOCK-MAPPING-START. - if self.add_indent(self.column): - mark = self.get_mark() - self.tokens.append(BlockMappingStartToken(mark, mark)) - - # Simple keys are allowed after '?' in the block context. - self.allow_simple_key = not self.flow_level - - # Reset possible simple key on the current level. - self.remove_possible_simple_key() - - # Add KEY. - start_mark = self.get_mark() - self.forward() - end_mark = self.get_mark() - self.tokens.append(KeyToken(start_mark, end_mark)) - - def fetch_value(self): - - # Do we determine a simple key? - if self.flow_level in self.possible_simple_keys: - - # Add KEY. - key = self.possible_simple_keys[self.flow_level] - del self.possible_simple_keys[self.flow_level] - self.tokens.insert(key.token_number-self.tokens_taken, - KeyToken(key.mark, key.mark)) - - # If this key starts a new block mapping, we need to add - # BLOCK-MAPPING-START. - if not self.flow_level: - if self.add_indent(key.column): - self.tokens.insert(key.token_number-self.tokens_taken, - BlockMappingStartToken(key.mark, key.mark)) - - # There cannot be two simple keys one after another. - self.allow_simple_key = False - - # It must be a part of a complex key. - else: - - # Block context needs additional checks. - # (Do we really need them? They will be catched by the parser - # anyway.) - if not self.flow_level: - - # We are allowed to start a complex value if and only if - # we can start a simple key. - if not self.allow_simple_key: - raise ScannerError(None, None, - "mapping values are not allowed here", - self.get_mark()) - - # If this value starts a new block mapping, we need to add - # BLOCK-MAPPING-START. It will be detected as an error later by - # the parser. - if not self.flow_level: - if self.add_indent(self.column): - mark = self.get_mark() - self.tokens.append(BlockMappingStartToken(mark, mark)) - - # Simple keys are allowed after ':' in the block context. - self.allow_simple_key = not self.flow_level - - # Reset possible simple key on the current level. - self.remove_possible_simple_key() - - # Add VALUE. - start_mark = self.get_mark() - self.forward() - end_mark = self.get_mark() - self.tokens.append(ValueToken(start_mark, end_mark)) - - def fetch_alias(self): - - # ALIAS could be a simple key. - self.save_possible_simple_key() - - # No simple keys after ALIAS. - self.allow_simple_key = False - - # Scan and add ALIAS. - self.tokens.append(self.scan_anchor(AliasToken)) - - def fetch_anchor(self): - - # ANCHOR could start a simple key. - self.save_possible_simple_key() - - # No simple keys after ANCHOR. - self.allow_simple_key = False - - # Scan and add ANCHOR. - self.tokens.append(self.scan_anchor(AnchorToken)) - - def fetch_tag(self): - - # TAG could start a simple key. - self.save_possible_simple_key() - - # No simple keys after TAG. - self.allow_simple_key = False - - # Scan and add TAG. - self.tokens.append(self.scan_tag()) - - def fetch_literal(self): - self.fetch_block_scalar(style='|') - - def fetch_folded(self): - self.fetch_block_scalar(style='>') - - def fetch_block_scalar(self, style): - - # A simple key may follow a block scalar. - self.allow_simple_key = True - - # Reset possible simple key on the current level. - self.remove_possible_simple_key() - - # Scan and add SCALAR. - self.tokens.append(self.scan_block_scalar(style)) - - def fetch_single(self): - self.fetch_flow_scalar(style='\'') - - def fetch_double(self): - self.fetch_flow_scalar(style='"') - - def fetch_flow_scalar(self, style): - - # A flow scalar could be a simple key. - self.save_possible_simple_key() - - # No simple keys after flow scalars. - self.allow_simple_key = False - - # Scan and add SCALAR. - self.tokens.append(self.scan_flow_scalar(style)) - - def fetch_plain(self): - - # A plain scalar could be a simple key. - self.save_possible_simple_key() - - # No simple keys after plain scalars. But note that `scan_plain` will - # change this flag if the scan is finished at the beginning of the - # line. - self.allow_simple_key = False - - # Scan and add SCALAR. May change `allow_simple_key`. - self.tokens.append(self.scan_plain()) - - # Checkers. - - def check_directive(self): - - # DIRECTIVE: ^ '%' ... - # The '%' indicator is already checked. - if self.column == 0: - return True - - def check_document_start(self): - - # DOCUMENT-START: ^ '---' (' '|'\n') - if self.column == 0: - if self.prefix(3) == u'---' \ - and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': - return True - - def check_document_end(self): - - # DOCUMENT-END: ^ '...' (' '|'\n') - if self.column == 0: - if self.prefix(3) == u'...' \ - and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': - return True - - def check_block_entry(self): - - # BLOCK-ENTRY: '-' (' '|'\n') - return self.peek(1) in u'\0 \t\r\n\x85\u2028\u2029' - - def check_key(self): - - # KEY(flow context): '?' - if self.flow_level: - return True - - # KEY(block context): '?' (' '|'\n') - else: - return self.peek(1) in u'\0 \t\r\n\x85\u2028\u2029' - - def check_value(self): - - # VALUE(flow context): ':' - if self.flow_level: - return True - - # VALUE(block context): ':' (' '|'\n') - else: - return self.peek(1) in u'\0 \t\r\n\x85\u2028\u2029' - - def check_plain(self): - - # A plain scalar may start with any non-space character except: - # '-', '?', ':', ',', '[', ']', '{', '}', - # '#', '&', '*', '!', '|', '>', '\'', '\"', - # '%', '@', '`'. - # - # It may also start with - # '-', '?', ':' - # if it is followed by a non-space character. - # - # Note that we limit the last rule to the block context (except the - # '-' character) because we want the flow context to be space - # independent. - ch = self.peek() - return ch not in u'\0 \t\r\n\x85\u2028\u2029-?:,[]{}#&*!|>\'\"%@`' \ - or (self.peek(1) not in u'\0 \t\r\n\x85\u2028\u2029' - and (ch == u'-' or (not self.flow_level and ch in u'?:'))) - - # Scanners. - - def scan_to_next_token(self): - # We ignore spaces, line breaks and comments. - # If we find a line break in the block context, we set the flag - # `allow_simple_key` on. - # The byte order mark is stripped if it's the first character in the - # stream. We do not yet support BOM inside the stream as the - # specification requires. Any such mark will be considered as a part - # of the document. - # - # TODO: We need to make tab handling rules more sane. A good rule is - # Tabs cannot precede tokens - # BLOCK-SEQUENCE-START, BLOCK-MAPPING-START, BLOCK-END, - # KEY(block), VALUE(block), BLOCK-ENTRY - # So the checking code is - # if <TAB>: - # self.allow_simple_keys = False - # We also need to add the check for `allow_simple_keys == True` to - # `unwind_indent` before issuing BLOCK-END. - # Scanners for block, flow, and plain scalars need to be modified. - - if self.index == 0 and self.peek() == u'\uFEFF': - self.forward() - found = False - while not found: - while self.peek() == u' ': - self.forward() - if self.peek() == u'#': - while self.peek() not in u'\0\r\n\x85\u2028\u2029': - self.forward() - if self.scan_line_break(): - if not self.flow_level: - self.allow_simple_key = True - else: - found = True - - def scan_directive(self): - # See the specification for details. - start_mark = self.get_mark() - self.forward() - name = self.scan_directive_name(start_mark) - value = None - if name == u'YAML': - value = self.scan_yaml_directive_value(start_mark) - end_mark = self.get_mark() - elif name == u'TAG': - value = self.scan_tag_directive_value(start_mark) - end_mark = self.get_mark() - else: - end_mark = self.get_mark() - while self.peek() not in u'\0\r\n\x85\u2028\u2029': - self.forward() - self.scan_directive_ignored_line(start_mark) - return DirectiveToken(name, value, start_mark, end_mark) - - def scan_directive_name(self, start_mark): - # See the specification for details. - length = 0 - ch = self.peek(length) - while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-_': - length += 1 - ch = self.peek(length) - if not length: - raise ScannerError("while scanning a directive", start_mark, - "expected alphabetic or numeric character, but found %r" - % ch.encode('utf-8'), self.get_mark()) - value = self.prefix(length) - self.forward(length) - ch = self.peek() - if ch not in u'\0 \r\n\x85\u2028\u2029': - raise ScannerError("while scanning a directive", start_mark, - "expected alphabetic or numeric character, but found %r" - % ch.encode('utf-8'), self.get_mark()) - return value - - def scan_yaml_directive_value(self, start_mark): - # See the specification for details. - while self.peek() == u' ': - self.forward() - major = self.scan_yaml_directive_number(start_mark) - if self.peek() != '.': - raise ScannerError("while scanning a directive", start_mark, - "expected a digit or '.', but found %r" - % self.peek().encode('utf-8'), - self.get_mark()) - self.forward() - minor = self.scan_yaml_directive_number(start_mark) - if self.peek() not in u'\0 \r\n\x85\u2028\u2029': - raise ScannerError("while scanning a directive", start_mark, - "expected a digit or ' ', but found %r" - % self.peek().encode('utf-8'), - self.get_mark()) - return (major, minor) - - def scan_yaml_directive_number(self, start_mark): - # See the specification for details. - ch = self.peek() - if not (u'0' <= ch <= u'9'): - raise ScannerError("while scanning a directive", start_mark, - "expected a digit, but found %r" % ch.encode('utf-8'), - self.get_mark()) - length = 0 - while u'0' <= self.peek(length) <= u'9': - length += 1 - value = int(self.prefix(length)) - self.forward(length) - return value - - def scan_tag_directive_value(self, start_mark): - # See the specification for details. - while self.peek() == u' ': - self.forward() - handle = self.scan_tag_directive_handle(start_mark) - while self.peek() == u' ': - self.forward() - prefix = self.scan_tag_directive_prefix(start_mark) - return (handle, prefix) - - def scan_tag_directive_handle(self, start_mark): - # See the specification for details. - value = self.scan_tag_handle('directive', start_mark) - ch = self.peek() - if ch != u' ': - raise ScannerError("while scanning a directive", start_mark, - "expected ' ', but found %r" % ch.encode('utf-8'), - self.get_mark()) - return value - - def scan_tag_directive_prefix(self, start_mark): - # See the specification for details. - value = self.scan_tag_uri('directive', start_mark) - ch = self.peek() - if ch not in u'\0 \r\n\x85\u2028\u2029': - raise ScannerError("while scanning a directive", start_mark, - "expected ' ', but found %r" % ch.encode('utf-8'), - self.get_mark()) - return value - - def scan_directive_ignored_line(self, start_mark): - # See the specification for details. - while self.peek() == u' ': - self.forward() - if self.peek() == u'#': - while self.peek() not in u'\0\r\n\x85\u2028\u2029': - self.forward() - ch = self.peek() - if ch not in u'\0\r\n\x85\u2028\u2029': - raise ScannerError("while scanning a directive", start_mark, - "expected a comment or a line break, but found %r" - % ch.encode('utf-8'), self.get_mark()) - self.scan_line_break() - - def scan_anchor(self, TokenClass): - # The specification does not restrict characters for anchors and - # aliases. This may lead to problems, for instance, the document: - # [ *alias, value ] - # can be interpteted in two ways, as - # [ "value" ] - # and - # [ *alias , "value" ] - # Therefore we restrict aliases to numbers and ASCII letters. - start_mark = self.get_mark() - indicator = self.peek() - if indicator == u'*': - name = 'alias' - else: - name = 'anchor' - self.forward() - length = 0 - ch = self.peek(length) - while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-_': - length += 1 - ch = self.peek(length) - if not length: - raise ScannerError("while scanning an %s" % name, start_mark, - "expected alphabetic or numeric character, but found %r" - % ch.encode('utf-8'), self.get_mark()) - value = self.prefix(length) - self.forward(length) - ch = self.peek() - if ch not in u'\0 \t\r\n\x85\u2028\u2029?:,]}%@`': - raise ScannerError("while scanning an %s" % name, start_mark, - "expected alphabetic or numeric character, but found %r" - % ch.encode('utf-8'), self.get_mark()) - end_mark = self.get_mark() - return TokenClass(value, start_mark, end_mark) - - def scan_tag(self): - # See the specification for details. - start_mark = self.get_mark() - ch = self.peek(1) - if ch == u'<': - handle = None - self.forward(2) - suffix = self.scan_tag_uri('tag', start_mark) - if self.peek() != u'>': - raise ScannerError("while parsing a tag", start_mark, - "expected '>', but found %r" % self.peek().encode('utf-8'), - self.get_mark()) - self.forward() - elif ch in u'\0 \t\r\n\x85\u2028\u2029': - handle = None - suffix = u'!' - self.forward() - else: - length = 1 - use_handle = False - while ch not in u'\0 \r\n\x85\u2028\u2029': - if ch == u'!': - use_handle = True - break - length += 1 - ch = self.peek(length) - handle = u'!' - if use_handle: - handle = self.scan_tag_handle('tag', start_mark) - else: - handle = u'!' - self.forward() - suffix = self.scan_tag_uri('tag', start_mark) - ch = self.peek() - if ch not in u'\0 \r\n\x85\u2028\u2029': - raise ScannerError("while scanning a tag", start_mark, - "expected ' ', but found %r" % ch.encode('utf-8'), - self.get_mark()) - value = (handle, suffix) - end_mark = self.get_mark() - return TagToken(value, start_mark, end_mark) - - def scan_block_scalar(self, style): - # See the specification for details. - - if style == '>': - folded = True - else: - folded = False - - chunks = [] - start_mark = self.get_mark() - - # Scan the header. - self.forward() - chomping, increment = self.scan_block_scalar_indicators(start_mark) - self.scan_block_scalar_ignored_line(start_mark) - - # Determine the indentation level and go to the first non-empty line. - min_indent = self.indent+1 - if min_indent < 1: - min_indent = 1 - if increment is None: - breaks, max_indent, end_mark = self.scan_block_scalar_indentation() - indent = max(min_indent, max_indent) - else: - indent = min_indent+increment-1 - breaks, end_mark = self.scan_block_scalar_breaks(indent) - line_break = u'' - - # Scan the inner part of the block scalar. - while self.column == indent and self.peek() != u'\0': - chunks.extend(breaks) - leading_non_space = self.peek() not in u' \t' - length = 0 - while self.peek(length) not in u'\0\r\n\x85\u2028\u2029': - length += 1 - chunks.append(self.prefix(length)) - self.forward(length) - line_break = self.scan_line_break() - breaks, end_mark = self.scan_block_scalar_breaks(indent) - if self.column == indent and self.peek() != u'\0': - - # Unfortunately, folding rules are ambiguous. - # - # This is the folding according to the specification: - - if folded and line_break == u'\n' \ - and leading_non_space and self.peek() not in u' \t': - if not breaks: - chunks.append(u' ') - else: - chunks.append(line_break) - - # This is Clark Evans's interpretation (also in the spec - # examples): - # - #if folded and line_break == u'\n': - # if not breaks: - # if self.peek() not in ' \t': - # chunks.append(u' ') - # else: - # chunks.append(line_break) - #else: - # chunks.append(line_break) - else: - break - - # Chomp the tail. - if chomping is not False: - chunks.append(line_break) - if chomping is True: - chunks.extend(breaks) - - # We are done. - return ScalarToken(u''.join(chunks), False, start_mark, end_mark, - style) - - def scan_block_scalar_indicators(self, start_mark): - # See the specification for details. - chomping = None - increment = None - ch = self.peek() - if ch in u'+-': - if ch == '+': - chomping = True - else: - chomping = False - self.forward() - ch = self.peek() - if ch in u'0123456789': - increment = int(ch) - if increment == 0: - raise ScannerError("while scanning a block scalar", start_mark, - "expected indentation indicator in the range 1-9, but found 0", - self.get_mark()) - self.forward() - elif ch in u'0123456789': - increment = int(ch) - if increment == 0: - raise ScannerError("while scanning a block scalar", start_mark, - "expected indentation indicator in the range 1-9, but found 0", - self.get_mark()) - self.forward() - ch = self.peek() - if ch in u'+-': - if ch == '+': - chomping = True - else: - chomping = False - self.forward() - ch = self.peek() - if ch not in u'\0 \r\n\x85\u2028\u2029': - raise ScannerError("while scanning a block scalar", start_mark, - "expected chomping or indentation indicators, but found %r" - % ch.encode('utf-8'), self.get_mark()) - return chomping, increment - - def scan_block_scalar_ignored_line(self, start_mark): - # See the specification for details. - while self.peek() == u' ': - self.forward() - if self.peek() == u'#': - while self.peek() not in u'\0\r\n\x85\u2028\u2029': - self.forward() - ch = self.peek() - if ch not in u'\0\r\n\x85\u2028\u2029': - raise ScannerError("while scanning a block scalar", start_mark, - "expected a comment or a line break, but found %r" - % ch.encode('utf-8'), self.get_mark()) - self.scan_line_break() - - def scan_block_scalar_indentation(self): - # See the specification for details. - chunks = [] - max_indent = 0 - end_mark = self.get_mark() - while self.peek() in u' \r\n\x85\u2028\u2029': - if self.peek() != u' ': - chunks.append(self.scan_line_break()) - end_mark = self.get_mark() - else: - self.forward() - if self.column > max_indent: - max_indent = self.column - return chunks, max_indent, end_mark - - def scan_block_scalar_breaks(self, indent): - # See the specification for details. - chunks = [] - end_mark = self.get_mark() - while self.column < indent and self.peek() == u' ': - self.forward() - while self.peek() in u'\r\n\x85\u2028\u2029': - chunks.append(self.scan_line_break()) - end_mark = self.get_mark() - while self.column < indent and self.peek() == u' ': - self.forward() - return chunks, end_mark - - def scan_flow_scalar(self, style): - # See the specification for details. - # Note that we loose indentation rules for quoted scalars. Quoted - # scalars don't need to adhere indentation because " and ' clearly - # mark the beginning and the end of them. Therefore we are less - # restrictive then the specification requires. We only need to check - # that document separators are not included in scalars. - if style == '"': - double = True - else: - double = False - chunks = [] - start_mark = self.get_mark() - quote = self.peek() - self.forward() - chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) - while self.peek() != quote: - chunks.extend(self.scan_flow_scalar_spaces(double, start_mark)) - chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) - self.forward() - end_mark = self.get_mark() - return ScalarToken(u''.join(chunks), False, start_mark, end_mark, - style) - - ESCAPE_REPLACEMENTS = { - u'0': u'\0', - u'a': u'\x07', - u'b': u'\x08', - u't': u'\x09', - u'\t': u'\x09', - u'n': u'\x0A', - u'v': u'\x0B', - u'f': u'\x0C', - u'r': u'\x0D', - u'e': u'\x1B', - u' ': u'\x20', - u'\"': u'\"', - u'\\': u'\\', - u'N': u'\x85', - u'_': u'\xA0', - u'L': u'\u2028', - u'P': u'\u2029', - } - - ESCAPE_CODES = { - u'x': 2, - u'u': 4, - u'U': 8, - } - - def scan_flow_scalar_non_spaces(self, double, start_mark): - # See the specification for details. - chunks = [] - while True: - length = 0 - while self.peek(length) not in u'\'\"\\\0 \t\r\n\x85\u2028\u2029': - length += 1 - if length: - chunks.append(self.prefix(length)) - self.forward(length) - ch = self.peek() - if not double and ch == u'\'' and self.peek(1) == u'\'': - chunks.append(u'\'') - self.forward(2) - elif (double and ch == u'\'') or (not double and ch in u'\"\\'): - chunks.append(ch) - self.forward() - elif double and ch == u'\\': - self.forward() - ch = self.peek() - if ch in self.ESCAPE_REPLACEMENTS: - chunks.append(self.ESCAPE_REPLACEMENTS[ch]) - self.forward() - elif ch in self.ESCAPE_CODES: - length = self.ESCAPE_CODES[ch] - self.forward() - for k in range(length): - if self.peek(k) not in u'0123456789ABCDEFabcdef': - raise ScannerError("while scanning a double-quoted scalar", start_mark, - "expected escape sequence of %d hexdecimal numbers, but found %r" % - (length, self.peek(k).encode('utf-8')), self.get_mark()) - code = int(self.prefix(length), 16) - chunks.append(unichr(code)) - self.forward(length) - elif ch in u'\r\n\x85\u2028\u2029': - self.scan_line_break() - chunks.extend(self.scan_flow_scalar_breaks(double, start_mark)) - else: - raise ScannerError("while scanning a double-quoted scalar", start_mark, - "found unknown escape character %r" % ch.encode('utf-8'), self.get_mark()) - else: - return chunks - - def scan_flow_scalar_spaces(self, double, start_mark): - # See the specification for details. - chunks = [] - length = 0 - while self.peek(length) in u' \t': - length += 1 - whitespaces = self.prefix(length) - self.forward(length) - ch = self.peek() - if ch == u'\0': - raise ScannerError("while scanning a quoted scalar", start_mark, - "found unexpected end of stream", self.get_mark()) - elif ch in u'\r\n\x85\u2028\u2029': - line_break = self.scan_line_break() - breaks = self.scan_flow_scalar_breaks(double, start_mark) - if line_break != u'\n': - chunks.append(line_break) - elif not breaks: - chunks.append(u' ') - chunks.extend(breaks) - else: - chunks.append(whitespaces) - return chunks - - def scan_flow_scalar_breaks(self, double, start_mark): - # See the specification for details. - chunks = [] - while True: - # Instead of checking indentation, we check for document - # separators. - prefix = self.prefix(3) - if (prefix == u'---' or prefix == u'...') \ - and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': - raise ScannerError("while scanning a quoted scalar", start_mark, - "found unexpected document separator", self.get_mark()) - while self.peek() in u' \t': - self.forward() - if self.peek() in u'\r\n\x85\u2028\u2029': - chunks.append(self.scan_line_break()) - else: - return chunks - - def scan_plain(self): - # See the specification for details. - # We add an additional restriction for the flow context: - # plain scalars in the flow context cannot contain ',', ':' and '?'. - # We also keep track of the `allow_simple_key` flag here. - # Indentation rules are loosed for the flow context. - chunks = [] - start_mark = self.get_mark() - end_mark = start_mark - indent = self.indent+1 - # We allow zero indentation for scalars, but then we need to check for - # document separators at the beginning of the line. - #if indent == 0: - # indent = 1 - spaces = [] - while True: - length = 0 - if self.peek() == u'#': - break - while True: - ch = self.peek(length) - if ch in u'\0 \t\r\n\x85\u2028\u2029' \ - or (not self.flow_level and ch == u':' and - self.peek(length+1) in u'\0 \t\r\n\x85\u2028\u2029') \ - or (self.flow_level and ch in u',:?[]{}'): - break - length += 1 - # It's not clear what we should do with ':' in the flow context. - if (self.flow_level and ch == u':' - and self.peek(length+1) not in u'\0 \t\r\n\x85\u2028\u2029,[]{}'): - self.forward(length) - raise ScannerError("while scanning a plain scalar", start_mark, - "found unexpected ':'", self.get_mark(), - "Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.") - if length == 0: - break - self.allow_simple_key = False - chunks.extend(spaces) - chunks.append(self.prefix(length)) - self.forward(length) - end_mark = self.get_mark() - spaces = self.scan_plain_spaces(indent, start_mark) - if not spaces or self.peek() == u'#' \ - or (not self.flow_level and self.column < indent): - break - return ScalarToken(u''.join(chunks), True, start_mark, end_mark) - - def scan_plain_spaces(self, indent, start_mark): - # See the specification for details. - # The specification is really confusing about tabs in plain scalars. - # We just forbid them completely. Do not use tabs in YAML! - chunks = [] - length = 0 - while self.peek(length) in u' ': - length += 1 - whitespaces = self.prefix(length) - self.forward(length) - ch = self.peek() - if ch in u'\r\n\x85\u2028\u2029': - line_break = self.scan_line_break() - self.allow_simple_key = True - prefix = self.prefix(3) - if (prefix == u'---' or prefix == u'...') \ - and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': - return - breaks = [] - while self.peek() in u' \r\n\x85\u2028\u2029': - if self.peek() == ' ': - self.forward() - else: - breaks.append(self.scan_line_break()) - prefix = self.prefix(3) - if (prefix == u'---' or prefix == u'...') \ - and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': - return - if line_break != u'\n': - chunks.append(line_break) - elif not breaks: - chunks.append(u' ') - chunks.extend(breaks) - elif whitespaces: - chunks.append(whitespaces) - return chunks - - def scan_tag_handle(self, name, start_mark): - # See the specification for details. - # For some strange reasons, the specification does not allow '_' in - # tag handles. I have allowed it anyway. - ch = self.peek() - if ch != u'!': - raise ScannerError("while scanning a %s" % name, start_mark, - "expected '!', but found %r" % ch.encode('utf-8'), - self.get_mark()) - length = 1 - ch = self.peek(length) - if ch != u' ': - while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-_': - length += 1 - ch = self.peek(length) - if ch != u'!': - self.forward(length) - raise ScannerError("while scanning a %s" % name, start_mark, - "expected '!', but found %r" % ch.encode('utf-8'), - self.get_mark()) - length += 1 - value = self.prefix(length) - self.forward(length) - return value - - def scan_tag_uri(self, name, start_mark): - # See the specification for details. - # Note: we do not check if URI is well-formed. - chunks = [] - length = 0 - ch = self.peek(length) - while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ - or ch in u'-;/?:@&=+$,_.!~*\'()[]%': - if ch == u'%': - chunks.append(self.prefix(length)) - self.forward(length) - length = 0 - chunks.append(self.scan_uri_escapes(name, start_mark)) - else: - length += 1 - ch = self.peek(length) - if length: - chunks.append(self.prefix(length)) - self.forward(length) - length = 0 - if not chunks: - raise ScannerError("while parsing a %s" % name, start_mark, - "expected URI, but found %r" % ch.encode('utf-8'), - self.get_mark()) - return u''.join(chunks) - - def scan_uri_escapes(self, name, start_mark): - # See the specification for details. - bytes = [] - mark = self.get_mark() - while self.peek() == u'%': - self.forward() - for k in range(2): - if self.peek(k) not in u'0123456789ABCDEFabcdef': - raise ScannerError("while scanning a %s" % name, start_mark, - "expected URI escape sequence of 2 hexdecimal numbers, but found %r" % - (self.peek(k).encode('utf-8')), self.get_mark()) - bytes.append(chr(int(self.prefix(2), 16))) - self.forward(2) - try: - value = unicode(''.join(bytes), 'utf-8') - except UnicodeDecodeError, exc: - raise ScannerError("while scanning a %s" % name, start_mark, str(exc), mark) - return value - - def scan_line_break(self): - # Transforms: - # '\r\n' : '\n' - # '\r' : '\n' - # '\n' : '\n' - # '\x85' : '\n' - # '\u2028' : '\u2028' - # '\u2029 : '\u2029' - # default : '' - ch = self.peek() - if ch in u'\r\n\x85': - if self.prefix(2) == u'\r\n': - self.forward(2) - else: - self.forward() - return u'\n' - elif ch in u'\u2028\u2029': - self.forward() - return ch - return u'' - -#try: -# import psyco -# psyco.bind(Scanner) -#except ImportError: -# pass - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/serializer.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/serializer.py deleted file mode 100644 index 15fdbb0c..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/serializer.py +++ /dev/null @@ -1,112 +0,0 @@ -# SPDX-License-Identifier: MIT - -__all__ = ['Serializer', 'SerializerError'] - -from error import YAMLError -from events import * -from nodes import * - -class SerializerError(YAMLError): - pass - -class Serializer(object): - - ANCHOR_TEMPLATE = u'id%03d' - - def __init__(self, encoding=None, - explicit_start=None, explicit_end=None, version=None, tags=None): - self.use_encoding = encoding - self.use_explicit_start = explicit_start - self.use_explicit_end = explicit_end - self.use_version = version - self.use_tags = tags - self.serialized_nodes = {} - self.anchors = {} - self.last_anchor_id = 0 - self.closed = None - - def open(self): - if self.closed is None: - self.emit(StreamStartEvent(encoding=self.use_encoding)) - self.closed = False - elif self.closed: - raise SerializerError("serializer is closed") - else: - raise SerializerError("serializer is already opened") - - def close(self): - if self.closed is None: - raise SerializerError("serializer is not opened") - elif not self.closed: - self.emit(StreamEndEvent()) - self.closed = True - - #def __del__(self): - # self.close() - - def serialize(self, node): - if self.closed is None: - raise SerializerError("serializer is not opened") - elif self.closed: - raise SerializerError("serializer is closed") - self.emit(DocumentStartEvent(explicit=self.use_explicit_start, - version=self.use_version, tags=self.use_tags)) - self.anchor_node(node) - self.serialize_node(node, None, None) - self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) - self.serialized_nodes = {} - self.anchors = {} - self.last_anchor_id = 0 - - def anchor_node(self, node): - if node in self.anchors: - if self.anchors[node] is None: - self.anchors[node] = self.generate_anchor(node) - else: - self.anchors[node] = None - if isinstance(node, SequenceNode): - for item in node.value: - self.anchor_node(item) - elif isinstance(node, MappingNode): - for key, value in node.value: - self.anchor_node(key) - self.anchor_node(value) - - def generate_anchor(self, node): - self.last_anchor_id += 1 - return self.ANCHOR_TEMPLATE % self.last_anchor_id - - def serialize_node(self, node, parent, index): - alias = self.anchors[node] - if node in self.serialized_nodes: - self.emit(AliasEvent(alias)) - else: - self.serialized_nodes[node] = True - self.descend_resolver(parent, index) - if isinstance(node, ScalarNode): - detected_tag = self.resolve(ScalarNode, node.value, (True, False)) - default_tag = self.resolve(ScalarNode, node.value, (False, True)) - implicit = (node.tag == detected_tag), (node.tag == default_tag) - self.emit(ScalarEvent(alias, node.tag, implicit, node.value, - style=node.style)) - elif isinstance(node, SequenceNode): - implicit = (node.tag - == self.resolve(SequenceNode, node.value, True)) - self.emit(SequenceStartEvent(alias, node.tag, implicit, - flow_style=node.flow_style)) - index = 0 - for item in node.value: - self.serialize_node(item, node, index) - index += 1 - self.emit(SequenceEndEvent()) - elif isinstance(node, MappingNode): - implicit = (node.tag - == self.resolve(MappingNode, node.value, True)) - self.emit(MappingStartEvent(alias, node.tag, implicit, - flow_style=node.flow_style)) - for key, value in node.value: - self.serialize_node(key, node, None) - self.serialize_node(value, node, key) - self.emit(MappingEndEvent()) - self.ascend_resolver() - diff --git a/src/collectors/python.d.plugin/python_modules/pyyaml2/tokens.py b/src/collectors/python.d.plugin/python_modules/pyyaml2/tokens.py deleted file mode 100644 index c5c4fb11..00000000 --- a/src/collectors/python.d.plugin/python_modules/pyyaml2/tokens.py +++ /dev/null @@ -1,105 +0,0 @@ -# SPDX-License-Identifier: MIT - -class Token(object): - def __init__(self, start_mark, end_mark): - self.start_mark = start_mark - self.end_mark = end_mark - def __repr__(self): - attributes = [key for key in self.__dict__ - if not key.endswith('_mark')] - attributes.sort() - arguments = ', '.join(['%s=%r' % (key, getattr(self, key)) - for key in attributes]) - return '%s(%s)' % (self.__class__.__name__, arguments) - -#class BOMToken(Token): -# id = '<byte order mark>' - -class DirectiveToken(Token): - id = '<directive>' - def __init__(self, name, value, start_mark, end_mark): - self.name = name - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - -class DocumentStartToken(Token): - id = '<document start>' - -class DocumentEndToken(Token): - id = '<document end>' - -class StreamStartToken(Token): - id = '<stream start>' - def __init__(self, start_mark=None, end_mark=None, - encoding=None): - self.start_mark = start_mark - self.end_mark = end_mark - self.encoding = encoding - -class StreamEndToken(Token): - id = '<stream end>' - -class BlockSequenceStartToken(Token): - id = '<block sequence start>' - -class BlockMappingStartToken(Token): - id = '<block mapping start>' - -class BlockEndToken(Token): - id = '<block end>' - -class FlowSequenceStartToken(Token): - id = '[' - -class FlowMappingStartToken(Token): - id = '{' - -class FlowSequenceEndToken(Token): - id = ']' - -class FlowMappingEndToken(Token): - id = '}' - -class KeyToken(Token): - id = '?' - -class ValueToken(Token): - id = ':' - -class BlockEntryToken(Token): - id = '-' - -class FlowEntryToken(Token): - id = ',' - -class AliasToken(Token): - id = '<alias>' - def __init__(self, value, start_mark, end_mark): - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - -class AnchorToken(Token): - id = '<anchor>' - def __init__(self, value, start_mark, end_mark): - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - -class TagToken(Token): - id = '<tag>' - def __init__(self, value, start_mark, end_mark): - self.value = value - self.start_mark = start_mark - self.end_mark = end_mark - -class ScalarToken(Token): - id = '<scalar>' - def __init__(self, value, plain, start_mark, end_mark, style=None): - self.value = value - self.plain = plain - self.start_mark = start_mark - self.end_mark = end_mark - self.style = style - diff --git a/src/collectors/python.d.plugin/python_modules/third_party/boinc_client.py b/src/collectors/python.d.plugin/python_modules/third_party/boinc_client.py deleted file mode 100644 index ec21779a..00000000 --- a/src/collectors/python.d.plugin/python_modules/third_party/boinc_client.py +++ /dev/null @@ -1,515 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- -# -# client.py - Somewhat higher-level GUI_RPC API for BOINC core client -# -# Copyright (C) 2013 Rodrigo Silva (MestreLion) <linux@rodrigosilva.com> -# Copyright (C) 2017 Austin S. Hemmelgarn -# -# SPDX-License-Identifier: GPL-3.0 - -# Based on client/boinc_cmd.cpp - -import hashlib -import socket -import sys -import time -from functools import total_ordering -from xml.etree import ElementTree - -GUI_RPC_PASSWD_FILE = "/var/lib/boinc/gui_rpc_auth.cfg" - -GUI_RPC_HOSTNAME = None # localhost -GUI_RPC_PORT = 31416 -GUI_RPC_TIMEOUT = 1 - -class Rpc(object): - ''' Class to perform GUI RPC calls to a BOINC core client. - Usage in a context manager ('with' block) is recommended to ensure - disconnect() is called. Using the same instance for all calls is also - recommended so it reuses the same socket connection - ''' - def __init__(self, hostname="", port=0, timeout=0, text_output=False): - self.hostname = hostname - self.port = port - self.timeout = timeout - self.sock = None - self.text_output = text_output - - @property - def sockargs(self): - return (self.hostname, self.port, self.timeout) - - def __enter__(self): self.connect(*self.sockargs); return self - def __exit__(self, *args): self.disconnect() - - def connect(self, hostname="", port=0, timeout=0): - ''' Connect to (hostname, port) with timeout in seconds. - Hostname defaults to None (localhost), and port to 31416 - Calling multiple times will disconnect previous connection (if any), - and (re-)connect to host. - ''' - if self.sock: - self.disconnect() - - self.hostname = hostname or GUI_RPC_HOSTNAME - self.port = port or GUI_RPC_PORT - self.timeout = timeout or GUI_RPC_TIMEOUT - - self.sock = socket.create_connection(self.sockargs[0:2], self.sockargs[2]) - - def disconnect(self): - ''' Disconnect from host. Calling multiple times is OK (idempotent) - ''' - if self.sock: - self.sock.close() - self.sock = None - - def call(self, request, text_output=None): - ''' Do an RPC call. Pack and send the XML request and return the - unpacked reply. request can be either plain XML text or a - xml.etree.ElementTree.Element object. Return ElementTree.Element - or XML text according to text_output flag. - Will auto-connect if not connected. - ''' - if text_output is None: - text_output = self.text_output - - if not self.sock: - self.connect(*self.sockargs) - - if not isinstance(request, ElementTree.Element): - request = ElementTree.fromstring(request) - - # pack request - end = '\003' - if sys.version_info[0] < 3: - req = "<boinc_gui_rpc_request>\n{0}\n</boinc_gui_rpc_request>\n{1}".format(ElementTree.tostring(request).replace(' />', '/>'), end) - else: - req = "<boinc_gui_rpc_request>\n{0}\n</boinc_gui_rpc_request>\n{1}".format(ElementTree.tostring(request, encoding='unicode').replace(' />', '/>'), end).encode() - - try: - self.sock.sendall(req) - except (socket.error, socket.herror, socket.gaierror, socket.timeout): - raise - - req = "" - while True: - try: - buf = self.sock.recv(8192) - if not buf: - raise socket.error("No data from socket") - if sys.version_info[0] >= 3: - buf = buf.decode() - except socket.error: - raise - n = buf.find(end) - if not n == -1: break - req += buf - req += buf[:n] - - # unpack reply (remove root tag, ie: first and last lines) - req = '\n'.join(req.strip().rsplit('\n')[1:-1]) - - if text_output: - return req - else: - return ElementTree.fromstring(req) - -def setattrs_from_xml(obj, xml, attrfuncdict={}): - ''' Helper to set values for attributes of a class instance by mapping - matching tags from a XML file. - attrfuncdict is a dict of functions to customize value data type of - each attribute. It falls back to simple int/float/bool/str detection - based on values defined in __init__(). This would not be needed if - Boinc used standard RPC protocol, which includes data type in XML. - ''' - if not isinstance(xml, ElementTree.Element): - xml = ElementTree.fromstring(xml) - for e in list(xml): - if hasattr(obj, e.tag): - attr = getattr(obj, e.tag) - attrfunc = attrfuncdict.get(e.tag, None) - if attrfunc is None: - if isinstance(attr, bool): attrfunc = parse_bool - elif isinstance(attr, int): attrfunc = parse_int - elif isinstance(attr, float): attrfunc = parse_float - elif isinstance(attr, str): attrfunc = parse_str - elif isinstance(attr, list): attrfunc = parse_list - else: attrfunc = lambda x: x - setattr(obj, e.tag, attrfunc(e)) - else: - pass - #print "class missing attribute '%s': %r" % (e.tag, obj) - return obj - - -def parse_bool(e): - ''' Helper to convert ElementTree.Element.text to boolean. - Treat '<foo/>' (and '<foo>[[:blank:]]</foo>') as True - Treat '0' and 'false' as False - ''' - if e.text is None: - return True - else: - return bool(e.text) and not e.text.strip().lower() in ('0', 'false') - - -def parse_int(e): - ''' Helper to convert ElementTree.Element.text to integer. - Treat '<foo/>' (and '<foo></foo>') as 0 - ''' - # int(float()) allows casting to int a value expressed as float in XML - return 0 if e.text is None else int(float(e.text.strip())) - - -def parse_float(e): - ''' Helper to convert ElementTree.Element.text to float. ''' - return 0.0 if e.text is None else float(e.text.strip()) - - -def parse_str(e): - ''' Helper to convert ElementTree.Element.text to string. ''' - return "" if e.text is None else e.text.strip() - - -def parse_list(e): - ''' Helper to convert ElementTree.Element to list. For now, simply return - the list of root element's children - ''' - return list(e) - - -class Enum(object): - UNKNOWN = -1 # Not in original API - - @classmethod - def name(cls, value): - ''' Quick-and-dirty fallback for getting the "name" of an enum item ''' - - # value as string, if it matches an enum attribute. - # Allows short usage as Enum.name("VALUE") besides Enum.name(Enum.VALUE) - if hasattr(cls, str(value)): - return cls.name(getattr(cls, value, None)) - - # value not handled in subclass name() - for k, v in cls.__dict__.items(): - if v == value: - return k.lower().replace('_', ' ') - - # value not found - return cls.name(Enum.UNKNOWN) - - -class CpuSched(Enum): - ''' values of ACTIVE_TASK::scheduler_state and ACTIVE_TASK::next_scheduler_state - "SCHEDULED" is synonymous with "executing" except when CPU throttling - is in use. - ''' - UNINITIALIZED = 0 - PREEMPTED = 1 - SCHEDULED = 2 - - -class ResultState(Enum): - ''' Values of RESULT::state in client. - THESE MUST BE IN NUMERICAL ORDER - (because of the > comparison in RESULT::computing_done()) - see html/inc/common_defs.inc - ''' - NEW = 0 - #// New result - FILES_DOWNLOADING = 1 - #// Input files for result (WU, app version) are being downloaded - FILES_DOWNLOADED = 2 - #// Files are downloaded, result can be (or is being) computed - COMPUTE_ERROR = 3 - #// computation failed; no file upload - FILES_UPLOADING = 4 - #// Output files for result are being uploaded - FILES_UPLOADED = 5 - #// Files are uploaded, notify scheduling server at some point - ABORTED = 6 - #// result was aborted - UPLOAD_FAILED = 7 - #// some output file permanent failure - - -class Process(Enum): - ''' values of ACTIVE_TASK::task_state ''' - UNINITIALIZED = 0 - #// process doesn't exist yet - EXECUTING = 1 - #// process is running, as far as we know - SUSPENDED = 9 - #// we've sent it a "suspend" message - ABORT_PENDING = 5 - #// process exceeded limits; send "abort" message, waiting to exit - QUIT_PENDING = 8 - #// we've sent it a "quit" message, waiting to exit - COPY_PENDING = 10 - #// waiting for async file copies to finish - - -class _Struct(object): - ''' base helper class with common methods for all classes derived from - BOINC's C++ structs - ''' - @classmethod - def parse(cls, xml): - return setattrs_from_xml(cls(), xml) - - def __str__(self, indent=0): - buf = '{0}{1}:\n'.format('\t' * indent, self.__class__.__name__) - for attr in self.__dict__: - value = getattr(self, attr) - if isinstance(value, list): - buf += '{0}\t{1} [\n'.format('\t' * indent, attr) - for v in value: buf += '\t\t{0}\t\t,\n'.format(v) - buf += '\t]\n' - else: - buf += '{0}\t{1}\t{2}\n'.format('\t' * indent, - attr, - value.__str__(indent+2) - if isinstance(value, _Struct) - else repr(value)) - return buf - - -@total_ordering -class VersionInfo(_Struct): - def __init__(self, major=0, minor=0, release=0): - self.major = major - self.minor = minor - self.release = release - - @property - def _tuple(self): - return (self.major, self.minor, self.release) - - def __eq__(self, other): - return isinstance(other, self.__class__) and self._tuple == other._tuple - - def __ne__(self, other): - return not self.__eq__(other) - - def __gt__(self, other): - if not isinstance(other, self.__class__): - return NotImplemented - return self._tuple > other._tuple - - def __str__(self): - return "{0}.{1}.{2}".format(self.major, self.minor, self.release) - - def __repr__(self): - return "{0}{1}".format(self.__class__.__name__, self._tuple) - - -class Result(_Struct): - ''' Also called "task" in some contexts ''' - def __init__(self): - # Names and values follow lib/gui_rpc_client.h @ RESULT - # Order too, except when grouping contradicts client/result.cpp - # RESULT::write_gui(), then XML order is used. - - self.name = "" - self.wu_name = "" - self.version_num = 0 - #// identifies the app used - self.plan_class = "" - self.project_url = "" # from PROJECT.master_url - self.report_deadline = 0.0 # seconds since epoch - self.received_time = 0.0 # seconds since epoch - #// when we got this from server - self.ready_to_report = False - #// we're ready to report this result to the server; - #// either computation is done and all the files have been uploaded - #// or there was an error - self.got_server_ack = False - #// we've received the ack for this result from the server - self.final_cpu_time = 0.0 - self.final_elapsed_time = 0.0 - self.state = ResultState.NEW - self.estimated_cpu_time_remaining = 0.0 - #// actually, estimated elapsed time remaining - self.exit_status = 0 - #// return value from the application - self.suspended_via_gui = False - self.project_suspended_via_gui = False - self.edf_scheduled = False - #// temporary used to tell GUI that this result is deadline-scheduled - self.coproc_missing = False - #// a coproc needed by this job is missing - #// (e.g. because user removed their GPU board). - self.scheduler_wait = False - self.scheduler_wait_reason = "" - self.network_wait = False - self.resources = "" - #// textual description of resources used - - #// the following defined if active - # XML is generated in client/app.cpp ACTIVE_TASK::write_gui() - self.active_task = False - self.active_task_state = Process.UNINITIALIZED - self.app_version_num = 0 - self.slot = -1 - self.pid = 0 - self.scheduler_state = CpuSched.UNINITIALIZED - self.checkpoint_cpu_time = 0.0 - self.current_cpu_time = 0.0 - self.fraction_done = 0.0 - self.elapsed_time = 0.0 - self.swap_size = 0 - self.working_set_size_smoothed = 0.0 - self.too_large = False - self.needs_shmem = False - self.graphics_exec_path = "" - self.web_graphics_url = "" - self.remote_desktop_addr = "" - self.slot_path = "" - #// only present if graphics_exec_path is - - # The following are not in original API, but are present in RPC XML reply - self.completed_time = 0.0 - #// time when ready_to_report was set - self.report_immediately = False - self.working_set_size = 0 - self.page_fault_rate = 0.0 - #// derived by higher-level code - - # The following are in API, but are NEVER in RPC XML reply. Go figure - self.signal = 0 - - self.app = None # APP* - self.wup = None # WORKUNIT* - self.project = None # PROJECT* - self.avp = None # APP_VERSION* - - @classmethod - def parse(cls, xml): - if not isinstance(xml, ElementTree.Element): - xml = ElementTree.fromstring(xml) - - # parse main XML - result = super(Result, cls).parse(xml) - - # parse '<active_task>' children - active_task = xml.find('active_task') - if active_task is None: - result.active_task = False # already the default after __init__() - else: - result.active_task = True # already the default after main parse - result = setattrs_from_xml(result, active_task) - - #// if CPU time is nonzero but elapsed time is zero, - #// we must be talking to an old client. - #// Set elapsed = CPU - #// (easier to deal with this here than in the manager) - if result.current_cpu_time != 0 and result.elapsed_time == 0: - result.elapsed_time = result.current_cpu_time - - if result.final_cpu_time != 0 and result.final_elapsed_time == 0: - result.final_elapsed_time = result.final_cpu_time - - return result - - def __str__(self): - buf = '{0}:\n'.format(self.__class__.__name__) - for attr in self.__dict__: - value = getattr(self, attr) - if attr in ['received_time', 'report_deadline']: - value = time.ctime(value) - buf += '\t{0}\t{1}\n'.format(attr, value) - return buf - - -class BoincClient(object): - - def __init__(self, host="", port=0, passwd=None): - self.hostname = host - self.port = port - self.passwd = passwd - self.rpc = Rpc(text_output=False) - self.version = None - self.authorized = False - - # Informative, not authoritative. Records status of *last* RPC call, - # but does not infer success about the *next* one. - # Thus, it should be read *after* an RPC call, not prior to one - self.connected = False - - def __enter__(self): self.connect(); return self - def __exit__(self, *args): self.disconnect() - - def connect(self): - try: - self.rpc.connect(self.hostname, self.port) - self.connected = True - except socket.error: - self.connected = False - return - self.authorized = self.authorize(self.passwd) - self.version = self.exchange_versions() - - def disconnect(self): - self.rpc.disconnect() - - def authorize(self, password): - ''' Request authorization. If password is None and we are connecting - to localhost, try to read password from the local config file - GUI_RPC_PASSWD_FILE. If file can't be read (not found or no - permission to read), try to authorize with a blank password. - If authorization is requested and fails, all subsequent calls - will be refused with socket.error 'Connection reset by peer' (104). - Since most local calls do no require authorization, do not attempt - it if you're not sure about the password. - ''' - if password is None and not self.hostname: - password = read_gui_rpc_password() or "" - nonce = self.rpc.call('<auth1/>').text - authhash = hashlib.md5('{0}{1}'.format(nonce, password).encode()).hexdigest().lower() - reply = self.rpc.call('<auth2><nonce_hash>{0}</nonce_hash></auth2>'.format(authhash)) - - if reply.tag == 'authorized': - return True - else: - return False - - def exchange_versions(self): - ''' Return VersionInfo instance with core client version info ''' - return VersionInfo.parse(self.rpc.call('<exchange_versions/>')) - - def get_tasks(self): - ''' Same as get_results(active_only=False) ''' - return self.get_results(False) - - def get_results(self, active_only=False): - ''' Get a list of results. - Those that are in progress will have information such as CPU time - and fraction done. Each result includes a name; - Use CC_STATE::lookup_result() to find this result in the current static state; - if it's not there, call get_state() again. - ''' - reply = self.rpc.call("<get_results><active_only>{0}</active_only></get_results>".format(1 if active_only else 0)) - if not reply.tag == 'results': - return [] - - results = [] - for item in list(reply): - results.append(Result.parse(item)) - - return results - - -def read_gui_rpc_password(): - ''' Read password string from GUI_RPC_PASSWD_FILE file, trim the last CR - (if any), and return it - ''' - try: - with open(GUI_RPC_PASSWD_FILE, 'r') as f: - buf = f.read() - if buf.endswith('\n'): return buf[:-1] # trim last CR - else: return buf - except IOError: - # Permission denied or File not found. - pass diff --git a/src/collectors/python.d.plugin/python_modules/third_party/mcrcon.py b/src/collectors/python.d.plugin/python_modules/third_party/mcrcon.py deleted file mode 100644 index a65a304b..00000000 --- a/src/collectors/python.d.plugin/python_modules/third_party/mcrcon.py +++ /dev/null @@ -1,74 +0,0 @@ -# Minecraft Remote Console module. -# -# Copyright (C) 2015 Barnaby Gale -# -# SPDX-License-Identifier: MIT - -import socket -import select -import struct -import time - - -class MCRconException(Exception): - pass - - -class MCRcon(object): - socket = None - - def connect(self, host, port, password): - if self.socket is not None: - raise MCRconException("Already connected") - self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self.socket.settimeout(0.9) - self.socket.connect((host, port)) - self.send(3, password) - - def disconnect(self): - if self.socket is None: - raise MCRconException("Already disconnected") - self.socket.close() - self.socket = None - - def read(self, length): - data = b"" - while len(data) < length: - data += self.socket.recv(length - len(data)) - return data - - def send(self, out_type, out_data): - if self.socket is None: - raise MCRconException("Must connect before sending data") - - # Send a request packet - out_payload = struct.pack('<ii', 0, out_type) + out_data.encode('utf8') + b'\x00\x00' - out_length = struct.pack('<i', len(out_payload)) - self.socket.send(out_length + out_payload) - - # Read response packets - in_data = "" - while True: - # Read a packet - in_length, = struct.unpack('<i', self.read(4)) - in_payload = self.read(in_length) - in_id = struct.unpack('<ii', in_payload[:8]) - in_data_partial, in_padding = in_payload[8:-2], in_payload[-2:] - - # Sanity checks - if in_padding != b'\x00\x00': - raise MCRconException("Incorrect padding") - if in_id == -1: - raise MCRconException("Login failed") - - # Record the response - in_data += in_data_partial.decode('utf8') - - # If there's nothing more to receive, return the response - if len(select.select([self.socket], [], [], 0)[0]) == 0: - return in_data - - def command(self, command): - result = self.send(2, command) - time.sleep(0.003) # MC-72390 workaround - return result diff --git a/src/collectors/python.d.plugin/samba/README.md b/src/collectors/python.d.plugin/samba/README.md deleted file mode 120000 index 3b63bbab..00000000 --- a/src/collectors/python.d.plugin/samba/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/samba.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/samba/integrations/samba.md b/src/collectors/python.d.plugin/samba/integrations/samba.md deleted file mode 100644 index 4d6f8fcc..00000000 --- a/src/collectors/python.d.plugin/samba/integrations/samba.md +++ /dev/null @@ -1,255 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/samba/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/samba/metadata.yaml" -sidebar_label: "Samba" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Storage, Mount Points and Filesystems" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# Samba - - -<img src="https://netdata.cloud/img/samba.svg" width="150"/> - - -Plugin: python.d.plugin -Module: samba - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -This collector monitors the performance metrics of Samba file sharing. - -It is using the `smbstatus` command-line tool. - -Executed commands: - -- `sudo -n smbstatus -P` - - -This collector is supported on all platforms. - -This collector only supports collecting metrics from a single instance of this integration. - -`smbstatus` is used, which can only be executed by `root`. It uses `sudo` and assumes that it is configured such that the `netdata` user can execute `smbstatus` as root without a password. - - -### Default Behavior - -#### Auto-Detection - -After all the permissions are satisfied, the `smbstatus -P` binary is executed. - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per Samba instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| syscall.rw | sendfile, recvfile | KiB/s | -| smb2.rw | readout, writein, readin, writeout | KiB/s | -| smb2.create_close | create, close | operations/s | -| smb2.get_set_info | getinfo, setinfo | operations/s | -| smb2.find | find | operations/s | -| smb2.notify | notify | operations/s | -| smb2.sm_counters | tcon, negprot, tdis, cancel, logoff, flush, lock, keepalive, break, sessetup | count | - - - -## Alerts - -There are no alerts configured by default for this integration. - - -## Setup - -### Prerequisites - -#### Enable the samba collector - -The `samba` collector is disabled by default. To enable it, use `edit-config` from the Netdata [config directory](/docs/netdata-agent/configuration/README.md), which is typically at `/etc/netdata`, to edit the `python.d.conf` file. - -```bash -cd /etc/netdata # Replace this path with your Netdata config directory, if different -sudo ./edit-config python.d.conf -``` -Change the value of the `samba` setting to `yes`. Save the file and restart the Netdata Agent with `sudo systemctl restart netdata`, or the [appropriate method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for your system. - - -#### Permissions and programs - -To run the collector you need: - -- `smbstatus` program -- `sudo` program -- `smbd` must be compiled with profiling enabled -- `smbd` must be started either with the `-P 1` option or inside `smb.conf` using `smbd profiling level` - -The module uses `smbstatus`, which can only be executed by `root`. It uses `sudo` and assumes that it is configured such that the `netdata` user can execute `smbstatus` as root without a password. - -- add to your `/etc/sudoers` file: - - `which smbstatus` shows the full path to the binary. - - ```bash - netdata ALL=(root) NOPASSWD: /path/to/smbstatus - ``` - -- Reset Netdata's systemd unit [CapabilityBoundingSet](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Capabilities) (Linux distributions with systemd) - - The default CapabilityBoundingSet doesn't allow using `sudo`, and is quite strict in general. Resetting is not optimal, but a next-best solution given the inability to execute `smbstatus` using `sudo`. - - - As the `root` user, do the following: - - ```cmd - mkdir /etc/systemd/system/netdata.service.d - echo -e '[Service]\nCapabilityBoundingSet=~' | tee /etc/systemd/system/netdata.service.d/unset-capability-bounding-set.conf - systemctl daemon-reload - systemctl restart netdata.service - ``` - - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/samba.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/samba.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| update_every | Sets the default data collection frequency. | 5 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | - -</details> - -#### Examples - -##### Basic - -A basic example configuration. - -<details open><summary>Config</summary> - -```yaml -my_job_name: - name: my_name - update_every: 1 - -``` -</details> - - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `samba` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin samba debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `samba` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep samba -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep samba /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep samba -``` - - diff --git a/src/collectors/python.d.plugin/samba/metadata.yaml b/src/collectors/python.d.plugin/samba/metadata.yaml deleted file mode 100644 index 09c04e7d..00000000 --- a/src/collectors/python.d.plugin/samba/metadata.yaml +++ /dev/null @@ -1,205 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: samba - monitored_instance: - name: Samba - link: https://www.samba.org/samba/ - categories: - - data-collection.storage-mount-points-and-filesystems - icon_filename: "samba.svg" - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: "" - keywords: - - samba - - file sharing - most_popular: false - overview: - data_collection: - metrics_description: "This collector monitors the performance metrics of Samba file sharing." - method_description: | - It is using the `smbstatus` command-line tool. - - Executed commands: - - - `sudo -n smbstatus -P` - supported_platforms: - include: [] - exclude: [] - multi_instance: false - additional_permissions: - description: | - `smbstatus` is used, which can only be executed by `root`. It uses `sudo` and assumes that it is configured such that the `netdata` user can execute `smbstatus` as root without a password. - default_behavior: - auto_detection: - description: "After all the permissions are satisfied, the `smbstatus -P` binary is executed." - limits: - description: "" - performance_impact: - description: "" - setup: - prerequisites: - list: - - title: Enable the samba collector - description: | - The `samba` collector is disabled by default. To enable it, use `edit-config` from the Netdata [config directory](/docs/netdata-agent/configuration/README.md), which is typically at `/etc/netdata`, to edit the `python.d.conf` file. - - ```bash - cd /etc/netdata # Replace this path with your Netdata config directory, if different - sudo ./edit-config python.d.conf - ``` - Change the value of the `samba` setting to `yes`. Save the file and restart the Netdata Agent with `sudo systemctl restart netdata`, or the [appropriate method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for your system. - - title: Permissions and programs - description: | - To run the collector you need: - - - `smbstatus` program - - `sudo` program - - `smbd` must be compiled with profiling enabled - - `smbd` must be started either with the `-P 1` option or inside `smb.conf` using `smbd profiling level` - - The module uses `smbstatus`, which can only be executed by `root`. It uses `sudo` and assumes that it is configured such that the `netdata` user can execute `smbstatus` as root without a password. - - - add to your `/etc/sudoers` file: - - `which smbstatus` shows the full path to the binary. - - ```bash - netdata ALL=(root) NOPASSWD: /path/to/smbstatus - ``` - - - Reset Netdata's systemd unit [CapabilityBoundingSet](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Capabilities) (Linux distributions with systemd) - - The default CapabilityBoundingSet doesn't allow using `sudo`, and is quite strict in general. Resetting is not optimal, but a next-best solution given the inability to execute `smbstatus` using `sudo`. - - - As the `root` user, do the following: - - ```cmd - mkdir /etc/systemd/system/netdata.service.d - echo -e '[Service]\nCapabilityBoundingSet=~' | tee /etc/systemd/system/netdata.service.d/unset-capability-bounding-set.conf - systemctl daemon-reload - systemctl restart netdata.service - ``` - configuration: - file: - name: python.d/samba.conf - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: update_every - description: Sets the default data collection frequency. - default_value: 5 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - examples: - folding: - enabled: true - title: "Config" - list: - - name: Basic - description: A basic example configuration. - config: | - my_job_name: - name: my_name - update_every: 1 - troubleshooting: - problems: - list: [] - alerts: [] - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: syscall.rw - description: R/Ws - unit: "KiB/s" - chart_type: area - dimensions: - - name: sendfile - - name: recvfile - - name: smb2.rw - description: R/Ws - unit: "KiB/s" - chart_type: area - dimensions: - - name: readout - - name: writein - - name: readin - - name: writeout - - name: smb2.create_close - description: Create/Close - unit: "operations/s" - chart_type: line - dimensions: - - name: create - - name: close - - name: smb2.get_set_info - description: Info - unit: "operations/s" - chart_type: line - dimensions: - - name: getinfo - - name: setinfo - - name: smb2.find - description: Find - unit: "operations/s" - chart_type: line - dimensions: - - name: find - - name: smb2.notify - description: Notify - unit: "operations/s" - chart_type: line - dimensions: - - name: notify - - name: smb2.sm_counters - description: Lesser Ops - unit: "count" - chart_type: stacked - dimensions: - - name: tcon - - name: negprot - - name: tdis - - name: cancel - - name: logoff - - name: flush - - name: lock - - name: keepalive - - name: break - - name: sessetup diff --git a/src/collectors/python.d.plugin/samba/samba.chart.py b/src/collectors/python.d.plugin/samba/samba.chart.py deleted file mode 100644 index 8eebcd60..00000000 --- a/src/collectors/python.d.plugin/samba/samba.chart.py +++ /dev/null @@ -1,144 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: samba netdata python.d module -# Author: Christopher Cox <chris_cox@endlessnow.com> -# SPDX-License-Identifier: GPL-3.0-or-later -# -# The netdata user needs to be able to be able to sudo the smbstatus program -# without password: -# netdata ALL=(ALL) NOPASSWD: /usr/bin/smbstatus -P -# -# This makes calls to smbstatus -P -# -# This just looks at a couple of values out of syscall, and some from smb2. -# -# The Lesser Ops chart is merely a display of current counter values. They -# didn't seem to change much to me. However, if you notice something changing -# a lot there, bring one or more out into its own chart and make it incremental -# (like find and notify... good examples). - -import re -import os - -from bases.FrameworkServices.ExecutableService import ExecutableService -from bases.collection import find_binary - -disabled_by_default = True - -update_every = 5 - -ORDER = [ - 'syscall_rw', - 'smb2_rw', - 'smb2_create_close', - 'smb2_info', - 'smb2_find', - 'smb2_notify', - 'smb2_sm_count' -] - -CHARTS = { - 'syscall_rw': { - 'options': [None, 'R/Ws', 'KiB/s', 'syscall', 'syscall.rw', 'area'], - 'lines': [ - ['syscall_sendfile_bytes', 'sendfile', 'incremental', 1, 1024], - ['syscall_recvfile_bytes', 'recvfile', 'incremental', -1, 1024] - ] - }, - 'smb2_rw': { - 'options': [None, 'R/Ws', 'KiB/s', 'smb2', 'smb2.rw', 'area'], - 'lines': [ - ['smb2_read_outbytes', 'readout', 'incremental', 1, 1024], - ['smb2_write_inbytes', 'writein', 'incremental', -1, 1024], - ['smb2_read_inbytes', 'readin', 'incremental', 1, 1024], - ['smb2_write_outbytes', 'writeout', 'incremental', -1, 1024] - ] - }, - 'smb2_create_close': { - 'options': [None, 'Create/Close', 'operations/s', 'smb2', 'smb2.create_close', 'line'], - 'lines': [ - ['smb2_create_count', 'create', 'incremental', 1, 1], - ['smb2_close_count', 'close', 'incremental', -1, 1] - ] - }, - 'smb2_info': { - 'options': [None, 'Info', 'operations/s', 'smb2', 'smb2.get_set_info', 'line'], - 'lines': [ - ['smb2_getinfo_count', 'getinfo', 'incremental', 1, 1], - ['smb2_setinfo_count', 'setinfo', 'incremental', -1, 1] - ] - }, - 'smb2_find': { - 'options': [None, 'Find', 'operations/s', 'smb2', 'smb2.find', 'line'], - 'lines': [ - ['smb2_find_count', 'find', 'incremental', 1, 1] - ] - }, - 'smb2_notify': { - 'options': [None, 'Notify', 'operations/s', 'smb2', 'smb2.notify', 'line'], - 'lines': [ - ['smb2_notify_count', 'notify', 'incremental', 1, 1] - ] - }, - 'smb2_sm_count': { - 'options': [None, 'Lesser Ops', 'count', 'smb2', 'smb2.sm_counters', 'stacked'], - 'lines': [ - ['smb2_tcon_count', 'tcon', 'absolute', 1, 1], - ['smb2_negprot_count', 'negprot', 'absolute', 1, 1], - ['smb2_tdis_count', 'tdis', 'absolute', 1, 1], - ['smb2_cancel_count', 'cancel', 'absolute', 1, 1], - ['smb2_logoff_count', 'logoff', 'absolute', 1, 1], - ['smb2_flush_count', 'flush', 'absolute', 1, 1], - ['smb2_lock_count', 'lock', 'absolute', 1, 1], - ['smb2_keepalive_count', 'keepalive', 'absolute', 1, 1], - ['smb2_break_count', 'break', 'absolute', 1, 1], - ['smb2_sessetup_count', 'sessetup', 'absolute', 1, 1] - ] - } -} - -SUDO = 'sudo' -SMBSTATUS = 'smbstatus' - - -class Service(ExecutableService): - def __init__(self, configuration=None, name=None): - ExecutableService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = CHARTS - self.rgx_smb2 = re.compile(r'(smb2_[^:]+|syscall_.*file_bytes):\s+(\d+)') - - def check(self): - smbstatus_binary = find_binary(SMBSTATUS) - if not smbstatus_binary: - self.error("can't locate '{0}' binary".format(SMBSTATUS)) - return False - - if os.getuid() == 0: - self.command = ' '.join([smbstatus_binary, '-P']) - return ExecutableService.check(self) - - sudo_binary = find_binary(SUDO) - if not sudo_binary: - self.error("can't locate '{0}' binary".format(SUDO)) - return False - command = [sudo_binary, '-n', '-l', smbstatus_binary, '-P'] - smbstatus = '{0} -P'.format(smbstatus_binary) - allowed = self._get_raw_data(command=command) - if not (allowed and allowed[0].strip() == smbstatus): - self.error("not allowed to run sudo for command '{0}'".format(smbstatus)) - return False - self.command = ' '.join([sudo_binary, '-n', smbstatus_binary, '-P']) - return ExecutableService.check(self) - - def _get_data(self): - """ - Format data received from shell command - :return: dict - """ - raw_data = self._get_raw_data() - if not raw_data: - return None - - parsed = self.rgx_smb2.findall(' '.join(raw_data)) - - return dict(parsed) or None diff --git a/src/collectors/python.d.plugin/samba/samba.conf b/src/collectors/python.d.plugin/samba/samba.conf deleted file mode 100644 index db15d4e9..00000000 --- a/src/collectors/python.d.plugin/samba/samba.conf +++ /dev/null @@ -1,60 +0,0 @@ -# netdata python.d.plugin configuration for samba -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -update_every: 5 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 1 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/spigotmc/README.md b/src/collectors/python.d.plugin/spigotmc/README.md deleted file mode 120000 index 66e5c9c4..00000000 --- a/src/collectors/python.d.plugin/spigotmc/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/spigotmc.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/spigotmc/integrations/spigotmc.md b/src/collectors/python.d.plugin/spigotmc/integrations/spigotmc.md deleted file mode 100644 index 2e5e6066..00000000 --- a/src/collectors/python.d.plugin/spigotmc/integrations/spigotmc.md +++ /dev/null @@ -1,250 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/spigotmc/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/spigotmc/metadata.yaml" -sidebar_label: "SpigotMC" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Gaming" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# SpigotMC - - -<img src="https://netdata.cloud/img/spigot.jfif" width="150"/> - - -Plugin: python.d.plugin -Module: spigotmc - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -This collector monitors SpigotMC server performance, in the form of ticks per second average, memory utilization, and active users. - - -It sends the `tps`, `list` and `online` commands to the Server, and gathers the metrics from the responses. - - -This collector is only supported on the following platforms: - -- Linux - -This collector supports collecting metrics from multiple instances of this integration, including remote instances. - - -### Default Behavior - -#### Auto-Detection - -By default, this collector will attempt to connect to a Spigot server running on the local host on port `25575`. - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per SpigotMC instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| spigotmc.tps | 1 Minute Average, 5 Minute Average, 15 Minute Average | ticks | -| spigotmc.users | Users | users | -| spigotmc.mem | used, allocated, max | MiB | - - - -## Alerts - -There are no alerts configured by default for this integration. - - -## Setup - -### Prerequisites - -#### Enable the Remote Console Protocol - -Under your SpigotMC server's `server.properties` configuration file, you should set `enable-rcon` to `true`. - -This will allow the Server to listen and respond to queries over the rcon protocol. - - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/spigotmc.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/spigotmc.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| update_every | Sets the default data collection frequency. | 1 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | -| name | Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. | | no | -| host | The host's IP to connect to. | localhost | yes | -| port | The port the remote console is listening on. | 25575 | yes | -| password | Remote console password if any. | | no | - -</details> - -#### Examples - -##### Basic - -A basic configuration example. - -```yaml -local: - name: local_server - url: 127.0.0.1 - port: 25575 - -``` -##### Basic Authentication - -An example using basic password for authentication with the remote console. - -<details open><summary>Config</summary> - -```yaml -local: - name: local_server_pass - url: 127.0.0.1 - port: 25575 - password: 'foobar' - -``` -</details> - -##### Multi-instance - -> **Note**: When you define multiple jobs, their names must be unique. - -Collecting metrics from local and remote instances. - - -<details open><summary>Config</summary> - -```yaml -local_server: - name : my_local_server - url : 127.0.0.1 - port: 25575 - -remote_server: - name : another_remote_server - url : 192.0.2.1 - port: 25575 - -``` -</details> - - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `spigotmc` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin spigotmc debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `spigotmc` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep spigotmc -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep spigotmc /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep spigotmc -``` - - diff --git a/src/collectors/python.d.plugin/spigotmc/metadata.yaml b/src/collectors/python.d.plugin/spigotmc/metadata.yaml deleted file mode 100644 index 5dea9f0c..00000000 --- a/src/collectors/python.d.plugin/spigotmc/metadata.yaml +++ /dev/null @@ -1,176 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: spigotmc - monitored_instance: - name: SpigotMC - link: "" - categories: - - data-collection.gaming - icon_filename: "spigot.jfif" - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: "" - keywords: - - minecraft server - - spigotmc server - - spigot - most_popular: false - overview: - data_collection: - metrics_description: | - This collector monitors SpigotMC server performance, in the form of ticks per second average, memory utilization, and active users. - method_description: | - It sends the `tps`, `list` and `online` commands to the Server, and gathers the metrics from the responses. - supported_platforms: - include: - - Linux - exclude: [] - multi_instance: true - additional_permissions: - description: "" - default_behavior: - auto_detection: - description: By default, this collector will attempt to connect to a Spigot server running on the local host on port `25575`. - limits: - description: "" - performance_impact: - description: "" - setup: - prerequisites: - list: - - title: Enable the Remote Console Protocol - description: | - Under your SpigotMC server's `server.properties` configuration file, you should set `enable-rcon` to `true`. - - This will allow the Server to listen and respond to queries over the rcon protocol. - configuration: - file: - name: "python.d/spigotmc.conf" - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: update_every - description: Sets the default data collection frequency. - default_value: 1 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - - name: name - description: > - Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed - running at any time. This allows autodetection to try several alternatives and pick the one that works. - default_value: "" - required: false - - name: host - description: The host's IP to connect to. - default_value: localhost - required: true - - name: port - description: The port the remote console is listening on. - default_value: 25575 - required: true - - name: password - description: Remote console password if any. - default_value: "" - required: false - examples: - folding: - enabled: true - title: "Config" - list: - - name: Basic - description: A basic configuration example. - folding: - enabled: false - config: | - local: - name: local_server - url: 127.0.0.1 - port: 25575 - - name: Basic Authentication - description: An example using basic password for authentication with the remote console. - config: | - local: - name: local_server_pass - url: 127.0.0.1 - port: 25575 - password: 'foobar' - - name: Multi-instance - description: | - > **Note**: When you define multiple jobs, their names must be unique. - - Collecting metrics from local and remote instances. - config: | - local_server: - name : my_local_server - url : 127.0.0.1 - port: 25575 - - remote_server: - name : another_remote_server - url : 192.0.2.1 - port: 25575 - troubleshooting: - problems: - list: [] - alerts: [] - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: spigotmc.tps - description: Spigot Ticks Per Second - unit: "ticks" - chart_type: line - dimensions: - - name: 1 Minute Average - - name: 5 Minute Average - - name: 15 Minute Average - - name: spigotmc.users - description: Minecraft Users - unit: "users" - chart_type: area - dimensions: - - name: Users - - name: spigotmc.mem - description: Minecraft Memory Usage - unit: "MiB" - chart_type: line - dimensions: - - name: used - - name: allocated - - name: max diff --git a/src/collectors/python.d.plugin/spigotmc/spigotmc.chart.py b/src/collectors/python.d.plugin/spigotmc/spigotmc.chart.py deleted file mode 100644 index 81370fb4..00000000 --- a/src/collectors/python.d.plugin/spigotmc/spigotmc.chart.py +++ /dev/null @@ -1,184 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: spigotmc netdata python.d module -# Author: Austin S. Hemmelgarn (Ferroin) -# SPDX-License-Identifier: GPL-3.0-or-later - -import platform -import re -import socket - -from bases.FrameworkServices.SimpleService import SimpleService -from third_party import mcrcon - -# Update only every 5 seconds because collection takes in excess of -# 100ms sometimes, and most people won't care about second-by-second data. -update_every = 5 - -PRECISION = 100 - -COMMAND_TPS = 'tps' -COMMAND_LIST = 'list' -COMMAND_ONLINE = 'online' - -ORDER = [ - 'tps', - 'mem', - 'users', -] - -CHARTS = { - 'tps': { - 'options': [None, 'Spigot Ticks Per Second', 'ticks', 'spigotmc', 'spigotmc.tps', 'line'], - 'lines': [ - ['tps1', '1 Minute Average', 'absolute', 1, PRECISION], - ['tps5', '5 Minute Average', 'absolute', 1, PRECISION], - ['tps15', '15 Minute Average', 'absolute', 1, PRECISION] - ] - }, - 'users': { - 'options': [None, 'Minecraft Users', 'users', 'spigotmc', 'spigotmc.users', 'area'], - 'lines': [ - ['users', 'Users', 'absolute', 1, 1] - ] - }, - 'mem': { - 'options': [None, 'Minecraft Memory Usage', 'MiB', 'spigotmc', 'spigotmc.mem', 'line'], - 'lines': [ - ['mem_used', 'used', 'absolute', 1, 1], - ['mem_alloc', 'allocated', 'absolute', 1, 1], - ['mem_max', 'max', 'absolute', 1, 1] - ] - } -} - -_TPS_REGEX = re.compile( - # Examples: - # §6TPS from last 1m, 5m, 15m: §a*20.0, §a*20.0, §a*20.0 - # §6Current Memory Usage: §a936/65536 mb (Max: 65536 mb) - r'^.*: .*?' # Message lead-in - r'(\d{1,2}.\d+), .*?' # 1-minute TPS value - r'(\d{1,2}.\d+), .*?' # 5-minute TPS value - r'(\d{1,2}\.\d+).*?' # 15-minute TPS value - r'(\s.*?(\d+)\/(\d+).*?: (\d+).*)?', # Current Memory Usage / Total Memory (Max Memory) - re.MULTILINE -) -_LIST_REGEX = re.compile( - # Examples: - # There are 4 of a max 50 players online: player1, player2, player3, player4 - # §6There are §c4§6 out of maximum §c50§6 players online. - # §6There are §c3§6/§c1§6 out of maximum §c50§6 players online. - # §6当前有 §c4§6 个玩家在线,最大在线人数为 §c50§6 个玩家. - # §c4§6 人のプレイヤーが接続中です。最大接続可能人数\:§c 50 - r'[^§](\d+)(?:.*?(?=/).*?[^§](\d+))?', # Current user count. - re.X -) - - -class Service(SimpleService): - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = CHARTS - self.host = self.configuration.get('host', 'localhost') - self.port = self.configuration.get('port', 25575) - self.password = self.configuration.get('password', '') - self.console = mcrcon.MCRcon() - self.alive = True - - def check(self): - if platform.system() != 'Linux': - self.error('Only supported on Linux.') - return False - try: - self.connect() - except (mcrcon.MCRconException, socket.error) as err: - self.error('Error connecting.') - self.error(repr(err)) - return False - - return self._get_data() - - def connect(self): - self.console.connect(self.host, self.port, self.password) - - def reconnect(self): - self.error('try reconnect.') - try: - try: - self.console.disconnect() - except mcrcon.MCRconException: - pass - self.console.connect(self.host, self.port, self.password) - self.alive = True - except (mcrcon.MCRconException, socket.error) as err: - self.error('Error connecting.') - self.error(repr(err)) - return False - return True - - def is_alive(self): - if any( - [ - not self.alive, - self.console.socket.getsockopt(socket.IPPROTO_TCP, socket.TCP_INFO, 0) != 1 - ] - ): - return self.reconnect() - return True - - def _get_data(self): - if not self.is_alive(): - return None - - data = {} - - try: - raw = self.console.command(COMMAND_TPS) - match = _TPS_REGEX.match(raw) - if match: - data['tps1'] = int(float(match.group(1)) * PRECISION) - data['tps5'] = int(float(match.group(2)) * PRECISION) - data['tps15'] = int(float(match.group(3)) * PRECISION) - if match.group(4): - data['mem_used'] = int(match.group(5)) - data['mem_alloc'] = int(match.group(6)) - data['mem_max'] = int(match.group(7)) - else: - self.error('Unable to process TPS values.') - if not raw: - self.error( - "'{0}' command returned no value, make sure you set correct password".format(COMMAND_TPS)) - except mcrcon.MCRconException: - self.error('Unable to fetch TPS values.') - except socket.error: - self.error('Connection is dead.') - self.alive = False - return None - - try: - raw = self.console.command(COMMAND_LIST) - match = _LIST_REGEX.search(raw) - if not match: - raw = self.console.command(COMMAND_ONLINE) - match = _LIST_REGEX.search(raw) - if match: - users = int(match.group(1)) - hidden_users = match.group(2) - if hidden_users: - hidden_users = int(hidden_users) - else: - hidden_users = 0 - data['users'] = users + hidden_users - else: - if not raw: - self.error("'{0}' and '{1}' commands returned no value, make sure you set correct password".format( - COMMAND_LIST, COMMAND_ONLINE)) - self.error('Unable to process user counts.') - except mcrcon.MCRconException: - self.error('Unable to fetch user counts.') - except socket.error: - self.error('Connection is dead.') - self.alive = False - return None - - return data diff --git a/src/collectors/python.d.plugin/spigotmc/spigotmc.conf b/src/collectors/python.d.plugin/spigotmc/spigotmc.conf deleted file mode 100644 index f0064ea2..00000000 --- a/src/collectors/python.d.plugin/spigotmc/spigotmc.conf +++ /dev/null @@ -1,66 +0,0 @@ -# netdata python.d.plugin configuration for spigotmc -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# update_every: 1 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 1 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# In addition to the above, spigotmc supports the following: -# -# host: localhost # The host to connect to. Defaults to the local system. -# port: 25575 # The port the remote console is listening on. -# password: '' # The remote console password. Most be set correctly. diff --git a/src/collectors/python.d.plugin/traefik/README.md b/src/collectors/python.d.plugin/traefik/README.md index 079f309c..f4574051 100644 --- a/src/collectors/python.d.plugin/traefik/README.md +++ b/src/collectors/python.d.plugin/traefik/README.md @@ -1,12 +1,3 @@ -<!-- -title: "Traefik monitoring with Netdata" -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/traefik/README.md" -sidebar_label: "traefik-python.d.plugin" -learn_status: "Published" -learn_topic_type: "References" -learn_rel_path: "Integrations/Monitor/Webapps" ---> - # Traefik collector Uses the `health` API to provide statistics. diff --git a/src/collectors/python.d.plugin/varnish/README.md b/src/collectors/python.d.plugin/varnish/README.md deleted file mode 120000 index 194be233..00000000 --- a/src/collectors/python.d.plugin/varnish/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/varnish.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/varnish/integrations/varnish.md b/src/collectors/python.d.plugin/varnish/integrations/varnish.md deleted file mode 100644 index 5850dcc4..00000000 --- a/src/collectors/python.d.plugin/varnish/integrations/varnish.md +++ /dev/null @@ -1,247 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/varnish/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/varnish/metadata.yaml" -sidebar_label: "Varnish" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Web Servers and Web Proxies" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# Varnish - - -<img src="https://netdata.cloud/img/varnish.svg" width="150"/> - - -Plugin: python.d.plugin -Module: varnish - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -This collector monitors Varnish metrics about HTTP accelerator global, Backends (VBE) and Storages (SMF, SMA, MSE) statistics. - -Note that both, Varnish-Cache (free and open source) and Varnish-Plus (Commercial/Enterprise version), are supported. - - -It uses the `varnishstat` tool in order to collect the metrics. - - -This collector is supported on all platforms. - -This collector only supports collecting metrics from a single instance of this integration. - -`netdata` user must be a member of the `varnish` group. - - -### Default Behavior - -#### Auto-Detection - -By default, if the permissions are satisfied, the `varnishstat` tool will be executed on the host. - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per Varnish instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| varnish.session_connection | accepted, dropped | connections/s | -| varnish.client_requests | received | requests/s | -| varnish.all_time_hit_rate | hit, miss, hitpass | percentage | -| varnish.current_poll_hit_rate | hit, miss, hitpass | percentage | -| varnish.cached_objects_expired | objects | expired/s | -| varnish.cached_objects_nuked | objects | nuked/s | -| varnish.threads_total | None | number | -| varnish.threads_statistics | created, failed, limited | threads/s | -| varnish.threads_queue_len | in queue | requests | -| varnish.backend_connections | successful, unhealthy, reused, closed, recycled, failed | connections/s | -| varnish.backend_requests | sent | requests/s | -| varnish.esi_statistics | errors, warnings | problems/s | -| varnish.memory_usage | free, allocated | MiB | -| varnish.uptime | uptime | seconds | - -### Per Backend - - - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| varnish.backend | header, body | kilobits/s | - -### Per Storage - - - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| varnish.storage_usage | free, allocated | KiB | -| varnish.storage_alloc_objs | allocated | objects | - - - -## Alerts - -There are no alerts configured by default for this integration. - - -## Setup - -### Prerequisites - -#### Provide the necessary permissions - -In order for the collector to work, you need to add the `netdata` user to the `varnish` user group, so that it can execute the `varnishstat` tool: - -``` -usermod -aG varnish netdata -``` - - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/varnish.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/varnish.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| instance_name | the name of the varnishd instance to get logs from. If not specified, the local host name is used. | | yes | -| update_every | Sets the default data collection frequency. | 10 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | -| name | Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. | | no | - -</details> - -#### Examples - -##### Basic - -An example configuration. - -```yaml -job_name: - instance_name: '<name-of-varnishd-instance>' - -``` - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `varnish` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin varnish debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `varnish` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep varnish -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep varnish /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep varnish -``` - - diff --git a/src/collectors/python.d.plugin/varnish/metadata.yaml b/src/collectors/python.d.plugin/varnish/metadata.yaml deleted file mode 100644 index d31c1cf6..00000000 --- a/src/collectors/python.d.plugin/varnish/metadata.yaml +++ /dev/null @@ -1,253 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: varnish - monitored_instance: - name: Varnish - link: https://varnish-cache.org/ - categories: - - data-collection.web-servers-and-web-proxies - icon_filename: 'varnish.svg' - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: '' - keywords: - - varnish - - varnishstat - - varnishd - - cache - - web server - - web cache - most_popular: false - overview: - data_collection: - metrics_description: | - This collector monitors Varnish metrics about HTTP accelerator global, Backends (VBE) and Storages (SMF, SMA, MSE) statistics. - - Note that both, Varnish-Cache (free and open source) and Varnish-Plus (Commercial/Enterprise version), are supported. - method_description: | - It uses the `varnishstat` tool in order to collect the metrics. - supported_platforms: - include: [] - exclude: [] - multi_instance: false - additional_permissions: - description: | - `netdata` user must be a member of the `varnish` group. - default_behavior: - auto_detection: - description: By default, if the permissions are satisfied, the `varnishstat` tool will be executed on the host. - limits: - description: '' - performance_impact: - description: '' - setup: - prerequisites: - list: - - title: Provide the necessary permissions - description: | - In order for the collector to work, you need to add the `netdata` user to the `varnish` user group, so that it can execute the `varnishstat` tool: - - ``` - usermod -aG varnish netdata - ``` - configuration: - file: - name: python.d/varnish.conf - description: '' - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: Config options - enabled: true - list: - - name: instance_name - description: the name of the varnishd instance to get logs from. If not specified, the local host name is used. - default_value: "" - required: true - - name: update_every - description: Sets the default data collection frequency. - default_value: 10 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - - name: name - description: Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. - default_value: '' - required: false - examples: - folding: - enabled: true - title: 'Config' - list: - - name: Basic - description: An example configuration. - folding: - enabled: false - config: | - job_name: - instance_name: '<name-of-varnishd-instance>' - troubleshooting: - problems: - list: [] - alerts: [] - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: varnish.session_connection - description: Connections Statistics - unit: "connections/s" - chart_type: line - dimensions: - - name: accepted - - name: dropped - - name: varnish.client_requests - description: Client Requests - unit: "requests/s" - chart_type: line - dimensions: - - name: received - - name: varnish.all_time_hit_rate - description: All History Hit Rate Ratio - unit: "percentage" - chart_type: stacked - dimensions: - - name: hit - - name: miss - - name: hitpass - - name: varnish.current_poll_hit_rate - description: Current Poll Hit Rate Ratio - unit: "percentage" - chart_type: stacked - dimensions: - - name: hit - - name: miss - - name: hitpass - - name: varnish.cached_objects_expired - description: Expired Objects - unit: "expired/s" - chart_type: line - dimensions: - - name: objects - - name: varnish.cached_objects_nuked - description: Least Recently Used Nuked Objects - unit: "nuked/s" - chart_type: line - dimensions: - - name: objects - - name: varnish.threads_total - description: Number Of Threads In All Pools - unit: "number" - chart_type: line - dimensions: - - name: None - - name: varnish.threads_statistics - description: Threads Statistics - unit: "threads/s" - chart_type: line - dimensions: - - name: created - - name: failed - - name: limited - - name: varnish.threads_queue_len - description: Current Queue Length - unit: "requests" - chart_type: line - dimensions: - - name: in queue - - name: varnish.backend_connections - description: Backend Connections Statistics - unit: "connections/s" - chart_type: line - dimensions: - - name: successful - - name: unhealthy - - name: reused - - name: closed - - name: recycled - - name: failed - - name: varnish.backend_requests - description: Requests To The Backend - unit: "requests/s" - chart_type: line - dimensions: - - name: sent - - name: varnish.esi_statistics - description: ESI Statistics - unit: "problems/s" - chart_type: line - dimensions: - - name: errors - - name: warnings - - name: varnish.memory_usage - description: Memory Usage - unit: "MiB" - chart_type: stacked - dimensions: - - name: free - - name: allocated - - name: varnish.uptime - description: Uptime - unit: "seconds" - chart_type: line - dimensions: - - name: uptime - - name: Backend - description: "" - labels: [] - metrics: - - name: varnish.backend - description: Backend {backend_name} - unit: "kilobits/s" - chart_type: area - dimensions: - - name: header - - name: body - - name: Storage - description: "" - labels: [] - metrics: - - name: varnish.storage_usage - description: Storage {storage_name} Usage - unit: "KiB" - chart_type: stacked - dimensions: - - name: free - - name: allocated - - name: varnish.storage_alloc_objs - description: Storage {storage_name} Allocated Objects - unit: "objects" - chart_type: line - dimensions: - - name: allocated diff --git a/src/collectors/python.d.plugin/varnish/varnish.chart.py b/src/collectors/python.d.plugin/varnish/varnish.chart.py deleted file mode 100644 index 506ad026..00000000 --- a/src/collectors/python.d.plugin/varnish/varnish.chart.py +++ /dev/null @@ -1,385 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: varnish netdata python.d module -# Author: ilyam8 -# SPDX-License-Identifier: GPL-3.0-or-later - -import re - -from bases.FrameworkServices.ExecutableService import ExecutableService -from bases.collection import find_binary - -ORDER = [ - 'session_connections', - 'client_requests', - 'all_time_hit_rate', - 'current_poll_hit_rate', - 'cached_objects_expired', - 'cached_objects_nuked', - 'threads_total', - 'threads_statistics', - 'threads_queue_len', - 'backend_connections', - 'backend_requests', - 'esi_statistics', - 'memory_usage', - 'uptime' -] - -CHARTS = { - 'session_connections': { - 'options': [None, 'Connections Statistics', 'connections/s', - 'client metrics', 'varnish.session_connection', 'line'], - 'lines': [ - ['sess_conn', 'accepted', 'incremental'], - ['sess_dropped', 'dropped', 'incremental'] - ] - }, - 'client_requests': { - 'options': [None, 'Client Requests', 'requests/s', - 'client metrics', 'varnish.client_requests', 'line'], - 'lines': [ - ['client_req', 'received', 'incremental'] - ] - }, - 'all_time_hit_rate': { - 'options': [None, 'All History Hit Rate Ratio', 'percentage', 'cache performance', - 'varnish.all_time_hit_rate', 'stacked'], - 'lines': [ - ['cache_hit', 'hit', 'percentage-of-absolute-row'], - ['cache_miss', 'miss', 'percentage-of-absolute-row'], - ['cache_hitpass', 'hitpass', 'percentage-of-absolute-row']] - }, - 'current_poll_hit_rate': { - 'options': [None, 'Current Poll Hit Rate Ratio', 'percentage', 'cache performance', - 'varnish.current_poll_hit_rate', 'stacked'], - 'lines': [ - ['cache_hit', 'hit', 'percentage-of-incremental-row'], - ['cache_miss', 'miss', 'percentage-of-incremental-row'], - ['cache_hitpass', 'hitpass', 'percentage-of-incremental-row'] - ] - }, - 'cached_objects_expired': { - 'options': [None, 'Expired Objects', 'expired/s', 'cache performance', - 'varnish.cached_objects_expired', 'line'], - 'lines': [ - ['n_expired', 'objects', 'incremental'] - ] - }, - 'cached_objects_nuked': { - 'options': [None, 'Least Recently Used Nuked Objects', 'nuked/s', 'cache performance', - 'varnish.cached_objects_nuked', 'line'], - 'lines': [ - ['n_lru_nuked', 'objects', 'incremental'] - ] - }, - 'threads_total': { - 'options': [None, 'Number Of Threads In All Pools', 'number', 'thread related metrics', - 'varnish.threads_total', 'line'], - 'lines': [ - ['threads', None, 'absolute'] - ] - }, - 'threads_statistics': { - 'options': [None, 'Threads Statistics', 'threads/s', 'thread related metrics', - 'varnish.threads_statistics', 'line'], - 'lines': [ - ['threads_created', 'created', 'incremental'], - ['threads_failed', 'failed', 'incremental'], - ['threads_limited', 'limited', 'incremental'] - ] - }, - 'threads_queue_len': { - 'options': [None, 'Current Queue Length', 'requests', 'thread related metrics', - 'varnish.threads_queue_len', 'line'], - 'lines': [ - ['thread_queue_len', 'in queue'] - ] - }, - 'backend_connections': { - 'options': [None, 'Backend Connections Statistics', 'connections/s', 'backend metrics', - 'varnish.backend_connections', 'line'], - 'lines': [ - ['backend_conn', 'successful', 'incremental'], - ['backend_unhealthy', 'unhealthy', 'incremental'], - ['backend_reuse', 'reused', 'incremental'], - ['backend_toolate', 'closed', 'incremental'], - ['backend_recycle', 'recycled', 'incremental'], - ['backend_fail', 'failed', 'incremental'] - ] - }, - 'backend_requests': { - 'options': [None, 'Requests To The Backend', 'requests/s', 'backend metrics', - 'varnish.backend_requests', 'line'], - 'lines': [ - ['backend_req', 'sent', 'incremental'] - ] - }, - 'esi_statistics': { - 'options': [None, 'ESI Statistics', 'problems/s', 'esi related metrics', 'varnish.esi_statistics', 'line'], - 'lines': [ - ['esi_errors', 'errors', 'incremental'], - ['esi_warnings', 'warnings', 'incremental'] - ] - }, - 'memory_usage': { - 'options': [None, 'Memory Usage', 'MiB', 'memory usage', 'varnish.memory_usage', 'stacked'], - 'lines': [ - ['memory_free', 'free', 'absolute', 1, 1 << 20], - ['memory_allocated', 'allocated', 'absolute', 1, 1 << 20]] - }, - 'uptime': { - 'lines': [ - ['uptime', None, 'absolute'] - ], - 'options': [None, 'Uptime', 'seconds', 'uptime', 'varnish.uptime', 'line'] - } -} - - -def backend_charts_template(name): - order = [ - '{0}_response_statistics'.format(name), - ] - - charts = { - order[0]: { - 'options': [None, 'Backend "{0}"'.format(name), 'kilobits/s', 'backend response statistics', - 'varnish.backend', 'area'], - 'lines': [ - ['{0}_beresp_hdrbytes'.format(name), 'header', 'incremental', 8, 1000], - ['{0}_beresp_bodybytes'.format(name), 'body', 'incremental', -8, 1000] - ] - }, - } - - return order, charts - - -def storage_charts_template(name): - order = [ - 'storage_{0}_usage'.format(name), - 'storage_{0}_alloc_objs'.format(name) - ] - - charts = { - order[0]: { - 'options': [None, 'Storage "{0}" Usage'.format(name), 'KiB', 'storage usage', 'varnish.storage_usage', 'stacked'], - 'lines': [ - ['{0}.g_space'.format(name), 'free', 'absolute', 1, 1 << 10], - ['{0}.g_bytes'.format(name), 'allocated', 'absolute', 1, 1 << 10] - ] - }, - order[1]: { - 'options': [None, 'Storage "{0}" Allocated Objects'.format(name), 'objects', 'storage usage', 'varnish.storage_alloc_objs', 'line'], - 'lines': [ - ['{0}.g_alloc'.format(name), 'allocated', 'absolute'] - ] - } - } - - return order, charts - - -VARNISHSTAT = 'varnishstat' - -re_version = re.compile(r'varnish-(?:plus-)?(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)') - - -class VarnishVersion: - def __init__(self, major, minor, patch): - self.major = major - self.minor = minor - self.patch = patch - - def __str__(self): - return '{0}.{1}.{2}'.format(self.major, self.minor, self.patch) - - -class Parser: - _backend_new = re.compile(r'VBE.([\d\w_.]+)\(.*?\).(beresp[\w_]+)\s+(\d+)') - _backend_old = re.compile(r'VBE\.[\d\w-]+\.([\w\d_-]+).(beresp[\w_]+)\s+(\d+)') - _default = re.compile(r'([A-Z]+\.)?([\d\w_.]+)\s+(\d+)') - - def __init__(self): - self.re_default = None - self.re_backend = None - - def init(self, data): - data = ''.join(data) - parsed_main = Parser._default.findall(data) - if parsed_main: - self.re_default = Parser._default - - parsed_backend = Parser._backend_new.findall(data) - if parsed_backend: - self.re_backend = Parser._backend_new - else: - parsed_backend = Parser._backend_old.findall(data) - if parsed_backend: - self.re_backend = Parser._backend_old - - def server_stats(self, data): - return self.re_default.findall(''.join(data)) - - def backend_stats(self, data): - return self.re_backend.findall(''.join(data)) - - -class Service(ExecutableService): - def __init__(self, configuration=None, name=None): - ExecutableService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = CHARTS - self.instance_name = configuration.get('instance_name') - self.parser = Parser() - self.command = None - self.collected_vbe = set() - self.collected_storages = set() - - def create_command(self): - varnishstat = find_binary(VARNISHSTAT) - - if not varnishstat: - self.error("can't locate '{0}' binary or binary is not executable by user netdata".format(VARNISHSTAT)) - return False - - command = [varnishstat, '-V'] - reply = self._get_raw_data(stderr=True, command=command) - if not reply: - self.error( - "no output from '{0}'. Is varnish running? Not enough privileges?".format(' '.join(self.command))) - return False - - ver = parse_varnish_version(reply) - if not ver: - self.error("failed to parse reply from '{0}', used regex :'{1}', reply : {2}".format( - ' '.join(command), re_version.pattern, reply)) - return False - - if self.instance_name: - self.command = [varnishstat, '-1', '-n', self.instance_name] - else: - self.command = [varnishstat, '-1'] - - if ver.major > 4: - self.command.extend(['-t', '1']) - - self.info("varnish version: {0}, will use command: '{1}'".format(ver, ' '.join(self.command))) - - return True - - def check(self): - if not self.create_command(): - return False - - # STDOUT is not empty - reply = self._get_raw_data() - if not reply: - self.error("no output from '{0}'. Is it running? Not enough privileges?".format(' '.join(self.command))) - return False - - self.parser.init(reply) - - # Output is parsable - if not self.parser.re_default: - self.error('cant parse the output...') - return False - - return True - - def get_data(self): - """ - Format data received from shell command - :return: dict - """ - raw = self._get_raw_data() - if not raw: - return None - - data = dict() - server_stats = self.parser.server_stats(raw) - if not server_stats: - return None - - stats = dict((param, value) for _, param, value in server_stats) - data.update(stats) - - self.get_vbe_backends(data, raw) - self.get_storages(server_stats) - - # varnish 5 uses default.g_bytes and default.g_space - data['memory_allocated'] = data.get('s0.g_bytes') or data.get('default.g_bytes') - data['memory_free'] = data.get('s0.g_space') or data.get('default.g_space') - - return data - - def get_vbe_backends(self, data, raw): - if not self.parser.re_backend: - return - stats = self.parser.backend_stats(raw) - if not stats: - return - - for (name, param, value) in stats: - data['_'.join([name, param])] = value - if name in self.collected_vbe: - continue - self.collected_vbe.add(name) - self.add_backend_charts(name) - - def get_storages(self, server_stats): - # Storage types: - # - SMF: File Storage - # - SMA: Malloc Storage - # - MSE: Massive Storage Engine (Varnish-Plus only) - # - # Stats example: - # [('SMF.', 'ssdStorage.c_req', '47686'), - # ('SMF.', 'ssdStorage.c_fail', '0'), - # ('SMF.', 'ssdStorage.c_bytes', '668102656'), - # ('SMF.', 'ssdStorage.c_freed', '140980224'), - # ('SMF.', 'ssdStorage.g_alloc', '39753'), - # ('SMF.', 'ssdStorage.g_bytes', '527122432'), - # ('SMF.', 'ssdStorage.g_space', '53159968768'), - # ('SMF.', 'ssdStorage.g_smf', '40130'), - # ('SMF.', 'ssdStorage.g_smf_frag', '311'), - # ('SMF.', 'ssdStorage.g_smf_large', '66')] - storages = [name for typ, name, _ in server_stats if typ.startswith(('SMF', 'SMA', 'MSE')) and name.endswith('g_space')] - if not storages: - return - for storage in storages: - storage = storage.split('.')[0] - if storage in self.collected_storages: - continue - self.collected_storages.add(storage) - self.add_storage_charts(storage) - - def add_backend_charts(self, backend_name): - self.add_charts(backend_name, backend_charts_template) - - def add_storage_charts(self, storage_name): - self.add_charts(storage_name, storage_charts_template) - - def add_charts(self, name, charts_template): - order, charts = charts_template(name) - - for chart_name in order: - params = [chart_name] + charts[chart_name]['options'] - dimensions = charts[chart_name]['lines'] - - new_chart = self.charts.add_chart(params) - for dimension in dimensions: - new_chart.add_dimension(dimension) - - -def parse_varnish_version(lines): - m = re_version.search(lines[0]) - if not m: - return None - - m = m.groupdict() - return VarnishVersion( - int(m['major']), - int(m['minor']), - int(m['patch']), - ) diff --git a/src/collectors/python.d.plugin/varnish/varnish.conf b/src/collectors/python.d.plugin/varnish/varnish.conf deleted file mode 100644 index 54bfe4de..00000000 --- a/src/collectors/python.d.plugin/varnish/varnish.conf +++ /dev/null @@ -1,66 +0,0 @@ -# netdata python.d.plugin configuration for varnish -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# update_every: 1 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 1 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# Additionally to the above, varnish also supports the following: -# -# instance_name: 'name' # the name of the varnishd instance to get logs from. If not specified, the host name is used. -# -# ---------------------------------------------------------------------- diff --git a/src/collectors/python.d.plugin/w1sensor/README.md b/src/collectors/python.d.plugin/w1sensor/README.md deleted file mode 120000 index c0fa9cd1..00000000 --- a/src/collectors/python.d.plugin/w1sensor/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/1-wire_sensors.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/w1sensor/integrations/1-wire_sensors.md b/src/collectors/python.d.plugin/w1sensor/integrations/1-wire_sensors.md deleted file mode 100644 index 15582879..00000000 --- a/src/collectors/python.d.plugin/w1sensor/integrations/1-wire_sensors.md +++ /dev/null @@ -1,201 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/w1sensor/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/w1sensor/metadata.yaml" -sidebar_label: "1-Wire Sensors" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Hardware Devices and Sensors" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# 1-Wire Sensors - - -<img src="https://netdata.cloud/img/1-wire.png" width="150"/> - - -Plugin: python.d.plugin -Module: w1sensor - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -Monitor 1-Wire Sensors metrics with Netdata for optimal environmental conditions monitoring. Enhance your environmental monitoring with real-time insights and alerts. - -The collector uses the wire, w1_gpio, and w1_therm kernel modules. Currently temperature sensors are supported and automatically detected. - -This collector is only supported on the following platforms: - -- Linux - -This collector supports collecting metrics from multiple instances of this integration, including remote instances. - - -### Default Behavior - -#### Auto-Detection - -The collector will try to auto detect available 1-Wire devices. - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per 1-Wire Sensors instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| w1sensor.temp | a dimension per sensor | Celsius | - - - -## Alerts - -There are no alerts configured by default for this integration. - - -## Setup - -### Prerequisites - -#### Required Linux kernel modules - -Make sure `wire`, `w1_gpio`, and `w1_therm` kernel modules are loaded. - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/w1sensor.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/w1sensor.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| update_every | Sets the default data collection frequency. | 5 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | -| name | Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. | | no | -| name_<1-Wire id> | This allows associating a human readable name with a sensor's 1-Wire identifier. | | no | - -</details> - -#### Examples - -##### Provide human readable names - -Associate two 1-Wire identifiers with human readable names. - -```yaml -sensors: - name_00000022276e: 'Machine room' - name_00000022298f: 'Rack 12' - -``` - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `w1sensor` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin w1sensor debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `w1sensor` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep w1sensor -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep w1sensor /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep w1sensor -``` - - diff --git a/src/collectors/python.d.plugin/w1sensor/metadata.yaml b/src/collectors/python.d.plugin/w1sensor/metadata.yaml deleted file mode 100644 index 7b076823..00000000 --- a/src/collectors/python.d.plugin/w1sensor/metadata.yaml +++ /dev/null @@ -1,119 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: w1sensor - monitored_instance: - name: 1-Wire Sensors - link: "https://www.analog.com/en/product-category/1wire-temperature-sensors.html" - categories: - - data-collection.hardware-devices-and-sensors - icon_filename: "1-wire.png" - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: "" - keywords: - - temperature - - sensor - - 1-wire - most_popular: false - overview: - data_collection: - metrics_description: "Monitor 1-Wire Sensors metrics with Netdata for optimal environmental conditions monitoring. Enhance your environmental monitoring with real-time insights and alerts." - method_description: "The collector uses the wire, w1_gpio, and w1_therm kernel modules. Currently temperature sensors are supported and automatically detected." - supported_platforms: - include: - - Linux - exclude: [] - multi_instance: true - additional_permissions: - description: "" - default_behavior: - auto_detection: - description: "The collector will try to auto detect available 1-Wire devices." - limits: - description: "" - performance_impact: - description: "" - setup: - prerequisites: - list: - - title: "Required Linux kernel modules" - description: "Make sure `wire`, `w1_gpio`, and `w1_therm` kernel modules are loaded." - configuration: - file: - name: python.d/w1sensor.conf - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: update_every - description: Sets the default data collection frequency. - default_value: 5 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - - name: name - description: Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. - default_value: "" - required: false - - name: name_<1-Wire id> - description: This allows associating a human readable name with a sensor's 1-Wire identifier. - default_value: "" - required: false - examples: - folding: - enabled: false - title: "Config" - list: - - name: Provide human readable names - description: Associate two 1-Wire identifiers with human readable names. - config: | - sensors: - name_00000022276e: 'Machine room' - name_00000022298f: 'Rack 12' - troubleshooting: - problems: - list: [] - alerts: [] - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: w1sensor.temp - description: 1-Wire Temperature Sensor - unit: "Celsius" - chart_type: line - dimensions: - - name: a dimension per sensor diff --git a/src/collectors/python.d.plugin/w1sensor/w1sensor.chart.py b/src/collectors/python.d.plugin/w1sensor/w1sensor.chart.py deleted file mode 100644 index 66797ced..00000000 --- a/src/collectors/python.d.plugin/w1sensor/w1sensor.chart.py +++ /dev/null @@ -1,97 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: 1-wire temperature monitor netdata python.d module -# Author: Diomidis Spinellis <http://www.spinellis.gr> -# SPDX-License-Identifier: GPL-3.0-or-later - -import os -import re - -from bases.FrameworkServices.SimpleService import SimpleService - -# default module values (can be overridden per job in `config`) -update_every = 5 - -# Location where 1-Wire devices can be found -W1_DIR = '/sys/bus/w1/devices/' - -# Lines matching the following regular expression contain a temperature value -RE_TEMP = re.compile(r' t=(-?\d+)') - -ORDER = [ - 'temp', -] - -CHARTS = { - 'temp': { - 'options': [None, '1-Wire Temperature Sensor', 'Celsius', 'Temperature', 'w1sensor.temp', 'line'], - 'lines': [] - } -} - -# Known and supported family members -# Based on linux/drivers/w1/w1_family.h and w1/slaves/w1_therm.c -THERM_FAMILY = { - '10': 'W1_THERM_DS18S20', - '22': 'W1_THERM_DS1822', - '28': 'W1_THERM_DS18B20', - '3b': 'W1_THERM_DS1825', - '42': 'W1_THERM_DS28EA00', -} - - -class Service(SimpleService): - """Provide netdata service for 1-Wire sensors""" - - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.order = ORDER - self.definitions = CHARTS - self.probes = [] - - def check(self): - """Auto-detect available 1-Wire sensors, setting line definitions - and probes to be monitored.""" - try: - file_names = os.listdir(W1_DIR) - except OSError as err: - self.error(err) - return False - - lines = [] - for file_name in file_names: - if file_name[2] != '-': - continue - if not file_name[0:2] in THERM_FAMILY: - continue - - self.probes.append(file_name) - identifier = file_name[3:] - name = identifier - config_name = self.configuration.get('name_' + identifier) - if config_name: - name = config_name - lines.append(['w1sensor_temp_' + identifier, name, 'absolute', - 1, 10]) - self.definitions['temp']['lines'] = lines - return len(self.probes) > 0 - - def get_data(self): - """Return data read from sensors.""" - data = dict() - - for file_name in self.probes: - file_path = W1_DIR + file_name + '/w1_slave' - identifier = file_name[3:] - try: - with open(file_path, 'r') as device_file: - for line in device_file: - matched = RE_TEMP.search(line) - if matched: - # Round to one decimal digit to filter-out noise - value = round(int(matched.group(1)) / 1000., 1) - value = int(value * 10) - data['w1sensor_temp_' + identifier] = value - except (OSError, IOError) as err: - self.error(err) - continue - return data or None diff --git a/src/collectors/python.d.plugin/w1sensor/w1sensor.conf b/src/collectors/python.d.plugin/w1sensor/w1sensor.conf deleted file mode 100644 index b60d2865..00000000 --- a/src/collectors/python.d.plugin/w1sensor/w1sensor.conf +++ /dev/null @@ -1,72 +0,0 @@ -# netdata python.d.plugin configuration for w1sensor -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -# update_every: 5 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 5 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# Additionally to the above, w1sensor also supports the following: -# -# name_<1-Wire id>: '<human readable name>' -# This allows associating a human readable name with a sensor's 1-Wire -# identifier. Example: -# name_00000022276e: 'Machine room' -# name_00000022298f: 'Rack 12' -# -# ---------------------------------------------------------------------- -# AUTO-DETECTION JOBS -# only one of them will run (they have the same name) diff --git a/src/collectors/python.d.plugin/zscores/README.md b/src/collectors/python.d.plugin/zscores/README.md deleted file mode 120000 index 159ce078..00000000 --- a/src/collectors/python.d.plugin/zscores/README.md +++ /dev/null @@ -1 +0,0 @@ -integrations/python.d_zscores.md
\ No newline at end of file diff --git a/src/collectors/python.d.plugin/zscores/integrations/python.d_zscores.md b/src/collectors/python.d.plugin/zscores/integrations/python.d_zscores.md deleted file mode 100644 index a5d2a7e4..00000000 --- a/src/collectors/python.d.plugin/zscores/integrations/python.d_zscores.md +++ /dev/null @@ -1,229 +0,0 @@ -<!--startmeta -custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/zscores/README.md" -meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/zscores/metadata.yaml" -sidebar_label: "python.d zscores" -learn_status: "Published" -learn_rel_path: "Collecting Metrics/Other" -most_popular: False -message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -endmeta--> - -# python.d zscores - -Plugin: python.d.plugin -Module: zscores - -<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> - -## Overview - -By using smoothed, rolling [Z-Scores](https://en.wikipedia.org/wiki/Standard_score) for selected metrics or charts you can narrow down your focus and shorten root cause analysis. - - -This collector uses the [Netdata rest api](/src/web/api/README.md) to get the `mean` and `stddev` -for each dimension on specified charts over a time range (defined by `train_secs` and `offset_secs`). - -For each dimension it will calculate a Z-Score as `z = (x - mean) / stddev` (clipped at `z_clip`). Scores are then smoothed over -time (`z_smooth_n`) and, if `mode: 'per_chart'`, aggregated across dimensions to a smoothed, rolling chart level Z-Score at each time step. - - -This collector is supported on all platforms. - -This collector supports collecting metrics from multiple instances of this integration, including remote instances. - - -### Default Behavior - -#### Auto-Detection - -This integration doesn't support auto-detection. - -#### Limits - -The default configuration for this integration does not impose any limits on data collection. - -#### Performance Impact - -The default configuration for this integration is not expected to impose a significant performance impact on the system. - - -## Metrics - -Metrics grouped by *scope*. - -The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. - - - -### Per python.d zscores instance - -These metrics refer to the entire monitored application. - -This scope has no labels. - -Metrics: - -| Metric | Dimensions | Unit | -|:------|:----------|:----| -| zscores.z | a dimension per chart or dimension | z | -| zscores.3stddev | a dimension per chart or dimension | count | - - - -## Alerts - -There are no alerts configured by default for this integration. - - -## Setup - -### Prerequisites - -#### Python Requirements - -This collector will only work with Python 3 and requires the below packages be installed. - -```bash -# become netdata user -sudo su -s /bin/bash netdata -# install required packages -pip3 install numpy pandas requests netdata-pandas==0.0.38 -``` - - - -### Configuration - -#### File - -The configuration file name for this integration is `python.d/zscores.conf`. - - -You can edit the configuration file using the `edit-config` script from the -Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory). - -```bash -cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata -sudo ./edit-config python.d/zscores.conf -``` -#### Options - -There are 2 sections: - -* Global variables -* One or more JOBS that can define multiple different instances to monitor. - -The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - -Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - -Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - - -<details open><summary>Config options</summary> - -| Name | Description | Default | Required | -|:----|:-----------|:-------|:--------:| -| charts_regex | what charts to pull data for - A regex like `system\..*/` or `system\..*/apps.cpu/apps.mem` etc. | system\..* | yes | -| train_secs | length of time (in seconds) to base calculations off for mean and stddev. | 14400 | yes | -| offset_secs | offset (in seconds) preceding latest data to ignore when calculating mean and stddev. | 300 | yes | -| train_every_n | recalculate the mean and stddev every n steps of the collector. | 900 | yes | -| z_smooth_n | smooth the z score (to reduce sensitivity to spikes) by averaging it over last n values. | 15 | yes | -| z_clip | cap absolute value of zscore (before smoothing) for better stability. | 10 | yes | -| z_abs | set z_abs: 'true' to make all zscores be absolute values only. | true | yes | -| burn_in | burn in period in which to initially calculate mean and stddev on every step. | 2 | yes | -| mode | mode can be to get a zscore 'per_dim' or 'per_chart'. | per_chart | yes | -| per_chart_agg | per_chart_agg is how you aggregate from dimension to chart when mode='per_chart'. | mean | yes | -| update_every | Sets the default data collection frequency. | 5 | no | -| priority | Controls the order of charts at the netdata dashboard. | 60000 | no | -| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no | -| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no | - -</details> - -#### Examples - -##### Default - -Default configuration. - -```yaml -local: - name: 'local' - host: '127.0.0.1:19999' - charts_regex: 'system\..*' - charts_to_exclude: 'system.uptime' - train_secs: 14400 - offset_secs: 300 - train_every_n: 900 - z_smooth_n: 15 - z_clip: 10 - z_abs: 'true' - burn_in: 2 - mode: 'per_chart' - per_chart_agg: 'mean' - -``` - - -## Troubleshooting - -### Debug Mode - - -To troubleshoot issues with the `zscores` collector, run the `python.d.plugin` with the debug option enabled. The output -should give you clues as to why the collector isn't working. - -- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on - your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. - - ```bash - cd /usr/libexec/netdata/plugins.d/ - ``` - -- Switch to the `netdata` user. - - ```bash - sudo -u netdata -s - ``` - -- Run the `python.d.plugin` to debug the collector: - - ```bash - ./python.d.plugin zscores debug trace - ``` - -### Getting Logs - -If you're encountering problems with the `zscores` collector, follow these steps to retrieve logs and identify potential issues: - -- **Run the command** specific to your system (systemd, non-systemd, or Docker container). -- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. - -#### System with systemd - -Use the following command to view logs generated since the last Netdata service restart: - -```bash -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep zscores -``` - -#### System without systemd - -Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: - -```bash -grep zscores /var/log/netdata/collector.log -``` - -**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. - -#### Docker Container - -If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: - -```bash -docker logs netdata 2>&1 | grep zscores -``` - - diff --git a/src/collectors/python.d.plugin/zscores/metadata.yaml b/src/collectors/python.d.plugin/zscores/metadata.yaml deleted file mode 100644 index e027562a..00000000 --- a/src/collectors/python.d.plugin/zscores/metadata.yaml +++ /dev/null @@ -1,187 +0,0 @@ -plugin_name: python.d.plugin -modules: - - meta: - plugin_name: python.d.plugin - module_name: zscores - monitored_instance: - name: python.d zscores - link: https://en.wikipedia.org/wiki/Standard_score - categories: - - data-collection.other - icon_filename: "" - related_resources: - integrations: - list: [] - info_provided_to_referring_integrations: - description: "" - keywords: - - zscore - - z-score - - standard score - - standard deviation - - anomaly detection - - statistical anomaly detection - most_popular: false - overview: - data_collection: - metrics_description: | - By using smoothed, rolling [Z-Scores](https://en.wikipedia.org/wiki/Standard_score) for selected metrics or charts you can narrow down your focus and shorten root cause analysis. - method_description: | - This collector uses the [Netdata rest api](/src/web/api/README.md) to get the `mean` and `stddev` - for each dimension on specified charts over a time range (defined by `train_secs` and `offset_secs`). - - For each dimension it will calculate a Z-Score as `z = (x - mean) / stddev` (clipped at `z_clip`). Scores are then smoothed over - time (`z_smooth_n`) and, if `mode: 'per_chart'`, aggregated across dimensions to a smoothed, rolling chart level Z-Score at each time step. - supported_platforms: - include: [] - exclude: [] - multi_instance: true - additional_permissions: - description: "" - default_behavior: - auto_detection: - description: "" - limits: - description: "" - performance_impact: - description: "" - setup: - prerequisites: - list: - - title: Python Requirements - description: | - This collector will only work with Python 3 and requires the below packages be installed. - - ```bash - # become netdata user - sudo su -s /bin/bash netdata - # install required packages - pip3 install numpy pandas requests netdata-pandas==0.0.38 - ``` - configuration: - file: - name: python.d/zscores.conf - description: "" - options: - description: | - There are 2 sections: - - * Global variables - * One or more JOBS that can define multiple different instances to monitor. - - The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values. - - Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition. - - Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified. - folding: - title: "Config options" - enabled: true - list: - - name: charts_regex - description: what charts to pull data for - A regex like `system\..*|` or `system\..*|apps.cpu|apps.mem` etc. - default_value: "system\\..*" - required: true - - name: train_secs - description: length of time (in seconds) to base calculations off for mean and stddev. - default_value: 14400 - required: true - - name: offset_secs - description: offset (in seconds) preceding latest data to ignore when calculating mean and stddev. - default_value: 300 - required: true - - name: train_every_n - description: recalculate the mean and stddev every n steps of the collector. - default_value: 900 - required: true - - name: z_smooth_n - description: smooth the z score (to reduce sensitivity to spikes) by averaging it over last n values. - default_value: 15 - required: true - - name: z_clip - description: cap absolute value of zscore (before smoothing) for better stability. - default_value: 10 - required: true - - name: z_abs - description: "set z_abs: 'true' to make all zscores be absolute values only." - default_value: "true" - required: true - - name: burn_in - description: burn in period in which to initially calculate mean and stddev on every step. - default_value: 2 - required: true - - name: mode - description: mode can be to get a zscore 'per_dim' or 'per_chart'. - default_value: per_chart - required: true - - name: per_chart_agg - description: per_chart_agg is how you aggregate from dimension to chart when mode='per_chart'. - default_value: mean - required: true - - name: update_every - description: Sets the default data collection frequency. - default_value: 5 - required: false - - name: priority - description: Controls the order of charts at the netdata dashboard. - default_value: 60000 - required: false - - name: autodetection_retry - description: Sets the job re-check interval in seconds. - default_value: 0 - required: false - - name: penalty - description: Indicates whether to apply penalty to update_every in case of failures. - default_value: yes - required: false - examples: - folding: - enabled: true - title: "Config" - list: - - name: Default - description: Default configuration. - folding: - enabled: false - config: | - local: - name: 'local' - host: '127.0.0.1:19999' - charts_regex: 'system\..*' - charts_to_exclude: 'system.uptime' - train_secs: 14400 - offset_secs: 300 - train_every_n: 900 - z_smooth_n: 15 - z_clip: 10 - z_abs: 'true' - burn_in: 2 - mode: 'per_chart' - per_chart_agg: 'mean' - troubleshooting: - problems: - list: [] - alerts: [] - metrics: - folding: - title: Metrics - enabled: false - description: "" - availability: [] - scopes: - - name: global - description: "These metrics refer to the entire monitored application." - labels: [] - metrics: - - name: zscores.z - description: Z Score - unit: "z" - chart_type: line - dimensions: - - name: a dimension per chart or dimension - - name: zscores.3stddev - description: Z Score >3 - unit: "count" - chart_type: stacked - dimensions: - - name: a dimension per chart or dimension diff --git a/src/collectors/python.d.plugin/zscores/zscores.chart.py b/src/collectors/python.d.plugin/zscores/zscores.chart.py deleted file mode 100644 index 1099b937..00000000 --- a/src/collectors/python.d.plugin/zscores/zscores.chart.py +++ /dev/null @@ -1,146 +0,0 @@ -# -*- coding: utf-8 -*- -# Description: zscores netdata python.d module -# Author: andrewm4894 -# SPDX-License-Identifier: GPL-3.0-or-later - -from datetime import datetime -import re - -import requests -import numpy as np -import pandas as pd - -from bases.FrameworkServices.SimpleService import SimpleService -from netdata_pandas.data import get_data, get_allmetrics - -priority = 60000 -update_every = 5 -disabled_by_default = True - -ORDER = [ - 'z', - '3stddev' -] - -CHARTS = { - 'z': { - 'options': ['z', 'Z Score', 'z', 'Z Score', 'zscores.z', 'line'], - 'lines': [] - }, - '3stddev': { - 'options': ['3stddev', 'Z Score >3', 'count', '3 Stddev', 'zscores.3stddev', 'stacked'], - 'lines': [] - }, -} - - -class Service(SimpleService): - def __init__(self, configuration=None, name=None): - SimpleService.__init__(self, configuration=configuration, name=name) - self.host = self.configuration.get('host', '127.0.0.1:19999') - self.charts_regex = re.compile(self.configuration.get('charts_regex', 'system.*')) - self.charts_to_exclude = self.configuration.get('charts_to_exclude', '').split(',') - self.charts_in_scope = [ - c for c in - list(filter(self.charts_regex.match, - requests.get(f'http://{self.host}/api/v1/charts').json()['charts'].keys())) - if c not in self.charts_to_exclude - ] - self.train_secs = self.configuration.get('train_secs', 14400) - self.offset_secs = self.configuration.get('offset_secs', 300) - self.train_every_n = self.configuration.get('train_every_n', 900) - self.z_smooth_n = self.configuration.get('z_smooth_n', 15) - self.z_clip = self.configuration.get('z_clip', 10) - self.z_abs = bool(self.configuration.get('z_abs', True)) - self.burn_in = self.configuration.get('burn_in', 2) - self.mode = self.configuration.get('mode', 'per_chart') - self.per_chart_agg = self.configuration.get('per_chart_agg', 'mean') - self.order = ORDER - self.definitions = CHARTS - self.collected_dims = {'z': set(), '3stddev': set()} - self.df_mean = pd.DataFrame() - self.df_std = pd.DataFrame() - self.df_z_history = pd.DataFrame() - - def check(self): - _ = get_allmetrics(self.host, self.charts_in_scope, wide=True, col_sep='.') - return True - - def validate_charts(self, chart, data, algorithm='absolute', multiplier=1, divisor=1): - """If dimension not in chart then add it. - """ - for dim in data: - if dim not in self.collected_dims[chart]: - self.collected_dims[chart].add(dim) - self.charts[chart].add_dimension([dim, dim, algorithm, multiplier, divisor]) - - for dim in list(self.collected_dims[chart]): - if dim not in data: - self.collected_dims[chart].remove(dim) - self.charts[chart].del_dimension(dim, hide=False) - - def train_model(self): - """Calculate the mean and stddev for all relevant metrics and store them for use in calulcating zscore at each timestep. - """ - before = int(datetime.now().timestamp()) - self.offset_secs - after = before - self.train_secs - - self.df_mean = get_data( - self.host, self.charts_in_scope, after, before, points=10, group='average', col_sep='.' - ).mean().to_frame().rename(columns={0: "mean"}) - - self.df_std = get_data( - self.host, self.charts_in_scope, after, before, points=10, group='stddev', col_sep='.' - ).mean().to_frame().rename(columns={0: "std"}) - - def create_data(self, df_allmetrics): - """Use x, mean, stddev to generate z scores and 3stddev flags via some pandas manipulation. - Returning two dictionaries of dimensions and measures, one for each chart. - - :param df_allmetrics <pd.DataFrame>: pandas dataframe with latest data from api/v1/allmetrics. - :return: (<dict>,<dict>) tuple of dictionaries, one for zscores and the other for a flag if abs(z)>3. - """ - # calculate clipped z score for each available metric - df_z = pd.concat([self.df_mean, self.df_std, df_allmetrics], axis=1, join='inner') - df_z['z'] = ((df_z['value'] - df_z['mean']) / df_z['std']).clip(-self.z_clip, self.z_clip).fillna(0) * 100 - if self.z_abs: - df_z['z'] = df_z['z'].abs() - - # append last z_smooth_n rows of zscores to history table in wide format - self.df_z_history = self.df_z_history.append( - df_z[['z']].reset_index().pivot_table(values='z', columns='index'), sort=True - ).tail(self.z_smooth_n) - - # get average zscore for last z_smooth_n for each metric - df_z_smooth = self.df_z_history.melt(value_name='z').groupby('index')['z'].mean().to_frame() - df_z_smooth['3stddev'] = np.where(abs(df_z_smooth['z']) > 300, 1, 0) - data_z = df_z_smooth['z'].add_suffix('_z').to_dict() - - # aggregate to chart level if specified - if self.mode == 'per_chart': - df_z_smooth['chart'] = ['.'.join(x[0:2]) + '_z' for x in df_z_smooth.index.str.split('.').to_list()] - if self.per_chart_agg == 'absmax': - data_z = \ - list(df_z_smooth.groupby('chart').agg({'z': lambda x: max(x, key=abs)})['z'].to_dict().values())[0] - else: - data_z = list(df_z_smooth.groupby('chart').agg({'z': [self.per_chart_agg]})['z'].to_dict().values())[0] - - data_3stddev = {} - for k in data_z: - data_3stddev[k.replace('_z', '')] = 1 if abs(data_z[k]) > 300 else 0 - - return data_z, data_3stddev - - def get_data(self): - - if self.runs_counter <= self.burn_in or self.runs_counter % self.train_every_n == 0: - self.train_model() - - data_z, data_3stddev = self.create_data( - get_allmetrics(self.host, self.charts_in_scope, wide=True, col_sep='.').transpose()) - data = {**data_z, **data_3stddev} - - self.validate_charts('z', data_z, divisor=100) - self.validate_charts('3stddev', data_3stddev) - - return data diff --git a/src/collectors/python.d.plugin/zscores/zscores.conf b/src/collectors/python.d.plugin/zscores/zscores.conf deleted file mode 100644 index 07d62ebe..00000000 --- a/src/collectors/python.d.plugin/zscores/zscores.conf +++ /dev/null @@ -1,108 +0,0 @@ -# netdata python.d.plugin configuration for example -# -# This file is in YaML format. Generally the format is: -# -# name: value -# -# There are 2 sections: -# - global variables -# - one or more JOBS -# -# JOBS allow you to collect values from multiple sources. -# Each source will have its own set of charts. -# -# JOB parameters have to be indented (using spaces only, example below). - -# ---------------------------------------------------------------------- -# Global Variables -# These variables set the defaults for all JOBs, however each JOB -# may define its own, overriding the defaults. - -# update_every sets the default data collection frequency. -# If unset, the python.d.plugin default is used. -update_every: 5 - -# priority controls the order of charts at the netdata dashboard. -# Lower numbers move the charts towards the top of the page. -# If unset, the default for python.d.plugin is used. -# priority: 60000 - -# penalty indicates whether to apply penalty to update_every in case of failures. -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. -# penalty: yes - -# autodetection_retry sets the job re-check interval in seconds. -# The job is not deleted if check fails. -# Attempts to start the job are made once every autodetection_retry. -# This feature is disabled by default. -# autodetection_retry: 0 - -# ---------------------------------------------------------------------- -# JOBS (data collection sources) -# -# The default JOBS share the same *name*. JOBS with the same name -# are mutually exclusive. Only one of them will be allowed running at -# any time. This allows autodetection to try several alternatives and -# pick the one that works. -# -# Any number of jobs is supported. -# -# All python.d.plugin JOBS (for all its modules) support a set of -# predefined parameters. These are: -# -# job_name: -# name: myname # the JOB's name as it will appear at the -# # dashboard (by default is the job_name) -# # JOBs sharing a name are mutually exclusive -# update_every: 1 # the JOB's data collection frequency -# priority: 60000 # the JOB's order on the dashboard -# penalty: yes # the JOB's penalty -# autodetection_retry: 0 # the JOB's re-check interval in seconds -# -# Additionally to the above, example also supports the following: -# -# - none -# -# ---------------------------------------------------------------------- -# AUTO-DETECTION JOBS -# only one of them will run (they have the same name) - -local: - name: 'local' - - # what host to pull data from - host: '127.0.0.1:19999' - - # what charts to pull data for - A regex like 'system\..*|' or 'system\..*|apps.cpu|apps.mem' etc. - charts_regex: 'system\..*' - - # Charts to exclude, useful if you would like to exclude some specific charts. - # Note: should be a ',' separated string like 'chart.name,chart.name'. - charts_to_exclude: 'system.uptime' - - # length of time to base calculations off for mean and stddev - train_secs: 14400 # use last 4 hours to work out the mean and stddev for the zscore - - # offset preceding latest data to ignore when calculating mean and stddev - offset_secs: 300 # ignore last 5 minutes of data when calculating the mean and stddev - - # recalculate the mean and stddev every n steps of the collector - train_every_n: 900 # recalculate mean and stddev every 15 minutes - - # smooth the z score by averaging it over last n values - z_smooth_n: 15 # take a rolling average of the last 15 zscore values to reduce sensitivity to temporary 'spikes' - - # cap absolute value of zscore (before smoothing) for better stability - z_clip: 10 # cap each zscore at 10 so as to avoid really large individual zscores swamping any rolling average - - # set z_abs: 'true' to make all zscores be absolute values only. - z_abs: 'true' - - # burn in period in which to initially calculate mean and stddev on every step - burn_in: 2 # on startup of the collector continually update the mean and stddev in case any gaps or initial calculations fail to return - - # mode can be to get a zscore 'per_dim' or 'per_chart' - mode: 'per_chart' # 'per_chart' means individual dimension level smoothed zscores will be aggregated to one zscore per chart per time step - - # per_chart_agg is how you aggregate from dimension to chart when mode='per_chart' - per_chart_agg: 'mean' # 'absmax' will take the max absolute value across all dimensions but will maintain the sign. 'mean' will just average. |