Merging upstream version 1.45.3+dfsg.

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

5 files changed, 690 insertions, 0 deletions

diff --git a/src/collectors/python.d.plugin/uwsgi/README.md b/src/collectors/python.d.plugin/uwsgi/README.md
new file mode 120000
index 000000000..44b855949
--- /dev/null
+++ b/src/collectors/python.d.plugin/uwsgi/README.md
@@ -0,0 +1 @@
+integrations/uwsgi.md
\ No newline at end of file
diff --git a/src/collectors/python.d.plugin/uwsgi/integrations/uwsgi.md b/src/collectors/python.d.plugin/uwsgi/integrations/uwsgi.md
new file mode 100644
index 000000000..e1b27ff1f
--- /dev/null
+++ b/src/collectors/python.d.plugin/uwsgi/integrations/uwsgi.md
@@ -0,0 +1,219 @@
+<!--startmeta
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/uwsgi/README.md"
+meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/uwsgi/metadata.yaml"
+sidebar_label: "uWSGI"
+learn_status: "Published"
+learn_rel_path: "Collecting Metrics/Web Servers and Web Proxies"
+most_popular: False
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE"
+endmeta-->
+
+# uWSGI
+
+
+<img src="https://netdata.cloud/img/uwsgi.svg" width="150"/>
+
+
+Plugin: python.d.plugin
+Module: uwsgi
+
+<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
+
+## Overview
+
+This collector monitors uWSGI metrics about requests, workers, memory and more.
+
+It collects every metric exposed from the stats server of uWSGI, either from the `stats.socket` or from the web server's TCP/IP socket.
+
+This collector is supported on all platforms.
+
+This collector supports collecting metrics from multiple instances of this integration, including remote instances.
+
+
+### Default Behavior
+
+#### Auto-Detection
+
+This collector will auto-detect uWSGI instances deployed on the local host, running on port 1717, or exposing stats on socket `tmp/stats.socket`.
+
+#### Limits
+
+The default configuration for this integration does not impose any limits on data collection.
+
+#### Performance Impact
+
+The default configuration for this integration is not expected to impose a significant performance impact on the system.
+
+
+## Metrics
+
+Metrics grouped by *scope*.
+
+The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.
+
+
+
+### Per uWSGI instance
+
+These metrics refer to the entire monitored application.
+
+This scope has no labels.
+
+Metrics:
+
+| Metric | Dimensions | Unit |
+|:------|:----------|:----|
+| uwsgi.requests | a dimension per worker | requests/s |
+| uwsgi.tx | a dimension per worker | KiB/s |
+| uwsgi.avg_rt | a dimension per worker | milliseconds |
+| uwsgi.memory_rss | a dimension per worker | MiB |
+| uwsgi.memory_vsz | a dimension per worker | MiB |
+| uwsgi.exceptions | exceptions | exceptions |
+| uwsgi.harakiris | harakiris | harakiris |
+| uwsgi.respawns | respawns | respawns |
+
+
+
+## Alerts
+
+There are no alerts configured by default for this integration.
+
+
+## Setup
+
+### Prerequisites
+
+#### Enable the uWSGI Stats server
+
+Make sure that you uWSGI exposes it's metrics via a Stats server.
+
+Source: https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html
+
+
+
+### Configuration
+
+#### File
+
+The configuration file name for this integration is `python.d/uwsgi.conf`.
+
+
+You can edit the configuration file using the `edit-config` script from the
+Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
+
+```bash
+cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
+sudo ./edit-config python.d/uwsgi.conf
+```
+#### Options
+
+There are 2 sections:
+
+* Global variables
+* One or more JOBS that can define multiple different instances to monitor.
+
+The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
+
+Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
+
+Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
+
+
+<details><summary>Config options</summary>
+
+| Name | Description | Default | Required |
+|:----|:-----------|:-------|:--------:|
+| update_every | Sets the default data collection frequency. | 5 | no |
+| priority | Controls the order of charts at the netdata dashboard. | 60000 | no |
+| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no |
+| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no |
+| name | The JOB's name as it will appear at the dashboard (by default is the job_name) | job_name | no |
+| socket | The 'path/to/uwsgistats.sock' | no | no |
+| host | The host to connect to | no | no |
+| port | The port to connect to | no | no |
+
+</details>
+
+#### Examples
+
+##### Basic (default out-of-the-box)
+
+A basic example configuration, one job will run at a time. Autodetect mechanism uses it by default. As all JOBs have the same name, only one can run at a time.
+
+<details><summary>Config</summary>
+
+```yaml
+socket:
+  name     : 'local'
+  socket   : '/tmp/stats.socket'
+
+localhost:
+  name     : 'local'
+  host     : 'localhost'
+  port     : 1717
+
+localipv4:
+  name     : 'local'
+  host     : '127.0.0.1'
+  port     : 1717
+
+localipv6:
+  name     : 'local'
+  host     : '::1'
+  port     : 1717
+
+```
+</details>
+
+##### Multi-instance
+
+> **Note**: When you define multiple jobs, their names must be unique.
+
+Collecting metrics from local and remote instances.
+
+
+<details><summary>Config</summary>
+
+```yaml
+local:
+  name     : 'local'
+  host     : 'localhost'
+  port     : 1717
+
+remote:
+  name     : 'remote'
+  host     : '192.0.2.1'
+  port     : 1717
+
+```
+</details>
+
+
+
+## Troubleshooting
+
+### Debug Mode
+
+To troubleshoot issues with the `uwsgi` collector, run the `python.d.plugin` with the debug option enabled. The output
+should give you clues as to why the collector isn't working.
+
+- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on
+  your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`.
+
+  ```bash
+  cd /usr/libexec/netdata/plugins.d/
+  ```
+
+- Switch to the `netdata` user.
+
+  ```bash
+  sudo -u netdata -s
+  ```
+
+- Run the `python.d.plugin` to debug the collector:
+
+  ```bash
+  ./python.d.plugin uwsgi debug trace
+  ```
+
+
diff --git a/src/collectors/python.d.plugin/uwsgi/metadata.yaml b/src/collectors/python.d.plugin/uwsgi/metadata.yaml
new file mode 100644
index 000000000..cdb090ac1
--- /dev/null
+++ b/src/collectors/python.d.plugin/uwsgi/metadata.yaml
@@ -0,0 +1,201 @@
+plugin_name: python.d.plugin
+modules:
+  - meta:
+      plugin_name: python.d.plugin
+      module_name: uwsgi
+      monitored_instance:
+        name: uWSGI
+        link: "https://github.com/unbit/uwsgi/tree/2.0.21"
+        categories:
+          - data-collection.web-servers-and-web-proxies
+        icon_filename: "uwsgi.svg"
+      related_resources:
+        integrations:
+          list: []
+      info_provided_to_referring_integrations:
+        description: ""
+      keywords:
+        - application server
+        - python
+        - web applications
+      most_popular: false
+    overview:
+      data_collection:
+        metrics_description: "This collector monitors uWSGI metrics about requests, workers, memory and more."
+        method_description: "It collects every metric exposed from the stats server of uWSGI, either from the `stats.socket` or from the web server's TCP/IP socket."
+      supported_platforms:
+        include: []
+        exclude: []
+      multi_instance: true
+      additional_permissions:
+        description: ""
+      default_behavior:
+        auto_detection:
+          description: "This collector will auto-detect uWSGI instances deployed on the local host, running on port 1717, or exposing stats on socket `tmp/stats.socket`."
+        limits:
+          description: ""
+        performance_impact:
+          description: ""
+    setup:
+      prerequisites:
+        list:
+          - title: Enable the uWSGI Stats server
+            description: |
+              Make sure that you uWSGI exposes it's metrics via a Stats server.
+              
+              Source: https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html
+      configuration:
+        file:
+          name: "python.d/uwsgi.conf"
+        options:
+          description: |
+            There are 2 sections:
+            
+            * Global variables
+            * One or more JOBS that can define multiple different instances to monitor.
+            
+            The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
+            
+            Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
+            
+            Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
+          folding:
+            title: "Config options"
+            enabled: true
+          list:
+            - name: update_every
+              description: Sets the default data collection frequency.
+              default_value: 5
+              required: false
+            - name: priority
+              description: Controls the order of charts at the netdata dashboard.
+              default_value: 60000
+              required: false
+            - name: autodetection_retry
+              description: Sets the job re-check interval in seconds.
+              default_value: 0
+              required: false
+            - name: penalty
+              description: Indicates whether to apply penalty to update_every in case of failures.
+              default_value: yes
+              required: false
+            - name: name
+              description: The JOB's name as it will appear at the dashboard (by default is the job_name)
+              default_value: job_name
+              required: false
+            - name: socket
+              description: The 'path/to/uwsgistats.sock'
+              default_value: no
+              required: false
+            - name: host
+              description: The host to connect to
+              default_value: no
+              required: false
+            - name: port
+              description: The port to connect to
+              default_value: no
+              required: false
+        examples:
+          folding:
+            enabled: true
+            title: "Config"
+          list:
+            - name: Basic (default out-of-the-box)
+              description: A basic example configuration, one job will run at a time. Autodetect mechanism uses it by default. As all JOBs have the same name, only one can run at a time.
+              config: |
+                socket:
+                  name     : 'local'
+                  socket   : '/tmp/stats.socket'
+                
+                localhost:
+                  name     : 'local'
+                  host     : 'localhost'
+                  port     : 1717
+                
+                localipv4:
+                  name     : 'local'
+                  host     : '127.0.0.1'
+                  port     : 1717
+                
+                localipv6:
+                  name     : 'local'
+                  host     : '::1'
+                  port     : 1717
+            - name: Multi-instance
+              description: |
+                > **Note**: When you define multiple jobs, their names must be unique.
+                
+                Collecting metrics from local and remote instances.
+              config: |
+                local:
+                  name     : 'local'
+                  host     : 'localhost'
+                  port     : 1717
+                
+                remote:
+                  name     : 'remote'
+                  host     : '192.0.2.1'
+                  port     : 1717
+    troubleshooting:
+      problems:
+        list: []
+    alerts: []
+    metrics:
+      folding:
+        title: Metrics
+        enabled: false
+      description: ""
+      availability: []
+      scopes:
+        - name: global
+          description: "These metrics refer to the entire monitored application."
+          labels: []
+          metrics:
+            - name: uwsgi.requests
+              description: Requests
+              unit: "requests/s"
+              chart_type: stacked
+              dimensions:
+                - name: a dimension per worker
+            - name: uwsgi.tx
+              description: Transmitted data
+              unit: "KiB/s"
+              chart_type: stacked
+              dimensions:
+                - name: a dimension per worker
+            - name: uwsgi.avg_rt
+              description: Average request time
+              unit: "milliseconds"
+              chart_type: line
+              dimensions:
+                - name: a dimension per worker
+            - name: uwsgi.memory_rss
+              description: RSS (Resident Set Size)
+              unit: "MiB"
+              chart_type: stacked
+              dimensions:
+                - name: a dimension per worker
+            - name: uwsgi.memory_vsz
+              description: VSZ (Virtual Memory Size)
+              unit: "MiB"
+              chart_type: stacked
+              dimensions:
+                - name: a dimension per worker
+            - name: uwsgi.exceptions
+              description: Exceptions
+              unit: "exceptions"
+              chart_type: line
+              dimensions:
+                - name: exceptions
+            - name: uwsgi.harakiris
+              description: Harakiris
+              unit: "harakiris"
+              chart_type: line
+              dimensions:
+                - name: harakiris
+            - name: uwsgi.respawns
+              description: Respawns
+              unit: "respawns"
+              chart_type: line
+              dimensions:
+                - name: respawns
diff --git a/src/collectors/python.d.plugin/uwsgi/uwsgi.chart.py b/src/collectors/python.d.plugin/uwsgi/uwsgi.chart.py
new file mode 100644
index 000000000..e4d900005
--- /dev/null
+++ b/src/collectors/python.d.plugin/uwsgi/uwsgi.chart.py
@@ -0,0 +1,177 @@
+# -*- coding: utf-8 -*-
+# Description: uwsgi netdata python.d module
+# Author: Robbert Segeren (robbert-ef)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+import json
+from copy import deepcopy
+
+from bases.FrameworkServices.SocketService import SocketService
+
+ORDER = [
+    'requests',
+    'tx',
+    'avg_rt',
+    'memory_rss',
+    'memory_vsz',
+    'exceptions',
+    'harakiri',
+    'respawn',
+]
+
+DYNAMIC_CHARTS = [
+    'requests',
+    'tx',
+    'avg_rt',
+    'memory_rss',
+    'memory_vsz',
+]
+
+# NOTE: lines are created dynamically in `check()` method
+CHARTS = {
+    'requests': {
+        'options': [None, 'Requests', 'requests/s', 'requests', 'uwsgi.requests', 'stacked'],
+        'lines': [
+            ['requests', 'requests', 'incremental']
+        ]
+    },
+    'tx': {
+        'options': [None, 'Transmitted data', 'KiB/s', 'requests', 'uwsgi.tx', 'stacked'],
+        'lines': [
+            ['tx', 'tx', 'incremental']
+        ]
+    },
+    'avg_rt': {
+        'options': [None, 'Average request time', 'milliseconds', 'requests', 'uwsgi.avg_rt', 'line'],
+        'lines': [
+            ['avg_rt', 'avg_rt', 'absolute']
+        ]
+    },
+    'memory_rss': {
+        'options': [None, 'RSS (Resident Set Size)', 'MiB', 'memory', 'uwsgi.memory_rss', 'stacked'],
+        'lines': [
+            ['memory_rss', 'memory_rss', 'absolute', 1, 1 << 20]
+        ]
+    },
+    'memory_vsz': {
+        'options': [None, 'VSZ (Virtual Memory Size)', 'MiB', 'memory', 'uwsgi.memory_vsz', 'stacked'],
+        'lines': [
+            ['memory_vsz', 'memory_vsz', 'absolute', 1, 1 << 20]
+        ]
+    },
+    'exceptions': {
+        'options': [None, 'Exceptions', 'exceptions', 'exceptions', 'uwsgi.exceptions', 'line'],
+        'lines': [
+            ['exceptions', 'exceptions', 'incremental']
+        ]
+    },
+    'harakiri': {
+        'options': [None, 'Harakiris', 'harakiris', 'harakiris', 'uwsgi.harakiris', 'line'],
+        'lines': [
+            ['harakiri_count', 'harakiris', 'incremental']
+        ]
+    },
+    'respawn': {
+        'options': [None, 'Respawns', 'respawns', 'respawns', 'uwsgi.respawns', 'line'],
+        'lines': [
+            ['respawn_count', 'respawns', 'incremental']
+        ]
+    },
+}
+
+
+class Service(SocketService):
+    def __init__(self, configuration=None, name=None):
+        super(Service, self).__init__(configuration=configuration, name=name)
+        self.order = ORDER
+        self.definitions = deepcopy(CHARTS)
+        self.url = self.configuration.get('host', 'localhost')
+        self.port = self.configuration.get('port', 1717)
+        # Clear dynamic dimensions, these are added during `_get_data()` to allow adding workers at run-time
+        for chart in DYNAMIC_CHARTS:
+            self.definitions[chart]['lines'] = []
+        self.last_result = {}
+        self.workers = []
+
+    def read_data(self):
+        """
+        Read data from socket and parse as JSON.
+        :return: (dict) stats
+        """
+        raw_data = self._get_raw_data()
+        if not raw_data:
+            return None
+        try:
+            return json.loads(raw_data)
+        except ValueError as err:
+            self.error(err)
+            return None
+
+    def check(self):
+        """
+        Parse configuration and check if we can read data.
+        :return: boolean
+        """
+        self._parse_config()
+        return bool(self.read_data())
+
+    def add_worker_dimensions(self, key):
+        """
+        Helper to add dimensions for a worker.
+        :param key: (int or str) worker identifier
+        :return:
+        """
+        for chart in DYNAMIC_CHARTS:
+            for line in CHARTS[chart]['lines']:
+                dimension_id = '{}_{}'.format(line[0], key)
+                dimension_name = str(key)
+
+                dimension = [dimension_id, dimension_name] + line[2:]
+                self.charts[chart].add_dimension(dimension)
+
+    @staticmethod
+    def _check_raw_data(data):
+        # The server will close the connection when it's done sending
+        # data, so just keep looping until that happens.
+        return False
+
+    def _get_data(self):
+        """
+        Read data from socket
+        :return: dict
+        """
+        stats = self.read_data()
+        if not stats:
+            return None
+
+        result = {
+            'exceptions': 0,
+            'harakiri_count': 0,
+            'respawn_count': 0,
+        }
+
+        for worker in stats['workers']:
+            key = worker['pid']
+
+            # Add dimensions for new workers
+            if key not in self.workers:
+                self.add_worker_dimensions(key)
+                self.workers.append(key)
+
+            result['requests_{}'.format(key)] = worker['requests']
+            result['tx_{}'.format(key)] = worker['tx']
+            result['avg_rt_{}'.format(key)] = worker['avg_rt']
+
+            # avg_rt is not reset by uwsgi, so reset here
+            if self.last_result.get('requests_{}'.format(key)) == worker['requests']:
+                result['avg_rt_{}'.format(key)] = 0
+
+            result['memory_rss_{}'.format(key)] = worker['rss']
+            result['memory_vsz_{}'.format(key)] = worker['vsz']
+
+            result['exceptions'] += worker['exceptions']
+            result['harakiri_count'] += worker['harakiri_count']
+            result['respawn_count'] += worker['respawn_count']
+
+        self.last_result = result
+        return result
diff --git a/src/collectors/python.d.plugin/uwsgi/uwsgi.conf b/src/collectors/python.d.plugin/uwsgi/uwsgi.conf
new file mode 100644
index 000000000..7d09e7330
--- /dev/null
+++ b/src/collectors/python.d.plugin/uwsgi/uwsgi.conf
@@ -0,0 +1,92 @@
+# netdata python.d.plugin configuration for uwsgi
+#
+# This file is in YaML format. Generally the format is:
+#
+# name: value
+#
+# There are 2 sections:
+#  - global variables
+#  - one or more JOBS
+#
+# JOBS allow you to collect values from multiple sources.
+# Each source will have its own set of charts.
+#
+# JOB parameters have to be indented (using spaces only, example below).
+
+# ----------------------------------------------------------------------
+# Global Variables
+# These variables set the defaults for all JOBs, however each JOB
+# may define its own, overriding the defaults.
+
+# update_every sets the default data collection frequency.
+# If unset, the python.d.plugin default is used.
+# update_every: 1
+
+# priority controls the order of charts at the netdata dashboard.
+# Lower numbers move the charts towards the top of the page.
+# If unset, the default for python.d.plugin is used.
+# priority: 60000
+
+# penalty indicates whether to apply penalty to update_every in case of failures.
+# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes.
+# penalty: yes
+
+# autodetection_retry sets the job re-check interval in seconds.
+# The job is not deleted if check fails.
+# Attempts to start the job are made once every autodetection_retry.
+# This feature is disabled by default.
+# autodetection_retry: 0
+
+# ----------------------------------------------------------------------
+# JOBS (data collection sources)
+#
+# The default JOBS share the same *name*. JOBS with the same name
+# are mutually exclusive. Only one of them will be allowed running at
+# any time. This allows autodetection to try several alternatives and
+# pick the one that works.
+#
+# Any number of jobs is supported.
+#
+# All python.d.plugin JOBS (for all its modules) support a set of
+# predefined parameters. These are:
+#
+# job_name:
+#     name: myname            # the JOB's name as it will appear at the
+#                             # dashboard (by default is the job_name)
+#                             # JOBs sharing a name are mutually exclusive
+#     update_every: 1         # the JOB's data collection frequency
+#     priority: 60000         # the JOB's order on the dashboard
+#     penalty: yes            # the JOB's penalty
+#     autodetection_retry: 0  # the JOB's re-check interval in seconds
+#
+# Additionally to the above, uwsgi also supports the following:
+#
+#     socket: 'path/to/uwsgistats.sock'
+#
+#  or
+#     host: 'IP or HOSTNAME' # the host to connect to
+#     port: PORT             # the port to connect to
+#
+# ----------------------------------------------------------------------
+# AUTO-DETECTION JOBS
+# only one of them will run (they have the same name)
+#
+
+socket:
+  name     : 'local'
+  socket   : '/tmp/stats.socket'
+
+localhost:
+  name     : 'local'
+  host     : 'localhost'
+  port     : 1717
+
+localipv4:
+  name     : 'local'
+  host     : '127.0.0.1'
+  port     : 1717
+
+localipv6:
+  name     : 'local'
+  host     : '::1'
+  port     : 1717