Adding upstream version 1.29.3.upstream/1.29.3upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

4 files changed, 693 insertions, 0 deletions

diff --git a/collectors/python.d.plugin/go_expvar/Makefile.inc b/collectors/python.d.plugin/go_expvar/Makefile.inc
new file mode 100644
index 0000000..74f50d7
--- /dev/null
+++ b/collectors/python.d.plugin/go_expvar/Makefile.inc
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+# THIS IS NOT A COMPLETE Makefile
+# IT IS INCLUDED BY ITS PARENT'S Makefile.am
+# IT IS REQUIRED TO REFERENCE ALL FILES RELATIVE TO THE PARENT
+
+# install these files
+dist_python_DATA       += go_expvar/go_expvar.chart.py
+dist_pythonconfig_DATA += go_expvar/go_expvar.conf
+
+# do not install these files, but include them in the distribution
+dist_noinst_DATA       += go_expvar/README.md go_expvar/Makefile.inc
+
diff --git a/collectors/python.d.plugin/go_expvar/README.md b/collectors/python.d.plugin/go_expvar/README.md
new file mode 100644
index 0000000..a73610e
--- /dev/null
+++ b/collectors/python.d.plugin/go_expvar/README.md
@@ -0,0 +1,319 @@
+<!--
+title: "Go applications monitoring with Netdata"
+custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/python.d.plugin/go_expvar/README.md
+sidebar_label: "Go applications"
+-->
+
+# Go applications monitoring with Netdata
+
+Monitors Go application that exposes its metrics with the use of `expvar` package from the Go standard library.  The package produces charts for Go runtime memory statistics and optionally any number of custom charts.
+
+The `go_expvar` module produces the following charts:
+
+1.  **Heap allocations** in kB
+
+    -   alloc: size of objects allocated on the heap
+    -   inuse: size of allocated heap spans
+
+2.  **Stack allocations** in kB
+
+    -   inuse: size of allocated stack spans
+
+3.  **MSpan allocations** in kB
+
+    -   inuse: size of allocated mspan structures
+
+4.  **MCache allocations** in kB
+
+    -   inuse: size of allocated mcache structures
+
+5.  **Virtual memory** in kB
+
+    -   sys: size of reserved virtual address space
+
+6.  **Live objects**
+
+    -   live: number of live objects in memory
+
+7.  **GC pauses average** in ns
+
+    -   avg: average duration of all GC stop-the-world pauses
+
+## Monitoring Go applications
+
+Netdata can be used to monitor running Go applications that expose their metrics with
+the use of the [expvar package](https://golang.org/pkg/expvar/) included in Go standard library.
+
+The `expvar` package exposes these metrics over HTTP and is very easy to use.
+Consider this minimal sample below:
+
+```go
+package main
+
+import (
+        _ "expvar"
+        "net/http"
+)
+
+func main() {
+        http.ListenAndServe("127.0.0.1:8080", nil)
+}
+```
+
+When imported this way, the `expvar` package registers a HTTP handler at `/debug/vars` that
+exposes Go runtime's memory statistics in JSON format. You can inspect the output by opening
+the URL in your browser (or by using `wget` or `curl`).
+
+Sample output:
+
+```json
+{
+"cmdline": ["./expvar-demo-binary"],
+"memstats": {"Alloc":630856,"TotalAlloc":630856,"Sys":3346432,"Lookups":27, <omitted for brevity>}
+}
+```
+
+You can of course expose and monitor your own variables as well.
+Here is a sample Go application that exposes a few custom variables:
+
+```go
+package main
+
+import (
+    "expvar"
+    "net/http"
+    "runtime"
+    "time"
+)
+
+func main() {
+
+    tick := time.NewTicker(1 * time.Second)
+    num_go := expvar.NewInt("runtime.goroutines")
+    counters := expvar.NewMap("counters")
+    counters.Set("cnt1", new(expvar.Int))
+    counters.Set("cnt2", new(expvar.Float))
+
+    go http.ListenAndServe(":8080", nil)
+
+    for {
+        select {
+        case <- tick.C:
+            num_go.Set(int64(runtime.NumGoroutine()))
+            counters.Add("cnt1", 1)
+            counters.AddFloat("cnt2", 1.452)
+        }
+    }
+}
+```
+
+Apart from the runtime memory stats, this application publishes two counters and the
+number of currently running Goroutines and updates these stats every second.
+
+In the next section, we will cover how to monitor and chart these exposed stats with
+the use of `netdata`s `go_expvar` module.
+
+### Using Netdata go_expvar module
+
+The `go_expvar` module is disabled by default. To enable it, edit `python.d.conf` (to edit it on your system run
+`/etc/netdata/edit-config python.d.conf`), and change the `go_expvar` variable to `yes`:
+
+```
+# Enable / Disable python.d.plugin modules
+#default_run: yes
+#
+# If "default_run" = "yes" the default for all modules is enabled (yes).
+# Setting any of these to "no" will disable it.
+# 
+# If "default_run" = "no" the default for all modules is disabled (no).
+# Setting any of these to "yes" will enable it.
+...
+go_expvar: yes
+...
+```
+
+Next, we need to edit the module configuration file (found at `/etc/netdata/python.d/go_expvar.conf` by default) (to
+edit it on your system run `/etc/netdata/edit-config python.d/go_expvar.conf`). The module configuration consists of
+jobs, where each job can be used to monitor a separate Go application. Let's see a sample job configuration:
+
+```
+# /etc/netdata/python.d/go_expvar.conf
+
+app1:
+  name : 'app1'
+  url  : 'http://127.0.0.1:8080/debug/vars'
+  collect_memstats: true
+  extra_charts: {}
+```
+
+Let's go over each of the defined options:
+
+```
+name: 'app1'
+```
+
+This is the job name that will appear at the Netdata dashboard.
+If not defined, the job_name (top level key) will be used.
+
+```
+url: 'http://127.0.0.1:8080/debug/vars'
+```
+
+This is the URL of the expvar endpoint. As the expvar handler can be installed
+in a custom path, the whole URL has to be specified. This value is mandatory.
+
+```
+collect_memstats: true
+```
+
+Whether to enable collecting stats about Go runtime's memory. You can find more
+information about the exposed values at the [runtime package docs](https://golang.org/pkg/runtime/#MemStats).
+
+```
+extra_charts: {}
+```
+
+Enables the user to specify custom expvars to monitor and chart.
+Will be explained in more detail below.
+
+**Note: if `collect_memstats` is disabled and no `extra_charts` are defined, the plugin will
+disable itself, as there will be no data to collect!**
+
+Apart from these options, each job supports options inherited from Netdata's `python.d.plugin`
+and its base `UrlService` class. These are:
+
+```
+update_every: 1          # the job's data collection frequency
+priority:     60000      # the job's order on the dashboard
+user:         admin      # use when the expvar endpoint is protected by HTTP Basic Auth
+password:     sekret     # use when the expvar endpoint is protected by HTTP Basic Auth
+```
+
+### Monitoring custom vars with go_expvar
+
+Now, memory stats might be useful, but what if you want Netdata to monitor some custom values
+that your Go application exposes? The `go_expvar` module can do that as well with the use of
+the `extra_charts` configuration variable.
+
+The `extra_charts` variable is a YaML list of Netdata chart definitions.
+Each chart definition has the following keys:
+
+```
+id:         Netdata chart ID
+options:    a key-value mapping of chart options
+lines:      a list of line definitions
+```
+
+**Note: please do not use dots in the chart or line ID field.
+See [this issue](https://github.com/netdata/netdata/pull/1902#issuecomment-284494195) for explanation.**
+
+Please see these two links to the official Netdata documentation for more information about the values:
+
+-   [External plugins - charts](/collectors/plugins.d/README.md#chart)
+-   [Chart variables](/collectors/python.d.plugin/README.md#global-variables-order-and-chart)
+
+**Line definitions**
+
+Each chart can define multiple lines (dimensions).
+A line definition is a key-value mapping of line options.
+Each line can have the following options:
+
+```
+# mandatory
+expvar_key: the name of the expvar as present in the JSON output of /debug/vars endpoint
+expvar_type: value type; supported are "float" or "int"
+id: the id of this line/dimension in Netdata
+
+# optional - Netdata defaults are used if these options are not defined
+name: ''
+algorithm: absolute
+multiplier: 1
+divisor: 100 if expvar_type == float, 1 if expvar_type == int
+hidden: False
+```
+
+Please see the following link for more information about the options and their default values:
+[External plugins - dimensions](/collectors/plugins.d/README.md#dimension)
+
+Apart from top-level expvars, this plugin can also parse expvars stored in a multi-level map;
+All dicts in the resulting JSON document are then flattened to one level.
+Expvar names are joined together with '.' when flattening.
+
+Example:
+
+```
+{
+    "counters": {"cnt1": 1042, "cnt2": 1512.9839999999983},
+    "runtime.goroutines": 5
+}
+```
+
+In the above case, the exported variables will be available under `runtime.goroutines`,
+`counters.cnt1` and `counters.cnt2` expvar_keys. If the flattening results in a key collision,
+the first defined key wins and all subsequent keys with the same name are ignored.
+
+## Enable the collector
+
+The `go_expvar` collector is disabled by default. To enable it, use `edit-config` from the Netdata [config
+directory](/docs/configure/nodes.md), which is typically at `/etc/netdata`, to edit the `python.d.conf` file.
+
+```bash
+cd /etc/netdata   # Replace this path with your Netdata config directory, if different
+sudo ./edit-config python.d.conf
+```
+
+Change the value of the `go_expvar` setting to `yes`. Save the file and restart the Netdata Agent with `sudo systemctl
+restart netdata`, or the appropriate method for your system, to finish enabling the `go_expvar` collector.
+
+## Configuration
+
+Edit the `python.d/go_expvar.conf` configuration file using `edit-config` from the Netdata [config
+directory](/docs/configure/nodes.md), which is typically at `/etc/netdata`.
+
+```bash
+cd /etc/netdata   # Replace this path with your Netdata config directory, if different
+sudo ./edit-config python.d/go_expvar.conf
+```
+
+The configuration below matches the second Go application described above.
+Netdata will monitor and chart memory stats for the application, as well as a custom chart of
+running goroutines and two dummy counters.
+
+```
+app1:
+  name : 'app1'
+  url  : 'http://127.0.0.1:8080/debug/vars'
+  collect_memstats: true
+  extra_charts:
+    - id: "runtime_goroutines"
+      options:
+        name: num_goroutines
+        title: "runtime: number of goroutines"
+        units: goroutines
+        family: runtime
+        context: expvar.runtime.goroutines
+        chart_type: line
+      lines:
+        - {expvar_key: 'runtime.goroutines', expvar_type: int, id: runtime_goroutines}
+    - id: "foo_counters"
+      options:
+        name: counters
+        title: "some random counters"
+        units: awesomeness
+        family: counters
+        context: expvar.foo.counters
+        chart_type: line
+      lines:
+        - {expvar_key: 'counters.cnt1', expvar_type: int, id: counters_cnt1}
+        - {expvar_key: 'counters.cnt2', expvar_type: float, id: counters_cnt2}
+```
+
+**Netdata charts example**
+
+The images below show how do the final charts in Netdata look.
+
+![Memory stats charts](https://cloud.githubusercontent.com/assets/15180106/26762052/62b4af58-493b-11e7-9e69-146705acfc2c.png)
+
+![Custom charts](https://cloud.githubusercontent.com/assets/15180106/26762051/62ae915e-493b-11e7-8518-bd25a3886650.png)
+
+[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fcollectors%2Fpython.d.plugin%2Fgo_expvar%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
diff --git a/collectors/python.d.plugin/go_expvar/go_expvar.chart.py b/collectors/python.d.plugin/go_expvar/go_expvar.chart.py
new file mode 100644
index 0000000..f9bbdc1
--- /dev/null
+++ b/collectors/python.d.plugin/go_expvar/go_expvar.chart.py
@@ -0,0 +1,253 @@
+# -*- coding: utf-8 -*-
+# Description: go_expvar netdata python.d module
+# Author: Jan Kral (kralewitz)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+from __future__ import division
+
+import json
+from collections import namedtuple
+
+from bases.FrameworkServices.UrlService import UrlService
+
+MEMSTATS_ORDER = [
+    'memstats_heap',
+    'memstats_stack',
+    'memstats_mspan',
+    'memstats_mcache',
+    'memstats_sys',
+    'memstats_live_objects',
+    'memstats_gc_pauses',
+]
+
+MEMSTATS_CHARTS = {
+    'memstats_heap': {
+        'options': ['heap', 'memory: size of heap memory structures', 'KiB', 'memstats',
+                    'expvar.memstats.heap', 'line'],
+        'lines': [
+            ['memstats_heap_alloc', 'alloc', 'absolute', 1, 1024],
+            ['memstats_heap_inuse', 'inuse', 'absolute', 1, 1024]
+        ]
+    },
+    'memstats_stack': {
+        'options': ['stack', 'memory: size of stack memory structures', 'KiB', 'memstats',
+                    'expvar.memstats.stack', 'line'],
+        'lines': [
+            ['memstats_stack_inuse', 'inuse', 'absolute', 1, 1024]
+        ]
+    },
+    'memstats_mspan': {
+        'options': ['mspan', 'memory: size of mspan memory structures', 'KiB', 'memstats',
+                    'expvar.memstats.mspan', 'line'],
+        'lines': [
+            ['memstats_mspan_inuse', 'inuse', 'absolute', 1, 1024]
+        ]
+    },
+    'memstats_mcache': {
+        'options': ['mcache', 'memory: size of mcache memory structures', 'KiB', 'memstats',
+                    'expvar.memstats.mcache', 'line'],
+        'lines': [
+            ['memstats_mcache_inuse', 'inuse', 'absolute', 1, 1024]
+        ]
+    },
+    'memstats_live_objects': {
+        'options': ['live_objects', 'memory: number of live objects', 'objects', 'memstats',
+                    'expvar.memstats.live_objects', 'line'],
+        'lines': [
+            ['memstats_live_objects', 'live']
+        ]
+    },
+    'memstats_sys': {
+        'options': ['sys', 'memory: size of reserved virtual address space', 'KiB', 'memstats',
+                    'expvar.memstats.sys', 'line'],
+        'lines': [
+            ['memstats_sys', 'sys', 'absolute', 1, 1024]
+        ]
+    },
+    'memstats_gc_pauses': {
+        'options': ['gc_pauses', 'memory: average duration of GC pauses', 'ns', 'memstats',
+                    'expvar.memstats.gc_pauses', 'line'],
+        'lines': [
+            ['memstats_gc_pauses', 'avg']
+        ]
+    }
+}
+
+EXPVAR = namedtuple(
+    "EXPVAR",
+    [
+        "key",
+        "type",
+        "id",
+    ]
+)
+
+
+def flatten(d, top='', sep='.'):
+    items = []
+    for key, val in d.items():
+        nkey = top + sep + key if top else key
+        if isinstance(val, dict):
+            items.extend(flatten(val, nkey, sep=sep).items())
+        else:
+            items.append((nkey, val))
+    return dict(items)
+
+
+class Service(UrlService):
+    def __init__(self, configuration=None, name=None):
+        UrlService.__init__(self, configuration=configuration, name=name)
+        # if memstats collection is enabled, add the charts and their order
+        if self.configuration.get('collect_memstats'):
+            self.definitions = dict(MEMSTATS_CHARTS)
+            self.order = list(MEMSTATS_ORDER)
+        else:
+            self.definitions = dict()
+            self.order = list()
+
+        # if extra charts are defined, parse their config
+        extra_charts = self.configuration.get('extra_charts')
+        if extra_charts:
+            self._parse_extra_charts_config(extra_charts)
+
+    def check(self):
+        """
+        Check if the module can collect data:
+        1) At least one JOB configuration has to be specified
+        2) The JOB configuration needs to define the URL and either collect_memstats must be enabled or at least one
+           extra_chart must be defined.
+
+        The configuration and URL check is provided by the UrlService class.
+        """
+
+        if not (self.configuration.get('extra_charts') or self.configuration.get('collect_memstats')):
+            self.error('Memstats collection is disabled and no extra_charts are defined, disabling module.')
+            return False
+
+        return UrlService.check(self)
+
+    def _parse_extra_charts_config(self, extra_charts_config):
+
+        # a place to store the expvar keys and their types
+        self.expvars = list()
+
+        for chart in extra_charts_config:
+
+            chart_dict = dict()
+            chart_id = chart.get('id')
+            chart_lines = chart.get('lines')
+            chart_opts = chart.get('options', dict())
+
+            if not all([chart_id, chart_lines]):
+                self.info('Chart {0} has no ID or no lines defined, skipping'.format(chart))
+                continue
+
+            chart_dict['options'] = [
+                chart_opts.get('name', ''),
+                chart_opts.get('title', ''),
+                chart_opts.get('units', ''),
+                chart_opts.get('family', ''),
+                chart_opts.get('context', ''),
+                chart_opts.get('chart_type', 'line')
+            ]
+            chart_dict['lines'] = list()
+
+            # add the lines to the chart
+            for line in chart_lines:
+
+                ev_key = line.get('expvar_key')
+                ev_type = line.get('expvar_type')
+                line_id = line.get('id')
+
+                if not all([ev_key, ev_type, line_id]):
+                    self.info('Line missing expvar_key, expvar_type, or line_id, skipping: {0}'.format(line))
+                    continue
+
+                if ev_type not in ['int', 'float']:
+                    self.info('Unsupported expvar_type "{0}". Must be "int" or "float"'.format(ev_type))
+                    continue
+
+                # self.expvars[ev_key] = (ev_type, line_id)
+                self.expvars.append(EXPVAR(ev_key, ev_type, line_id))
+
+                chart_dict['lines'].append(
+                    [
+                        line.get('id', ''),
+                        line.get('name', ''),
+                        line.get('algorithm', ''),
+                        line.get('multiplier', 1),
+                        line.get('divisor', 100 if ev_type == 'float' else 1),
+                        line.get('hidden', False)
+                    ]
+                )
+
+            self.order.append(chart_id)
+            self.definitions[chart_id] = chart_dict
+
+    def _get_data(self):
+        """
+        Format data received from http request
+        :return: dict
+        """
+
+        raw_data = self._get_raw_data()
+        if not raw_data:
+            return None
+
+        data = json.loads(raw_data)
+
+        expvars = dict()
+        if self.configuration.get('collect_memstats'):
+            expvars.update(self._parse_memstats(data))
+
+        if self.configuration.get('extra_charts'):
+            # the memstats part of the data has been already parsed, so we remove it before flattening and checking
+            #   the rest of the data, thus avoiding needless iterating over the multiply nested memstats dict.
+            del (data['memstats'])
+            flattened = flatten(data)
+
+            for ev in self.expvars:
+                v = flattened.get(ev.key)
+
+                if v is None:
+                    continue
+
+                try:
+                    if ev.type == 'int':
+                        expvars[ev.id] = int(v)
+                    elif ev.type == 'float':
+                        expvars[ev.id] = float(v) * 100
+                except ValueError:
+                    self.info('Failed to parse value for key {0} as {1}, ignoring key.'.format(ev.key, ev.type))
+                    return None
+
+        return expvars
+
+    @staticmethod
+    def _parse_memstats(data):
+
+        memstats = data['memstats']
+
+        # calculate the number of live objects in memory
+        live_objs = int(memstats['Mallocs']) - int(memstats['Frees'])
+
+        # calculate GC pause times average
+        # the Go runtime keeps the last 256 GC pause durations in a circular buffer,
+        #  so we need to filter out the 0 values before the buffer is filled
+        gc_pauses = memstats['PauseNs']
+        try:
+            gc_pause_avg = sum(gc_pauses) / len([x for x in gc_pauses if x > 0])
+        # no GC cycles have occured yet
+        except ZeroDivisionError:
+            gc_pause_avg = 0
+
+        return {
+            'memstats_heap_alloc': memstats['HeapAlloc'],
+            'memstats_heap_inuse': memstats['HeapInuse'],
+            'memstats_stack_inuse': memstats['StackInuse'],
+            'memstats_mspan_inuse': memstats['MSpanInuse'],
+            'memstats_mcache_inuse': memstats['MCacheInuse'],
+            'memstats_sys': memstats['Sys'],
+            'memstats_live_objects': live_objs,
+            'memstats_gc_pauses': gc_pause_avg,
+        }
diff --git a/collectors/python.d.plugin/go_expvar/go_expvar.conf b/collectors/python.d.plugin/go_expvar/go_expvar.conf
new file mode 100644
index 0000000..4b821cd
--- /dev/null
+++ b/collectors/python.d.plugin/go_expvar/go_expvar.conf
@@ -0,0 +1,108 @@
+# netdata python.d.plugin configuration for go_expvar
+#
+# This file is in YaML format. Generally the format is:
+#
+# name: value
+#
+# There are 2 sections:
+#  - global variables
+#  - one or more JOBS
+#
+# JOBS allow you to collect values from multiple sources.
+# Each source will have its own set of charts.
+#
+# JOB parameters have to be indented (using spaces only, example below).
+
+# ----------------------------------------------------------------------
+# Global Variables
+# These variables set the defaults for all JOBs, however each JOB
+# may define its own, overriding the defaults.
+
+# update_every sets the default data collection frequency.
+# If unset, the python.d.plugin default is used.
+# update_every: 1
+
+# priority controls the order of charts at the netdata dashboard.
+# Lower numbers move the charts towards the top of the page.
+# If unset, the default for python.d.plugin is used.
+# priority: 60000
+
+# penalty indicates whether to apply penalty to update_every in case of failures.
+# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes.
+# penalty: yes
+
+# autodetection_retry sets the job re-check interval in seconds.
+# The job is not deleted if check fails.
+# Attempts to start the job are made once every autodetection_retry.
+# This feature is disabled by default.
+# autodetection_retry: 0
+
+# ----------------------------------------------------------------------
+# JOBS (data collection sources)
+#
+# Any number of jobs is supported.
+#
+# All python.d.plugin JOBS (for all its modules) support a set of
+# predefined parameters. These are:
+#
+# job_name:
+#     name: myname            # the JOB's name as it will appear at the
+#                             # dashboard (by default is the job_name)
+#                             # JOBs sharing a name are mutually exclusive
+#     update_every: 1         # the JOB's data collection frequency
+#     priority: 60000         # the JOB's order on the dashboard
+#     penalty: yes            # the JOB's penalty
+#     autodetection_retry: 0  # the JOB's re-check interval in seconds
+#
+# Additionally to the above, this plugin also supports the following:
+#
+#     url: 'http://127.0.0.1/debug/vars'       # the URL of the expvar endpoint
+#
+# As the plugin cannot possibly know the port your application listens on, there is no default value. Please include
+# the whole path of the endpoint, as the expvar handler can be installed in a non-standard location.
+#
+# if the URL is password protected, the following are supported:
+#
+#     user: 'username'
+#     pass: 'password'
+#
+#     collect_memstats: true        # enables charts for Go runtime's memory statistics
+#     extra_charts: {}              # defines extra data/charts to monitor, please see the example below
+#
+# If collect_memstats is disabled and no extra charts are defined, this module will disable itself, as it has no data to
+# collect.
+#
+# Please visit the module wiki page for more information on how to use the extra_charts variable:
+#
+# https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin/go_expvar
+#
+# Configuration example
+# ---------------------
+
+#app1:
+#  name : 'app1'
+#  url  : 'http://127.0.0.1:8080/debug/vars'
+#  collect_memstats: true
+#  extra_charts:
+#    - id: "runtime_goroutines"
+#      options:
+#        name: num_goroutines
+#        title: "runtime: number of goroutines"
+#        units: goroutines
+#        family: runtime
+#        context: expvar.runtime.goroutines
+#        chart_type: line
+#      lines:
+#        - {expvar_key: 'runtime.goroutines', expvar_type: int, id: runtime_goroutines}
+#    - id: "foo_counters"
+#      options:
+#        name: counters
+#        title: "some random counters"
+#        units: awesomeness
+#        family: counters
+#        context: expvar.foo.counters
+#        chart_type: line
+#      lines:
+#        - {expvar_key: 'counters.cnt1', expvar_type: int, id: counters_cnt1}
+#        - {expvar_key: 'counters.cnt2', expvar_type: float, id: counters_cnt2}
+